JP6767664B1 - Information processing systems, information processing equipment and programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently perform voice notification to a person located in a predetermined space while suppressing an increase in system scale. An information processing system 1-1 includes an information processing device 1, a camera 2, and a speaker 3. The camera 2 photographs a person located in the space. The speaker 3 is non-integrated with the camera 2, has voice directivity, and is rotationally driven based on an instruction from the control unit 1a. The control unit 1a in the information processing device 1 identifies the target person from the captured image taken by the camera 2, detects the head position of the target person, and emits a sound toward the head position of the speaker 3. Calculate the rotation angle. Then, the control unit 1a selects a voice pattern to be emitted to the target person, rotates the speaker 3 at a rotation angle, and outputs the selected voice pattern from the speaker 3. [Selection diagram] Fig. 1

Description

The present invention relates to information processing systems, information processing devices and programs.

With the progress of information processing technology and the increase in resolution of surveillance cameras, a system has been developed that detects a person from an image taken by a surveillance camera and emits sound from a speaker. By constructing such a system in the store, for example, it is possible to prevent crime against a suspicious person who has invaded the store, to contact a store clerk, and the like.

Japanese Unexamined Patent Publication No. 2017-215806

In the above system, the surveillance camera and the speaker are conventionally integrated, and the orientation of the surveillance camera and the orientation of the speaker are the same. However, in order to notify a person located in the space of voice in such a system, a plurality of integrated surveillance cameras / speakers must be arranged, which causes a problem that the system scale increases and it is inefficient. There is.

In one aspect, the present invention provides an information processing system, an information processing device, and a program capable of efficiently performing voice notification to a person located in a predetermined space while suppressing an increase in the system scale. The purpose is.

An information processing system is provided to solve the above problems. The information processing system identifies the target person from the camera, the speaker that is non-integrated with the camera and is driven to rotate with directionality, and the captured image taken by the camera, and detects the head position of the target person. A control unit that calculates the rotation angle of the speaker to emit sound toward the head position, selects the sound pattern to be emitted to the target person, rotates the speaker at the rotation angle, and outputs the sound pattern from the speaker. The control unit associates the two-dimensional image of the captured image with the three-dimensional space, detects the coordinates of the feet and overhead of the target person from the two-dimensional image, and three-dimensionalizes the coordinates of the feet and overhead. A two-dimensional image that maps to space, detects the position of the ear by subtracting a predetermined value from the overhead height of the target person based on the overhead coordinates mapped to the three-dimensional space, and sets the position of the ear as the head position of the target person. Detects multiple coordinates of the target person's feet at regular time intervals to acquire time-series coordinate data, calculates the movement amount of the target person after a lapse of a predetermined time from the coordinate data, and heads based on the movement amount. The position of the part is updated, the delay time from the detection of the target person to the output of the voice pattern from the speaker is maintained, and the movement amount is calculated including the delay time in the predetermined time. Further, the control unit calculates the first rotation angle at which the speaker is directed to the detected head position, and when the movement destination of the target person is not predicted, the control unit rotates the speaker at the first rotation angle and transmits the voice pattern to the speaker. When predicting the movement destination of the target person by outputting from, the second rotation angle for pointing the speaker to the updated head position is calculated, the speaker is rotated at the first rotation angle, and the first rotation angle is calculated. After the rotation of the speaker is completed, the speaker is rotated at the second rotation angle while outputting the sound pattern from the speaker.

Further, in order to solve the above problems, an information processing device that executes the same control as the above information processing system is provided.
Further, a program for causing the computer to execute the same control as the information processing system is provided.

According to one aspect, it is possible to suppress an increase in the system scale and efficiently perform voice notification to a person located in a predetermined space.

It is a figure for demonstrating an example of the information processing system of 1st Embodiment. It is a figure which shows an example of the structure of the information processing system of 2nd Embodiment. It is a figure which shows an example of the structure of a speaker. It is a figure which shows an example of the hardware configuration of an information processing apparatus. It is a figure which shows an example of the operation sequence from taking a picture of a person to making a voice call. It is a figure which shows an example of the operation sequence from taking a picture of a person to making a voice call. It is a figure which shows an example of the operation sequence from taking a picture of a person to making a voice call. It is a figure which shows an example of the voice pattern table. It is a figure for demonstrating the position of a camera and a person in 3D space. It is a figure for demonstrating the position of a camera and a person in 3D space. It is a flowchart which shows an example of the whole operation from the detection of a person to the call. It is a flowchart which shows an example of the detection process of a head position. It is a flowchart which shows an example of estimation of the movement speed of a target person, and update of a head position. It is a flowchart which shows an example of the calculation process of the rotation angle of a speaker. It is a flowchart which shows an example of the calculation process of the rotation angle with the movement of a target person. It is a flowchart which shows an example of the rotation drive of a speaker, and the operation of a voice call. It is a flowchart which shows an example of the rotation drive of a speaker, and the operation of a voice call.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram for explaining an example of the information processing system of the first embodiment. The information processing system 1-1 includes an information processing device 1, a camera 2, and a speaker 3. The information processing device 1 includes a control unit 1a and a storage unit 1b.

The camera 2 monitors and photographs a person located in a predetermined space. The speaker 3 is non-integrated with the camera 2, has sound directivity, and is rotationally driven to output sound based on an instruction from the control unit 1a.

The control unit 1a performs image analysis based on AI (Artificial Intelligence) processing on the captured image captured by the camera 2. Further, the control unit 1a performs drive control and audio output control for the speaker 3 based on the image analysis result. The storage unit 1b stores various data required for processing by the control unit 1a.
Each process of the control unit 1a and the storage unit 1b is realized, for example, by executing a predetermined program by a processor (not shown) included in the information processing device 1.

The operation flow of the control unit 1a will be described.
[Step S1] The control unit 1a identifies the target person from the captured image captured by the camera 2.
[Step S2] The control unit 1a detects the head position of the target person.

[Step S3] The control unit 1a calculates the rotation angle of the speaker 3 for emitting sound toward the head position.
[Step S4] The control unit 1a selects a voice pattern suitable for the target person.
[Step S5] The control unit 1a rotates the speaker 3 at the calculated rotation angle, and outputs the selected voice pattern from the speaker 3.

As described above, in the information processing system 1-1, the head of the target person calculated from the captured image captured by the camera 2 using the speaker 3 which is not integrated with the camera 2 and has directivity and is rotationally driven. The speaker 3 is rotated toward the part position, and the speaker 3 outputs the sound to the target person. As a result, the number of speakers installed can be reduced, so that it is possible to suppress an increase in the system scale and efficiently perform voice notification to a person located in a predetermined space.

[Second Embodiment]
Next, the second embodiment will be described. In the following description, giving a voice notification to the target person may be referred to as a voice call.

FIG. 2 is a diagram showing an example of the configuration of the information processing system according to the second embodiment. The information processing system 1-2 includes an information processing device 10, a camera 20-1, ..., 20-n (collectively referred to as a camera 20), a speaker 30, a terminal 41 (for maintenance management), and a terminal 42 (for maintenance management). It includes (for notification), AP (access point) 50, hub (Hub) 61 and PoE (Power over Ethernet) hub 62 (Ethernet is a registered trademark).

The information processing device 10 includes a control unit 11 and a storage unit 12. The control unit 11 has the function of the control unit 1a of FIG. 1, and the storage unit 12 has the function of the storage unit 1b of FIG. The speaker 30 has the function of the speaker 3 of FIG.

The hub 61 has ports p1, ..., P4, and the PoE hub 62 has ports p11, p12-1 ..., P12-n. Ports p1, ..., P4 and port p11 are ports to which, for example, a 1 Gbit / s communication line can be connected. Ports p12-1 ..., P12-n are ports to which, for example, a 100 Mbit / s communication line can be connected.

The port p1 of the hub 61 and the port p11 of the PoE hub 62 are connected by a LAN (Local Area Network) cable L1. The PoE hub 62 supplies power through the LAN cable L1 of category 5e or higher used for Ethernet communication.

Therefore, by connecting the camera 20 to the PoE hub 62, external power such as an AC (Alternating Current) adapter becomes unnecessary, and power can be supplied only by the LAN cable L1 that performs data communication. Therefore, the camera 20 can be installed outdoors or in a place where power supply is difficult, such as the ceiling.

On the other hand, to the hub 61, the terminal 41 is connected to the port p2, the information processing device 10 is connected to the port p3, and the AP50 is connected to the port p4. Further, the PoE hub 62 is connected to the cameras 20-1, ..., 20-n to the ports p12-1 ..., P12-n, respectively. The terminal 42 and the speaker 30 are wirelessly connected to the AP 50.

<Speaker configuration>
FIG. 3 is a diagram showing an example of the speaker configuration. The speaker 30 includes an audio output unit 31 and a rotation mechanism unit 32. The voice output unit 31 has a voice propagation function using ultrasonic waves, and performs directional output of voice.

The rotation mechanism unit 32 has a biaxial rotation mechanism in the horizontal direction and the vertical direction. The horizontal motor rotation mechanism of the rotation mechanism unit 32 rotates the audio output unit 31 in the plus direction (arrow h1) and the minus direction (arrow h2) with reference to 0 degrees of the horizontal axis h.

Further, the vertical motor rotation mechanism of the rotation mechanism unit 32 rotates the audio output unit 31 in the plus direction (arrow v1) and the minus direction (arrow v2) with reference to 0 degree of the vertical axis v. .. A mounting component 33 for mounting the speaker 30 on a wall surface or the like is provided on the upper surface of the rotation mechanism unit 32, and the speaker 30 is provided with a wireless LAN communication function (not shown).

<Hardware configuration>
FIG. 4 is a diagram showing an example of the hardware configuration of the information processing device. The information processing device 10 is totally controlled by the processor (computer) 100. The processor 100 realizes the function of the control unit 11.

A memory 101, an input / output interface 102, and a network interface 104 are connected to the processor 100 via a bus 103. The processor 100 may be a multiprocessor. The processor 100 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic Device). Further, the processor 100 may be a combination of two or more elements of the CPU, MPU, DSP, ASIC, and PLD.

The memory 101 includes the function of the storage unit 12 and is used as the main storage device of the information processing device 10. At least a part of an OS (Operating System) program or an application program to be executed by the processor 100 is temporarily stored in the memory 101. In addition, various data required for processing by the processor 100 are stored in the memory 101.

The memory 101 is also used as an auxiliary storage device for the information processing device 10, and stores OS programs, application programs, and various data. The memory 101 may include a flash memory, a semiconductor storage device such as an SSD (Solid State Drive), or a magnetic recording medium such as an HDD (Hard Disk Drive) as an auxiliary storage device.

Peripheral devices connected to the bus 103 include an input / output interface 102 and a network interface 104. A monitor (for example, LED (Light Emitting Diode), LCD (Liquid Crystal Display), etc.) that functions as a display device that displays the status of the information processing device 10 according to an instruction from the processor 100 can be connected to the input / output interface 102.

Further, the input / output interface 102 can be connected to an information input device such as a keyboard or a mouse, and transmits a signal sent from the information input device to the processor 100.
Furthermore, the input / output interface 102 also functions as a communication interface for connecting peripheral devices. For example, the input / output interface 102 can be connected to an optical drive device that reads data recorded on an optical disk by using laser light or the like. Optical discs include DVD (Digital Versatile Disc), Blu-ray Disc (registered trademark), CD-ROM (Compact Disc Read Only Memory), CD-R (Recordable) / RW (Rewritable), and the like.

Further, the input / output interface 102 can be connected to a memory device or a memory reader / writer. The memory device is a recording medium equipped with a communication function with the input / output interface 102. A memory reader / writer is a device that writes data to or reads data from a memory card. A memory card is a card-type recording medium.

The network interface 104 connects to the network and controls the network interface. For example, a NIC (Network Interface Card), a wireless LAN card, or the like can also be used. The data received by the network interface 104 is output to the memory 101 and the processor 100.

With the hardware configuration as described above, the processing function of the information processing apparatus 10 can be realized. For example, the information processing device 10 can perform the processing of the present invention by each processor 100 executing a predetermined program.

The information processing device 10 realizes the processing function of the present invention, for example, by executing a program recorded on a computer-readable recording medium. The program that describes the processing content to be executed by the information processing apparatus 10 can be recorded on various recording media.

For example, a program to be executed by the information processing device 10 can be stored in the auxiliary storage device. The processor 100 loads at least a part of the program in the auxiliary storage device into the main storage device and executes the program.

It can also be recorded on a portable recording medium such as an optical disk, a memory device, or a memory card. The program stored in the portable recording medium can be executed after being installed in the auxiliary storage device, for example, under the control of the processor 100. The processor 100 can also read and execute the program directly from the portable recording medium.

<Voice operation sequence>
FIG. 5 is a diagram showing an example of an operation sequence from shooting a person to speaking out.
The operation sequence when calling out to a suspicious person is shown.
[Step S11] A person enters the store.
[Step S11a] The camera 20 photographs a person who has entered the store.
[Step S11b] The camera 20 transmits a photographed image of a person to the control unit 11.
[Step S11c] The control unit 11 analyzes the captured image by AI processing to detect and track a person.

[Step S12] A person performs some suspicious action.
[Step S12a] The camera 20 captures the suspicious behavior of the person.
[Step S12b] The camera 20 transmits a captured image of the suspicious behavior to the control unit 11.
[Step S12c] The control unit 11 recognizes in advance a pattern of normal behavior (or suspicious behavior) as a behavior pattern of a person, and determines the behavior pattern based on the received captured image. Then, when a behavior pattern (or a suspicious behavior pattern) that deviates from the normal behavior is detected, it is determined that the person is a suspicious person.

[Step S13] The control unit 11 notifies the notification terminal 42 that a suspicious person has been found.
[Step S14] The terminal 42 displays on the screen that a suspicious person has entered the store.
[Step S15] The control unit 11 performs 3D (Dimension) space mapping, head position detection, rotation angle calculation, and voice pattern selection in order to call out to a suspicious person from the speaker 30.

The 3D space mapping process maps the position of a person in 3D space. The head position detection process detects the coordinates of the head position of a person in 3D space. The rotation angle calculation process calculates the rotation angle of the speaker 30 so that the speaker 30 faces the detected head position of the person. In the voice pattern selection process, a voice pattern at the time of speaking is selected from a plurality of sound sources (a specific example of the voice pattern will be described later in FIG. 8).

[Step S16] The control unit 11 transmits a voice call command (calculated rotation angle and selected voice pattern) to the speaker 30.
[Step S17] The speaker 30 is driven to the instructed rotation angle based on the received voice command.

[Step S18] The speaker 30 emits a voice of the instructed voice pattern based on the received voice command to speak to the suspicious person. The suspicious person will notice the voice.

FIG. 6 is a diagram showing an example of an operation sequence from shooting a person to speaking out. The operation sequence when calling out to a specific person is shown. The specific person is a person other than a suspicious person, and corresponds to, for example, a general customer who has visited the store.

[Step S21] A person enters the store.
[Step S21a] The camera 20 photographs a person who has entered the store.
[Step S21b] The camera 20 transmits a photographed image of a person to the control unit 11.
[Step S21c] The control unit 11 analyzes the captured image by AI processing to detect and track a specific person. The control unit 11 recognizes the behavior pattern of the person in advance, and determines the behavior pattern based on the received captured image. For example, when an action pattern of normal behavior is detected, it is determined that the person is a specific person.

[Step S22] The control unit 11 notifies the notification terminal 42 that a specific person has been found.
[Step S23] The terminal 42 displays on the screen that a specific person has entered the store.
[Step S24] The control unit 11 performs 3D space mapping, head position detection, rotation angle calculation, and voice pattern selection in order to call out to a specific person from the speaker 30.

[Step S25] The control unit 11 transmits a voice call command (calculated rotation angle and selected voice pattern) to the speaker 30.
[Step S26] The speaker 30 is driven to the instructed rotation angle based on the received voice command.

[Step S27] The speaker 30 emits a voice of the instructed voice pattern based on the received voice command, and speaks to a specific person. A specific person will notice the call.

FIG. 7 is a diagram showing an example of an operation sequence from shooting a person to speaking out. The operation sequence when calling out to a specific person in a specific area is shown. The specific person in the specific area corresponds to, for example, a clerk in the sales floor in the store.

[Step S31] A person enters the store.
[Step S31a] The camera 20 photographs a person who has entered the store.
[Step S31b] The camera 20 transmits a photographed image of a person to the control unit 11.
[Step S31c] The control unit 11 analyzes the captured image by AI processing, detects a specific person, and maps it to the 3D space. It also tracks a specific person in 3D space.

[Step S32] A person enters a specific area.
[Step S32a] The camera 20 takes a picture of a person in a specific area.
[Step S32b] The camera 20 transmits a photographed image of a person to the control unit 11.
[Step S32c] The control unit 11 determines that a specific person is present in the specific area.

[Step S33] The control unit 11 notifies the notification terminal 42 that a specific person in the specific area has been found.
[Step S34] The terminal 42 displays on the screen that a specific person is in the specific area.

[Step S35] The control unit 11 performs each process of head position detection, rotation angle calculation, and voice pattern selection in order to call out to a specific person from the speaker 30.
[Step S36] The control unit 11 transmits a voice command (calculated rotation angle and selected voice pattern) to the speaker 30.
[Step S37] The speaker 30 is driven to the instructed rotation angle based on the received voice command.

[Step S38] The speaker 30 emits a voice of the instructed voice pattern based on the received voice command, and speaks to a specific person in the specific area. A specific person in a specific area will notice the call.

<Voice pattern>
FIG. 8 is a diagram showing an example of a voice pattern table. The voice pattern table 12a has items of a person, a voice file, and a voice pattern (voice content), and the data structure of the table is stored in the storage unit 12.

As the table contents, for example, when the person is a suspicious person, the audio file includes the audio file 1. wav, audio file 2. wav, audio file 3. wav is registered. Audio file 1. The voice pattern of wav is "Welcome", voice file 2. The audio pattern of wav is "Customers are waiting in the xx area", audio file 3. As for the voice pattern of wav, the voice "Thank you for your purchase" is registered.

When the person is a specific person (for example, a man in his thirties), the audio file includes the audio file 4. wav is registered. Audio file 4. As for the voice pattern of wav, the voice "○○ products are recommended" is registered.

Further, when the person is a specific person (for example, a clerk) in the specific area, the audio file includes the audio file 5. wav is registered. Audio file 5. As for the voice pattern of wav, the voice "Please come to xx" is registered.
As described above, in the voice pattern table 12a, voices suitable for speaking to the target person are registered.

<Position of camera and person in 3D space>
9 and 10 are diagrams for explaining the positions of the camera and the person in the 3D space. Note that FIG. 10 is a representation of the image of FIG. 9 in the xz plane. In FIG. 9, the feet of the target person are located at the coordinates A (x1, y1, z1 = 0) in the 3D xyz space. Further, the camera 20 is located at the coordinates (x2, y2, z2).

In FIG. 10, the target person is located at the coordinates (x1, z1), and the camera 20 is located at (x2, z2). The overhead coordinates of the target person are (x1, H), and the point where the line segment drawn above the target person's head from the camera 20 intersects the x-axis is the coordinate B (x3, z3 = 0).

<Flow chart>
Next, the detailed operation will be described with reference to the flowcharts of FIGS. 11 to 17. FIG. 11 is a flowchart showing an example of the overall operation from the detection of a person to the calling.
[Step S41] The control unit 11 activates the image analysis process by the AI process.
[Step S42] The control unit 11 detects a person from the captured image of the camera 20 and determines whether or not the detected person is the target person for speaking. In the case of the target person to be called, the process proceeds to step S43, and if it is not the target person, the person detection and the determination process are repeated.

[Step S43] The control unit 11 detects the head position of the target person in the 3D space.
[Step S44] The control unit 11 determines whether or not to predict the movement destination of the target person. If the movement destination is predicted, the process proceeds to step S45, and if the movement destination is not predicted, the process proceeds to step S46.

[Step S45] The control unit 11 estimates the moving speed of the target person and updates the head position.
[Step S46] The control unit 11 calculates the rotation angle of the speaker 30.
[Step S47] The control unit 11 selects a voice pattern suitable for the target person by using the voice pattern table 12a.

[Step S48] The control unit 11 determines whether or not to speak while tracking the target person. If the call is made while tracking, the process proceeds to step S49, and if the call is made without tracking, the process proceeds to step S50a.
[Step S49] The control unit 11 calculates the rotation angle of the speaker 30 as the target person moves. The process proceeds to step S50b.

[Step S50a] The speaker 30 is driven to a rotation angle instructed by the control unit 11, and also speaks to the target person in a voice pattern instructed by the control unit 11.
[Step S50b] The speaker 30 is driven to a rotation angle in accordance with the movement of the target person instructed by the control unit 11, and also speaks to the target person in a voice pattern instructed by the control unit 11.

FIG. 12 is a flowchart showing an example of the head position detection process. The detailed flow of step S43 of FIG. 11 is shown.
[Step S43a] The control unit 11 associates the camera screen corrected by calibration of the camera 20 with the 3D space.

[Step S43b] The control unit 11 detects the target person from the captured image captured by the camera 20 and acquires the coordinates in the captured image of the target person. When a person is detected, for example, the position of the person is indicated by a rectangle (rectangular information).

[Step S43c] The control unit 11 detects the coordinates of the feet from the rectangular information of the target person. For example, the midpoint of the lower base of the rectangle indicating the position of the person is calculated, and the midpoint is used as the coordinates of the feet.
[Step S43d] The control unit 11 maps the detected foot coordinates onto the coordinates in the 3D space (corresponding to the coordinates A in FIG. 9).

[Step S43e] The control unit 11 calculates overhead coordinates from the rectangular information of the target person. For example, the midpoint of the upper base of the rectangle indicating the position of the person is calculated, and the midpoint is used as the overhead coordinates.
[Step S43f] The control unit 11 regards the overhead coordinates of the 2D image (captured image) as above the floor in the 3D image, and maps the overhead coordinates in the 3D space (corresponding to the coordinates B in FIG. 10).

[Step S43g] The control unit 11 extracts a z component equal to the x component of the coordinate A in the line segment connecting the coordinates B and the coordinates of the camera 20 (the y component may be used instead of the x component).
[Step S43h] The control unit 11 sets the ear height H of the target person to a position that is a predetermined length lower (for example, 20 cm lower) with respect to the extracted z component.
[Step S43i] The control unit 11 sets the coordinate value in which the z component in the coordinate A is the height of the ear as the head position, and determines the head position as the coordinates for pointing the speaker 30 (referred to as the coordinate C).

In this way, the control unit 11 maps the captured image to the 3D space, obtains the overhead position of the target person in the 3D space, obtains the ear position from the overhead position, and sets the ear position to the head position. And. As a result, since the speaker 30 is rotated toward the head position, the sound from the speaker 30 can be clearly heard by the target person.

FIG. 13 is a flowchart showing an example of estimating the moving speed of the target person and updating the head position. The detailed flow of step S45 of FIG. 11 is shown.
[Step S45a] The control unit 11 detects a plurality of coordinates of the feet of the target person in the 2D image for the past several seconds.

[Step S45b] The control unit 11 converts the detected coordinates of the past feet into the coordinates in the 3D space. As a result, time-series coordinate data including the coordinate A is obtained.
[Step S45c] The control unit 11 estimates the amount of movement of the target person in 3D space after t seconds based on the time-series coordinate data. For example, the moving speed between coordinates obtained from the time-series coordinate data is obtained as a vector in three directions of xyz, and then the average value is obtained for each component (moving speed Va). Then, by multiplying the movement speed Va by the time t, the movement amount dL after t seconds can be estimated.

However, t seconds corresponds to the delay time from the time when the target person is detected to the time when the sound is output. It is assumed that t seconds is obtained by a system test or the like in advance and is retained as a set value in advance.

[Step S45d] The control unit 11 adds the movement amount dL after t seconds in the xyz direction with respect to the coordinate A. As a result, the coordinates (A + dL) can be obtained. The coordinates (A + dL) are the coordinates of the feet of the target person to be called (referred to as coordinates A2). Further, by setting the z component of the coordinate A2 to the height H of the ear, this becomes the head position after movement and becomes the coordinates of the target to which the speaker 30 is directed (update of the coordinate C of the head position).

Here, when speaking to a person, a delay time occurs from the person detection to the output of the voice from the speaker 30. If this delay time is not taken into consideration, even if a person is called to a person, if the person is moving, it is possible that the person is no longer present.

As described above, the control unit 11 detects a plurality of coordinates of the feet of the target person from the two-dimensional image at regular time intervals, acquires time-series coordinate data, and heads based on the movement amount calculated from the coordinate data. Update the department position. As a result, the position of the target person after movement can be detected with high accuracy.

Further, the control unit 11 calculates the movement amount including the delay time from the detection of the target person to the output of the voice pattern from the speaker 30. As a result, even if the person is moving, the speaker 30 can output the voice toward the head of the person after the movement, and the accuracy of speaking can be improved.

FIG. 14 is a flowchart showing an example of the calculation process of the rotation angle of the speaker. The detailed flow of step S46 of FIG. 11 is shown.
[Step S46a] The control unit 11 subtracts the installation coordinates of the speaker 30 from the coordinates C (head position) in the 3D space. This subtraction process vectorizes the coordinates C centered on the speaker 30, and the subtraction result is called a vector S.

[Step S46b] The control unit 11 calculates the horizontal rotation angle (horizontal rotation angle) from the horizontal components (x component and y component) of the vector S. The formula for calculating the horizontal rotation angle is the following formula (1).

[Step S46c] The control unit 11 calculates the r component from the x component and the y component, using the rotation direction component when rotating at the horizontal rotation angle obtained by the equation (1) as a new r component. The formula for calculating the r component is the following formula (2).

[Step S46d] The control unit 11 calculates a vertical rotation angle (vertical rotation angle) from the above r component and the z component which is a vertical component of the vector S. The formula for calculating the vertical rotation angle is the following formula (3).

The control unit 11 can easily and accurately calculate the rotation angle of the speaker 30 by obtaining the horizontal rotation angle and the vertical rotation angle by using the above calculation formula.

FIG. 15 is a flowchart showing an example of the calculation process of the rotation angle accompanying the movement of the target person. The detailed flow of step S49 of FIG. 11 is shown.
[Step S49a] The control unit 11 determines the reproduction time t2 of the voice pattern selected when speaking.

[Step S49b] The control unit 11 multiplies the movement speed Va by the time t2, and calculates the multiplication result as the movement amount.
[Step S49c] The control unit 11 adds the calculated movement amount to the coordinates A2 (coordinates of the feet after the movement), and obtains the head position with the z component as the height H of the ear (referred to as the coordinates Ca). This head position becomes the coordinates of the head of the target person at the end of the voice call.

[Step S49d] The control unit 11 subtracts the installation coordinates of the speaker 30 from the coordinates Ca. This corresponds to the vectorization of the coordinates Ca centered on the speaker 30, and the subtraction result is the vector Sa.

[Step S49e] The control unit 11 calculates a horizontal rotation angle (horizontal rotation angle) from the x and y components of the vector Sa using the equation (1).
[Step S49f] Using the equation (2), the control unit 11 newly sets the horizontal rotation angle direction as the r component, and calculates the r component from the x component and the y component.
[Step S49g] The control unit 11 calculates the rotation angle (vertical rotation angle) in the vertical direction from the r component and z component of the vector Sa.

FIG. 16 is a flowchart showing an example of the rotational drive of the speaker and the operation of speaking. The detailed flow of step S50a of FIG. 11 is shown. The horizontal rotation angle of the speaker 30 calculated based on the coordinates C (first head position) described in FIG. 14 is defined as the horizontal rotation angle a1, and the vertical rotation angle of the speaker 30 calculated based on the coordinates C is vertical. The rotation angle is b1.

[Step S50a1] The control unit 11 transmits the calculated horizontal rotation angle a1 and vertical rotation angle b1 (first rotation angle) to the speaker 30 and the selected voice pattern to the speaker 30.
[Step S50a2] The speaker 30 is rotationally driven at a horizontal rotation angle a1 and a vertical rotation angle b1.
[Step S50a3] When the rotation drive is completed, the speaker 30 speaks to the target person with the instructed voice pattern.

FIG. 17 is a flowchart showing an example of the rotational drive of the speaker and the operation of speaking. The detailed flow of step S50b of FIG. 11 is shown. The horizontal rotation angle of the speaker 30 calculated based on the coordinates Ca (head position after movement) described in FIG. 15 is defined as the horizontal rotation angle a2, and the vertical rotation angle of the speaker 30 calculated based on the coordinates Ca is defined as the horizontal rotation angle a2. The vertical rotation angle is b2.

[Step S50b1] The control unit 11 transmits the calculated horizontal rotation angle a1 and vertical rotation angle b1 (first rotation angle) to the speaker 30 and the selected voice pattern to the speaker 30.
[Step S50b2] The control unit 11 transmits the calculated horizontal rotation angle a2 and vertical rotation angle b2 (second rotation angle) and the information of the time t2 to the speaker 30 to the speaker 30. The time t2 is the reproduction time of the voice pattern in consideration of the delay as described above.

[Step S50b3] The speaker 30 is rotationally driven at a horizontal rotation angle a1 and a vertical rotation angle b1 (first rotation angle).
[Step S50b4] After the rotation drive of the horizontal rotation angle a1 and the vertical rotation angle b1 is completed, the speaker 30 speaks with the instructed voice pattern and at the reproduction time (time t2) of the transmitted voice pattern. Further, the speaker 30 is rotationally driven at a horizontal rotation angle a2 and a vertical rotation angle b2 (second rotation angle) while speaking out.

[Step S50b5] The speaker 30 speaks. Further, when the speaker 30 finishes speaking, the rotation drive stops at the same time or after the end.
In this way, the control unit 11 subtracts the coordinates of the installation position of the speaker 30 in the 3D space from the coordinates of the head position to calculate the vector of the coordinates of the head position centered on the speaker 30, and the vector is horizontal. The horizontal rotation angle of the speaker 30 is calculated based on the directional component.
Then, the vertical rotation angle of the speaker 30 is calculated based on the rotation direction component when the speaker 30 is rotated at the horizontal rotation angle and the vertical component of the vector, and the horizontal rotation angle and the vertical rotation angle are set to the rotation of the speaker 30. Let it be an angle. As a result, the rotation angle of the speaker 30 having the biaxial rotation mechanism in the horizontal direction and the vertical direction can be accurately obtained.

Further, when the control unit 11 calculates the first rotation angle (horizontal rotation angle a1 and vertical rotation angle b1) for directing the speaker 30 to the detected head position and does not predict the movement destination of the target person, the speaker 30 is rotated by the first rotation angle to output an audio pattern from the speaker 30.
Further, when predicting the movement destination of the target person, the second rotation angle (horizontal rotation angle a2 and vertical rotation angle b2) for directing the speaker 30 to the updated head position is calculated, and the speaker 30 is used as the first rotation angle. It is rotated at a rotation angle, and after the rotation of the first rotation angle is completed, the speaker 30 is rotated at the second rotation angle while outputting an audio pattern from the speaker 30.
As a result, the speaker 30 is controlled so as to follow the movement of the target person. Therefore, when the target person moves, the voice from the speaker 30 does not reach the target person, and the target person misses the voice. It is possible to prevent such a situation.

As described above, in the information processing system 1-2 of the second embodiment, the captured image taken by the camera 20 is used by the speaker 30 which is not integrated with the camera 20 and is rotationally driven with directivity. The speaker 30 is rotated toward the head position of the target person calculated from the above, and the speaker 30 outputs the sound to the target person. As a result, the number of speakers installed can be reduced, so that it is possible to suppress an increase in the system scale and efficiently perform voice notification to a person located in a predetermined space.

The processing functions of the information processing systems 1-1 and 1-2 of the present invention described above can be realized by a computer. In this case, a program that describes the processing contents of the functions that the information processing systems 1-1 and 1-2 should have is provided. By executing the program on a computer, the above processing function is realized on the computer.

The program that describes the processing content can be recorded on a computer-readable recording medium. Computer-readable recording media include magnetic storage units, optical disks, opto-magnetic recording media, semiconductor memories, and the like. The magnetic storage unit includes a hard disk device (HDD), a flexible disk (FD), a magnetic tape, and the like. Optical discs include CD-ROM / RW and the like. The magneto-optical recording medium includes MO (Magneto Optical disk) and the like.

When a program is distributed, for example, a portable recording medium such as a CD-ROM on which the program is recorded is sold. It is also possible to store the program in the storage unit of the server computer and transfer the program from the server computer to another computer via the network.

The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage unit. Then, the computer reads the program from its own storage unit and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute the processing according to the program.

In addition, the computer can sequentially execute processing according to the received program each time the program is transferred from the server computer connected via the network. Further, at least a part of the above processing functions can be realized by an electronic circuit such as a DSP, ASIC, or PLD.

Although the embodiment has been illustrated above, the configuration of each part shown in the embodiment can be replaced with another having the same function. Further, any other components or processes may be added. Further, any two or more configurations (features) of the above-described embodiments may be combined.

1-1 Information processing system 1 Information processing device 1a Control unit 1b Storage unit 2 Camera 3 Speaker

Claims (4)

With the camera
A speaker that is not integrated with the camera and has directivity and is driven to rotate,
The target person is identified from the captured image taken by the camera, the head position of the target person is detected, the rotation angle of the speaker for emitting a sound toward the head position is calculated, and the target person A control unit that selects a voice pattern to be emitted from the speaker, rotates the speaker at the rotation angle, and outputs the voice pattern from the speaker.
With
The control unit
The two-dimensional image of the captured image is associated with the three-dimensional space, the coordinates of the feet and overhead of the target person are detected from the two-dimensional image, and the coordinates of the feet and the coordinates of the overhead are converted into the three-dimensional space. The position of the ear is detected by subtracting a predetermined value from the overhead height of the target person based on the overhead coordinates mapped to the three-dimensional space, and the position of the ear is the position of the head of the target person. age,
A plurality of coordinates of the foot of the target person are detected from the two-dimensional image at regular time intervals to acquire time-series coordinate data, and the movement amount of the target person after a lapse of a predetermined time is calculated from the coordinate data. Then, the head position is updated based on the movement amount, and the head position is updated.
The delay time from the detection of the target person to the output of the voice pattern from the speaker is held, and the movement amount is calculated by including the delay time in the predetermined time.
The control unit
The first rotation angle at which the speaker is directed to the detected head position is calculated.
When the movement destination of the target person is not predicted, the speaker is rotated at the first rotation angle to output the voice pattern from the speaker.
When predicting the movement destination of the target person, a second rotation angle at which the speaker is directed to the updated head position is calculated, the speaker is rotated at the first rotation angle, and the first rotation angle is used. After the rotation of the rotation angle is completed, the speaker is rotated at the second rotation angle while outputting the sound pattern from the speaker.
Information processing system. The control unit
The coordinates of the installation position of the speaker in the three-dimensional space are subtracted from the coordinates of the head position to calculate the vector of the coordinates of the head position centered on the speaker.
The horizontal rotation angle of the speaker is calculated based on the horizontal component of the vector.
The vertical rotation angle of the speaker is calculated based on the rotation direction component when the speaker is rotated at the horizontal rotation angle and the vertical component of the vector.
The information processing system according to claim 1, wherein the horizontal rotation angle and the vertical rotation angle are the rotation angles of the speaker. The target person is identified from the captured image taken by the camera, the head position of the target person is detected, and the head position is relative to a speaker that is non-integrated with the camera and has directivity and is rotationally driven. A control unit that calculates the rotation angle of the speaker for emitting sound toward the target person, selects an audio pattern to be emitted to the target person, rotates the speaker at the rotation angle, and outputs the audio pattern from the speaker. When,
A storage unit that stores the voice pattern and
With
The control unit
The two-dimensional image of the captured image is associated with the three-dimensional space, the coordinates of the feet and overhead of the target person are detected from the two-dimensional image, and the coordinates of the feet and the coordinates of the overhead are converted into the three-dimensional space. The position of the ear is detected by subtracting a predetermined value from the overhead height of the target person based on the overhead coordinates mapped to the three-dimensional space, and the position of the ear is the position of the head of the target person. age,
A plurality of coordinates of the foot of the target person are detected from the two-dimensional image at regular time intervals to acquire time-series coordinate data, and the movement amount of the target person after a lapse of a predetermined time is calculated from the coordinate data. Then, the head position is updated based on the movement amount, and the head position is updated.
The delay time from the detection of the target person to the output of the voice pattern from the speaker is maintained, and the movement amount is calculated by including the delay time in the predetermined time.
The control unit
The first rotation angle at which the speaker is directed to the detected head position is calculated.
When the movement destination of the target person is not predicted, the speaker is rotated at the first rotation angle to output the voice pattern from the speaker.
When predicting the movement destination of the target person, a second rotation angle for directing the speaker to the updated head position is calculated, the speaker is rotated at the first rotation angle, and the first rotation angle is used. After the rotation of the rotation angle is completed, the speaker is rotated at the second rotation angle while outputting the sound pattern from the speaker.
Information processing device. On the computer
Identify the target person from the captured image taken by the camera,
The head position of the target person is detected,
The rotation angle of the speaker for emitting sound toward the head position with respect to the speaker which is not integrated with the camera and is driven to rotate with directivity is calculated.
Select the voice pattern to be emitted to the target person,
The speaker is rotated at the rotation angle to output the voice pattern from the speaker.
The two-dimensional image of the captured image is associated with the three-dimensional space, the coordinates of the feet and overhead of the target person are detected from the two-dimensional image, and the coordinates of the feet and the coordinates of the overhead are converted into the three-dimensional space. The position of the ear is detected by subtracting a predetermined value from the overhead height of the target person based on the overhead coordinates mapped to the three-dimensional space, and the position of the ear is the position of the head of the target person. age,
A plurality of coordinates of the foot of the target person are detected from the two-dimensional image at regular time intervals to acquire time-series coordinate data, and the movement amount of the target person after a lapse of a predetermined time is calculated from the coordinate data. Then, the head position is updated based on the movement amount, and the head position is updated.
The delay time from the detection of the target person to the output of the voice pattern from the speaker is held, and the movement amount is calculated by including the delay time in the predetermined time.
The first rotation angle at which the speaker is directed to the detected head position is calculated.
When the movement destination of the target person is not predicted, the speaker is rotated at the first rotation angle to output the voice pattern from the speaker.
When predicting the movement destination of the target person, a second rotation angle at which the speaker is directed to the updated head position is calculated, the speaker is rotated at the first rotation angle, and the first rotation angle is used. After the rotation of the rotation angle is completed, the speaker is rotated at the second rotation angle while outputting the sound pattern from the speaker.
A program that executes processing.

Images