JP2000132693A - Device and method for processing picture, and providing medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make easily measurable a direction which a person faces. SOLUTION: A matching calculating part 42 detects the positions of both eyes from the picture of an inputted man's face and calculates the distance between the both detected eyes. A calibration part 43 stores the distance A between the eyes, which is calculated by the matching part 42 from the picture where the person looks at a camera in front of it. An angle calculating part 44 substitutes the distance (d) calculated by the matching calculating part 4 and the distance A stored in the calibration part 43 in θ=ACOS-1d and, then, calculates the angle θ from the reference point of the direction which the person faces.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, an image processing method, and a providing medium. The present invention relates to an image processing apparatus and method for measuring a direction in which a person is facing, and a providing medium.

[0002]

2. Description of the Related Art There has been proposed a remote conference system in which a plurality of conference rooms are connected via a network and a conference is held as if a single table is surrounded. Each conference room in such a system is provided with a display for displaying an image of a participant other than the user (referred to as participant A) and a speaker for outputting the participant's remark displayed on the display. . The same number of video cameras and speakers as the number of participants excluding the participant A are provided.

[0005] Each conference room is also provided with a video camera for capturing an image of a participant in the conference room and a microphone for capturing audio. The video camera and microphone are installed near (mainly at the top of) a display provided in each conference room. The video of the participant A captured by the video camera and the sound captured by the microphone are output to a display and speaker corresponding to the participant A provided at each venue.

[0004] If a microphone having a directivity is used as the microphone for taking in the speech of the participant A, the participant A can be a participant who wants to convey the speech, in other words, a display on which the participant is projected. When speaking toward the microphone, the microphone installed at the position facing the microphone can capture the utterance at a larger level than other microphones. Therefore, for example, when the speech of the participant A is output from the speakers installed in each conference room, the output level differs for each conference room. Thereby, the participant who has heard the sound of the high output level can intuitively understand that the participant A is talking to himself.

[0005]

In the above-described configuration of the conference room, a plurality of video cameras are installed, and the direction of the participant's face is measured using these video cameras. Determine which conference room the participant is talking to, and transmit the participant's statement or increase the output level only to the conference room corresponding to the result of the determination. Can be transmitted.

However, in order to measure the orientation of the participant's face in this way, it is necessary to perform a complicated process such as three-dimensional measurement. For the measurement, template matching is used. When the face direction is measured by the template matching, the images of the face of the participant at each fixed angle are stored in advance, and the stored image (template) is
The orientation of the face is measured by calculating the degree of correlation with the image captured at that time. For this reason, there is a problem that a large amount of memory for storing an image as a template is required, and the amount of calculation for calculating the degree of correlation increases.

The present invention has been made in view of such a situation, and detects a position of an eye from a captured image of a participant, and changes the distance between the detected eyes.
It measures the direction in which the participant is facing.

[0008]

According to a first aspect of the present invention, there is provided an image processing apparatus, comprising: an image pickup means for picking up an image of a subject; and two points matching a predetermined pattern are extracted from the image picked up by the image pickup means. An extracting unit, a first calculating unit that calculates a distance between the two points extracted by the extracting unit, and a second calculating unit that calculates an angle of the subject using the distance between the two points calculated by the first calculating unit. 2 calculation means.

According to a seventh aspect of the present invention, there is provided an image processing method comprising: an image capturing step of capturing an image of a subject; an extracting step of extracting two points matching a predetermined pattern from the image captured in the image capturing step; A first calculation step of calculating the distance between the two points extracted in step (a), and a second calculation step of calculating the angle of the subject using the distance between the two points calculated in the first calculation step. It is characterized by including.

[0010] The providing medium according to claim 8 includes an imaging step of capturing an image of a subject, an extraction step of extracting two points matching a predetermined pattern from the image captured in the imaging step, and an extraction step. A first calculation step of calculating the distance between the two extracted points; and a second calculation step of calculating the angle of the subject using the distance between the two points calculated in the first calculation step. A computer readable program for causing an image processing apparatus to execute a process is provided.

An image processing apparatus according to claim 1 and claim 7.
In the image processing method described in (1) and the providing medium described in (8), an image of a subject is captured, and two points that match a predetermined pattern are extracted from the captured image, and the two extracted points are extracted. The distance between is calculated and the calculated 2
The angle of the subject is calculated using the distance between the points.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below. In order to clarify the correspondence between each means of the invention described in the claims and the following embodiments, each means is described. When the features of the present invention are described by adding the corresponding embodiment (however, an example) in parentheses after the parentheses, the result is as follows. However, of course, this description does not mean that each means is limited to those described.

An image processing apparatus according to a first aspect of the present invention includes an image pickup means (for example, a front video camera 31-21 in FIG. 4) for picking up an image of a subject and a predetermined pattern from the image picked up by the image pickup means. Extraction means for extracting two matching points (for example, the matching calculation unit 42 in FIG. 5) and first calculation means for calculating the distance between the two points extracted by the extraction means (for example, step S3 in FIG. 6) And second calculating means (for example, step S8 in FIG. 6) for calculating the angle of the subject using the distance between the two points calculated by the first calculating means.

According to a third aspect of the present invention, the image processing apparatus includes a storage unit (for example, the calibration unit 43 in FIG. 5) for storing the distance A between the two points calculated from the front image of the subject by the first calculation unit. In addition, the second calculating means includes a distance A
And the distance d calculated by the first calculating means, θ = A
It is characterized in that the angle θ is calculated by substituting into the equation of COS ⁻¹ d.

According to a fourth aspect of the present invention, there is provided an image processing apparatus, comprising: a second extracting unit (for example, a matching calculation unit 52 shown in FIG. 12) for extracting one point which coincides with the second pattern from an image captured by the imaging unit; -1, 52-2), and the distance between the midpoint of the distance between the two points calculated by the first calculating means and the one point extracted by the second extracting means is calculated. A normalizing unit (for example, distance normalizing units 53-1 and 53-2 in FIG. 12) that normalizes the distance between the two points calculated by the first calculating unit using the value. The calculation means calculates the angle of the subject using the distance between the two points normalized by the normalization means.

According to a sixth aspect of the present invention, in the image processing apparatus, the imaging means is two video cameras installed at a predetermined angle α, and a normal image is taken from one of the video cameras. The value normalized by the conversion means is distance d ₁ ,
From the captured image in the other video camera, when the normalized value by the normalization means and the distance d _2, the angle alpha,
A third calculating means (for example, step S25 in FIG. 13) for calculating the maximum distance A between the two points using the distance d ₁ and the distance d ₂ is further provided. The angle of the subject is calculated using the distance A calculated by the third calculating means.

FIG. 1 shows a configuration of a video conference system to which the image processing apparatus of the present invention is applied. In this specification, a system refers to an entire device including a plurality of devices. As shown in FIG.
A plurality (four in this embodiment) of communication centers 1-
For example, ISDN (Integrated Service Diode)
(Gital Network). Each communication center is, for example, as shown in FIG.
One conference room as shown in FIG.

In the conference room shown in FIG. 2, one table 10, one chair, and three display devices are provided. For example, in a conference room of the communication center 1-4, a chair is arranged at a position of number 4 shown in FIG. 2, and a display device is arranged at positions of numbers 1 to 3. Further, in the conference room of the communication center 1-3, a chair is arranged at the position of No. 3 shown in FIG. 2, and a display device is arranged at the positions of Nos. 1, 2, and 4. In the conference room of the communication center 1-2, the chair is arranged at the position of No. 2 shown in FIG. 2, and the display device is arranged at the positions of Nos. 1, 3 and 4. Further, in the conference room of the communication center 1-1, a chair is arranged at a position of number 1 shown in FIG. 2, and a display device is arranged at positions of numbers 2 to 4.

Further, in the display device arranged in the conference room of the communication center 1-4, the number 1 shown in FIG.
Is displayed on the display device arranged at the position of the communication center 1-1, and the display device arranged at the position of No. 2 is displayed on the display device arranged at the position of the communication center 1-1.
An image of the participant of the communication center 1-3 is displayed on the display device arranged at the position of the number 3 by displaying an image of the participant No. 2. Similarly, in other communication centers, a video image of the participant of the corresponding communication center is displayed on a display device arranged at a position other than the chair where the participant sits.

As described above, in the conference room of each communication center, the chair for the participant to sit at a specific position of the communication center is arranged, and for the others, the participants of the other communication centers are displayed. Display device to be arranged. Therefore, by configuring the conference room in this way, the arrangement of the participants is the same in the conference room of any communication center. That is, it is as if four participants were actually arranged at specific positions with the table 10 at the center. However, in the conference rooms of each communication center, all the participants other than the actual participants themselves are displayed, and the same conference state is realized in any conference room.

Next, the details of each communication center will be described with reference to FIG.
This will be described with reference to FIG. Although the arrangement of the display devices is slightly different in each of the communication centers, they have almost the same configuration. Therefore, only the communication center 1-4 will be described here, and the other communication centers 1-1 to 1-3 will be described. Description is omitted.

First, in the conference room of the communication center 1-4, as shown in FIG. 2, a chair is arranged at a position of No. 4 and display devices are arranged at positions of Nos. 1 to 3, respectively. I have. Therefore, the participant 24 shown in FIG.
Will be sitting on the chair located at position 4 in FIG. In addition, in order to capture the image of the participant 24, each of the display devices 21 to 23 has a left side video camera 31-22 installed on the left side of the participant 24,
Front video camera 3 installed in front of participant 24
1-2, and a right side video camera 31-23 installed on the right side of the participant 24. Further, a microphone 32 for capturing the speech of the participant 24
-21 to 32-23 (hereinafter, a microphone 32-21)
When it is not necessary to individually discriminate among the microphones 32 to 23, the microphone 32 is simply described. Speaker units 33-21 to 33-2 for outputting sounds respectively supplied from other communication centers.
3 and display units 34-21 to 34-23 for displaying an image corresponding to the sound.

The speaker units 33-21 to 33-23 and the display units 34-21 to 34-23 output images transmitted from the communication centers 1-1 to 1-3 and sound corresponding to the images, respectively. It has been made like that.
That is, for example, an image of a participant of the communication center 1-1 is displayed on the display unit 34-21 of the display device 21, and a statement of the participant is output from the speaker unit 33-21. ing. Further, an image of a participant of the communication center 1-2 is displayed on the display unit 34-22 of the display device 22, and the speaker unit 3 is displayed.
From 3-22, the utterance of the participant is output. Further, an image of the participant of the communication center 1-3 is displayed on the display part 34-23 of the display device 23, and the utterance of the participant is output from the speaker part 33-23. .

The front video camera 31-21 disposed on the display device 21 is connected to the communication center 1-4.
Of the participant 24, the microphone 32-21 captures the utterance of the participant 24, and the image and the utterance of the participant 24 are supplied to the communication center 1-1. Further, the left side video camera 31-22 installed on the display device 22 photographs the participant 24 of the communication center 1-4, and the microphone 32-22 captures the participant's remark, and the participant 24 Of the communication center 1
-2. Further, the video camera 31-23 installed on the display device 23 is connected to the communication center 1
-4 participant 24 is photographed, and microphones 32-23 are taken.
Captures the speech of the participant, and the image and speech of the participant 24 are supplied to the communication center 1-3.

Then, as shown in FIG. 3, the display devices 21 to 23 are configured such that the participant 24 displays the display units 34-21 to 34-2 of the respective display devices 21 to 23.
3 so that they can be seen in the predetermined position shown in FIG.

In the conference room having such a configuration, a participant (corresponding display devices 21 to 23) to which the participant 24 pays attention is determined, and the participant (hereinafter, appropriately referred to as a talker) is determined. By emphasizing the remarks or images, it is possible to provide a video conferencing system that is more realistic and easy to use. Hereinafter, measurement of the face direction of the participant 24 performed to determine the interlocutor that the participant 24 focuses on will be described.

FIG. 4 shows the configuration of an apparatus for measuring the direction of the face of the participant 24. In FIG. 4, a front video camera 31-21 is used as a video camera for imaging the participant 24. Front video camera 31
-21 is connected to an arithmetic unit 41 that calculates the direction of the face from the obtained image of the participant 24.

FIG. 5 is a block diagram showing the internal configuration of the arithmetic unit 41. Image data captured by the front video camera 31-21 and input to the arithmetic unit 41 is input to the matching calculation unit 42. The matching calculation unit 42
An eye portion is extracted from the input image data of the participant 24, and the distance between the extracted eyes is calculated. The details will be described later. Of the distance between the eyes output from the matching calculation unit 42, a value calculated from the image data input to be used as the reference value is output to the calibration unit 43 and stored.

The angle calculation unit 44 uses the reference value output from the calibration unit 43 and the value of the distance between the eyes output from the matching calculation unit 42 to calculate the angle in the direction in which the participant 24 is facing. Is calculated. The calculated angle is
For example, it controls the volume of sound output from the speaker units 33-21 to 33-23, or controls the display unit 34-2.
It is used as data when calculating a control amount for controlling the images 1 to 34-23.

The operation of the arithmetic unit 41 will be described with reference to the flowchart of FIG. In step S1, an initial image is input. The participant 24 causes the front video camera 31-21 to capture an image when the participant 24 faces the front with respect to the front video camera 31-21 as an initial image. In step S2, the matching calculation unit 42 of the arithmetic device 41 detects the position of the eye from the obtained front image of the participant 24.

FIG. 7 is a diagram for explaining how to detect the position of the eye. The portion 81 is a black eye portion of the eyeball, and the portions 82 to 85 are skin color portions around the eyeball. As described above, only the eyes have a pattern in which the central part is black and the four parts around the face are flesh-colored. By detecting the part corresponding to this pattern, both eyes of the participant 24 are detected. It is possible to detect. Specifically, the portion 81 has a condition that the luminance Y is small and the color differences U and V are around 128, and the portions 82 to 85 have values obtained by dividing the color difference U by the luminance Y from −0.1 to 0.0. And the value obtained by dividing the color difference V by the luminance Y is a value between 0.1 and 0.3. Each of the portions 81 to 85 has a size of a predetermined number of pixels, for example, a size of 3 × 3 pixels,
When it is determined that a certain number or more of the range of each part is the color, the color is detected as the position of the eye.

In FIG. 7, portions 81 to 84 have the same size, and only portion 85 has, for example, 3 × 15 pixels.
Different sizes. The reason why the portion 85 is vertically elongated is to distinguish it from other portions. Further, in consideration of the time when the face is tilted, it is not necessary to detect both the portion 83 and the portion 84, and it is sufficient that one of them is detected as the skin color. FIG. 8 shows an example of an eye detected using such a pattern. As shown in FIG. 8, the positions of both eyes can be determined by detecting a portion having the pattern shown in FIG. 7 from the input face image.

Thus, in step S2,
When both eyes are detected, the process proceeds to step S3. Step S
In step 3, the distance between straight lines connecting the centers of the parts 81 of both eyes detected in step S2 is calculated. The calculated distance between both eyes is stored in the calibration unit 43 as a calibration value in step S4.

After the calibration is performed as described above, in step S5, a new image of the participant 24 is captured by the front video camera 31-21. The processes of steps S6 and S7 are performed on the captured image by the matching calculation unit 42. Since the processes are the same as the processes of steps S2 and S3, the description thereof is omitted. .

In step S8, the direction (angle) of the face of the participant 24 is calculated. This calculation is performed according to the following equation. θ = COS ⁻¹ d / A (1) Here, θ indicates the angle of the face direction of the participant 24 with respect to the front video camera 31-21, and A is stored in the calibration unit 43 in step S4. Is the distance between both eyes calculated, and d is the distance between both eyes calculated in step S7.

FIG. 9 is a diagram for explaining that the angle θ is obtained by the equation (1). FIG. 9A illustrates a case where the participant 24 faces the front of the front video camera 31-21 to input an initial image (when θ = 0), and the both eyes calculated at that time are shown. The distance between them is a distance A. Then, as shown in FIG. 9B, when the participant 24 tilts the face by the angle θ,
It can be seen that the distance between the eyes calculated from the image captured by the front video camera 31-21 is the distance d.

From FIG. 9B, it can be seen that a triangle is formed in which the distance A and the distance d are two sides and the angle between them is the angle θ. Therefore, the equation of COSθ = (d / A) holds, and the above equation (1) is derived from this equation.

Returning to the description of the flowchart of FIG. 6, when the angle of the face to which the participant 24 faces is calculated in step S8, the process proceeds to step S9, and the calculated angle is output from the speaker unit 33. The sound is output to a device for controlling the sound volume and the image projected on the display unit 34.

By repeating the processing of steps S5 to S9, the direction of the face of the participant 24 is detected at any time during the conference, and the speaker unit 33 and the display unit 34 are controlled based on the detection result, thereby realizing a real It is possible to provide a video conference system with a feeling.

FIG. 10 is a graph showing the result of calculating the direction (angle) of the face of the participant 24 from an image captured by one video camera as described above. The graph shown in FIG. 10 has an angle on the vertical axis and the number of frames on the horizontal axis, and the participant 24 shakes his face between (−45) degrees and 45 degrees, and the image at that time is represented by 210 frames. The image is taken, the angle is calculated for each frame, and the result is plotted.

In the above description, the front video camera 31-21, the left side video camera 31-22, and the video camera 31- provided in the conference room.
Of the three video cameras of 21R, only the front video camera 31-21 is provided with the arithmetic unit 41 and the participant 2
Although the face direction of No. 4 is calculated, the arithmetic unit 41 may be attached to all three devices, and the face direction of the participant 24 may be detected by each device. By doing so, even when the front video camera 31-21 cannot detect both eyes, it is possible to detect one of the other two cameras. As a result, the participant 24 Can be calculated. This means that the angle of the participant 24 can be calculated in a wide range.

As described above, in order to take an image of the participant 24 with one video camera and calculate the face direction from the captured image, it is necessary to hold the calibration value. However, by using two video cameras, it is possible to calculate the face direction of the participant 24 without holding the calibration value.
Shows a configuration example of the device in such a case.

In the configuration example shown in FIG. 11, as two front video cameras, a video camera 31-21L disposed on the right side toward the participant 24 and a video camera 31-21R disposed on the right side. Are connected to the arithmetic unit 51, respectively. The inside of the arithmetic unit 51 is configured as shown in FIG. That is, the image of the participant 24 captured by the video camera 31-21L is input to the matching calculation unit 52-1.
Participant 24 imaged by video camera 31-21R
Are input to the matching calculation unit 52-2.

The data output from matching calculating section 52-1 is input to distance normalizing section 53-1. The data output from matching calculating section 52-2 is input to distance normalizing section 53-2. You. Further, the distance normalizing unit 53-
The data output from the first and the second 53-2 are calculated by an angle calculator 54.
Is output to

Next, the operation of the arithmetic unit 51 will be described with reference to the flowchart of FIG. Step S
At 21, an image of the face of the participant 24 captured by the video camera 31-21 L and the right side camera 31-23 is input to the arithmetic unit 51. Video camera 3
The image captured in 1-21L is matched
-1 and the image captured by the video camera 31-21R is input to the matching calculation unit 52-2.

The matching calculation units 52-1 and 52-2 are:
In step S22, by performing the same processing as the matching calculation unit 42 (FIG. 5), the eye part of the participant 24 is detected, and the distance between the detected eyes is calculated. Further, in step S23, the matching calculation units 52-1 and 52-2 determine the distance normalization units 53-1 and 53-1 described later.
In order to calculate the distance used for the normalization performed by 3-2, the mouth part of the participant 24 is also detected. The mouth is detected by detecting a red portion 91 within a range of a predetermined number of pixels in the face, as shown in FIG. Using the position of the mouth thus detected,
The matching calculation units 52-1 and 52-2 calculate the distance between the midpoint of the distance between the eyes and the mouth.

The distance between the eyes calculated by the matching calculation unit 52-1 (hereinafter referred to as distance 1) and the distance between the center of the distance between the eyes and the mouth (hereinafter referred to as distance 2) Is
Output to distance normalizing section 53-1. Similarly, the distance 1 and the distance 2 calculated by the matching calculation unit 52-2 are output to the distance normalization unit 53-2. Distance normalizing unit 53-
In step S24, 1, 53-2 respectively
Normalization is performed by dividing the input distance 1 by the distance 2. That is, since the distance 1 between the human eyes and the distance 2 from the midpoint of the distance between the eyes to the mouth are constant without change in the case of the same person, the result of dividing the distance 1 by the distance 2 Is always a constant value (ratio).

FIG. 15 shows an example of the eyes and the mouth detected from the image of the participant 24. As described above, the eyes and the mouth can be detected by the above-described method.

The values normalized by the distance normalizing sections 53-1 and 53-2 are output to the angle calculating section 54. In step S25, the angle calculation unit 54 calculates the maximum distance A between the two eyes (the value stored in the calibration unit 43 in the embodiment described with reference to FIG. 5) from the input value, and calculates the value. The angle of the participant A is calculated using the distance A obtained.

First, the calculation of the distance A will be described. Here, as shown in FIG.
L and the video camera 31-21R are arranged. That is, the angles dI ₁ and dI ₂ are respectively set to the video camera 31-21L or the video camera 31-21R.
Of the participant 24 with respect to the front of the participant 24. Here, the angle dI ₁ with respect to the front of the participants 24 of the video camera 31-21L and minus 45 degrees, the angle dI ₂ to 45 degrees with respect to the front of the participants 24 of the video camera 31-21R. And the video camera 31 of the participant 24
The angle with respect to −21L is the angle θ ₁ , and the video camera 31−
The angle with respect to 21R is θ ₂ . Note that the angle (θ ₁ + θ
Value of ₂₎ is the same size as the angle (dI ₁ + dI _2).

When the video cameras 31 are arranged as described above, that is, when the video cameras 31-21L and 31-21R are installed at an angle of 90 degrees, both of the participants 24 The maximum distance A between the eyes is
It is calculated based on the following equation.

(Equation 1) In this equation (2), d ₁ is the video camera 31-2.
The interocular distance calculated from the image of the face of the participant 24 captured in 1L, and d ₂ is the video camera 31-21R.
Is the distance between the eyes calculated from the image of the face of the participant 24 captured in the step (b). However, the distances d ₁ and d ₂ between both eyes are the normalized values calculated in step S24, respectively.

The equation (2) is obtained by eliminating the angles θ1 and θ2 from the following equations (3) to (5). θ ₁ + θ ₂ = 90 degrees (3) COS θ ₁ = (d ₁ / A) (4) COS θ ₂ = (d ₂ / A) (5)

By the way, when the angle between the video camera 31-21L and the video camera 31-21R is 45 degrees,
That is, when the angle (θ ₁ + θ ₂ ) = 45 degrees, the distance A
Is calculated according to the following equation.

(Equation 2) Of course, the present invention can be applied to any size of the angle (θ ₁ + θ ₂ ).

The direction of the face of the participant 24 is calculated by substituting the thus calculated distance A and one of the distances d ₁ and d ₂ into the equation (1). Then, the calculated face direction is determined in step S26.
The sound is output to a device for controlling the volume of the sound output from the speaker unit 33 and the image displayed on the display unit 34. The processing of steps S21 to S26
The process is repeated at any time during the conference, and when the conference is terminated (the power supply of the video conference system is turned off), the process of this flowchart is also terminated as an interrupt process.

As described above, by using two video cameras, even if the calibration values are not held,
The direction of the face of the participant 24 can be calculated. Also, when the participant 24 moves back and forth with respect to the video camera (zooms), the distance between the eyes changes, but the eyes and mouth are detected, and the obtained distances are used. By normalizing, it is possible to calculate the direction of the face of the participant 24 without being affected by such a change.

FIG. 17 is a graph showing a result obtained by measuring the orientation of the face of the participant 24 using two video cameras as described above. Similar to FIG. 10, it is a graph in which the vertical axis shows an angle and the horizontal axis shows a frame.
Reference numeral 4 denotes a graph in which the face is swung between (−45) degrees and 45 degrees, the image at that time is captured in 210 frames, the angle is calculated for each frame, and plotted.

As described above, even if the calibration values are not held, by using two video cameras,
The direction of the face of the participant 24 can be calculated, and the calculation can be performed without the need to perform complicated calculations or enormous calculations.

In the above description, the present invention is applied to a video conference system, but can be applied to other devices and the like. Further, in the above description, the case where there are four communication centers has been described, but the present invention is not limited to this, and the number of communication centers may be further increased,
You may reduce it.

In this specification, a medium for providing a user with a computer program for executing the above-mentioned processing includes an information recording medium such as a magnetic disk and a CD-ROM, and a network such as the Internet and a digital satellite. Transmission media is also included.

[0060]

As described above, the image processing apparatus according to claim 1, the image processing method according to claim 7, and the image processing apparatus according to claim 8.
According to the providing medium described in (1), an image of a subject is captured, two points that match a predetermined pattern are extracted from the captured image, a distance between the extracted two points is calculated, and the calculated distance between the two points is calculated. Since the angle of the subject is calculated using the distance, it is possible to easily detect the direction in which the subject is facing.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration example of a video conference system to which an image processing apparatus according to the present invention is applied.

FIG. 2 is a diagram showing a state of a conference room of each communication center in the video conference system of the present invention.

FIG. 3 is a diagram showing an arrangement state of a display device in a communication center in FIG. 1;

FIG. 4 is a diagram illustrating a configuration example of a device that calculates the direction of a participant's face;

FIG. 5 is a block diagram showing an internal configuration of the arithmetic unit.

FIG. 6 is a flowchart illustrating an operation of the arithmetic device.

FIG. 7 is a diagram illustrating eye detection.

FIG. 8 is a diagram illustrating a detected eye.

FIG. 9 is a diagram illustrating calculation of an angle.

FIG. 10 is a graph in which calculated angles are plotted.

FIG. 11 is a diagram illustrating a configuration example of a device that calculates the direction of a participant's face.

FIG. 12 is a block diagram illustrating an internal configuration of a calculation device.

FIG. 13 is a flowchart illustrating an operation of the arithmetic device.

FIG. 14 is a diagram illustrating eye detection.

FIG. 15 is a diagram illustrating a detected eye.

FIG. 16 is a diagram illustrating calculation of an angle.

FIG. 17 is a graph in which calculated angles are plotted.

[Explanation of symbols]

1-1 to 1-6 communication center, 2 networks,
21-23 display device, 35-21 front video camera, 35-22 left side video camera,
35-23 Right Side Video Camera, 36-21 to 3
6-23 microphone, 37-21 to 37-23
Display unit, 38-21 to 38-23 speaker unit, 41 arithmetic unit, 42 matching calculation unit, 43 calibration unit, 44 angle calculation unit, 51 arithmetic unit, 52-1, 52-2 matching calculation unit, 53-1 , 53-2 distance normalization unit,
54 angle calculator

Continued on the front page (72) Inventor Junichi Ishibashi 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Daisuke Kikuchi 6-35-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Stock In-house F-term (reference) 2F065 AA03 AA07 AA31 BB05 BB07 CC16 DD00 DD06 DD07 FF04 FF61 JJ03 JJ05 JJ19 JJ26 NN20 QQ00 QQ23 QQ24 QQ25 QQ26 QQ28 QQ38 QQ42 SS02 SS13 5B057 BA02 BC06 DC02 BC02 DC02 AC04 AC06 AC22 5L096 BA18 CA05 EA13 FA66 FA67 GA38

Claims (8)

[Claims] An imaging unit that captures an image of a subject; an extraction unit that extracts two points that match a predetermined pattern from an image captured by the imaging unit; A first calculating means for calculating a distance between the two points, and a second calculating means for calculating an angle of the subject using a distance between two points calculated by the first calculating means. Image processing device. 2. The method according to claim 1, wherein the two points extracted by the extracting means are eyes, and the pattern has a black point as a center, and upper and lower portions thereof and either one of a left portion and a right portion is a flesh color pattern. The image processing apparatus according to claim 1, wherein: 3. The image processing apparatus according to claim 1, further comprising a storage unit configured to store a distance A between the two points calculated from the front image of the subject by the first calculation unit, wherein the second calculation unit stores the distance A and the first The image processing apparatus according to claim 1, wherein the angle θ is calculated by substituting the distance d calculated by the calculating means into the equation θ = COS ⁻¹ d / A. 4. A second extracting means for extracting one point that matches a second pattern from an image captured by the image capturing means, and a distance between the two points calculated by the first calculating means. A normal that calculates the distance between the midpoint and one point extracted by the second extracting means, and normalizes the distance between the two points calculated by the first calculating means with the calculated value. 2. The apparatus according to claim 1, further comprising a conversion unit, wherein the second calculation unit calculates the angle of the subject using a distance between the two points normalized by the normalization unit. 3. Image processing device. 5. The one extracted by the second extracting means.
The image processing apparatus according to claim 4, wherein the dot is a mouth, and the second pattern is a red pattern in a predetermined size range. 6. The imaging unit is two video cameras installed at a predetermined angle α, and is normalized by the normalization unit from an image captured by one of the video cameras. When the value is distance d ₁ and the value normalized by the normalizing means is distance d ₂ from an image captured by the other of the video cameras, the angle α, the distance d ₁ , and the distance d _And a third calculating means for calculating a maximum distance A of the distance between the two points by using the distance A, wherein the second calculating means uses the distance A calculated by the third calculating means. The image processing apparatus according to claim 4, wherein an angle of the subject is calculated. 7. An image capturing step of capturing an image of a subject, an extracting step of extracting two points matching a predetermined pattern from the image captured in the image capturing step, and a step between the two points extracted in the extracting step A first calculation step of calculating the distance between the two points, and a second calculation step of calculating the angle of the subject using the distance between the two points calculated in the first calculation step. Image processing method. 8. An image capturing step of capturing an image of a subject, an extracting step of extracting two points that match a predetermined pattern from the image captured in the image capturing step, and a step between the two points extracted in the extracting step. Image processing, comprising: a first calculation step of calculating a distance between two points; and a second calculation step of calculating an angle of the subject using the distance between the two points calculated in the first calculation step. A providing medium for providing a computer-readable program to be executed by an apparatus.