JPH1042273A - Three-dimensional position recognition utilization system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide the respective kinds of three-dimensional position recogni tion utilization systems for effectively utilizing the depth information after proposing a technique for accurately obtaining the depth of a subject from two-dimensional moving images (that are moving images photographed from a single-eye camera) or three-dimensional images (that are images photographed from a stereo camera) first. SOLUTION: The movement information on a screen of the subject is extracted from input video images (process 1). At this point, the method of block matching or the like is used. Then, the real movement of the subject in a three-dimensional space is calculated (process 2). Since the input video images are the projection of original three-dimensional movement, the movement of three-dimensional display is obtained from the movement of a plurality of representative points by inverse transformation. As a result, sine the coordinate of the subject is discriminated, the depth information of the subject is acquired (process 3). By using the depth, the respective kinds of systems are constructed (process 4).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system for recognizing a three-dimensional position of an object, and more particularly to a system for recognizing the three-dimensional position of a subject projected on a two-dimensional image taken from a monocular camera or a three-dimensional image taken from a stereo camera. It relates to various systems that recognize and use this.

[0002]

2. Description of the Related Art In the field of television technology, there has been conventionally known a technique for generating a three-dimensional image, that is, a pseudo three-dimensional image based on the depth of a two-dimensional image. As an example of this, Japanese Patent Publication No. 55-36240 discloses a stereoscopic image display device using externally provided depth information. In addition, magazine PIXEL (No. 128) (1
A pseudo three-dimensional image system using depth information has been proposed on pages 97 to 102 of May 1, 993). Furthermore, Japanese Unexamined Patent Publication No. Hei 4-504333 (WO8)
8/04804) also discloses a method of implementing pseudo stereoscopic vision using depth information.

[0003] The idea of generating depth information from the grasp of the correspondence between frames is also known. For example, in Japanese Patent Application Laid-Open No. 7-71940, as “conventional technology”, (1) a point or a line is associated between two images captured by a stereo camera, and an actual scene space (three-dimensional space) is obtained. Technology for estimating the positions of points and lines in a computer, (2)
It has been pointed out that there is a technique of continuously photographing a subject while moving a camera and tracking feature points on an image to estimate an actual position of each feature point in a scene space.

[0004]

As described above, a technique for generating a pseudo three-dimensional image using depth information is known, but depth information is used for image processing other than generation of a pseudo three-dimensional image. There are not many suggestions.

Accordingly, an object of the present invention is to first provide a two-dimensional moving image (ie, a moving image taken from a monocular camera) or a three-dimensional image (ie, an image taken from a stereo camera).
It is an object of the present invention to provide a technology for accurately obtaining the depth of an object, that is, a subject, and to provide various three-dimensional position recognition systems that effectively use the depth information.

[0006]

[Means for Solving the Problems]

(1) The three-dimensional position recognition and use system of the present invention is a system for recognizing and using the three-dimensional position of an object, comprising extraction means for extracting the depth of a photographed object, and also using the extracted depth. And guard.

According to this configuration, when an object is photographed, the depth of the object is extracted by the extracting means.
The extraction means may be configured in any form such as hardware, software, or a mixture thereof. “Extraction” includes concepts such as detection and calculation, and refers to recognizing or specifying depth. If the depth of the object is known, the movement of the object can be known, and this is used for security.

(2) In the case of (1), a predetermined process may be performed when an object approaches. As an example of the predetermined processing, the object is enlarged and photographed, a warning is issued to a security official, the lighting of a security area is brightened, and any processing is performed as long as it is processing related to security. Is also good.

(3) In another system of the present invention, preprocessing for segment matching in computer vision technology is performed based on the extracted depth.
“Segment matching” refers to an image comparison process performed by focusing on some characteristic of the region in order to grasp the correspondence between the regions. “Preprocessing” refers to preparation for smoothly performing segment matching.

(4) In the case of (3), the preprocessing may be a process of narrowing a search area in segment matching using the depth of an object. That is, in the segment matching, an image region corresponding between two frames is to be found, but at this time, a region likely to correspond can be estimated in advance from the depth. Based on this estimation, the area for which the correspondence relationship is to be searched is narrowed down.

(5) In another system of the present invention, an instruction from a user is recognized based on the extracted depth and accepted. Since the user's instruction is considered to involve some operation, the content of the instruction is specified based on the depth of the place where this operation occurs.

(6) In the case of (5), the movement of a portion having the smallest depth in the photographed object may be determined as a user's instruction and accepted.

(7) In still another system of the present invention,
Display control of the pseudo stereoscopic image is performed based on the extracted depth. As an example of display control, there is adjustment of a position at which a pseudo stereoscopic image is correctly perceived.

(8) In the case of (7), it is further determined that there is a viewer of the pseudo stereoscopic image at the position of the photographed object,
Display position control may be performed so that the pseudo three-dimensional image is favorably displayed at this position. In particular, if the position of the viewer's head is specified in combination with image processing techniques such as contour detection together with the depth, it is possible to control such that the pseudo three-dimensional image is displayed most effectively at that head position. is there.

[0015]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described with reference to the drawings.

In a first embodiment, a technique will be described in which depth information is extracted from a two-dimensional moving image captured by a monocular camera and is used as a distance sensor of a security system.

In the second embodiment, the same processing as in the first embodiment is performed based on an image from a multi-lens camera, that is, a stereo camera. In the third to fifth embodiments, depth information is extracted as in the first and second embodiments, and is used for controlling computer vision, a user interface of a computer system, and pseudo three-dimensional display, respectively.

Embodiment 1 FIG . In the present embodiment, first, a technique for generating a correct stereoscopic image based on depth information by applying a computer vision technique to an image processing field such as a television field will be described.

In the present embodiment, the motion of a two-dimensional moving image is detected, the three-dimensional motion between the scene of the moving image and the photographing viewpoint is calculated from the motion, and the relative three-dimensional motion and each part of the image are calculated. Depth information is derived by calculating a relative distance from the shooting viewpoint to each part of the image based on the movement.

In other words, this technique selects a plurality of video frames (hereinafter simply referred to as “frames”) from a two-dimensional moving image to be processed, and determines a two-dimensional position between these frames. The relative positional relationship occupied by each part of the image in the actual three-dimensional space is derived from the displacement of the image, and the depth is determined according to the result. That is, the three-dimensional movement of each part of the image is calculated from the displacement of the two-dimensional position, and the position coordinates of each part of the image in the three-dimensional space are calculated from the movement by the principle of triangulation. Determine the depth according to Here, the “frame” is a unit of image processing, and is a concept including, for example, a frame picture and a field picture in MPEG.

In the case of a two-dimensional moving image, since the "plurality of video frames" are frames shot at different times, they are hereinafter referred to as "different time frames" and are simultaneously shot by a multi-lens camera described later. A plurality of frames, that is, “simultaneous frames” are distinguished. "Two-dimensional displacement of position" refers to displacement of a position on a frame plane.
In the first embodiment, since the different time frame becomes a problem,
"Two-dimensional displacement of position" refers to a change in position over time, that is, movement. It should be noted that, in the case of the same-time frame described later, the “two-dimensional position displacement” indicates a position shift between a plurality of frames.

FIG. 1 is a diagram showing main steps for generating a three-dimensional display image according to the present embodiment. In the present embodiment, depth information is extracted from the original two-dimensional image according to the following steps, and is used. Here, steps 1 to 3 relate to the extraction of depth information by analyzing a video, and step 4 relates to its use, particularly in the first embodiment for use in a security system. First, the outline of each step will be described.

[Step 1] Extraction of two-dimensional motion information The motion information of the subject contained in the video is extracted. This motion information is two-dimensional information. The display screen is set on a coordinate plane, and the movement of the subject on the screen is described in two-dimensional coordinates.

In this step, the correspondence between video frames is detected in order to grasp the movement of the subject. A plurality of representative points are set in advance in a video frame at time t (hereinafter referred to as “frame t”) which is a reference frame, and a different time t ′
The corresponding points of the respective representative points in the target frame (hereinafter referred to as “frame t ′”) are tracked. Frames t and t '
Form inter-time frames, which need not be temporally adjacent frames. The feature of this process is
The point is that two-dimensional motion information can be extracted from the motion not only in the horizontal direction but also in any direction of the subject. In this specification, hereinafter, t and t 'each indicate time.

In the present specification, the term "frame" is hereinafter referred to as "frame".
Broadly refers to the overall video composition unit such as the field,
For example, it does not refer to only one frame of a television receiver composed of 525 scanning lines or one screen of a personal computer composed of 640 × 480 pixels. The representative point may be set not only in the frame t but also in both the frames t and t ′.

[Step 2] Calculation of Three-Dimensional Motion Information When the two-dimensional motion of the subject is determined, actual motion information of the subject in the three-dimensional space (hereinafter, also referred to as three-dimensional motion information) is calculated. At this time, by taking a large number of pairs of representative points and corresponding points, the motion actually caused by the subject is described by both translational and rotational motion components.

[Step 3] Acquisition of Depth Information If the actual state of movement of the subject is known, the relative positional relationship of the subject at each time can be determined. If this relationship is known, depth information of the subject or each part thereof (hereinafter, also simply referred to as “each part of the image”) is obtained.

[Step 4] Use of Depth Information A security system is constructed based on the depth information. For example,
If there is an object approaching a certain place within a certain distance, processing such as issuing a warning is performed.

The above is an outline. Hereinafter, each step will be described in detail.

[Step 1] Extraction of Two-Dimensional Motion Information FIG. 2 is a flowchart for detecting the correspondence between video frames. Each step shown in the figure will be described.

(S10) Setting a representative point in the frame t As shown in FIG. 3, first, a representative point is set in the reference frame t. In the figure, a frame t is divided by a mesh every 8 × 8 pixels, and each representative point is placed at a grid point. Here, the i-th representative point from the left and the j-th representative point from the top are Pt
(I, j), and the time t 'with respect to Pt (i, j).
Is described as Pt '(i, j). Also, if necessary, the x and y coordinates of Pt (i, j)
(I, j) x and Pt (i, j) y.

In this step, the representative points are not limited to the lattice points, but may be arranged at arbitrary positions. In extreme cases, all pixels can be used as representative points.

(S11) Setting of Corresponding Point Candidate Area For example, in consideration of a representative point Pt (6, 4) shown in FIG. 3, an area where Pt ′ (6, 4) can exist is set in advance. This is based on the assumption that Pt '(6,4) is in the vicinity of Pt (6,4), except when the motion of the image is more abrupt than a certain limit. In the present embodiment, for example, Pt (6,
It is assumed that Pt '(6, 4) is included in the area of 100 × 60 pixels in the vicinity of 4), and the amount of calculation at the time of detecting Pt' (6, 4) is reduced.

The following application is possible for this step.

1. When the image is moving relatively violently, t 'is determined so that frames t and t' are adjacent to each other. The change in the position of the representative point is suppressed to a minimum, and the possibility that the corresponding point does not enter the area is also minimized. However, as a matter of course, a method in which the candidate area is set to the entire screen in advance may be considered. In that case,
Although the amount of calculation increases, the possibility that a corresponding point is missed due to a large motion of the image decreases.

2. In the present embodiment, it is simply assumed that Pt ′ (6, 4) is near Pt (6, 4).
When the movement trajectory in a plurality of frames of t (6, 4) is found, it is possible to determine a candidate area on an extension of this trajectory. If the motion of the video is constant to some extent, narrowing down the corresponding point candidate areas by this method is very effective.

(S12) Calculation of Dissimilarity in Corresponding Point Candidate Area Next, the position of the corresponding point is specifically determined from the candidate area. However, in this step, contrary to the previous step, a problem occurs when the motion of the video is too slow. This is because if the movement is small, it becomes difficult to extract the motion information, and the extracted information may include a large error.

In such a case, t 'is selected in advance so that the frames t and t' are separated to some extent. At this time, the amount of change in each part of the image is statistically processed, and for example, t 'may be selected such that the magnitude of the change or the variance of the amount of change exceeds a predetermined value. As another method, even if t 'is selected such that the sum of the motions of the feature points (described later) exceeds a predetermined value, or t' is such that the variance of the motion of the feature points exceeds a predetermined value. Good.

In this step, the dissimilarity is calculated by block matching between frames t and t 'in order to determine the position of the corresponding point. The sum of the squared errors of the density, that is, the dissimilarity between the neighboring block around a certain point in the corresponding point candidate area and the neighboring block around the representative point, is calculated. Determine the point.

FIG. 4 is a diagram showing the state of block matching. In this embodiment, nine pixels are defined as one block, and the central pixel is a representative point.

In the figure, first, Pt is placed on frame t.
A block 1 containing (i, j) is taken, and a block 2 containing a tentative candidate Pt '(i, j) for the corresponding point is taken on the frame t' side. Here, generally, if the pixel value of the pixel (x, y) at time t is described as It (x, y), the dissimilarity (E1
)

E1 = ΣΣ ｛It (Pt (i, j) x + u, Pt (i, j) y + v) -It '(Pt' (i, j) x + u, Pt '(i, j) y + v)｝ ² (Equation 1) Here, two Σs relate to u and v. These take values u = -1,0,1 v = -1,0,1 respectively, and the sum of square errors for a total of 9 pixels can be calculated for the temporary Pt '(i, j). Then Pt '(i,
j) may be moved little by little in the candidate area, and a point at which E1 becomes minimum may be determined as a corresponding point.

FIG. 5 is a schematic diagram showing the value of E1 in the vertical direction for each Pt '(i, j). In this case, the point Q at which the dissimilarity has a steep peak is determined as the corresponding point. . Thereafter, corresponding points are similarly determined for other representative points.

This step has the following applications or modifications.

1. Here, the density 2
Although the squared error has been calculated, in the case of a color image, the sum of squared errors of each density of RGB, that is, E1 _R + E1 _G + E1 _B may be used as the dissimilarity. This is the density from other color spaces,
For example, it may be an HVC concentration. Instead of the square error, a simple absolute value of the error, that is, the sum of the residuals may be employed. 2. In this step, one block has 9 pixels, but it is usually desirable to define a block by a relatively large number of pixels. For example, when assuming a high-resolution screen of an ordinary personal computer or workstation, as a result of an experiment, good results have been obtained with a block of about 16 × 16 pixels.

(S13) Determination of Initial Position of Corresponding Point In the previous step, the corresponding point could be determined at first. However, at this stage, the position of the corresponding point is not always correct. Although the corresponding points can be found relatively correctly for the boundaries and edges of the subject, the position of the corresponding points should be considered to contain a considerable error, for example, in an image part with little change. In FIG. 5, it may be said that E1 does not take a clear peak. FIG. 6 is a diagram showing the relationship between the corresponding points and the representative points obtained as a result of the previous step. As shown in FIG. 6, the corresponding points are satisfactorily obtained for characteristic points such as a house and a tree, especially their contours. However, the error is large for the sky and the ground.

In this step and the next step, the position of the corresponding point is corrected. In this step, first, the concept of the initial position of the corresponding point is introduced, and the initial position is actually determined. Next, in the next step, the position accuracy is improved by repeated calculation.

The following policy can be considered for determining the corresponding point initial position in this step.

1. Treat all the corresponding points found in the previous step equally. The positions of all the corresponding points are input to the next step as their initial positions.

2. Provide a difference in handling of corresponding points The positions of corresponding points (hereinafter referred to as “feature points”) that seem to be in a correct position to some extent from the beginning are used as initial positions as they are, and corresponding points that are not (hereinafter referred to as “non-feature points”) Is determined based on that of the feature points. Here, the following points can be assumed as the feature points, but these points often coincide in reality. Note that the original representative points corresponding to these corresponding points are also referred to as feature points in this specification.

(1) Corresponding point where E1 in the previous step shows a clear peak This is because the positional accuracy of such a corresponding point is generally high.

(2) Corresponding point at a position where many orthogonal edge components exist This is because the position of the corresponding point seems to be fairly correct in a portion such as a corner of a building. (3) Corresponding points whose positions are stably changing in frames t, t ',... Here, the stability of the change can be said to be the uniformity of the motion vector. Here, a corresponding point whose moving direction and moving distance are constant as the frame progresses is selected. Specifically, for example, a corresponding point whose variation of the motion vector is equal to or less than a predetermined value is selected. This is because such corresponding points should be accurately tracked, and it can be determined that they have a correct correspondence with the representative points. However, for example, when the camera that shoots the video moves irregularly, the determination is made in consideration of the influence.

When feature points are selected in this way, they can be used as initial positions as they are, and the initial positions of non-feature points can be determined by interpolating the positions of the feature points or sequentially starting from the vicinity of the feature points. That is, since the position accuracy of the non-feature point in the previous step is low, the initial position of the non-feature point is given geometrically from the highly accurate feature point. Of course, the method of the previous step can also be used effectively when finding the feature point of (3).

The method of determining the corresponding point initial position based on the selection of the feature point has been described above. Alternatively, the initial value of the corresponding point may be obtained by using a dynamic programming method (dynamic programming).

(S14) Corresponding point improvement processing An equation is introduced to evaluate the validity of the position of the corresponding point, and the position accuracy is improved by iterative calculation. In step S12, equation 1 for evaluating the degree of dissimilarity is introduced. Here, an equation for evaluating the validity of the relative positional relationship between corresponding points is introduced, and these two evaluation results are integrated to improve the position. Plan.

FIG. 7 is a diagram for explaining the principle of evaluating the relative position. In the figure, each point represents a corresponding point. Considering Pt '(i, j) in the figure, the following four corresponding points

## EQU2 ## Pt '(i-1, j), Pt' (i + 1, j), Pt '(i, j-
1) and Pt '(i, j + 1) are adjacent. It is reasonable to consider that Pt '(i, j) usually lies near the centroid of these four points. This is based on an empirical rule that, even when each part of the image moves, the relative positional relationship is substantially maintained if viewed microscopically in pixel units. Note that mathematically speaking this property, (i, j)
The function Pt '(i, j) of the above is substantially zero.

Therefore, the center of gravity of the above four points is defined as (St '(i, j) x, S
t '(i, j) y)

Equation 3] E2 = {Pt '(i, j) x-St' (i, j) x} 2 + {Pt '(i, j) y-St' (i, j) y} 2 ( Formula 2 ) Is the formula for evaluating the validity of the relative position. Considering only this equation, the position of the corresponding point becomes the most desirable state when E2 becomes the minimum value.

In this step, the evaluation results of Expressions 1 and 2 are added with an appropriate coupling constant k, and E represented by E = E1 / N + kE2 (Expression 3) is used as the final evaluation expression ( N is the number of pixels included in one block defined at the time of block matching). That is, first, E is calculated for each corresponding point, then the sum ΣE of E of all corresponding points is calculated, and the position of each corresponding point is changed little by little so that ΣE becomes the minimum value. The improvement processing is performed until the value of ΣE converges or until the repetition calculation reaches a predetermined upper limit number. More specifically, when changing the position of each corresponding point, one of the following methods may be performed.

(1) Method of Solving the Euler Equation Corresponding points are obtained by numerically solving the Euler equation showing the condition that ΣE is an extreme value, here a minimum value. This technique itself is known. This is because the direction to be improved is found from the image inclination information in the block including each representative point and the pixel difference information between the corresponding blocks, and based on this, the position of the corresponding point is gradually moved from the initial position. Find a solution.

(2) Fixed Search Method First, in the corresponding point candidate area, the E of the corresponding point to be improved is
Find a point where is minimum, and use this as a new corresponding point. At this time, it is characterized in that the search is performed by regarding the positions of other points as immovable. This process is sequentially performed on all corresponding points.

(3) Mixing Method According to the method (2), the position of the corresponding point can be determined with the accuracy of the pixel unit. On the other hand, according to (1), the position can be theoretically obtained with an accuracy of a pixel unit or less. Therefore, it is also possible to first obtain the correspondence with the accuracy of the pixel unit by the method (2), and thereafter to improve the accuracy by applying the Euler equation.

According to the experiment, when compared with the same precision, a preferable solution is obtained in a shorter time than the method (1) by the method (2).

FIG. 8 is a diagram showing the result of performing the improvement processing of this step on the corresponding point candidates in FIG. According to an experiment, in the case of a color image, k was about 5 to 200, and a good result was obtained. FIG. 6 and FIG. 8 are schematic diagrams, but as a result of the experiment, improvements close to those in these figures were actually observed.

The above is the details of the step 1. The feature of this step is that two-dimensional motion information can be extracted from the motion of the subject in any direction. This is an advantage of grasping the movement based on the concept of the representative point and the corresponding point, and enables a wider application than the conventional technique of detecting the movement in the horizontal direction and determining the time difference.

This step has the following applications or modifications.

1. When deriving E2, consider not only the four points in the up, down, left, and right, but also the center of gravity of a total of eight points including four points in the oblique direction.
Which combination is optimal depends on the type of video, so it is desirable to appropriately determine the combination by experiment.

2. The evaluation based on Expression 3 is preferentially performed from the corresponding point where the evaluation result based on only E1 is not good. This is because a corresponding point having a poor result of E1 is generally considered to have a large position error, and it is desirable to improve the position of such a corresponding point early and significantly.

3. Geometric information is also used for position improvement.
For a region having a geometric characteristic in the frame t, for example, a plurality of representative points that have formed a straight line, the positions are corrected so that their corresponding points also form a straight line. This is because a portion that appears to be a straight line on an image is likely to be a straight line even in the actual three-dimensional space, while a straight line in the three-dimensional space should be a straight line even in the frame t ′. Originally, the depth changes uniformly along the straight line, and the change along the straight line can be easily grasped visually, so that the improvement effect by this method is great. The other geometric information may be an edge of an image area.

4. A corresponding point is obtained for another frame. In this step, the frame t 'with respect to the frame t
Are obtained, the corresponding points in the third frame t '' are also obtained, and the averaged motion of each part of the video can be obtained. This method does not improve the position of the corresponding point in the frame t '. By taking corresponding points in many frames, the movement of each part of the video is statistically determined from the positions of the corresponding points and the time at which the frames were taken.

[Step 2] Calculation of Three-Dimensional Motion Information In step 1, two-dimensional motion of each part of the image on the screen was determined. In step 2, a three-dimensional movement of each part is calculated from this information. The image is obtained by projecting the actual movement of the subject on a plane, and in this step, the original movement is derived from the positional relationship between the representative point and the corresponding point.

In general, the motion of a subject in a three-dimensional space can be described as a combination of a translational motion and a rotational motion. Here, first, a calculation method in the case where the motion is composed of only translational motion will be described, and then a generalized method will be outlined later.

1. FIG. 9 is a diagram showing the correspondence between the movement of a certain point P on the screen and the actual movement in a three-dimensional space. In the figure, the two-dimensional coordinates on the screen are represented by capital letters X and the like, and the actual three-dimensional coordinates are represented by small letters x and the like. The x and y axes of the three-dimensional coordinates are represented on the screen and the z-axis is represented in the depth direction. I have. Further, the distance from the viewpoint to the screen is set to 1.

As shown in this figure, P (X, Y) moves on the screen to P ′ (X ′, Y ′), during which point, in the three-dimensional space, S (x, y, z) To S (x ', y',
z '). here,

If (x ′, y ′, z ′) = (x, y, z) + (a, b, c), the distance to the screen is 1, so that X = x / z, Y = y / z X ′ = x ′ / z ′ and Y ′ = y ′ / z ′. If this is solved, X '= (Xz + a) / (z + c) Y' = (Yz + b) / (z + c), so z is eliminated and the following equation is obtained.

[0073]

(A−X ′ c) (Y′−Y) = (b−Y ′ c) (X′−X) (Equation 4) Since Equation 4 is represented by motion information on the screen, Step 1
The unknowns a, b, and c can be determined based on the information obtained in (1). However, in this case, in reality, in the case where an object having a size of k times moves at a speed of k times at a position separated by k times, the value of k (scale factor) cannot be determined. , c, it is possible to determine only their ratio. Mathematically speaking, even if three sets of (X, Y) and (X ', Y') are given, the rank (order) of the coefficient matrix when this simultaneous equation is represented as a matrix is 2 at most, a, b, and c can be determined only as relative values. Therefore, in this step, a and b are temporarily represented by normalizing c = 1. This is because the process in the next step can be performed using only the ratio.

As another solution to the translation, the error e from

E = e (a-X′c) (Y′-Y) − (b-Y′c) (X′-X)｝ ² = ｛(Y′-Y) a- (X′- X) b− (XY′−X′Y) c｝ ² (Equation 5), and the sum Σe of e is obtained for all the correspondences between the representative points and the corresponding points, and a, b, c may be calculated from the following equation.

D (Σe) / da = 0 (formula 6) d (Σe) / db = 0 (formula 7) d (ｄe) / dc = 0 (formula 8) More specifically, formulas 6 to 8 are Each is expanded to the following form.

[0076]

７ (Y′-Y) ² -bΣ (X'-X) (Y'-Y) -cΣ (Y'-Y) (XY'-X'Y) = 0 (Equation 9)- a Σ (X'-X) (Y'-Y) + bΣ (X'-X) ² + cΣ (X'-X) (XY'-X'Y) = 0 (Equation 10) -a Σ (Y '-Y) (XY'-X'Y) + b Σ (X'-X) (XY'-X'Y) + c Σ (XY'-X'Y) ² = 0 (Equation 11) It is an example of the calculation method regarding exercise.

2. When Motion Includes Rotational Motion Rotational motion can be described by three displacements in the x, y, and z directions and three angles of rotation about each axis, for example, α, β, γ. The rotation angle can be expressed by an Euler angle or a roll pitch method.

Here, the total of six variables may be determined. However, since the scale factor is not determined as described above, a certain variable is set to 1 and the ratio of each variable is obtained.
Theoretically, the movement can be described if there are five sets of representative points and corresponding points.

It should be noted here that, depending on how the pairs are set, the state of the movement may not be obtained by the linear solution method. When such a case is considered, it is known that the number of sets should be eight or more. For the grounds that can describe the rotational motion by the linear solution from the eight sets of changes, see, for example, “About the linear solution of shape recognition by monocular stereoscopic vision from motion” (Exit / Akiba, Transactions of the Society of Instrument and Control Engineers vol.26,
No. 6, 714/720 (1990)).

[Step 3] Acquisition of Depth Information In step 2, the relative amount of three-dimensional movement of each part of the image was determined. In step 3, depth information of each part is derived from the relative amount. In this step, for the sake of explanation, it is assumed that the subject is stationary and the camera that captures the subject moves. Since the relative motion between the subject and the camera becomes a problem during image processing, a good result can be obtained by this assumption.

The motion of a certain part of the image is represented by (x ′, y ′, z ′) = R (x, y, z) + (a, b, c) using the rotation matrix R and the translation vector (a, b, c). c) this inverse transformation,

(X, y, z) = R ⁻¹ {(x ′, y ′, z ′) − (a, b, c)} (Equation 12) is considered as the motion of the camera.

FIG. 10 is a diagram for explaining the principle of deriving the three-dimensional coordinates of the point P from the three-dimensional movement of the camera and the movement of the point P on the screen. As can be seen from the figure, this principle is generally known as the principle of triangulation,
When the direction of the point P is viewed from the point, the actual position of the point P (point S in the figure) exists at the intersection of these two lines of sight.

In the figure, it is assumed that the camera has moved in accordance with Expression 12 between the times t and t 'as shown by arrows. In frame t, point S is projected on point Pt, and in t ', point S is projected on point Pt'. Point S is located at the intersection of two straight lines Lt and Lt 'in the figure.

Here, the angles θt and θt ′ formed by the direction of the camera and Lt and Lt ′ are known, while the moving direction and the distance of the camera are known, so that the three-dimensional coordinates of the point S can be obtained. Becomes From these coordinates, the depth information of each part of the image is determined.

It should be noted that c = 1 as described above.
For this normalization, the obtained coordinates are also expanded or compressed at a constant rate. However, also in this case, the depth information is correct because the depth information is uniformly expanded and compressed.

The above is the outline of the present step. In this step, it is necessary to consider an error up to the previous step. Due to the error,
This is because Lt and Lt 'do not normally intersect in calculation. In consideration of such circumstances, in this step, the z coordinate of the midpoint between the closest points of both straight lines is approximated to the depth value of the point S. This will be described using mathematical expressions.

The direction vectors of Lt and Lt ′ are (u, v, w) and (u ′, v ′, w ′), respectively. Here, both straight lines are expressed by the real number parameters α and β as follows: Lt: (x, y, z) + α (u, v, w) Lt ′: (x ′, y ′, z ′) + β (u ′, v ′, w ′) (Equation 13). Therefore,

## EQU9 ## e = {(x + βu)-(x ′ + αu ′)} ² + {(y + βv)-(y ′ +
αv ′)} ² + {(z + βw) − (z ′ + αw ′)} ^2, and α and β that minimize e are de / dα = 0, de /
It is determined from dβ = 0. That is,

(U ² + v ² + w ² ) α- (uu '+ vv' + ww ') β + (x-x') u + (y-y ') v + (z-z') w = 0 (u ' ² + v' ² + w ' ² ) β- (uu' + vv '+ ww') α + (x-x ') u' + (y-y ') v' + (z-z ') w' = 0 to obtain α and β, and finally the depth value of the point S,

[Equation 11] {(z + αw) + (z ′ + βw ′)} / 2 Here, if the error is 0, this coordinate coincides with the z coordinate of the intersection of both straight lines.

As another method, these two straight lines are once perspectively projected on the screen of the frame t, and the projected closest point z is projected.
Coordinates can also be determined. Here, Lt is projected to one point which is a representative point, while Lt 'is generally projected to a straight line. If Lt ′ is expressed by Equation 13, the x, y coordinates of each point after projection can be obtained by dividing those of each point on Lt ′ by its z coordinate, as x = f (x ′ + βu ′). ) / (z ′ + βw ′) (Equation 14) y = f (y ′ + βv ′) / (z ′ + βw ′) (Equation 15) Here, f is the distance from the viewpoint to the screen of the frame t, and may be handled as f = 1 in practice. If β is eliminated from Expressions 14 and 15, a straight line after projection (hereinafter referred to as Li) is obtained as follows.

Kx + my + f n = 0 where k = v'z'-w'y ', m = w'x'-u'z', and n = u'y'-v'x '.

The closest point to be obtained is L from the representative point Pt.
It is the intersection (hereinafter referred to as D) of the perpendicular drawn to i and Li, and its coordinates are x = (m ² X-kn-kmY) / (k ² + m ² ) (Equation 16) y = (k ² Y-mn-kmX) / (k ² + m ² ). Here, a point on Lt ′ corresponding to point T is represented by E
If (x '', y '', z ''), the point E is obtained by substituting equation 16 into equation 14 to obtain β, and substituting this into the equation Lt '. Here, β is β = (xz′−fx ′) / (fu′−xw ′). Therefore, this is substituted into Expression 13 to obtain the z coordinate z ″ of the point E.
Is obtained as z '' = z '+ w'(xz'-fx') / (fu'-xw'). This may be used as the depth value of the point S.

If the above-mentioned depth has a negative value due to an error in the image processing (when the point S is located behind the camera), the calculation result cannot be relied on. At this time, processing such as interpolation from a nearby representative point having a positive depth value is performed.

[0092] Regardless of which method is used,
The obtained depth of each part of the image may be given as a numerical value for each representative point, for example. FIG. 11 is a diagram showing a state in which a numerical value is given to each representative point in a frame t.
t (2, 3) and Pt (4, 3) each have a depth of 10
0 and 200, indicating that the actual position of the latter is twice as far as the former.

[Step 4] Use of Depth Information A security system is constructed according to the depth information obtained in Step 3. As an example, a system using a monitoring camera and a PC can be considered. That is, a surveillance camera is installed in an area where a person or any other object is prohibited from approaching, and a video taken by the surveillance camera is sent to a PC for analysis to extract the depth of the subject. Processing such as issuing a warning sound via a PC, notifying a security guard, brightening the lighting of the area, or recording an image of the subject when the depth of the subject falls below a predetermined value due to approaching the subject. I do.

The security system includes a system for measuring the distance from the reflection of a target object by applying a laser or ultrasonic wave to the object. In that case, a scanning mechanism for shaking the laser or the like is required. In the present embodiment, such a scanning mechanism is unnecessary. Other security systems detect infrared light from objects, but cannot detect objects that do not produce heat. In this sense, the present embodiment, which is not related to heat, is advantageous.

In the security system, for example, only an image area of an object approaching within a certain distance D can be cut out from the entire image. This is from the whole image,
What is necessary is just to select an area whose depth is within D. Processing such as enlarging the region cut out in this way or observing the movement following the movement of the region is also possible.

Embodiment 2 In the first embodiment, it is assumed that the input video is captured by the monocular camera. here,
A case in which a stereo image from a multi-lens camera is used as an input image will be described focusing on differences from the first embodiment.

FIG. 12 shows main steps of the second embodiment. The main differences between FIG. 1 and FIG. 1 of the first embodiment are as follows.

1. "Motion information" in step 1 is "displacement information"
The first embodiment deals with the different time frame, but the second embodiment
So basically we deal with the same time frame. In the case of the same time, since movement cannot be defined for the subject, information on the displacement of the subject between the frames at the same time, that is, information on the displacement is extracted instead.

2. Step 2 becomes unnecessary There is no step corresponding to step 2 “calculation of three-dimensional motion information” in FIG. This is because, in the case of a multi-view, since the shooting is performed in the state of FIG. 10 from the beginning, depth information can be directly obtained by the principle of triangulation.

When using a multi-lens camera system in which the relative positional relationship between a plurality of cameras may be out of order, it is better to perform self-calibration to correct the out of order. In this case, step 2 is used as a self-calibration step. For the self-calibration method, see, for example, Tomita and Takahashi "Self-calibration of stereo camera" (Information Processing Vol.31, No.5 (1990)
650-659), JP-A-02-138671, JP-A-02-138672 and the like. Hereinafter, steps 1 to 3 of the second embodiment will be described.

[Step 1] Extraction of Two-Dimensional Displacement Information In the description of the first embodiment, in addition to replacing “movement” with “displacement”, a set of frames t and t ′ may be replaced with frames 1 and 2. Frames 1 and 2 respectively indicate images taken from left and right cameras 1 and 2 constituting a stereo camera, and the shooting time is fixed at t. In the second embodiment,
The final image can be obtained only from at least these two frames. That is, in the case of multi-view photography, the input may be a still image. In addition, the differences from Step 1 of Embodiment 1 are as follows.

(1) In S11 (setting of corresponding point candidate area) in the first embodiment, selection of a different time frame or narrowing of the corresponding point candidate area is performed based on the intensity of the motion of the image or the movement trajectory of each part. The calculation amount of point detection processing was reduced. In the second embodiment, the method of narrowing down is changed as follows, and an effective calculation amount is similarly reduced.

First, it is assumed that the multi-lens camera is installed horizontally as usual. At this time, the y-coordinates (vertical coordinates) of the corresponding points are substantially equal. Considering this assumption, an error associated with image processing, and a camera installation error, the corresponding point candidate area is limited to a horizontally long band-like area. Further, the frame t '
If the difference between the positions of the corresponding representative points at (t ′ = t−1) is x, the corresponding point search area in the frame t can also be limited to the neighborhood where the difference is also x.

(2) In S12 (calculation of dissimilarity in the corresponding point candidate area) of the first embodiment, statistical processing is introduced when the motion of the video is too slow. However, this operation is not required in the second embodiment. .

(3) As in S12 of the first embodiment, block matching is performed in the second embodiment to determine the position of the corresponding point. Here, it may be better to employ biased block matching. Biased block matching works effectively when the cameras constituting the multi-view camera have different characteristics. For example, assuming that the camera 2 transmits an image that is more bluish than the camera 1, after subtracting a certain amount of the blue (B) component from the color density of the frame 2, that is, after subtracting the color deflection constant α _B , Block matching should be performed. If such processing is not performed, there is a possibility that the meaning of Expression 3 for adding E1 and E2 may be lost. Actually, when the color density is represented by RGB, for example, the color deflection constants α _R and α _G should be subtracted not only for the blue (B), but also for the red (R) and the green (G). Biased block matching has a calibration function of matching the characteristics of two cameras when quantifying similarity between images. Biased block matching guarantees the adequacy of evaluating the similarity of images and the validity of the relative position at the same stage.

The biased block matching will be described using equations based on FIG. 4 and equation (1). Here, Pt (i, j) used in the first embodiment is simply expressed as P1 and P2 corresponding to frames 1 and 2, and It (i, j) is similarly expressed as I1 and I2. At this time, Equation 1 is

[Equation 12] E1 = {I1 (P1x + u, P1y + v) −I2 (P2x + u, P2y + v)} ² (Equation 17) This expression represents normal block matching for a grayscale image.

On the other hand, in biased block matching, equation 17 is obtained by

E1 = {I1 (P1x + u, P1y + v) −I2 (P2x + u, P2y + v) −α} ² (Equation 18) In the case of a color image, α is α _R , α _G or α _B , and E obtained in each of the RGB images is
Matching is performed with the sum of 1, that is, E1 _R + E1 _G + E1 _B. Further considering the legibility, if I1 (P1x + u, P1y + v) is simply expressed as I1, and I2 (P2x + u, P2y + v) is simply expressed as I2,
Equation 18 is given by E1 = ΣΣ (I1-I2-α) ² (Equation 19). I1 and I2 are functions of u and v, while α is a constant.

Consider the optimal value of α. Since the cameras 1 and 2 should have photographed the same subject, the images of the frames 1 and 2 include substantially the same contents except for the displacement of each part of the image. That is, as the characteristics of the camera become closer, E
The value of 1 becomes smaller. Conversely, from this fact, it can be seen that α should be a value that minimizes E1. Equation 19 is

Equation 14] E1 = ΣΣ {(I1-I2 ) 2 -2 α (I1-I2) + α 2} = ΣΣ (I1-I2) 2 -2 αΣΣ (I1-I2) + ΣΣα 2 ( Equation 20) area total of Assuming that the number of pixels is N, since ΣΣ1 = N,
Equation 20 is

[Equation 15] E1 = {(I1-I2) ² −2 α} (I1-I2) + Nα ² (Equation 21) Therefore, since dE1 / dα = −2ΣΣ (I1-I2) + 2Nα, when α = {(I1-I2)} / N (Equation 22), E1 becomes the minimum. This α is rephrased as the average value of the color density difference of each pixel between the two regions to be subjected to block matching. Substituting equation 22 into equation 21 and calculating,

## EQU16 ## Since E1 = {(I1-I2) ² -{(I1-I2)} ² / N (Equation 23), Equation 23 can be calculated in biased block matching. According to the adoption of Expression 23, if cameras 1 and 2 capture completely the same subject, E1 becomes almost zero. On the other hand, at this time, since E2 is also substantially equal to 0 from Equation 2, biased block matching has an effect of matching the origin of the determination of the video similarity with the determination of the validity of the relative position. Thereafter, the best matching may be searched through the same processing as in the first embodiment.

Here, of course, RG such as HVC concentration
A density in a color space other than the B density may be adopted. Block matching may be performed based on a first-order error, that is, a residual error instead of a square error. If the value of the correction value α given by Expression 22 exceeds a certain range, the biased block matching may be stopped. Camera 1, 2
Even if the same subject is photographed, the angle of photographing is different. Therefore, even if the characteristics of these cameras are completely the same, the captured images naturally have some differences. If all this is corrected, the value of E1 becomes unnecessarily small,
There may be cases where a correct evaluation cannot be made.

When the biased block matching is stopped, the evaluation value based on the ordinary block matching may be used as the evaluation value of the video similarity. The value after the correction may be used as the evaluation value of the video similarity. The evaluation value in that case is calculated by the following equation.

E1 = ΣΣ (I1-I2) ^2- ｛ΣΣ (I1-I2
)｝ ² / N + Nx ² where x = | ΣΣ (I 1 −I 2) / N | −T.

(4) In S13 of the first embodiment (determination of the initial position of the corresponding point), the different time frame t,
At t ',..., a point whose position is stably changed is selected, but the selection criterion is weighted here.

FIG. 13 is a diagram showing criteria for selecting feature points introduced in the second embodiment. In the figure, F10 to F1
3 are different time frames taken by the camera 1, and F20 to F22 are different time frames taken by the camera 2. Each set of two right and left pieces indicates the same time frame. Here, focusing on a certain point P, the movement of the position of the point P between different time frames is represented by
Are represented by a vector Bn (n: natural number).

Under the above settings, in Embodiment 2, points satisfying the following criteria are selected as feature points.

(A) Whether the vector Bn is substantially constant,
In addition to the above, a criterion of (b) that the vector An is almost constant or that it changes almost constant is added, and a point satisfying both (a) and (b) is selected as a feature point. Is also good.

(B) corresponds to the condition introduced in the first embodiment. As described above, in multi-view imaging, depth information can be obtained only from the same time frame. However, accurate understanding of the correspondence between videos as a premise is another problem, and information between frames at different times should be actively used. Points satisfying the above two conditions at the same time are considered to be tracked quite accurately, and thus provide important clues to extraction of two-dimensional displacement information. However, if the input is a still image, the corresponding point can be obtained by a known dynamic programming method (dynamic programming).

[Step 2] Acquisition of Depth Information Depth information of each part is derived from the displacement of each part of the video obtained in Step 1. In the case of multi-view, since the state of FIG. 10 is realized at a certain time t, depth information may be obtained by the method of step 3 of the first embodiment.

It should be noted here that since the positional relationship between the photographing cameras is fixed, if this relationship and the magnification or the focal length of the camera are known, the scale factor c determined to be undetermined in the first embodiment is also included. In the second embodiment, correct depth information is obtained.

[Step 3] Use of Depth Information Processing equivalent to step 4 (use of depth information) in the first embodiment may be performed.

Embodiment 3 In the first and second embodiments, a security system is constructed using the extracted depth information. However, in the third embodiment, the depth information is used for a part of the computer vision technology.

In a research field called computer vision, a technique for estimating a three-dimensional structure and a three-dimensional motion of a target is mainly studied for automatic control of a robot. Specifically, for autonomous running of the robot, there is a method of correctly grasping the distance to the object by photographing the object from a stereo camera or photographing the object while moving the monocular camera. Some aspects of this approach are described, for example, on page 57 of "1990 Image Coding Symposium (PCSJ90)".

When grasping the distance to an object in computer vision, segment matching is often performed. Segment matching is common to block matching in the sense that matching between image regions is performed, but differs from ordinary block matching in that matching is performed on some characteristic region of an image.
This is because the three-dimensional position of the object is specified with higher accuracy than the ordinary block matching by performing the processing on the characteristic region.

FIG. 14 is a diagram showing a method of performing segment matching on edge segments. FIG. 3A shows an image shot from the left camera among the stereo cameras, and the edge 120 is now a search target. on the other hand,
FIG. 2B shows the image of the right camera, which includes a plurality of edges 122 to 122.
126 are present. In the case of a real image, it is easy to imagine that there are many more edges. In the actual processing, first, a region having an edge is selected from the image of the right camera, and detailed segment matching is performed for each of these regions. In the case of controlling a robot, very high accuracy is required for recognizing the position of an object, so that the time required for segment matching is generally very long.

This embodiment aims to solve this problem. That is, first, steps 1 to 3 of the first embodiment are performed to obtain depth information of each part of the image. At this time, since it is not necessary to select a characteristic region such as an edge, the processing time is relatively short. If the depth is determined with a certain degree of accuracy in this way, it is possible to narrow down the search area by segment matching based on the depth information.

FIG. 15 is a diagram for explaining the principle of narrowing the search area. An edge 130 included in the image of the left camera shown in FIG. 9A is a search target edge, and this edge is to be searched for from the image of the right camera in FIG. Edge 1
Assume that 30 is located at x0 from the right end of the image of the left camera. Here, it is assumed that the cameras are installed in parallel.

On this assumption, the value of the depth z of the edge 130 mathematically uniquely determines at which position in the image of the right camera the edge 130 should be seen. For example, if z is infinity, there should be no parallax. Therefore, in FIGS. 7A and 7B, the image is seen at the same position separated by the camera interval, that is, at the position of x0 + 1 (1 is the camera interval on the image) from the right end of the video. become. This position is indicated by a straight line L1 in FIG. As z decreases from this position, the edge 130 gradually moves to the left in FIG. Here, z = 100, 5
Taking the values 0 and 10 as examples, straight lines L2, L3,
This is schematically shown at a position L4.

On the other hand, apart from such mathematical considerations, the depth of each part of the image has been found in steps 1 to 3 of the first embodiment. Therefore, if a position where the depths match is obtained from the depth distribution obtained by the method of the first embodiment and the distribution in FIG. 15B, the probability that this position is a segment corresponding to the edge 130 is high. Area 132 in FIG. 15B
Is on the straight line L3 at z = 50 and the depth of the region is determined to be a value of about 50 by the steps 1 to 3, the region 1
There is a high possibility that a corresponding area of the edge 130 exists near 32. Therefore, it is sufficient to perform detailed segment matching focusing on the area 132.

As described above, according to the present embodiment, the segment
Not only can the processing time for matching be significantly reduced,
It is also possible to reduce erroneous correspondences in an image in which many similar segments exist.

Embodiment 4 FIG . In the present embodiment, the depth information extracted in the first and second embodiments is used for a user interface of a computer system, specifically, for inputting an instruction.

As is well known, when a user inputs an instruction to a computer, the user often uses a keyboard, a mouse, a touch panel, or the like. Although user interfaces that do not use such input devices have also been studied, conventionally, in general, a model of the shape of a human face or hand is held in a computer, and this model is used in combination with the facial expression of the actual user's face or hand. They tried to read the instructions by comparing them with shapes. However, in reality, it is difficult to have an accurate model, or matching of so-called "soft models" such as hands has problems in terms of accuracy and processing time,
Practical application was difficult. In the present embodiment, a relatively simple user interface of the computer system is realized based on the depth information.

FIG. 16 is a schematic diagram showing the configuration of the system of this embodiment and a method of using the system. In FIG.
0 instructs the PC 142. A camera 144 for photographing the user 140 is installed in a part of the PC 142. With this configuration, the user 140 uses the fingertip 146 to perform simple operations such as up, down, left, and right. The image of the user 140 taken from the camera 144 is processed inside the PC 142 according to steps 1 to 3 of the first or second embodiment.
Depth information is calculated.

Here, the user's fingertip 146 is
4, the fingertip 146 can be recognized by selecting an image portion having the minimum depth. Thereafter, when the fingertip 146 moves, the video portion having the minimum depth is tracked to thereby enable the fingertip 146 to move.
You can know the movement of. The PC 142 has a fingertip 146
If it is known that the cursor has moved in a certain direction, a process such as moving the cursor on the screen in that direction may be performed.
For example, if 6 draws a circle, some processing may be executed as the “OK” sign.

Embodiment 5 FIG . In the present embodiment, the depth information extracted in the first and second embodiments is used for display control of the three-dimensional display device.

The stereoscopic display of an image is realized by several methods. A typical example is a display device in which a lenticular lens is superimposed on a screen. The lenticular lens is provided with a large number of fine arc-shaped convex portions continuously on the surface,
Due to the refraction effect of this lens, only the pixels constituting the image for the right eye can be seen from the right eye, and only the pixels constituting the image for the left eye can be seen from the left eye. The right-eye image and the left-eye image are provided with parallax, so that stereoscopic viewing is possible.

A problem with a display device using a lenticular lens is that if the position of the viewer's head deviates even a little, it will be in a state of inverted stereoscopic vision (also called reverse vision). Inverse stereoscopic viewing is a state in which a pixel for the right eye is seen from the right eye, and a pixel for the right eye is seen from the left eye. To address this problem, head tracking technology is known. Head tracking technology is based on “Basics of 3D video” (supervised by Izumi Takehiro, NH
As introduced on pages 154 and 155 of K Broadcasting Research Institute, Ohmsha), the movement of the viewer's head is detected by infrared, magnetism, ultrasonic waves, etc., and linked to the movement of the head for the right eye And the pixel for the left eye are appropriately replaced. In addition, it is described that an infrared sensor is provided above the display device to detect the movement of the viewer's head and mechanically change the position of the lenticular screen.

However, these head tracking techniques require a scanning mechanism and an array structure for infrared rays and ultrasonic waves. In the case of magnetic detection, it is necessary for the observer to attach a sensor to the head.

The system according to the present embodiment solves such a problem. In the present embodiment, as shown in FIG. 16 of the fourth embodiment, a camera is attached to a display device that performs stereoscopic display, and a viewer is photographed from the camera. Since the three-dimensional position including the depth of the viewer is determined from the image of the camera, display control suitable for the position of the viewer is performed.

As an example of the control method, when the depth of the viewer is small, that is, when the viewer is close to the display device, the parallax of the left and right images to be stereoscopically displayed is adjusted to be small as a whole.
Consideration is given so that the subject forms an image on the other side of the display surface (that is, the screen position) as viewed from the viewer. Conversely, when the viewer's depth is large, it is necessary to adjust the image so that it is closer to this side than the display surface, or to increase the range of parallax to be given so that the difference in the sense of depth between subjects becomes clear. Action can be taken.

FIG. 17 is a block diagram of the present system. As shown in the figure, this system employs a left display panel 158 for viewing with the left eye and a right display panel 160 for viewing with the right eye as display devices. Further, a camera 152 for photographing the viewer and a depth extracting unit 162 for extracting the depth of the viewer's head from the video are provided.

The image to be displayed is given to the input terminal 150. This image is an image for the left eye, and the system generates a right-eye image by displacing pixels from the image.
Parallax is generated by the displacement of the pixel, and stereoscopic viewing is enabled.
The image input from the input terminal 150 is stored in the buffer memory 1
54 and the right-eye image generation unit 156.
The buffer memory 154 functions to absorb a processing delay caused by the right-eye image generation unit 156. The left display panel 158 displays the image output from the buffer memory 154, and the right display panel 160 displays the image displaced by the right-eye image generation unit 156.

The depth of the viewer's head extracted by the depth extraction unit 162 is input to the right-eye image generation unit 156.
Here, when the depth is small, the displacement amount of the pixel is set small, and when the depth is large, the displacement amount is set large. As a result, good stereoscopic vision is realized according to the position of the viewer. Although only the image for the right eye is generated here, the image for the left eye may be generated by adding displacement to the pixels of the original image.

[0142]

When the system for utilizing three-dimensional position recognition according to the present invention is used as a security system, a scanning mechanism required for a security system that looks at the reflection of infrared rays and ultrasonic waves is unnecessary, and unlike a sensor that senses infrared rays, heat is not used. Even objects that do not emit can be detected.

When the system for utilizing three-dimensional position recognition according to the present invention is used for preprocessing of computer vision, it is possible to reduce the time required for segment matching and reduce erroneous correspondence.

When the three-dimensional position recognition utilizing system of the present invention is used for a user interface of a computer, an input device such as a keyboard is not required. Also, there is no need to hold facial expressions, hand shapes, and the like as templates or models. Furthermore, the processing time is shorter than the recognition of facial expressions.

When the three-dimensional position recognition utilizing system of the present invention is used for controlling the display of a pseudo three-dimensional image, the hardware configuration for knowing the position of the observer is simplified.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating main steps for generating a three-dimensional display image according to a first embodiment.

FIG. 2 is a flowchart for detecting a correspondence between video frames.

FIG. 3 is a diagram showing how a representative point is set in a reference frame t.

FIG. 4 is a diagram showing a state of block matching.

FIG. 5 is a schematic diagram showing the value of E1 in the vertical direction for each provisional corresponding point Pt ′ (i, j).

FIG. 6 is a diagram showing a relationship between corresponding points and representative points obtained as a result of step S12.

FIG. 7 is a diagram illustrating a principle of evaluating a relative position of a corresponding point.

8 is a diagram illustrating a result of performing the improvement processing of this step on the corresponding point candidates in FIG. 6;

FIG. 9 is a diagram showing a correspondence between a movement of a certain point P on the screen and a movement in a three-dimensional space.

FIG. 10 is a diagram illustrating the principle of deriving the three-dimensional coordinates of a point P from the three-dimensional movement of the camera and the movement of a point P on the screen.

FIG. 11 is a diagram showing a state where numerical values are given to respective representative points in a frame t.

FIG. 12 is a diagram showing main steps of a second embodiment.

FIG. 13 is a diagram illustrating a feature point selection criterion introduced in the second embodiment.

FIG. 14 is a diagram illustrating a method of performing segment matching on an edge segment.

FIG. 15 is a diagram illustrating a principle of narrowing a search area according to the third embodiment.

FIG. 16 is a schematic diagram illustrating a configuration of a system according to a fourth embodiment and a method of using the system.

FIG. 17 is a configuration diagram of a system according to a fifth embodiment.

[Explanation of symbols]

120, 130 Edge to be searched, 122 to 126
Edge, 132 area, 140 user, 142 P
C, 144 camera, 146 fingertip, 150 input terminals,
152 camera, 154 buffer memory, 156 image generator for right eye, 158 left display panel, 16
0 Right display panel, 162 depth extraction unit.

 ──────────────────────────────────────────────────続 き Continuation of the front page (72) Inventor Tsutomu Arakawa 2-5-5 Keihanhondori, Moriguchi-shi, Osaka Sanyo Electric Co., Ltd.

Claims (8)

[Claims] 1. A system for recognizing and using a three-dimensional position of an object, comprising: extracting means for extracting the depth of a photographed object, wherein security is performed based on the extracted depth. System to do. 2. The system according to claim 1, wherein the system performs a predetermined process when an object approaches. 3. A system for recognizing and using a three-dimensional position of an object, comprising extraction means for extracting the depth of a photographed object, and performing segment matching in computer vision technology based on the extracted depth. A system for performing pre-processing for 4. The system according to claim 3, wherein the preprocessing is processing for narrowing a search area in segment matching using a depth of an object. 5. A system for recognizing and using a three-dimensional position of an object, comprising extraction means for extracting a depth of a photographed object, and recognizing an instruction from a user based on the extracted depth. A system characterized by receiving this. 6. The system according to claim 5, wherein a movement of a portion having the smallest depth among the photographed objects is determined as a user's instruction and accepted. 7. A system for recognizing and using a three-dimensional position of an object, comprising extraction means for extracting a depth of a photographed object, and controlling display of a pseudo three-dimensional image based on the extracted depth. A system characterized by performing. 8. The display device according to claim 7, wherein it is determined that a viewer of the pseudo three-dimensional image exists at the position of the photographed object, and display position control is performed so that the pseudo three-dimensional image is favorably displayed at this position. system.