KR101779564B1 - Method and Apparatus for Motion Recognition - Google Patents

Abstract

A motion recognition apparatus and method therefor are provided. According to an aspect of the present invention, at least one image sensor is used to sense a motion occurrence part of a subject to output an event, track a motion locus of a motion occurrence area using an event, The operation of the device can be controlled.

Description

[0001] The present invention relates to a method and apparatus for motion recognition,

The present invention relates to a motion recognition apparatus and a method thereof, and it is possible to track a motion locus by sensing a motion occurrence part of a subject and to control the motion of the device according to the track motion locus.

A user interface (UI) for interaction between a human and an electronic device includes a keyboard, a mouse, and a touch panel. For example, in the case of a touch technology used in a touch panel, a user must touch the screen to operate the UI. As the number of touch increases, the screen of the panel may be damaged, and the user may feel uncomfortable hygiene by direct contact. Accordingly, there is a need to improve user convenience by providing an intuitive interfacing technique that enhances interaction between humans and electronic devices naturally.

In one aspect, there is provided an image processing apparatus comprising: an image sensor that senses a motion occurrence portion of a subject and outputs events; A motion tracking unit for tracking a motion trajectory of a motion-generated portion using a time-space correlation diagram of output events; And an operation pattern determining unit that determines an operation pattern of a portion where the motion is generated from the traced motion trajectory.

And a light source unit for outputting light toward the subject to sense motion of the subject.

At this time, the light source unit can output infrared light.

A light measuring unit for measuring intensity of light; And a light source for outputting light when the intensity of the light measured by the light measuring unit is smaller than a predetermined light intensity.

A parallax calculating unit for calculating a parallax between image sensors using a histogram of events output from a plurality of image sensors when there are a plurality of image sensors; And a synthesizer for synthesizing events output from the plurality of image sensors based on the calculated parallax.

And a depth calculating unit for calculating a depth of a portion where motion is generated using the calculated parallax.

Each of the image sensors is made up of a plurality of sensing units, and the target sensing units corresponding to a part of the plurality of sensing units where the motion occurs can output an event.

The motion tracking unit includes: a plurality of space-time correlators, each of which calculates a space-time correlation between target sensing units using events input from the target sensing units; And a motion trajectory tracing unit for tracing a motion trajectory of a portion where motion occurs, using the calculated high and low values of the space-time correlation.

The plurality of space time correlators each have an internal state value indicating a space-time correlation, and each time an event is input, the internal state value is incremented, and the increased internal state value is compared with a preset threshold value to determine a space- can do.

Meanwhile, the motion recognition apparatus may further include a light measuring unit for measuring light intensity, and the plurality of space-time correlators may set the threshold value in consideration of the light intensity.

The motion recognition device may further include a depth calculating unit that calculates a depth of the motion-generating part using the parallax between the image sensors when the image sensor is plural, wherein the plurality of space- The threshold may be set in consideration of the depth of the portion where the motion occurs.

The motion trajectory tracking unit groups space-time correlators having a high time-space correlation among a plurality of space-time correlators to generate one cluster-type corresponding to a part where motion occurs, The movement locus can be traced by connecting the center position of the cluster.

The plurality of space-time correlators are each mapped to the divided regions of the image sensor, and the divided regions may overlap with at least one surrounding region.

And an operation control unit for outputting a control command with reference to the determined operation pattern.

In another aspect of the present invention, there is provided a method of detecting a motion of a subject, the method comprising sensing motion of a subject photographed by the image sensor and outputting events; Tracking motion trajectories of a motion-generating portion using the time-space correlation diagrams of output events; And determining an action pattern of a motion-generated portion from the traced motion trajectory.

The method may further include outputting light toward the subject to sense motion of the subject before outputting the events.

Measuring the intensity of light prior to outputting the events; And outputting light when the intensity of the light measured by the light measuring unit is smaller than a preset light amount.

Calculating a parallax between image sensors using a histogram of events output from image sensors of a plurality of cameras when there are a plurality of image sensors; And synthesizing events output from the plurality of image sensors based on the calculated disparity; And calculating the depth of the portion where the motion has occurred using the calculated parallax.

Wherein each of the plurality of space-time correlators, using events input from the target sensing units, each calculating a space-time correlation between target sensing units, The movement trajectory of the portion where the motion occurs can be tracked.

Wherein the plurality of space time correlators each have an internal state value indicating a space-time correlation, and the calculating step increases an internal state value each time events are input, compares the increased internal state value with a preset threshold value, Can be determined.

The method may further include measuring light intensity, and the calculating step may set the threshold value in consideration of the light intensity.

The method may further include calculating a depth of a portion where the motion occurs by using a parallax between the image sensors when the image sensor has a plurality of images, The threshold value can be set.

The step of tracing the movement trajectory includes: grouping space time correlators having a high space-time correlation among a plurality of space-time correlators to generate one cluster - a shape corresponding to a part in which motion occurs; Calculating a center position of the generated cluster; And tracking the motion locus by linking the calculated center position and the previously calculated center position.

And outputting a control command with reference to the determined operation pattern.

According to another aspect of the present invention, there is provided an image processing apparatus comprising: an image sensor that senses a motion of a subject and outputs events; And a controller for tracking motion trajectories of the motion-generating portion using the time-space correlation diagrams of the output events, determining an operation pattern of the motion-generated portion from the traced motion trajectory, referring to the determined motion pattern At least one processor configured to identify an input signal of a corresponding user is provided.

According to the motion recognition apparatus and method, an image sensor can detect only a moving object except for a fixed image by outputting an event only in a sensing unit having a change in light intensity, unlike an existing sensor that outputs an image at a predetermined time interval. Accordingly, the motion recognition apparatus and method can omit the image preprocessing process to recognize a moving object, and do not require high computing power required in a conventional motion recognition system.

In addition, according to the motion recognition apparatus and method thereof, motion of an object can be recognized at a very high speed of 1 ms level, so that motion can be recognized in real time.

In addition, according to the motion recognition apparatus and method, depth information of an object in which a motion occurs can be known by using a plurality of image sensors. The depth information can be used for motion recognition with a depth sense such as a pushing operation for pushing the hand toward the body. The depth information can also be used to remove background noise. For example, when the depth information of the moving person and the camera is known, the motion occurring at a distance farther than the moving person is ignored, so that the motion can be accurately recognized even when the motion occurs behind the person.

Fig. 1 shows a configuration example of a first motion recognition device.
2 is a diagram showing a configuration example of the first motion tracking unit.
3 is a diagram showing an example of an array and a reception field of the sensing units constituting the first image sensor.
4 is a diagram for explaining a spatial relation between first through nth space time correlators. Fig.
5 is a diagram illustrating an example of generating first through third clusters by grouping first through nth space time correlators.
FIG. 6 is a diagram for explaining an example of tracking a motion locus of a motion-generated portion using the generated cluster.
FIG. 7 is a diagram illustrating a series of processes for tracking motion trajectories from output events.
Fig. 8 shows a configuration example of the second motion recognition device.
9 is a diagram showing an example of events output from the first and second image sensors of the first and second photographing units.
FIG. 10 is a diagram for explaining a process of obtaining a time difference from a histogram of events.
11 is a flowchart for explaining an operation recognition method of the first motion recognition device.
12 is a flowchart for explaining an operation recognition method of the second operation recognition device.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Fig. 1 shows an example of the configuration of the first motion recognition device 100. Fig.

Referring to FIG. 1, a first motion recognition apparatus 100 for recognizing an operation pattern according to a motion of a subject includes a first imaging unit 110, a first motion tracking unit 120, a first motion pattern determination unit 130 ), A first pattern storage unit 140, and a first operation control unit 150.

The first photographing unit 110 may be a camera capable of photographing a subject and photographing a moving image or a still image. The first imaging unit 110 includes an optical system (not shown) including one or more lenses and a first image sensor 111. The subject can be anyone who can move, animals, robots, and so on.

The first image sensor 111 senses a part where a motion occurs in a subject and outputs events. The first image sensor 111 includes a plurality of sensing units arrayed as shown in Fig. 3, which will be described later. The sensing units may be provided in units of pixels of an image. For example, when the image output from the first photographing unit 110 is 60x60, the sensing units may be provided in a 60x60 form.

The sensing units may be light receiving elements. When the subject or a part of the subject moves, the intensity of the light sensed by the sensing units of the first image sensor 111 changes. In the case of the first image sensor 111, the target sensing units in which the intensity variation of the light among the sensing units is sensed output an event. That is, the target sensing units corresponding to the portion of the sensing unit where the motion occurs output an event. The target sensing unit is a sensing unit for outputting an event.

The event includes information such as the time at which the event occurred, the position of the sensing unit outputting the event, polarity, and the like. The polarity is 'on' when the intensity of the light received by the sensing unit increases and the intensity of the received light is 'off' when an event occurs. At this time, each sensing unit can output an event when the intensity change amount of the light is larger or smaller than the set reference value.

The first motion tracking unit 120 may track a motion trajectory of a motion-generated portion using spatial-temporal correlations of events output from the first image sensor 111.

FIG. 2 is a diagram illustrating an example of the configuration of the first motion tracking unit 120. Referring to FIG.

Referring to FIG. 2, the first motion tracking unit 120 includes first to nth spatial correlation correlators 121-1, 121-2, ..., 121-n, where n is a positive number, And a trajectory tracking unit 123.

The first to nth space time correlators 121-1, 121-2, ..., 121-n may calculate the space-time correlation between the target sensing units using the events input from the target sensing units, respectively .

Hereinafter, in order to explain the space-time correlation, a reception field will be described first with reference to FIG. 3, and a first space time correlator 121-1 will be described as an example.

3 is a diagram showing an example of an array and a reception field of the sensing units constituting the first image sensor 111. As shown in FIG.

Referring to FIG. 3, the first space time correlator 121-1 receives events output from the sensing units constituting a specific region of the first image sensor 111. FIG. The specific area means the reception field immediately. An area that is electrically connected to the first space-time correlator 121-1 and occupied by the sensing units capable of outputting an event to the first space-time correlator 121-1 is referred to as a reception field. The receive field may have a size of m x m (where m is a positive number). Accordingly, the first space time correlator 121-1 may receive an event from at least one of the sensing units of the reception field. The second to nth space time correlators 121-2, ..., 121-n may also be connected to the sensing units of the corresponding reception field to receive events.

Referring again to FIG. 2, the first to nth space time correlators 121-1, 121-2,..., 121-n each have an internal state value indicating a current space-time correlation. Each internal state value may be equal to or different from each other, and may be, for example, a voltage value. The internal state value can be determined by the current internal state value and the newly input event. When an event is input, the internal state value increases. If there is no event input, the internal state value may decrease as the set time passes. Reducing the internal state value can minimize the bandwidth load of the memory storing the internal state value.

The first to nth space time correlators 121-1, 121-2, ..., 121-n increase the internal state value each time events are input from the corresponding first image sensor 111 , The increased internal state value can be compared with the set threshold to determine the high or low of the space-time correlation. The space-time correlation is a temporal correlation and a spatial correlation between events input to the first to nth space-time correlators 121-1, 121-2, ..., and 121-n, respectively.

(1) is a relational expression for calculating an output value output from the nth space time correlator 121-n (n = 1, 2, 3, ...) according to an input event.

는 임계값이다. 이벤트 세트는 시간 t에서 발생하는 이벤트들에 대한 단순 정보를 나타낸다. 예를 들어, 처음 t=0라는 시간에 이벤트가 세 개가 발생했다면 e(0) = {e_1, e_2, e_3}이다. e_n(n=은 n=1, 2, 3, …)은 n번째 타겟 센싱 유닛의 이벤트이다. Q _n (t) is the internal state value of the n th space-time correlator 121-n at time t, Q _n (t _prev ) is the internal state value of the n th space-time correlator 121-n at the previous time, ), I.e., the current internal state value, t _prev , is the time at which the event set occurred most recently among the previously input event sets. Also, e (t) is an event set input at time t, out _n (t) is an output value of the nth space time correlator 121-n,

Is a threshold value. The event set represents simple information about events occurring at time t. For example, if three events occurred at the time t = 0 for the first time, e (0) = {e_1, e_2, e_3}. e_n (n = n = 1, 2, 3, ...) is an event of the nth target sensing unit.

When the n-th space-time correlator (121-n) is an event it is input from one of the target sensing unit of the sensing unit of a receiving field, corresponding to the time t,, thereby increasing the previous internal states of the value (Q _n (t _prev)) . The degree of increase is affected by the weight set in the target sensing unit that generated the event. Therefore, when a plurality of events are input at the same time, the rate of increase also increases. The weights may be set differently for each sensing unit.

)을 초과하면, 제n시공간 상관기(121-n)로 이벤트를 출력한 타겟 센싱 유닛들 간의 시공간 상관도가 높은 것으로 판단하고, 1을 출력한다. 또한, 제n시공간 상관기(121-n)는 내부 상태값(Q_n(t))이 설정된 임계값(

)보다 작거나 같으면, 시공간 상관도가 낮은 것으로 판단하고, 0을 출력할 수 있다. '1'은 'high' 또는 'true' 형태로도 대신 출력되며, '0'은 'low' 또는 'false' 형태로 대신 출력될 수 있다. 선택적으로, 제n시공간 상관기(121-n)의 출력이 '1'인 된 후, 제n시공간 상관기(121-n)는 제n시공간 상관기(121-n)의 내부 상태값을 설정된 양만큼 감소시킬 수 있다.The first to nth space time correlators 121-1, 121-2, ..., 121-n output different output values to the first motion trajectory tracking unit 123 according to the spatial and temporal correlation. That is, the nth space time correlator 121-n determines whether the internal state value Q _n (t)

, It is determined that the time-space correlation degree between the target sensing units outputting the event to the nth space time correlator 121-n is high, and 1 is output. Also, the nth space time correlator 121-n determines whether or not the internal state value Q _n (t)

), It is determined that the space-time correlation is low, and 0 can be output. '1' can also be output in the form of 'high' or 'true', and '0' can be output in the form of 'low' or 'false' instead. Alternatively, the nth space time correlator 121-n may reduce the internal state value of the nth space time correlator 121-n by a predetermined amount after the output of the nth space time correlator 121-n becomes '1' .

According to the above description, the internal state value Q _n (t) calculated by the n-th time-space correlator 121-n is the spatial state of the events input from the target sensing units to the nth space- Associations and temporal associations. For example, if events are continuously input to the nth time-space correlator 121-n from one target sensing unit, the internal state value Q _n (t) may represent a temporal association between incoming events. Also, if two events are simultaneously input from the two target sensing units to the nth space time correlator 121-n, i.e., the two target sensing units are close and connected to the same nth space time correlator 121-n If so, the two events have a high spatial relevance. Since two events are simultaneously input from two target sensing units, the two events also have temporal correlation.

On the other hand, the first to n-th space time correlators 121-1, 121-2, ..., 121-n may have a spatial relationship with each other.

FIG. 4 is a diagram for explaining a spatial relationship between the first through n-th space time correlators 121-1, 121-2,..., 121-n.

Referring to FIG. 4, the first image sensor 111 may be logically divided into a plurality of reception fields. The segmented receive fields overlap with at least one surrounding area. The first to nth space time correlators 121-1, 121-2, ..., 121-n may be mapped to the divided receive fields of the first image sensor 111, respectively. The sensing units located in each reception field output an event to the corresponding one of the first to n-th spacetime correlators 121-1, 121-2, ..., 121-n.

In FIG. 4, C (i, j) is the center coordinate of the reception field located at the center of the first image sensor 111. C (i-1, j) is the center coordinate of the reception field whose center coordinate is shifted by '1' in the x-axis direction (that is, the horizontal direction). In the case of FIG. 4, '1' means that three pixels are spaced apart, which is changeable.

As shown in FIG. 4, some overlap of the receive fields means that the same event can be simultaneously output to at least two space-time correlators. The spatial correlation between the first to n th space time correlators 121-1, 121-2, ..., 121-n or the spatial correlation between the reception fields can be given by having the reception fields partially overlap each other have. The spatial correlation between the first to nth space time correlators 121-1, 121-2, ..., 121-n may have an effect on tracking motion trajectories.

Referring back to FIG. 2, the first motion locator tracking unit 123 estimates the output values of the space-time correlation calculated from the first to nth space time correlators 121-1, 121-2, ..., and 121-n, That is, it is possible to track the motion locus of the portion where the motion occurs by using the high and low.

The first movement trajectory tracking unit 123 generates one cluster by grouping the space time correlators having high time-space correlation among the first to nth space time correlators 121-1, 121-2, ..., 121-n . Specifically, the first motion trajectory tracking unit 123 compares space-time correlators outputting '1' among the first through n-th space-time correlators 121-1, 121-2, ..., and 121- As shown in FIG. That is, the first motion locator tracking unit 123 generates a cluster by grouping the space time correlators overlapping each other. The output of '1' means high spatial and temporal correlation.

5 illustrates a method of generating first through third clusters (cluster n, n = 1, 2, 3) by grouping first through nth space time correlators 121-1, 121-2, Fig.

Referring to FIG. 5, a first cluster (cluster 1) is generated by grouping two reception fields 51 and 53 overlapping with each other. The second cluster (cluster 2) is generated by grouping 10 reception fields overlapping each other. The third cluster (cluster 3) is generated by grouping seven reception fields overlapping each other.

On the other hand, the first movement trajectory tracking unit 123 can know the position of a portion where the motion occurs using the cluster. That is, the first image sensor 111 outputs only the target sensing unit having a change in the luminous intensity of the light, so that the part where the motion occurs can be found using the space-time correlation between the events. For this, the first motion trajectory tracking unit 123 calculates the center position of the cluster, and tracks the motion locus by linking the calculated center position with the center position of the previously calculated cluster. The cluster may have a shape corresponding to the portion where the motion occurs.

The first motion trajectory tracking unit 123 may multiply the position of each space-time correlator included in the cluster by a constant, and set the average value of the multiplication results as the center position of the cluster. The location of the space-time correlator may be a location representative of the area covered by the space-time correlator. The constant may be various values such as an internal state value (Q (t)) calculated by space time correlators, the number of events input to space time correlators during a specific time interval, and the like.

For example, a cluster is generated by the grouping of the first and second space-time correlators 121-1 and 121-2, the internal state value of the first space-time correlator 121-1 is Q1, the second space- 121-2) is Q2. The position representative of the first space-time correlator 121-1 is (x1, y1), and the position representative of the second space-time correlator 121-2 is (x2, y2). The first movement trajectory tracking unit 123 can calculate the center position of the cluster using Equation (2).

In Equation (2), x 'is the x coordinate of the center position of the cluster, and y' is the y coordinate of the center position.

FIG. 6 is a diagram for explaining an example of tracking a motion locus of a motion-generated portion using the generated cluster.

The first movement trajectory tracking unit 123 calculates the center position of the cluster each time a cluster is generated. Referring to FIG. 6, the first motion locator tracking unit 123 sequentially generates clusters at time t1, t2, and t3, and calculates the center position of each cluster. The first movement trajectory tracking unit 123 tracks the movement trajectory by connecting the calculated center positions. That is, the motion trajectory of the motion-generating portion can be tracked by calculating the position of the grouped space-time correlators (i.e., grouped receive fields).

FIG. 7 is a diagram illustrating a series of processes for tracking motion trajectories from output events.

Referring to FIG. 7, a white point is an event of 'off' polarity generated by a decrease in light intensity, and a black point is an event of 'on' polarity generated by an increase in light brightness. The portion where the events are gathered is the portion where the motion is sensed by the first image sensor 111. The larger the density of events, the greater the change in motion. The first to nth space time correlators 121-1, 121-2, ..., 121-n output an output value indicating the temporal correlation of the space-time correlation using input events. The first motion locator tracking unit 123 generates a cluster by grouping space time correlators outputting '1' among the first to nth space time correlators 121-1, 121-2, ..., and 121-n, The center position p of the cluster is calculated. The first motion trajectory tracking unit 123 may track the motion trajectory by connecting the calculated center position to the previously calculated center position. In FIG. 7, 'object' is a region corresponding to a cluster as a part where a motion occurs, for example, a hand.

Referring back to FIG. 1, the first motion pattern determiner 130 may determine an operation pattern of a motion-affected portion from a motion trajectory tracked by the first motion-tracking unit 120. FIG. In particular, the first operation pattern determination unit 130 obtains the feature components for expressing the operation pattern from the traced movement trajectory, compares the feature components with the operation patterns stored in the first pattern storage unit 140, It is possible to determine the operation pattern of the generated portion. The feature parameters include various parameters such as the position of the cluster, the direction in which the cluster moves, and the angle of movement of the cluster.

The first pattern storage unit 140 may store values of a plurality of feature components and a corresponding operation pattern.

The first operation control unit 150 may identify the input signal of the user by referring to the determined operation pattern, and may output a control command to the apparatus for controlling the apparatus (not shown) according to the input signal of the identified user. The device may be provided at a place capable of performing wired / wireless communication with the first motion recognition device 100, or the first motion recognition device 100 may be provided in the device.

On the other hand, the first image sensor 111, which outputs an event according to the movement of the subject, is affected by the ambient brightness. That is, when the ambient brightness is dark, .

The first motion recognition apparatus 100 may further include one or both of the first light source unit 160 and the first light measurement unit 170 to reduce the influence of the ambient brightness.

First, when only the first light source unit 160 is added to the first motion recognition device 100, the first light source unit 160 outputs light toward the subject to accurately sense the motion of the subject even when it is dark. At this time, the output light may be infrared light.

Next, when both the first light source unit 160 and the first light measurement unit 170 are added to the first motion recognition apparatus 100, the first light measurement unit 170 measures the intensity of light. The first light source unit 160 outputs light when the intensity of the light measured by the first light measurement unit 170 is smaller than the predetermined light intensity.

First, when only the first light measuring unit 170 is added to the first motion recognizing apparatus 100, the first light measuring unit 170 measures the intensity of light. The first to nth space time correlators 121-1, 121-2, ..., and 121-n set threshold values in consideration of light intensity before determining the high and low of the space-time correlation . That is, when the intensity of the light is weak, the first to nth space time correlators 121-1, 121-2, ..., 121-n set the threshold value relatively low, and when the intensity of the light is strong, It can be set relatively high.

Further, in the example of Fig. 1, the first motion recognition device 100 may further include a control unit (not shown). The control unit (not shown) can control the overall operation of the first motion recognition device 100. [ The control unit (not shown) may perform the functions of the first motion tracking unit 120, the first motion pattern determination unit 130, and the first motion control unit 150. The control unit (not shown), the first motion tracking unit 120, the first operation pattern determination unit 130, and the first operation control unit 150 are separately shown to distinguish the respective functions. Accordingly, the control unit (not shown) includes at least one processor configured to perform the functions of the first motion tracking unit 120, the first motion pattern determination unit 130, and the first motion control unit 150 . In addition, the control unit (not shown) may include at least one processor configured to perform some of the functions of the first motion tracking unit 120, the first motion pattern determination unit 130, and the first motion control unit 150, . ≪ / RTI >

Fig. 8 shows a configuration example of the second motion recognition device 200. Fig.

Referring to FIG. 8, the second motion recognition device 200 for recognizing an operation pattern according to the movement of a subject includes a first imaging unit 811, a second imaging unit 812, a third imaging unit 813, The second motion pattern determination unit 860, the second pattern storage unit 870, and the second motion estimation unit 860. The motion estimation unit 820, the depth calculation unit 830, the composition unit 840, the second motion tracking unit 850, And a control unit 880.

The second motion pattern determination unit 860, the second pattern storage unit 870, and the second motion control unit 880. The first to third imaging units 811, 812, and 813, the second motion tracking unit 850, Is similar to the first imaging unit 110, the first motion tracking unit 120, the first motion pattern determination unit 130, the first pattern storage unit 140, and the first motion control unit 150 of FIG. 1 A detailed description thereof will be omitted.

The second motion recognition apparatus 200 may include a plurality of photographing units, and three photographing units 811, 812 and 813 are illustrated in FIG. The first to third photographing units 811, 812 and 813 photograph the same or different subjects and have first to third image sensors 811a, 812a and 813a, respectively. The first to third image sensors 811a, 812a, and 813a sense an area where a motion occurs, output an event, and perform operations similar to those of the first image sensor 111. [

The parallax calculating unit 820 calculates parallax between the first to third image sensors 811a, 812a and 813a using the histogram of the events output from the first to third image sensors 811a, 812a and 813a disparity can be calculated. For example, in a case where two first and second photographing units 811 and 812 are provided, and the same subject is photographed, the first photographing unit 811 and the second photographing unit 812 are respectively disposed in the left and right And therefore, the calculated disparity can be a binocular disparity.

Meanwhile, when motion recognition is performed using the first to third photographing units 811, 812, and 813, the second motion recognition device 800 can three-dimensionally recognize the motion of the motion occurrence portion . In other words, the second motion recognition device 800 recognizes not only the two-dimensional position information of the upper and lower parts, the left and right parts of the motion occurrence part, but also the depth of the first to third photographing parts 811, 812, Can be obtained.

9 is a diagram showing an example of events output from the first and second image sensors 811a and 812a of the first and second photographing units 811 and 812. [

Referring to FIG. 9, the first event is an event output from the first image sensor 811a, and the second event is an event output from the second image sensor 812a. It can be seen that the distribution patterns of the first event and the second event are similar.

The parallax calculating unit 820 calculates a first histogram for the first event and a second histogram for the second event on the basis of the x-axis address.

FIG. 10 is a diagram for explaining a process of obtaining a time difference from a histogram of events.

10, the x-axis address is the x-axis coordinate of the positions of the sensing units constituting the first image sensor 811a, Bins is the same x-axis address (x _i ) among the target sensing units that simultaneously generate the first event, Lt; / RTI > The first histogram is the result of calculating Bins for all x-axis addresses. This is also the same for the second histogram, so a detailed description will be omitted.

The parallax calculating unit 820 can obtain the value of the binocular parallax using the correlation between the first histogram and the second histogram. That is, the parallax calculating unit 820 moves one of the first histogram and the second histogram, for example, the first histogram in a predetermined offset interval in the x-axis direction, and then calculates the difference between the first histogram and the second histogram Obtain a correlation. The correlation shows the degree of resemblance of the shapes of the first and second histograms, and if the correlation is 1, it means that they match each other. The parallax calculating unit 820 determines the binocular parallax as the distance to which the first histogram has moved when the parallax coincides most with the first and second histograms, that is, when the correlation is closest to 1 or 1.

The depth calculating unit 830 can calculate the depth of a portion where a motion occurs using the calculated parallax, for example, binocular parallax. An algorithm for calculating the depth from the disparity can use one of a variety of well known techniques. The larger the calculated parallax, the closer the moving parts are to the first and second photographing units 811 and 812. [

The combining unit 840 may combine the events output from the first and second image sensors 812a based on the parallax calculated by the parallax calculating unit 820 and output one image of the events. For example, the synthesizing unit 840 synthesizes the first and second image signals outputted from the first and second image sensors 811a and 812a so as to best overlap the parts with the greatest motion using the binocular parallax calculated by FIG. And the second event.

The second motion tracking unit 850 can determine a motion occurrence portion and track a motion trajectory from the video events output from the combining unit 840 using a plurality of space time correlators. The distribution of the events constituting the image output from the synthesis unit 840 can be divided into a dense region and a less dense region. The second motion tracking unit 850 may track motion trajectories using events of the largest density of events. The larger the density of events, the greater the degree of motion.

The second motion tracking unit 850 may take into account the depth of the portion where the motion is calculated through the depth calculating unit 830 when tracking the motion locus. More specifically,

The time-space correlators included in the second motion-tracking unit 850 set thresholds in consideration of the depth of the motion-occurring portion before determining the high-low of the space-time correlation. That is, if the depth of the motion-generating part is large, the space-time correlator included in the second motion-tracking unit 850 sets the threshold value relatively low, and if the depth of the motion- Can be set.

The second motion pattern determination unit 860 can determine an operation pattern of a motion portion from the motion locus tracked by the second motion tracking unit 850. Particularly, the second operation pattern determination unit 860 can determine the operation pattern of the portion where the motion occurs, using the operation patterns stored in the second pattern storage unit 870.

The second operation control unit 880 can identify the input signal of the user by referring to the determined operation pattern and output a control command for controlling the device (not shown) according to the input signal of the identified user.

On the other hand, the first to third image sensors 811a, 812a, and 813a that output events according to the movement of the subject are affected by the ambient brightness, that is, the first to third image sensors 811a, 812a, ), The relative output event is reduced when the surrounding brightness is dark.

The second motion recognition device 200 may further include one or both of the second light source unit 890 and the second light measurement unit 895 to reduce the influence of the ambient brightness.

First, when only the second light source unit 890 is added to the second motion recognition device 200, the second light source unit 890 outputs light toward the subject in order to accurately sense motion of the subject even when it is dark. At this time, the output light may be infrared light.

Next, when both the second light source unit 890 and the second light measurement unit 895 are added to the first motion recognition apparatus 100, the second light measurement unit 895 measures the intensity of light. The second light source unit 890 outputs light when the intensity of the light measured by the second light measurement unit 895 is smaller than the predetermined light amount.

First, when only the second light measuring unit 895 is added to the second motion recognizing apparatus 200, the second light measuring unit 895 measures the intensity of light. The space time correlator included in the second motion tracking unit 850 sets a threshold value in consideration of the light intensity before determining the high and low of the space-time correlation. That is, if the intensity of the light is weak, the time-space correlators included in the second motion tracking unit 850 may set the threshold value relatively low and set the threshold value relatively high when the intensity of the light is strong.

In the example of Fig. 8, the second motion recognition device 800 may further include a control unit (not shown). The control unit (not shown) can control the overall operation of the second motion recognition device 800. The control unit (not shown) includes a parallax calculating unit 820, a depth calculating unit 830, a composing unit 840, a second motion tracking unit 850, a second motion pattern determining unit 860, And can perform the function of the control unit 880. A depth calculating unit 830, a combining unit 840, a second motion tracking unit 850, a second motion pattern determining unit 860, and a second motion controlling unit (not shown) 880) are shown separately for explaining the respective functions. Accordingly, the control unit (not shown) includes a parallax calculating unit 820, a depth calculating unit 830, a composing unit 840, a second motion tracking unit 850, a second motion pattern determining unit 860, And at least one processor configured to perform each function of the processor 880. The control unit (not shown) includes a parallax calculating unit 820, a depth calculating unit 830, a composing unit 840, a second motion tracking unit 850, a second motion pattern determining unit 860, And may include at least one processor configured to perform some of the functions of each of the controllers 880.

11 is a flowchart for explaining an operation recognition method of the first motion recognition device.

Each step of Fig. 11 can be operated by a control unit (not shown) or a processor (not shown) of the first motion recognition apparatus 100 of Fig.

In step 1110, the first motion recognition device measures the intensity of light when it includes a light measuring part,

In step 1112, The first motion recognition device outputs light toward the subject to sense motion of the subject. At this time, the light output of step 1112 may occur every time the subject is photographed, or may be output only when the intensity of light measured in step 1110 is less than a predetermined light amount.

Steps 1110 and 1112 may be optional and may not be included in the motion recognition method, or may include only a portion of the two steps.

In step 1114, the first motion recognition device captures a subject using a camera, and outputs the events to the corresponding space time correlators through a sensing unit corresponding to the part where the motion occurred during the shooting of the subject. The camera has an image sensor in which a plurality of sensing units are arrayed, and the image sensor allows the sensing unit of the part where the motion occurs to output an event through the amount of change in light intensity.

In step 1116, a threshold, which is a scale for determining the high and low of the space-time correlation, is set. At this time, the threshold value may be fixed to a preset value by the experiment, or may be varied according to the intensity of the light measured in step 1110.

In step 1118, the first motion recognition device uses the input events to increase the internal state value of space time correlators. The internal state value may be a measure of the time-space correlation between events.

In step 1120, the first motion recognition device compares the internal state value and the threshold value for each space-time correlators.

If it is determined in step 1120 that the increased internal state value exceeds the set threshold value, the first motion recognition apparatus determines that the space-time correlation of the space-time correlator is high and outputs the output value '1' in step 1122.

If it is determined in step 1120 that the increased internal state value is less than the preset threshold value, the first motion recognition device determines that the space-time correlation of the space-time correlator is low and outputs the output value '0' in step 1124.

In step 1126, the first motion recognition device generates clusters by grouping space time correlators having an output value of '1'. At this time, the first motion recognition apparatus groups the space time correlators overlapping with the surrounding space-time correlators into one cluster.

In step 1128, the first motion recognition device calculates the center position of each cluster. For example, the first motion recognition apparatus can obtain the center position of the cluster using the position and the constant of each space-time correlator included in the cluster.

In step 1130, the first motion recognition device tracks the motion locus of the part where the motion occurs by associating the calculated center position with the center position of the previously calculated cluster.

In step 1132, the first motion recognition device extracts the feature components for expressing the motion pattern from the tracked motion trajectory.

In step 1134, the first motion recognition device compares the extracted feature components with pre-stored motion patterns to determine an motion pattern of a motion-generated part, and refers to the determined motion pattern to perform motion of the device (not shown) Can be controlled. The device (not shown) may be connected to the first motion recognition device, or a first motion recognition device may be provided in the device.

12 is a flowchart for explaining an operation recognition method of the second operation recognition device.

Each step of Fig. 12 can be operated by a control unit (not shown) or a processor (not shown) of the second motion recognition device 800 of Fig.

In step 1210, the second motion recognition device measures the intensity of the light when it includes a light measuring part,

In step 1212. The first motion recognition device outputs light toward the subject to sense motion of the subject. At this time, the light output of step 1212 may be generated each time a subject is photographed, or may be output only when the intensity of light measured in step 1210 is less than a predetermined light amount.

Steps 1210 and 1212 may be optional steps or may not be included in the motion recognition method, or may include only a part of the two steps.

In operation 1214, the second motion recognition device captures a subject using a plurality of cameras, and outputs events to the space-time correlators through a target sensing unit corresponding to a portion in which motion occurs in the subject.

In operation 1216, the second motion recognition device generates a histogram of events output from the image sensors provided in the plurality of cameras.

In step 1218, the second motion recognition apparatus moves one of the generated histograms in the x-axis direction at a predetermined offset interval, and obtains the correlation between the histograms after being moved.

In step 1220, the second motion recognition apparatus time-lags the distance (i.e., the offset) in which the histogram moves in the x-axis direction when the correlation is greatest, i.e., the closeness is 1 or 1.

In step 1222, the second motion recognition device can calculate the depth of the portion where the motion occurs using the calculated parallax.

In operation 1224, the second motion recognition apparatus may synthesize events input from operation 1214 on the basis of the calculated time difference, and output one image including the events.

In step 1226, a threshold value, which is a scale for determining the high and low of the space-time correlation, is set. At this time, the threshold value may be fixed to a preset value by an experiment, or may be variable according to the intensity of light measured in step 1210.

When a threshold value is set, the second motion recognition apparatus proceeds to step 1118 of FIG. 11 to track motion trajectory and control operation of the motion . That is, the operation after the events are synthesized is similar to the operation described with reference to FIG. 11, and a detailed description thereof will be omitted.

The above-described first and second motion recognition apparatuses can be applied to replace an existing remote controller in an apparatus using various application programs such as a TV, a computer, and a game console.

The methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

100: first motion recognition device 110: first photographing part
120: first motion tracking unit 130: first motion pattern determining unit
140: first pattern storage unit 150: first operation control unit
160: first light source unit 170: first light measuring unit
121-1, 121-2, ... , 121-n: first to nth space time correlators

Claims (28)

An image sensor that senses a part where a motion occurs in a subject and outputs events;
A motion tracking unit for tracking a motion trajectory of a portion where the motion occurs using the time-space correlation of the output events; And
An operation pattern determination unit for determining an operation pattern of a portion where the motion has occurred from the tracked motion trajectory,
And a motion recognition device. The method according to claim 1,
And a light source unit for outputting light toward the subject to sense motion of the subject,
Further comprising: 3. The method of claim 2,
The light source unit includes:
Infrared light,
Motion recognition device. The method according to claim 1,
A light measuring unit for measuring intensity of light; And
When the intensity of the light measured by the light measuring unit is smaller than a preset light amount,
Further comprising: The method according to claim 1,
A parallax calculating unit for calculating a parallax between the image sensors using a histogram of events output from the plurality of image sensors when the image sensors are plural; And
And a synthesizer for synthesizing events output from the plurality of image sensors on the basis of the calculated parallax,
Further comprising: 6. The method of claim 5,
And a depth calculating unit for calculating a depth of the portion where the motion is generated using the calculated parallax,
Further comprising: The method according to claim 1,
Wherein the image sensor comprises a plurality of sensing units and target sensing units corresponding to a portion in which the motion has occurred out of the plurality of sensing units output the events. 8. The method of claim 7,
The motion-
A plurality of space-time correlators for respectively calculating a space-time correlation between the target sensing units using events input from the target sensing units; And
And a motion trajectory tracking unit for tracking the motion trajectory of the part in which the motion occurs using the calculated high and low values of the space-
And a motion recognition device. 9. The method of claim 8,
Wherein the plurality of space time correlators each have an internal state value indicating a time-space correlation between the target sensing units, incrementing the internal state value each time the events are input, And determines the level of the time-space correlation between the target sensing units. 10. The method of claim 9,
And a light measuring unit for measuring intensity of light,
Wherein the plurality of space-
The threshold value is set in consideration of the intensity of the light
Motion recognition device. 10. The method of claim 9,
Further comprising a depth calculating unit for calculating a depth of a portion where the motion occurs by using a parallax between the plurality of image sensors when the image sensors are plural,
Wherein the plurality of space-
The threshold value is set in consideration of the depth of the portion where the motion occurs
Motion recognition device. 9. The method of claim 8,
Wherein the motion trajectory tracking unit generates one cluster - a shape corresponding to a portion where the motion occurs, by grouping space time correlators having high temporal correlation between the target sensing units among the plurality of space time correlators, Wherein the motion locus of the part in which the motion is generated is correlated with the center position of the cluster that has been previously calculated and the center position of the previously calculated cluster. 9. The method of claim 8,
Wherein the plurality of space-time correlators are each mapped to the divided regions of the image sensor, and the divided regions overlap with at least one surrounding region. The method according to claim 1,
An operation control unit for outputting a control command with reference to the determined operation pattern,
Further comprising: Sensing events of a subject photographed by the image sensor and outputting events;
Tracing a motion trajectory of the motion-generated portion using the time-space correlation diagram of the output events; And
Determining an action pattern of the motion-generated portion from the traced motion trajectory
Gt; a < / RTI > 16. The method of claim 15,
Prior to outputting the events,
And outputting light toward the subject to sense motion of the subject
Further comprising the steps of: 16. The method of claim 15,
Prior to outputting the events,
Measuring the intensity of light; And
When the intensity of the measured light is smaller than the predetermined light amount,
Further comprising the steps of: 16. The method of claim 15,
Calculating a parallax between the image sensors using a histogram of events output from the plurality of image sensors when the image sensors are plural; And
Synthesizing events output from the plurality of image sensors based on the calculated parallax;
Further comprising the steps of: 19. The method of claim 18,
Calculating a depth of the motion-generated portion using the calculated parallax
Further comprising the steps of: 16. The method of claim 15,
Wherein the image sensor comprises a plurality of sensing units,
Wherein, in the step of including the events, target sensing units corresponding to a portion of the plurality of sensing units where the motion occurs output the events. 21. The method of claim 20,
Calculating a space-time correlation between the target sensing units, using a plurality of time-space correlators, the events input from the target sensing units
Further comprising:
Wherein the tracing comprises:
Wherein the movement trajectory of the portion where the motion occurs is tracked using the calculated high and low values of the space-time correlation. 22. The method of claim 21,
Wherein the plurality of space time correlators each have an internal state value indicating a space-time correlation between the target sensing units,
Wherein the calculating step comprises:
Wherein the internal state value is incremented each time the events are input, and the increase or decrease in the space-time correlation between the target sensing units is determined by comparing the increased internal state value with a preset threshold value. 23. The method of claim 22,
Further comprising the step of measuring the intensity of light,
Wherein the calculating step comprises:
The threshold value is set in consideration of the intensity of the light
Motion recognition method. 23. The method of claim 22,
Further comprising the step of calculating a depth of a portion where the motion occurs by using a parallax between the plurality of image sensors when the image sensors are plural,
Wherein the calculating step comprises:
The threshold value is set in consideration of the depth of the portion where the motion occurs
Motion recognition method. 22. The method of claim 21,
The step of tracking the motion trajectory includes:
Grouping space-time correlators having a high time-space correlation between the target sensing units among the plurality of space-time correlators to generate one cluster - a shape corresponding to a portion where the motion occurs;
Calculating a center position of the generated cluster; And
Tracking the motion locus of the part in which the motion has occurred by associating the calculated center position with the center position of the previously calculated cluster
Gt; a < / RTI > 22. The method of claim 21,
Wherein the plurality of space-time correlators are each mapped to the divided regions of the image sensor, and wherein the divided regions overlap with at least one surrounding region. 16. The method of claim 15,
Outputting a control command with reference to the determined operation pattern
Further comprising the steps of: An image sensor that senses a part where a motion occurs in a subject and outputs events; And
A motion trajectory of a portion where the motion is generated by using a time-space correlation diagram of the output events, determining an operation pattern of the motion-generated portion from the traced motion trajectory, referring to the determined motion pattern, At least one processor configured to identify a user's input signal,
And an operation input unit for inputting an operation input of the user.

Images