JP2011198161A - Object recognition system, and monitoring system and watching system using the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an object recognition system capable of improving segmentation and position estimating accuracy, improving recognition accuracy of attitude and operation as a result, and detecting the occurrence of an abnormal situation with high reliability, and to provide a monitoring system and a watching system.SOLUTION: The object recognition system includes a plurality of same kind or different kinds of observing device including at least two of a one-dimensional sensor for acquiring one-dimensional data of an object, a two-dimensional sensor for acquiring two-dimensional data of the object, and a three-dimensional sensor for acquiring three-dimensional data of the object, and a recognizing device for recognizing the object in association with the one-dimensional data, two-dimensional data and three-dimensional data obtained from the observing device.

Description

  To ensure security, such as the discovery and tracking of suspicious persons (suspicious persons) such as terrorists, tails, pickles, fights, etc. in public places such as subways, stations, airports, etc. In order to detect abnormalities by detecting and recognizing human postures / motions / movements and absence of owners as an essential technology for surveillance systems or monitoring systems for elderly people, children, sick people, etc. in buildings such as homes and schools The present invention relates to a monitoring system and a monitoring system using multi-sensor information integration technology.

  Most conventional surveillance systems use fixedly installed surveillance cameras. On the other hand, with regard to the monitoring system for elderly people and children using portable sensors, there are some that have children equipped with GPS-equipped mobile phones, and there are no others. There are many examples where multiple cameras are installed, but there is no practical use of a modality of different media types, for example, a method that complementarily improves reliability using a visible light camera and an accelerometer, etc. It was. Most conventional surveillance technologies use only one media type (mainly a video camera), so even if it can cover a wide range, detection of suspicious persons, suspicious objects, and detection of abnormal situations is human. I had to rely on visual inspection. The present invention reinforces these drawbacks and aims to realize a highly reliable automatic monitoring system and watching system.

  Other than surveillance cameras, suspicious persons and suspicious objects are typically detected by X-ray inspection at airports, visual inspection by officials, thermography inspection to detect fever, facial photographs and fingerprint verification. The method of effectively integrating information from dissimilar media such as cameras and accelerometers is still just the beginning, and past research has focused on the automatic detection of stationary objects and specific moving people using image processing algorithms. It is limited to motion recognition (walking, running, falling, etc.) in a relatively stable environment that is not so complicated, and has not yet reached a level where a practical 24-hour monitoring system and watching system can be realized.

Teddy Ko, “A survey on behavior analysis in videosurveillance for homeland security applications,” 37th IEEE Applied Imagery Pattern RecognitionWorkshop, Washington, DC, USA, pp.1-8, Oct. 15-17, 2008.

  However, the conventional monitoring system and watching system have the following problems. In other words, there are many cases where segmentation is difficult, inaccurate, or person tracking fails when only a single piece of media information is used, and there is a problem that accuracy and reliability for detecting an object are lacking. there were.

In addition, there are many cases where high accuracy is required in the position estimation of the object, but it has been difficult to achieve without using expensive equipment. In addition, camera information is often used in human posture / motion recognition systems, but when it is considered including monitoring and watching, scenes that are difficult to judge from images often appear. Ensuring accuracy and high reliability is a major issue.

The present invention is an object recognition system proposed to solve such problems, and is a plurality of the same type and / or different types for observing a monitoring area and / or a monitoring area (hereinafter referred to as an observation area). And a recognizing unit for recognizing an object in the observation region by associating a plurality of data obtained from the respective observation units. As the observation means, a one-dimensional sensor that acquires one-dimensional data, a two-dimensional sensor that acquires two-dimensional data, and a three-dimensional sensor that acquires three-dimensional data can be used. Examples of the one-dimensional sensor include an accelerometer, an angular accelerometer, and an inclinometer. An example of the two-dimensional sensor is a camera that captures an image of an object. In general, a visible light camera is often used, but a near-infrared camera, a far-infrared camera, a thermography, or the like can also be used. Examples of the three-dimensional sensor include a range sensor that acquires distance image data in a three-dimensional space, a three-dimensional position sensor, a three-dimensional motion sensor, or a three-dimensional distance measuring sensor.

  There were many cases where segmentation was difficult or inaccurate with a single piece of media information, or people tracking failed, but according to the present invention, a request to execute segmentation with higher accuracy is required. Can respond. This problem is solved by extracting the object region by associating the one-dimensional data, the two-dimensional data, and the three-dimensional data, and recognizing the object using the result. Here, cutting out the object region is called segmentation.

Further, according to the present invention, the plurality of observation means are different from each other in at least one of data type, installation location, and observation direction, and the recognition means is obtained from the plurality of observation means. There may be included superimposing means for associating and superimposing the position data, and position detecting means for detecting the position of the object based on a portion where a plurality of position data overlap as a result of the superimposing means being superimposed. According to this, by using a plurality of sensors, it is possible to achieve high-accuracy position estimation, which was impossible with a single sensor, and to realize a more accurate monitoring system and monitoring system. Spread.

In the present invention, the observation means includes the one-dimensional sensor, the two-dimensional sensor, or the three-dimensional sensor.
Based on the one-dimensional data obtained from the one-dimensional sensor, the recognition means detects the direction and / or movement of the object, and the like, and the two-dimensional data or 3 obtained from the two-dimensional sensor or the three-dimensional sensor. Based on the dimensional data, a posture detecting means for detecting the posture and / or orientation and / or motion of the object, a direction and a state of movement detected by the orientation detecting means, a direction detected by the posture detecting means and the like And / or state detection means for detecting the state of the object from the posture and / or movement. Camera information is often used in human posture / motion recognition systems, but when monitoring and watching are taken into consideration, a high-accuracy and high-reliability recognition system is ensured even in situations where it is difficult to judge from images alone. can do.

Further, the present invention provides an upright state determination unit that determines whether or not a person who is an object is standing based on data obtained from a one-dimensional sensor, a two-dimensional sensor, or a three-dimensional sensor. When the standing state determination means is determined to be in a standing state, it is determined whether the person is stationary, walking or running based on the moving speed obtained from each sensor. Moving state determination means for performing the above-described operation. The moving speed can be obtained from the one-dimensional sensor, the two-dimensional sensor, or the three-dimensional sensor.

Further, according to the present invention, based on two-dimensional data or three-dimensional data, a number determination unit that determines whether there is one person or two or more people within a certain range in the observation area, and a number determination unit When it is determined that the person is one and the movement state determination unit determines that the person is in a walking state, the movement efficiency calculation unit that calculates the movement efficiency, and the movement efficiency is 1 or less and the first The movement efficiency determination means for determining whether the movement efficiency determination means is less than or equal to the threshold, is less than or equal to the first threshold and is greater than or equal to the second threshold, and is less than or equal to the second threshold. If it is determined that the threshold value is greater than or equal to the threshold value, it is determined that the state is normal, and if it is determined that the threshold value is equal to or less than the second threshold value and is equal to or greater than the second threshold value, it is determined that the person is drinking, If it is determined that the movement recognition means It can also be provided with.

Here, the movement efficiency is a parameter that indicates how efficiently the movement is primarily performed, and takes a value in the range of 0 to 1. Specifically, the ratio of the travel distance to the displacement distance ( (Hereinafter referred to as SR), the difference between the position before the T frame and the current position, and the ratio of the total distance moved between them (hereinafter referred to as DR) can also be used. Details of the meanings of SR and DR will be described later. It is also possible to extract the meaningless movements from unnecessary movements and irregularities of the limbs, useless movements such as bending the waist, shaking the head and neck, and movements of the upper and lower bodies, and using the ratio.

When it is determined that the number of persons determination means is two or more and the movement state determination means is determined to be a walking state, a second movement efficiency calculation means for calculating movement efficiency, and the movement efficiency Is less than or equal to the first threshold or less than or equal to the first threshold, and the second movement efficiency determination means and the second movement efficiency determination means are less than or equal to 1 If it is determined that it is greater than or equal to the threshold, "If you are walking side by side, you are walking together" or "If the person behind is walking a certain distance from the first person, If it is determined that it is a “tailing state or an abnormal state that may be a suspicious person” and it is determined to be less than or equal to the first threshold value, there is a possibility of “playing, fighting, fighting” Second motion recognition means for determining.

When it is determined that the number of persons determination means is two or more and the movement state determination means is determined to be in a running state, a third movement efficiency calculation means for calculating movement efficiency, and the movement efficiency Is less than or equal to the first threshold or less than or equal to the first threshold, and the third movement efficiency determination means and the third movement efficiency determination means are less than or equal to 1 If it is judged that it is above the threshold, “If you are running side by side, run together with jogging etc. to be in a normal state” or “Run behind the person at a certain distance from the first person. If it is determined that it is a chasing state or an abnormal state that may be a suspicious person, and if it is determined to be less than or equal to the first threshold value, play a state of “running or chasing” Judging that there is a possibility of `` fighting, fighting, fighting '' A third operation recognition means that may comprise

Furthermore, the present invention, when it is determined that the standing state determination means is not in a standing state, the shape measure and the like calculation means for calculating the shape measure and / or the motion measure, and the shape measure calculated by the shape measure etc. It is also possible to provide posture detecting means for detecting a human posture based on the motion measure.

In homes, public buildings, school routes, etc., detect whether there are abnormalities such as sudden fall, cramping, or illness of sick people, children, elderly people, etc. It can be used to build a monitoring system for reporting.

In order to detect and track suspicious individuals, discover abnormal situations in the person being watched over, etc., it is strongly desired to develop a simple and highly accurate posture / motion recognition method as a post-processing after human candidate areas are detected. , A 24-hour automatic monitoring system and a monitoring system can be realized.

By using a plurality of sensors, segmentation and position estimation accuracy can be improved. As a result, posture / motion recognition accuracy is improved, and it is possible to detect an abnormal situation with higher reliability. In addition, since a three-dimensional position can be captured, accurate tracking is possible.

When detecting a person or object that moves a certain amount of time, in a crowded place, shielding often occurs, and there are many cases where it cannot be detected well due to the effects of lighting fluctuations. Highly robust object detection is possible. Since the preprocessing is stably performed with high accuracy, recognition accuracy and position estimation accuracy of the posture / motion of a person are improved, and correct tracking of a suspicious person or the like becomes possible.

Overall system configuration including processing means Process flow diagram of the entire system Diagram showing parameters related to bounding box Tracking when you change direction as soon as you pass Explanatory drawing of the three-dimensional distance measuring sensor in the first embodiment The figure explaining the segmentation by combined use of the image and 3D ranging sensor in a 1st embodiment The figure explaining the difference of the shielding area by a plurality of sensors The figure which shows the principle of the position estimation in 2nd Embodiment The figure which shows the attitude | position and operation | movement in 3rd Embodiment Explanatory drawing in the case of using together the fixed sensor and portable sensor in 3rd Embodiment The figure which shows the determination method of the standing state of the person in 4th Embodiment The figure which shows the classification | category method of the standing state of the person in 4th Embodiment The figure which shows the classification | category method when it is not the standing state of the person in 4th Embodiment The figure which shows the difference in the movement of the person in 4th Embodiment

  Hereinafter, a first embodiment of the present invention will be described based on the drawings and mathematical formulas. FIG. 1 is a configuration diagram of the entire system including processing means in the present embodiment. The system 100 includes a plurality of sensors 101 and 102 of the same type or different types that observe an object, and a recognition unit 103 that recognizes the object in the observation region by associating a plurality of data obtained from the sensors 101 and 102. ing. The sensors 101 and 102 have a function as observation means for observing a background and an object within a certain time, and are composed of a fixed sensor 101, a portable sensor 102, and the like.

  The following sensors may be used as observation means for observing scenes to be monitored and watched over within a certain period of time. The types of sensors are “0 dimension: position measurement GPS, temperature, humidity”, “1 dimension: accelerometer (3 axes), angular accelerometer (3 axes), inclinometer (3 directions, can be included in the angular accelerometer. ), Microphone (acoustic) ”,“ 2D: normal visible light camera, near infrared camera, far infrared camera, thermography ”,“ 3D: 3D data (range sensor, 3D position sensor, 3D motion sensor, 3D ranging sensor) ”and the like. Here, we will explain mainly using images for the purpose of original use and intuitive understanding, but the same applies to cases where observation data such as other media (such as 3D data) is missing. it can. Also, not only zero-order approximation but also linear approximation or higher-order approximation can be used to cope with various fluctuations, and the least mean square error (LMS) or maximum is used to minimize the error. A technique for obtaining an optimum value by minimizing an error may be properly used as necessary. The recognizing means 103 is composed of, for example, a personal computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), etc., and the CPU is a ROM or HDD. A function as a recognition unit is performed by executing a program stored in the above.

  Next, the flow of the entire system in the object recognition system (or monitoring system, watching system) of this embodiment will be briefly described. FIG. 2 is a processing flow diagram of the entire system in the present embodiment. First, the background as the initial value is obtained before the observation. Here, the background means a thing that is stationary for a certain period after the observation starts before the start of observation, regardless of the person or object, and the initial background is the background at the start of observation, the short-time background. A long-time background is distinguished by the length of stationary time. That is, even if it appears from the middle, it may be present in the scene for a certain period of time and remain stationary.

As shown in FIG. 2, the data from the sensor 200 is received (S201), and the background is extracted and updated (S202). First, an object region is extracted based on a background difference, an inter-frame difference or the like (S203), finely adjusted in position and size (S204), segmented (S205), and region features are extracted (S206). . Position estimation in a three-dimensional real space is performed (S207), and the posture and movement of a target stationary object or a moving object that moves in a certain manner are detected, and a suspicious object is detected (S208 to S213). Specifically, it is determined whether it is an object or a person (S208). If it is an object, it is determined as a suspicious object (S209), and tracking of the person who brought this suspicious object is started (S210). If it is a person, an abnormal situation is detected from the recognition result of posture / motion (S211 and S212), and if an abnormality is found, a notification or the like is taken (S213). A series of processing is completed by these procedures, but the above processing is repeated during monitoring and watching time.

In the present embodiment, the object is a stationary object or an object that moves in a certain manner and is a target of detection / recognition, and includes a person and an object. The target stationary object is a thing that does not exist at a certain time, for example, at the beginning of observation, regardless of a person or an object, and has existed in the scene from the middle and has been stationary since then. Point to. For example, it refers to something that someone has left behind, deliberately left behind, or a person who walks and sits in a chair. Those that existed from the beginning of the observation are not included. In addition, even if it appears from the middle, it may be present in the scene for a certain period of time and remain stationary.

As shown in FIG. 3, the bounding box is a minimum rectangle or a rectangular parallelepiped region surrounding a certain object, but the minimum one cannot always be extracted. The center of gravity is one parameter that represents the position of the candidate object region, and other parameters that represent the position of the candidate object region include the position of the upper left corner of the bounding box, the top point, and the bottom point. It is also possible to select etc. The silhouette is a black and white representation of an object (person, object) region extracted by background difference or the like. The three-dimensional data may collectively refer to distance image data in a three-dimensional space, and can be acquired from a range sensor, a three-dimensional position sensor, a three-dimensional motion sensor, a three-dimensional distance sensor, or the like.
Here, the object candidate area is an area before it is determined to be the object area, and the object area is determined based on the candidate area. Parameters that reflect the position and motion of the object include the silhouette of the object, the center of gravity of the bounding box, the maximum and minimum values of the area, the absolute value of the background difference, and a motion measure described later. Taking an image as an example, if an area on the image corresponding to this parameter is obtained, the position and size of the object can be known. The object parameter is a parameter that reflects the object, and the background parameter is a parameter that reflects the background. Specific examples include pixel values and 3D position data. Both are collectively referred to as region parameters. Taking an image as an example, if the area on the image corresponding to the target object position parameter is obtained, the target position, the target area, the size, and the like can be known.

  If a plurality of data obtained from the plurality of sensors are used in association with each other, FIG. 4 shows an example that can be easily performed using a 3D distance measuring sensor even when tracking is difficult with an image.

  A recognition unit that recognizes an object by associating a plurality of data obtained from the plurality of sensors will be described. When using a plurality of sensors, so-called segmentation, which cuts out an object region, is important. FIG. 5 is an explanatory diagram of a three-dimensional distance sensor, and FIG. 6 is a diagram for explaining segmentation by using an image and a three-dimensional distance sensor in combination with this embodiment. First, in the segmentation, by associating data from a plurality of sensors, both data can be used complementarily. In the present example, segmentation can be performed with high accuracy by using both the shape and the image. For example, the valid data range in FIG. 6A corresponds to the position of the girl in FIG. 6B, and accurate segmentation can be performed even if the color of the clothes is the same as the color of the surrounding wall. At that time, bidirectional spatial mapping can be considered. Specifically, for example, 2D image data may be superimposed on 3D curved surface data, or 3D ranging image data may be superimposed on a 2D image. In FIG. 6, the solid line on the image indicates the scan point.

  When the camera and the three-dimensional distance measuring sensor are not at the same position, shielding may occur at different positions as shown in FIG. 7, but this can be dealt with by giving a margin to the segment area. Further, if the relative positions of each other are known in advance, the shielding position can be found by calculation. If a sensor capable of obtaining three-dimensional data such as a three-dimensional distance measuring sensor is used, distance data can be obtained directly without using stereo vision, and a result can be obtained at high speed with high reliability. Shielding occurs basically at the same time as long as it is an observation value obtained from a sensor installed at the same location.

If the place is different, the occurrence of the shielding is different in time and place, so that complementary use is possible. In other words, shielding occurs at a certain sensor S ₁ (including cases where data loss or discontinuity is observed), and if shielding is not observed at sensor S ₂ (observation data is continuous), the target object is continuous. The shielding of S ₁ is interpreted as temporary, and the data of the shielding occurrence section can be obtained by prediction by linear interpolation or the like. Further, when the occurrence of shielding of the other sensor (S ₂ ) can be estimated from the information of _one sensor (S ₁ ), the accuracy is improved by excluding the data of S ₂ in that section and performing approximation. In general, linear approximation is simple and effective.

In FIG. 6, a solid line indicates a valid range of data from the three-dimensional distance measuring sensor, and a broken line indicates an invalid range (undecided) along the radial direction. Accurate segmentation is indispensable for preprocessing of recognition, and if there is data from a plurality of directions, more precise correspondence can be made, and more accurate segmentation is possible. By combining in this way, the position where a person is present can be known, and a 3D shape can also be acquired. Therefore, it is possible to determine abnormality based on information from an image and information from a three-dimensional distance measuring sensor. For example, if the height suddenly changes and becomes half or less, the possibility of slipping and falling is large, and this can be confirmed by an image. Although it can be replaced by a three-dimensional data such as a range sensor instead of the three-dimensional distance measuring sensor, it is generally a more expensive system. Further, since a sudden movement is caused by a fall or the like, a change can be observed from an accelerometer, an angular accelerometer, an inclinometer or the like.

Applications from the perspective of safe driving support are also conceivable. For example, an in-vehicle camera is used to extract a blind spot area such as a car pillar or a tree, and a three-dimensional position and direction of another vehicle near the own vehicle is detected, and an image from an outside camera or a fixed camera is used. If the image in the blind spot can be synthesized and presented to the driver, it will help improve safety. Based on the same idea, it is possible to construct an image acquisition / recognition system without a blind spot.

Image composition is possible by replacing the background image. In the chroma key, blue is often used as the background, so blue objects are interpreted as the background.However, if a point at an infinite distance or a distance above a certain distance is considered as the background, there is no such failure, and image composition can be performed. It becomes possible.

Next, a second embodiment of the present invention will be described. Differences from the first embodiment will be mainly described, and description of common points will be omitted. In the present embodiment, the plurality of sensors are different in at least one of data type, installation location, and observation direction. The recognizing means includes superimposing means and position detecting means to be described later.

In general, the error distribution (spreading method) varies depending on the installation location and type of the sensor, and a highly accurate position estimation method is performed using the difference. First, the superimposing means superimposes a plurality of position data obtained from a plurality of sensors with different error distributions in association with each other. FIG. 8 shows a state in which high accuracy is achieved by adopting an overlapping portion of regions where there are few errors in estimated positions from a plurality of sensors in the present embodiment. A broken line indicates a position estimated value from Img ₁ , and a dotted line indicates a position estimated value from Img _{2. A} more accurate position is obtained as a common part of both. However, the ellipse indicates a region within an estimation error range. In addition, the accuracy is increased by adopting a common area from a plurality of sensors. This example is an example of images, but can be extended between an image and three-dimensional data. The present invention can be similarly applied to data between visible light and infrared images, between images having different resolutions, and between a plurality of sensors. In particular, as shown in the example of FIG. 8, a method of improving the position estimation accuracy by mutual recognition (measurement) between the object-mounted sensor and the sensors installed in the vicinity is effective. By doing so, the accuracy increases because the direction of error spread is generally different. In other words, it is important to create sensor installations that have different distributions. Position estimation is indispensable for tracking an object. As described above, the position of the target object can be detected by the position detection unit based on the overlapping portion of the plurality of position data superimposed by the superimposition unit.

Next, a third embodiment of the present invention will be described. Differences from the first and second embodiments will be mainly described, and description of common points will be omitted. In the present embodiment, the posture state of an object can be detected by using a one-dimensional sensor such as an accelerometer, an angular accelerometer, or an inclinometer and a camera or a three-dimensional distance measuring sensor in combination.

  In this embodiment, in order to realize a robust and highly accurate human posture / motion recognition system that is resistant to shielding and illumination fluctuations, not only images but also different media type sensor information is used in combination. For example, it is difficult to determine whether it is lying on its back or lying down when it falls, but it can be reliably determined by using it in combination with an accelerometer, angular accelerometer, or inclinometer. If the data from the sensor worn by the subject is obtained, information such as which direction the body has fallen easily and which direction it has fallen can be obtained. On the other hand, if information from cameras and three-dimensional ranging sensors installed at a plurality of locations are integrated, much information can be obtained in the same manner. Here, information integration is performed in order to improve accuracy and reliability by using the same type and / or different types of sensors in a complementary manner. The orientation detection means detects the orientation and / or movement of the object from data such as an accelerometer, an angular accelerometer, and an inclinometer, and the orientation detection means processes the data of the camera or the three-dimensional sensor to process the object. Detect posture and / or orientation and / or movement. The state detection means detects the state of the object by using these pieces of information in a complementary manner, and will be described below with an example.

  Hereinafter, a posture detection method for detecting the posture state of an object will be described. When an accelerometer, angular accelerometer, or inclinometer is attached to the upper body, it is easy to recognize posture changes and movements such as “bend the waist, lower the waist, bend the upper body, tilt the upper body, and crouch.” However, there are some operations that do not appear so distinct, such as staggering, stumble, dragging, and slowing down, and must be used in combination with images or 3D data. Since accelerometers, angular accelerometers, and inclinometers are not shielded, they can be detected regardless of viewing direction or location. FIG. 9 shows the posture and movement in the present embodiment. For “walking, running, stumbling, bending the waist, sitting” in the upper stage, from the image and the three-dimensional data, the movement speed is “stop, It can be distinguished by “slow and fast”, shape measure “aspect ratio, moment ratio, height (H), Top”. Here, the shape measure is a monosashi that measures the shape of a thing, for example, the shape itself, size, circularity, elongation, height and width, moment, aspect ratio, moment ratio, contour line, Fourier descriptor, There are histograms, HOG, etc. A HOG (Histgrams of Oriented Gradients) feature is a gradient-based feature, and a Fourier descriptor (Fourier Descriptor) is a feature representing the shape of a boundary curve of an object. A motion measure is a monosashi that seeks a difference in motion, and includes speed, acceleration, angular velocity, angular acceleration, inclination angle, movement efficiency (details will be described later), and the like. As described above, the direction and / or movement of the object can be detected based on data such as an accelerometer, an angular accelerometer, and an inclinometer.

In addition, although it is easy to distinguish between the upper and lower tiers, in the lower part of the figure, for “falling (downside down), falling (facing up), falling (sideways), resting”, whether you have fallen or are resting? It is difficult to determine whether or not, but it can be easily known from the data of accelerometers, angular accelerometers and inclinometers. Even when only images and three-dimensional data are used, it is possible to determine the posture and motion of the object, that is, whether it falls or rests from the progress to this state. If it suddenly changed, it was a fall, and if it was slow, it just lay down for rest. When it falls, a large angular acceleration is observed due to the rotation. However, if the orientation is simply changed, the rotation will occur, but the height will remain unchanged (as seen from the image and 3D data), and the observed angular acceleration data will also differ. In addition, when standing still, “whether standing, sitting, or lying” is still, so it can be judged relatively easily from images and 3D data, but accelerometers, angular accelerometers, It cannot be judged from the inclinometer.

FIG. 9 shows an example, and the sensor has such advantages and disadvantages. By using different sensor information in an integrated manner, a weak recognition can be compensated and a highly reliable recognition result can be obtained. As described above, the state of the object can be detected based on data such as an accelerometer, an angular accelerometer, and an inclined shape, and a plurality of different data such as images and three-dimensional data.

  There is another aspect to sensor information integration. That is, the above-described integration of different types of sensor information can realize high-accuracy recognition by integrating the same type of sensors. FIG. 10 is an example in one embodiment, but the situation can be recognized from two images of a fixed camera and a portable camera. For example, when the camera falls, a sudden change is observed in both cameras, indicating that an abnormal situation has occurred. Mobile camera images change dramatically, and the direction and position of people can be determined by what is reflected. At the same time, an abnormal situation is confirmed from the sensor. This situation is also observed from the information of the one-dimensional sensor. In this way, by performing information integration with a combination of “fixed, mobile” × “same type / different type” × “multiple sensors”, it is possible to reliably detect a fall. Here, an example of a fall was given for the sake of simplicity, but the present invention can be similarly applied to detection of other abnormal situations. At this time, the occurrence of abnormality can be understood from the longitudinal acceleration in the sensor data during walking. For example, Fig. 10 shows, “When you are walking normally, a periodic change (1), after turning around the corner and running (2) turning to tin (3) ⇒ an abnormality occurred. Indicates that the reliability is improved by making a judgment together with the portable and / or fixed camera image. "

The following equation shows the definition of a moment that is generally used as a commonly used shape feature, but any parameter that can express the dispersion around a point or an axis can be used instead of the moment. For example, histogram dispersion along a certain axis can be used.

The moment around the center of gravity is expressed as

The vertical and horizontal moments passing through the center of gravity are expressed as follows:

Here, all moment parameters can be used independently, but the ratio ((4), (8)) is more effective. In the above two types of moment parameters, the moment calculation of (2), (3), (6), (7) is divided into upper and lower here, but when using for human detection, the ratio of height is set The ratio is divided into three parts (head, torso, leg) (can be divided into two parts around the waist, or divided into some meaningful parts) and calculated for each range (for each interval) Height: k ₁ * H to k ₂ * H, 0 ≦ k ₁ <k ₂ ≦ 1). In this way, even if there is partial shielding, the visible portion can be used and a certain degree of judgment can be made, so that a robust system that is strong against shielding can be constructed. As described above, the moment can be substituted by dispersion or the like.

In addition, the following equation is used as a motion measure.

The above is a two-dimensional story, but it can be similarly defined in a three-dimensional space. SR and DR may be defined using the reciprocals of equations (8) and (9), but here take values in the range of 0 to 1 (0 ≦ SR ≦ 1, 0 ≦ DR ≦ 1). Define as above. In general, a frame means an image, but it can also be applied to other multimedia contents such as three-dimensional data. Content is a general term for data obtained from sensors, and when content changes with time, data at each time is called a frame. For example, taking a moving image as an example, a plurality of frames (still images) are collected to form one moving image. In general, a frame is often used for an image, but here, the same term is used for other contents.

  Next, a fourth embodiment of the present invention will be described. Differences from the above-described embodiment will be mainly described, and description of common points will be omitted. In the present embodiment, an upright state determination unit is provided that determines whether or not a person who is an object is standing based on image data from a camera.

First, the determination method by the standing state determination means will be described. Human posture / motion recognition means are broadly classified into “standing” and “other”. Here, the “standing” state is called the standing state. First, using the silhouette binarized from the background difference or the like, the posture is first classified into the standing posture and the rest as shown in FIG. This process can be understood at a glance without taking the movement into consideration, and can be easily and accurately discriminated from one silhouette. Although it is possible to use a plurality of sheets for the purpose of removing noise and improving accuracy, they can be identified from a single silhouette. After that, by using the information of multiple sensors in a complementary and integrated manner and setting detailed discrimination rules (details will be described later) that match each posture and movement, the other states can be further classified (small) In this way, it is characterized by improving the accuracy by narrowing down the possibilities, at the same time achieving high speed, and ensuring reliability by combining parameters obtained from a plurality of sensors. FIGS. 12 and 13 show examples of standing items and other detailed classification items in this embodiment, respectively. Here, a description has been given using silhouettes, but the present invention can be applied not only to images but also to three-dimensional data.

  A case where the standing state determination unit determines that a person is standing will be described. As an example of the present embodiment, an example of a detailed determination rule in a “standing” state is shown. “Stop” does not move, “walk, run” can be distinguished by the difference in speed, and “running, full power” can be distinguished by the difference in speed (quasi-cycle of movement). That is, based on the moving speed, it can be determined whether the person is in a stationary state, a walking state, or a traveling state. “Standing” can be categorized as “Standing, Walking, Running, etc.” (Normal), “Drowsy” (Wandering), “Drunk (Drunken)” (Abnormal), The situation is shown in FIG. The movement state determination means classifies the movement from the data from the one-dimensional sensor or the camera image three-dimensional sensor using a movement measure. Here, an image is used for intuitive understanding, but data from other media (such as three-dimensional data and an accelerometer) may be used. I haven't seen any paper that strictly distinguishes “walking, drunk”, but here we define it as follows: “Wandering” means repeating the same thing without the purpose, meaningless, and the action of the same person appearing in the same place many times, often turning the neck well. In addition, “drunk state (drunk)” does not repeat as many times as “uro”, but it is not stable, shakes hands, seems to stop for a moment, and starts walking again. The difference from the average is quite large.

An illustration of a typical trajectory of movement is shown in Figure (a). As an example, a rule that distinguishes the following operations can be created.
(1)
Normal walking, running: SR ≒ 1 or slightly less than 1 1 ≧ SR ≧ Th ₁
When you walk, you can distinguish running (running, full power, etc.) by the difference in moving speed.
(2)
Drunk: …… Th ₁ ≧ SR ≧ Th ₂
(3) Wandering: ... Th ₂ ≧ SR ≧ 0

However, SR should not be greater than 1 in principle (SR ≦ 1). The movement distance is more accurate when converted to a three-dimensional real space rather than an image, and data such as three-dimensional data or an accelerometer may be used. The threshold is determined by experiment. Instead of SR, the difference (D) between the position before the T frame and the current position, and the ratio of the total distance (S) moved between them (DR = D / S) may be used. As described above, SR is equal to or less than 1 and equal to or greater than the first threshold (Th ₁ ), equal to or less than the first threshold and equal to or greater than the second threshold (Th ₂ ), or equal to or less than the second threshold. It is possible to determine “normal state”, “drunk state”, and “drinking state”.

A case where the standing state determination means determines that the person is not standing will be described. In this case, the shape measure and the motion measure of the object can be calculated, and the human posture can be recognized based on the calculated shape measure and the like. In the present embodiment, parameters including the bounding box aspect ratio, moment ratio, height, width, and histogram gravity center position are used as the shape measure. Details will be described below.

As an example of the present embodiment, an example of a detailed determination rule of “other than standing” is given below. When squat from standing state, the center of gravity G _y of the entire bounding box, or silhouette (t) is lowered (e.g., change from G _y in about half 1 / 2G _y), jumping the up (e.g., about twice 2 * G _y ). An appropriate threshold is set in actual use. As can be seen from FIG. 13, it can also be determined that the silhouette disappears from the upper part or the lower part of the first bounding box. This means that if a three-dimensional distance measuring sensor is used, the upper (upper body part) data disappears when squatting, and the lower (lower body part) data disappears when jumping. Here, the loss of data means that a background appears.

Changes in parameters representing posture are used for instantaneous motion recognition, and some examples are given below.
MR: Medium to small, AR: Medium to small, H: Medium to large, W: Medium to small, _Gy : Medium to high (sudden)
Turn right MR: Small, AR: Small, H: Large, W: Small, _Gy : High Sit MR: Small → Medium, AR: Small → Medium H: Large → Medium, W: Small → Medium, _Gy : High → Middle Lying MR: Medium → Large, AR: Medium → Large, H: Medium → Small, W: Medium → Large, _Gy : Medium → Low (slow)
Jump Top, Bottom: Medium → High (Sudden), G _y : → High (Sudden)
It moves rapidly to a higher position along with the center of gravity of the bounding box.
Depending on how you raise your feet, you can take various postures, so decide how to raise your feet.
MR: High → Medium, AR: High → Medium, H: Large → Medium, W: Small → Medium, _Gy : High → Medium (Normal)
Top, Bottom: High → Medium (Normal)
Along with the center of gravity of the bounding box, it moves to a low position at a normal speed.
Apply different rules depending on whether you are facing front or side.

Here, represents the width of the whole body of the bounding box is W, the height H, the aspect ratio AR, the moment ratio MR, or y-direction center of gravity of the histogram in G _y. The change speed represents a sudden change, a slow change, and a normal change at a normal speed. Similarly, rules can be created using bounding boxes and silhouette features. Here, in order to express relative values, qualitative expressions such as “large, medium, small, high, medium, low, and normal” are used, but actually, a threshold value is specifically set. Generally speaking, the aspect ratio of the bounding box (AR: W / H) and the moment ratio _{(MR: m 20 / m 02} , m i (90) / m i (0)) is always show the same trend, More stable results are often obtained using the moment ratio. In addition, in the same standing state, it can be determined that MR and AR are small, but the threshold value for that is different depending on the front direction and the horizontal direction. Whether it is front or sideways can be investigated using (1) silhouette histogram, (2) silhouette shape, (3) objectivity, etc.

As a measure of symmetry with respect to the vertical axis, a cross-correlation coefficient of the following equation 5 can be used. A front-facing image has high symmetry (a large cross-correlation coefficient is close to 1), and in a horizontal direction, the symmetry is lower than the front (the cross-correlation coefficient is small and close to 0). The maximum value Max of the normalized cross-correlation coefficient is obtained while translating the image X ′ obtained by horizontally inverting the binary image X (i, j) in the bounding box. If the value is larger than a certain threshold, the symmetry is obtained. Is high. If the symmetry is high, it is determined to face forward, otherwise it is determined to face sideways. Reliability can be increased by using in combination with other parameters.

When it is determined that the number of persons determination means is two or more and when the movement state determination means is determined to be a walking state, the movement efficiency determination means calculates the movement efficiency, and the motion recognition means determines the value. If it is determined that the value is equal to or less than 1 and equal to or greater than the first threshold, it is determined that the condition is “a normal state walking together” or “an abnormal state that may be a tailing state or a suspicious person”. If it is determined that the value is equal to or less than the threshold value, it is determined that there is a possibility of “playing, fighting, fighting”. Further, the determination is made together with the shape measure and the motion measure. For example, if you move your limbs quickly, it is judged that there is a high possibility of fighting and fighting, and alerts are called.

When it is determined that the number of persons determination means is two or more and the movement state determination means is determined to be in a running state, the movement efficiency determination means calculates the movement efficiency, and the motion recognition means determines the value. If it is determined that the value is equal to or less than 1 and equal to or greater than the first threshold, it is determined that the state is “normal state running together”, “following state, or abnormal state with possibility of suspicious person”, When it is determined that the value is equal to or less than the first threshold value, it is determined that there is a possibility of “playing or running away”. Further, the determination is made together with the shape measure and the motion measure. For example, if you go around the same place, you are chasing and playing, so if you leave it a little away and move linearly at high speed, there is a high possibility of “running / chasing” Judge and call for vigilance.

When it is determined that the standing state determination unit is not in a standing state, the shape measure calculation unit calculates the shape measure and / or the motion measure, and the posture detection unit calculates the shape measure calculation unit and the shape measure calculation unit. The human posture is detected based on the shape measure and / or the motion measure calculated by.

Regarding position estimation, the difference in the error distribution of each sensor is different.For example, if one is spread in the east-west direction and the other is in the north-south direction, both areas have the same error variance. As a result, the position can be estimated with higher accuracy.

  Regarding the detection and tracking of a plurality of people, it is easy to separate them by installing a camera, a sensor, etc. as high as possible so as not to overlap as much as possible.

  Portable sensors such as small cameras, GPS, accelerometers, angular accelerometers, and inclinometers can be worn with as little physical restraint as possible, and must be small, power-saving, and inexpensive. For sensors that are assumed to be used fixedly, such as distance measuring sensors, it is necessary that the installation location be ensured at an optimal position so that detection is possible with high accuracy with less shielding.

100 network 101 fixed sensor 102 portable sensor 103 recognition means

Claims (12)

A plurality of same and / or different observation means for observing a monitoring area and / or a monitoring area (hereinafter referred to as an observation area);
An object recognition system comprising recognition means for recognizing an object in an observation region by associating a plurality of data obtained from each of the observation means. At least two of the one-dimensional sensor that acquires the one-dimensional data of the object, the two-dimensional sensor that acquires the two-dimensional data of the object, and the three-dimensional sensor that acquires the three-dimensional data of the object. Including more than one,
2. The object recognition system according to claim 1, wherein the recognition means recognizes the object by associating at least two or more of one-dimensional data, two-dimensional data, and three-dimensional data obtained from the observation means. The one-dimensional sensor is an accelerometer, an angular accelerometer, or an inclinometer,
The two-dimensional sensor consists of a camera that captures the observation area,
The 3D sensor consists of a range sensor, 3D position sensor, 3D motion sensor, or 3D ranging sensor.
The recognition means includes one-dimensional data obtained from an accelerometer, an angular accelerometer, or an inclinometer, an image captured by a camera, a range sensor, a three-dimensional position sensor, a three-dimensional motion sensor, or a three-dimensional distance sensor. The object recognition system according to claim 2, wherein at least two or more pieces of position data in a three-dimensional space obtained from are recognized in association with each other. The plurality of observation means are different in at least one of data type, installation location, and observation direction,
The recognition means is based on a portion obtained by superimposing and superimposing a plurality of position data having different error distributions obtained from the plurality of observation means, and a portion where the plurality of position data overlap as a result of the superimposition means being superimposed. The object recognition system according to claim 1, further comprising position detection means for detecting the position of the object. The observation means includes the one-dimensional sensor and the two-dimensional sensor or the three-dimensional sensor,
The recognition means is
Detecting means for detecting the direction and / or movement of the object based on the one-dimensional data obtained from the one-dimensional sensor;
A detection means for detecting the posture and / or orientation and / or movement of an object based on two-dimensional data or three-dimensional data obtained from a two-dimensional sensor or a three-dimensional sensor;
5. A state detection unit that detects a state of an object from a direction and a state of movement detected by the direction detection unit and a posture and / or direction and / or movement detected by the movement detection unit. The object recognition system according to any one of the above. Based on the data obtained from the two-dimensional sensor or the three-dimensional sensor, standing state determination means for determining whether or not the person who is the object is standing;
When it is determined that the standing state determination means is in a standing state, based on the moving speed obtained from each sensor, it is determined whether the person is in a stationary state, a walking state, or a traveling state. State determination means;
An object recognition system according to any one of claims 1 to 5. Based on two-dimensional data or three-dimensional data, it is determined that there is one person determination means and one person determination means for determining whether one person is nearby or two or more persons within a certain range in the observation area. A movement efficiency calculation means for calculating a parameter (hereinafter referred to as movement efficiency) indicating how much a person has moved when the movement state determination means is in a walking state. A movement efficiency determination means for determining whether the movement efficiency is equal to or less than 1 and equal to or greater than the first threshold, equal to or less than the first threshold, equal to or greater than the second threshold, and equal to or less than the second threshold;
When the movement efficiency determination means determines that the moving efficiency determination means is 1 or less and is equal to or greater than the first threshold, it is determined to be in a normal state, and when it is determined that the movement efficiency determination means is equal to or less than the first threshold and is equal to or greater than the second threshold, An action recognizing means for determining that there is a heel state when it is determined that there is a second threshold or less;
An object recognition system according to claim 6. A second movement efficiency calculating means for calculating movement efficiency when the number of persons determining means is determined to be two or more and the moving state determining means is determined to be a walking state;
A second movement efficiency determination means for determining whether the movement efficiency is equal to or less than 1 and equal to or greater than a first threshold, or equal to or less than a first threshold;
When the second movement efficiency determination means determines that the value is equal to or less than 1 and equal to or greater than the first threshold, it is “normal state walking together” or “an abnormal state that may be a tail state or a suspicious person” And the second motion recognition means for determining that there is a possibility of "playing, fighting, fighting",
An object recognition system according to claim 6. A third movement efficiency calculating means for calculating the movement efficiency when the number of persons determining means is determined to be two or more and the moving state determining means is determined to be in a running state;
Third movement efficiency determination means for determining whether the movement efficiency is equal to or less than 1 and equal to or greater than a first threshold, or equal to or less than a first threshold;
If the third movement efficiency determination means determines that the value is equal to or less than 1 and equal to or greater than the first threshold, the “normal state running together”, “the state of chasing, or the possibility of a suspicious person” Third action recognition means for determining that there is a possibility of “playing or running away” when it is determined that the state is “a certain abnormal state” and is equal to or less than the first threshold; ,
An object recognition system according to claim 6. When it is determined that the standing state determination means is not in a standing state, a shape measure and the like calculation means for calculating a shape measure and / or a motion measure;
Posture detecting means for detecting a human posture based on the shape measure and / or the motion measure calculated by the shape measure calculation means;
An object recognition system according to claim 6 or 7. A monitoring system using the object recognition system according to claim 1. A watching system using the object recognition system according to claim 1.