JP6740614B2 - Object detection device and image display device including the object detection device - Google Patents

Description

The present invention relates to an object detection device and an image display device including the object detection device.

In recent years, equipped with an object detection device that detects the target object based on image information, functions such as writing characters and figures in the projected image projected on the screen, and operations such as enlargement, reduction, and page turning of the projected image. A so-called interactive projector device having a function of executing has been developed. These functions use the operator's finger touching the screen, the pen and pointing stick held by the operator as the input operation unit, and detect information such as the position and movement of the input operation unit, and the detection result. Is realized by sending to a computer or the like.

As such a projector device, distance information of a screen serving as a background object and distance information of an input operation unit are respectively acquired by a distance image sensor, and the position and movement of the input operation unit with respect to the screen are three-dimensionally obtained from the information. A technique has been proposed (see, for example, Patent Document 1). Known range image sensors include a range image sensor using a stereo camera, a TOF (Time-of-Flight) sensor, and a sensor based on a pattern projection method.

On the other hand, there is also disclosed an intrusion detection device that detects an intruding object into the surveillance space by using distance information (for example, see Patent Document 2). In Patent Document 2, distance information of each of a background object and an intruding object is measured for the purpose of reducing malfunctions due to pixels incapable of ranging and improving intrusion detection accuracy of an intruder, and one of those information is measured. A technique is disclosed in which, when there is a pixel with a reduced distance accuracy, the pixel is also extracted as a distance-changed pixel to determine the presence or absence of an intruding object.

By the way, a background object such as a screen or a building and a target object such as an operator's finger or an intruding object have different reflectances with respect to the measurement light used by the distance image sensor, and therefore, the position or movement of the target object based on the distance information. In order to obtain such information with high accuracy, it is necessary to consider the difference in reflectance.

The present invention has been made in view of the above circumstances, and an object thereof is to detect information regarding a target object with high accuracy based on distance information.

In order to achieve the above object, an object detection device according to the present invention is an object detection device that has a distance measuring unit and acquires distance information within a predetermined visual field from the distance measuring unit. The control unit controls a condition when acquiring the distance information about the field of view, and the control unit acquires the first distance information about the field of view under the first condition suitable for the reference object existing in the field of view. , The distance measuring unit is controlled to acquire the second distance information in the visual field under the second condition suitable for the target object existing in the visual field, and the reliability of the value regarding the reference object in the second distance information is determined. If it is determined that the reliability is high, the reference distance from the reference position in the distance measuring unit to the reference object and the target distance from the reference position in the distance measuring unit to the target object are obtained from the second distance information, and the reliability is calculated. If it is determined that is low, the reference distance from the reference position in the distance measuring unit to the reference object is obtained based on the first distance information, and the target distance from the reference position in the distance measuring unit to the target object is the second distance. It is characterized in that the state of the target object with respect to the reference object is detected based on the reference distance and the target distance obtained from the information.

According to the present invention, it is possible to detect information about an object with high accuracy based on distance information.

FIG. 1 is a schematic diagram schematically showing the configuration of an image projection system including an image projection device having a state determination device as an object detection device according to the first embodiment of the present application, and an image information device. It is the schematic which showed the structure of the image projection apparatus shown in FIG. 1 roughly. FIG. 3 is an optical diagram for explaining a configuration example of a projection unit shown in FIG. 1. It is the schematic which showed the example of 1 structure of the ranging part using the method based on a pattern projection method. It is the schematic which showed the example of 1 structure of the ranging part using the system based on a stereo method. It is the schematic which showed the example of 1 structure of the ranging part using the system based on the TOF method. FIG. 6 is an explanatory diagram for explaining an example of the state determination of the input operation unit by the state determination unit, showing a state in which the input operation unit is not present near the projection surface. FIG. 9 is an explanatory diagram for explaining an example of the state determination of the input operation unit by the state determination unit, showing a state in which the input operation unit exists near the projection surface. It is a flow chart which shows an example of the state judging processing performed with a state judging device. 6B is a flowchart showing an example of depth map generation processing in the state determination processing of FIG. 6A. It is a figure which shows the relationship between a brightness value and a ranging error. It is a figure which shows the relationship between a brightness value and data reliability. It is explanatory drawing for demonstrating the production|generation procedure of the depth map for a state determination by a depth map production|generation part. It is a perspective view which shows an example of the electronic blackboard apparatus which concerns on 2nd Embodiment of this application. It is a perspective view which shows an example of the digital signage apparatus which concerns on 3rd Embodiment of this application.

(First embodiment)
An embodiment of an image projection system (interactive projection system) according to the first embodiment of the present application will be described below with reference to the drawings. As shown in FIGS. 1 and 2, an image projection system 100 according to this embodiment includes an image projection device (an example of an image display device) 1 having a state determination device (an example of an object detection device) 30, and an image management device 2. And is configured. In the image projection system 100 of the present embodiment, the image projection device 1 projects an image on a predetermined projection plane 3 based on an image signal input from the image management device 2.

The image projection device 1 is arranged so that its front surface (front surface) faces the projection surface 3. In this embodiment, one surface of the dedicated screen is used as the projection surface 3, but the board surface, wall surface, etc. of the whiteboard may be used as the projection surface 3. In the present embodiment, the horizontal direction (horizontal direction) of the projection surface 3 is the X direction, the vertical direction (vertical direction) of the projection surface 3 is the Y direction, and the X direction and the direction orthogonal to the Y direction are the projection surface 3 and the image. The direction in which the projection device 1 faces is the Z direction.

The operator (user) of the image projection apparatus 1 uses the finger as the input operation unit 4 to perform a pointing operation in the vicinity of the projection surface 3 or an operation of touching (contacting) the projection surface 3 to perform projection. An input operation is performed on the projection image projected on the surface 3. The input operation unit 4 is not limited to the operator's finger, and a pen, a pointing stick, or the like may be used as the input operation unit 4. In the image projection apparatus 1, the input operation unit 4 is used as a target object of state detection, state detection (information detection) of its position, shape, movement, and the like is performed, and image operation processing corresponding to the state detection result is performed to perform interactive operation. To realize various operation functions.

The image management device 2 holds a plurality of image information, and transmits the image information of the projection target image and the like to the image projection device 1 based on the instruction of the operator. As the image management device 2, a personal computer (PC), a tablet terminal or the like in which a predetermined program is installed can be used.

The image projection device 1 and the image management device 2 are bidirectionally connected via a general-purpose cable 5 such as a dedicated cable or a USB (Universal Serial Bus) cable. The image management device 2 and the image projection device 1 can be bidirectionally connected by wireless communication according to a known wireless communication protocol. If the image management device 2 has a removable recording medium interface such as a USB memory or an SD card, the image stored in the recording medium can be the projection target image.

The image projection apparatus 1 according to the present embodiment is a so-called interactive projector apparatus, and includes a state determination device 30 having a distance measuring unit 10 and a control unit 20 and a projection unit 40, as shown in FIG. To be done. These are integrally housed in the housing 1a (see FIG. 1), but can be configured separately.

The projection unit 40 has a function of projecting and displaying an optical image of an image on a predetermined projection area 3 a of the projection surface 3. By inputting image information from the image management device 2 to the image projection device 1, the projection unit 40 can project various images.

As shown in FIG. 3, the projection unit 40 includes a light source 41, a wavelength filter 42, a fly-eye lens 43, dichroic mirrors 44R, 44G and 44B, a total reflection mirror 45, collimator lenses 46R, 46G and 46B, and a spatial light modulator 47R. , 47G, 47B, a dichroic prism 48, and a projection lens 49. The projection unit 40 is connected to the projection control unit 50, and its operation is controlled by the projection control unit 50. The projection control unit 50 can be configured by an information processing device including a CPU, a RAM, a ROM and the like provided in the image projection device 1.

Light from a light source 41 such as a lamp light source is incident on a fly-eye lens 43 after unnecessary light components such as infrared light (IR) and ultraviolet light (UV) are removed by a wavelength filter 42. The light that has passed through the fly-eye lens 43 enters the dichroic mirrors 44R, 44G, and 44B, and is separated into red (R light), green (G light), and blue (B light) color components, respectively. The light (R light, G light, B light) separated into each color component is converted into parallel light by the collimator lenses 46R, 46G, 46B, and then enters the spatial light modulation elements 47R, 47G, 47B.

The spatial light modulators 47R, 47G, and 47B are, for example, transmissive liquid crystal display devices that modulate according to image signals, and the transmitted light is spatially modulated. The spatially modulated light of each color component (R light, G light, B light) is combined by the dichroic prism 48 and projected by the projection lens 49. By having such a configuration, the projection unit 40 projects and displays the optical image of the image on the projection surface 3.

The configuration of the projection unit 40 is not limited to the above-described configuration, and a DMD (Digital Micromirror Device) or a reflective liquid crystal display element can be used as the spatial modulation element. Further, the light source 41 is not limited to the lamp light source, and an LED light source or an LD light source can be used.

In the state determination device 30, the control unit 20 detects the state of the target object with respect to the reference object within the predetermined visual field from the distance measuring unit 10 based on the distance information (depth map) acquired by the distance measuring unit 10. In the present embodiment, the input operation state is detected based on the position, the shape, the movement, and the like with respect to the projection surface 3 that is the reference object, with the input operation unit 4 that is the target object as the input state detection target. Therefore, the state determination device 30 of this embodiment is also an input operation detection device.

The distance measuring unit 10 has a field of view toward the object to be measured, and has a function of measuring the three-dimensional coordinates of each unit in the field of view. A depth map in the visual field is acquired based on the measured three-dimensional coordinates. In the present embodiment, the projection surface 3 and the input operation unit 4 are the measurement target objects of the distance measuring unit 10.

As the distance measuring method, for example, a method based on a pattern projection method in which a reference pattern is projected on an object to be measured and three-dimensional coordinates are obtained from the pattern shape in the photographed image, or a parallax between two or more photographed images with different viewpoints is used to measure the object. A method based on a stereo method for obtaining three-dimensional coordinates of an object, a method based on a Time-of-Flight (TOF) method for measuring a distance from a flight time of light to an object to be measured, and the like can be used. Hereinafter, an example of the distance measuring unit 10 (the distance measuring units 10A, 10B, and 10C) using each method will be described with reference to FIGS. 4A to 4C. By using such distance measuring units 10A, 10B, and 10C, it is possible to perform three-dimensional shape measurement at a low cost with a small number of parts.

FIG. 4A is a schematic diagram showing a configuration example of the distance measuring unit 10A (10) using a method based on the pattern projection method. A distance measuring unit 10A shown in FIG. 4A includes a projection unit 11 that projects a predetermined pattern serving as a reference on a measurement target object, an image capturing unit 12 that captures the measurement target object on which the pattern is projected, and the captured measurement target object. And a three-dimensional coordinate calculation unit 13 that calculates three-dimensional coordinates from the captured image, and an exposure condition control unit 14 that sets exposure conditions for the projection unit 11 and the imaging unit 12, and the exposure conditions are stored. An exposure condition storage unit 15 is provided. The projection unit 11 projects the pattern in a predetermined visual field range, and the projection unit 11 and the imaging unit 12 are separated by a predetermined distance. The exposure condition is a condition (parameter) for setting the brightness value of the captured image captured by the image capturing unit 12 to an appropriate value. Specifically, the frame rate, the exposure time, the number of exposures, the sensitivity of the image capturing unit 12, the brightness of the pattern projected by the projecting unit 11, and the like when the image capturing unit 12 performs the image capturing operation.

In the exposure condition storage unit 15, in the brightness image obtained in the process of acquiring the depth map, the first exposure for making the brightness of the background object (an example of the reference object) a brightness value suitable for distance measurement of this background object. The conditions and the second exposure condition for the brightness of the target object (an example of the target object) to have a brightness value suitable for distance measurement of the target object are stored in advance. The exposure condition control unit 14 acquires the first exposure condition and the second exposure condition from the exposure condition storage unit 15.

In addition, a setting value table in which the setting values of the first and second exposure conditions suitable for the measurement target object such as the background object and the target object are set is stored in the exposure condition storage unit 15, Correspondingly, an appropriate setting value may be used. It should be noted that each exposure condition is not limited to being acquired from the exposure condition storage unit 15, and the exposure condition may be set each time via the image management device 2 or another PC.

The projection unit 11 can be configured by, for example, a light source and a projection optical system provided with a mask that forms a pattern. It is desirable to use a light source that emits near infrared light as the light source, but the light source is not limited to this. The imaging unit 12 can be configured by an imaging optical system, a CCD image sensor, an imaging device such as a CMOS, for example. The three-dimensional coordinate calculation unit 13 and the exposure condition control unit 14 can be configured by an information processing device including a CPU, a RAM, a ROM, a memory such as a flash memory provided in the image projection apparatus 1, and the like. The exposure condition storage unit 15 can be provided in the memory of the information processing device.

In the distance measuring unit 10A having the above-described configuration, the exposure condition control unit 14 gives set values for each exposure condition, the projection unit 11 projects a pattern on the measurement target object based on each set value, and the imaging unit 12 Take a captured image. Based on the captured image, the three-dimensional coordinate calculation unit 13 calculates three-dimensional coordinates and measures the distance of the measurement target object.

FIG. 4B is a schematic diagram showing a configuration example of the distance measuring unit 10B (10) using a method based on the stereo method. The distance measuring unit 10B illustrated in FIG. 4B includes two or more image capturing units 12 that capture an image of a measurement target object, and a three-dimensional coordinate calculation unit that calculates three-dimensional coordinates from two or more captured images of the captured measurement target object. 13, and an exposure condition control unit 14 that sets exposure conditions for the imaging unit 12, and an exposure condition storage unit 15 that stores the exposure conditions. Here, the plurality of imaging units 12 have different fields of view, and the captured images that are captured have different viewpoints (have parallax).

The image capturing unit 12, the exposure condition control unit 14, and the exposure condition storage unit 15 in FIG. 4B can have the same configuration as the corresponding configuration in the distance measuring unit 10A in FIG. 4A.

In the distance measuring unit 10B having the above-described configuration, the exposure condition control unit 14 gives the set value of each exposure condition, and the plurality of image capturing units 12 capture the captured image of the measurement target object based on each set value. The three-dimensional coordinate calculation unit 13 calculates three-dimensional coordinates based on the parallax of the plurality of captured images, and measures the distance of the measurement target object.

FIG. 4C is a schematic diagram showing a configuration example of the distance measuring unit 10C (10) using a method based on the TOF method. The distance measuring unit 10C illustrated in FIG. 4C includes an illuminating unit 16 that illuminates the measurement target object with illumination light, a light receiving unit 17 that receives reflected light from the measurement target object irradiated with the illumination light, and illumination light and reflected light. And a three-dimensional coordinate calculation unit 13 for calculating the distance to the object to be measured, and an exposure condition control unit for setting exposure conditions for the illumination unit 16 and the light receiving unit 17. 14 and an exposure condition storage unit 15 in which the exposure conditions are stored.

The illumination unit 16 can be configured by, for example, a light source, an irradiation optical system, or the like. The light source is preferably a light source that emits near infrared light, but is not limited to this. The light receiving unit 17 can be configured by, for example, an image pickup optical system, a CCD image sensor, an image pickup device such as a CMOS. In the distance measuring unit 10 in FIG. 4C, the exposure condition is a condition (parameter) for setting the brightness value in the output signal output from the light receiving unit 17 to be an appropriate value. Specifically, the frame rate, the light receiving time, the number of times the light is received, the sensitivity of the light receiving unit 17, the brightness of the illumination light emitted by the illumination unit 16, and the like when the light receiving unit 17 performs the light receiving operation. Further, the first and second exposure conditions can be set from the image management device 2 each time.

In the distance measuring unit 10C having the above-described configuration, the exposure condition control unit 14 gives a set value for each exposure condition, and the illumination unit 16 irradiates the measurement target object with illumination light based on each set value to measure the measurement target object. The light receiving unit 17 receives the reflected light from. The three-dimensional coordinate calculation unit 13 calculates three-dimensional coordinates based on the phase difference between the illumination light and the reflected light, and measures the distance of the measurement target object.

The control unit 20 detects the state of the target object as the target object based on the distance information (depth map) acquired by the distance measuring unit 10. The control unit 20 can be configured by an information processing device including a CPU provided in the image projection device 1, a memory such as a RAM, a ROM, and a flash memory, and the like.

Next, the function of the control unit 20 will be described with reference to FIG. As shown in FIG. 2, the control unit 20 includes a depth map storage unit 21, a reliability determination unit 22, a depth map generation unit 23, and a state determination unit 24.

The depth map storage unit 21 has a function of storing the depth map acquired by the distance measuring unit 10. In the first embodiment, the depth map acquired by the distance measuring unit 10 under the first exposure condition suitable for the background object is stored in the depth map storage unit 21 as the first depth map (first distance information). .. In addition, the second depth map (second distance information) acquired by the distance measuring unit 10 under the second exposure condition suitable for the target object, various parameters such as a threshold value, and the like are also stored. In the present embodiment, the background object is the projection surface 3 on which an image is projected as shown in FIG. 1, and a dedicated screen is used. The target object is the input operation unit 4 including the finger of the operator who performs an input operation on the projection image.

The reliability determination unit 22 has a function of determining the reliability of the value regarding the background object in the second depth map acquired by the distance measuring unit 10. For example, there is a method of determining the data reliability of the distance measurement value data of each pixel regarding the background object based on the luminance value of the luminance image obtained in the process of the second depth map acquisition by the distance measurement unit 10.

The depth map generation unit 23 has a function of complementing the second depth map acquired by the distance measurement unit 10 under the second exposure condition suitable for the target object based on the determination result of the reliability determination unit 22. This is because the state determination unit 24 performs the state determination with the value of the second depth map for the region determined to have high reliability, and the region determined to have the low reliability corresponds to the region of the first depth map. This is because the value is used to determine the state. That is, the depth map generation unit 23 determines the distance measurement value data of the first depth map stored in the depth map storage unit 21 for the portion determined to have low data reliability in the determination result of the reliability determination unit 22. Complement with. As a result, a new state determination depth map with high reliability of the distance measurement value data is generated.

The state determination unit 24 determines the state of the target object with respect to the background object based on the second depth map generated by the depth map generation unit 23 and the depth map for state determination. Specifically, the state determination unit 24 determines the position information, shape information, movement, etc. of the input operation unit 4 with respect to the projection plane 3.

For example, when the distance between the input operation unit 4 and the projection surface 3 which is the background object is less than a predetermined threshold value, the input operation unit 4 is defined as being in contact with the projection surface 3. Based on this definition, the input information of the operation on the projection area 3a is detected. Furthermore, by reflecting this input information on the projection unit 40 through the image management device 2 that manages the image information, an interactive image projection system that interactively reflects the input information on the projected image can be realized.

5A and 5B specifically show an example of the state determination of the input operation unit 4 by the state determination unit 24. In the state determination device 30, the state detection boundary M is set at a position separated by a predetermined distance with reference to the detection surface N of the projection surface 3 which is the background object, based on the distance measurement result of the distance measurement unit 10. The back surface (front surface) of the finger, which is the input operation unit 4, is used as the detection surface L of the input operation unit 4 by the distance measuring unit 10.

As shown in FIG. 5B, when the detection surface L of the input operation unit 4 exists in the area α between the state detection boundary M and the detection surface N of the projection surface 3, the input operation unit 4 exists near the projection surface 3. Then, it can be determined. On the other hand, as shown in FIG. 5A, the detection surface L of the input operation unit 4 is on the front side of the state detection boundary M (the region β between the state detection boundary M and the state determination device 30) when viewed from the state determination device 30. When it exists, it can be determined that the input operation unit 4 does not exist near the projection surface 3.

Consider such a difference in state determination as a difference in input operation. For example, it is determined that the operator touches the projection surface 3 in a state as shown in FIG. 5B and intends to operate the projection image, and the result is reflected in the image projection by the projection unit 40. Thereby, the input operation information by the input operation unit 4 can be reflected in the projection image.

Here, as described above, when the depth map is acquired under the condition that the distance measurement error due to the difference in reflectance or the like is large in either the projection plane 3 or the input operation unit 4, the state determination device 30 determines the state. It is easy to make a mistake. However, in the state determination device 30 of the first embodiment according to the present application, the first depth map is acquired using the first exposure condition suitable for the projection surface 3 that is the background object, and the input operation unit that is the object of interest is acquired. The second depth map is acquired using the second exposure condition suitable for No. 4. Then, after the second depth map is interpolated by the first depth map, the state of the input operation unit 4 is determined. Therefore, the state determination can be performed with high accuracy.

Hereinafter, an example of the state determination process (state determination method) performed by the state determination device 30 will be described with reference to the flowcharts of FIGS. 6A and 6B. In the state determination process, the input operation unit 4 such as the operator's finger is detected based on the depth map, and the state, that is, the input from the input operation unit 4 is detected from the position, shape, movement, and the like with respect to the projection plane 3. Since the operation information is detected, the state determination process is also an input operation detection process.

In the image projection system 100, the state determination process is performed at any time while the image projection device 1 is projecting an image onto a predetermined projection plane 3 based on an image signal input from the image management device 2. ..

When the start of the state determination process is instructed by turning on the power of the image projection apparatus 1, selecting an operation menu, or the like, first, the control unit 20 first determines whether the first depth map for the projection surface 3 that is the background object has been acquired. It is determined (step S100). If the first depth map has already been stored in the depth map storage unit 21 and has been acquired (Yes), the process proceeds to step S120, and if not acquired (No), the process proceeds to step S110.

Steps S110 to S112 are steps for previously acquiring and storing the depth map in the state where the input operation unit 4 does not exist, that is, the first depth map for the projection surface 3 which is the background object. This is a so-called initialization process, which is a pre-process for detecting the state of the target object (input operation unit 4) in real time thereafter. Therefore, the state determination process is preferably performed immediately after the start is instructed, and is preferably performed at a time timing when no object other than the background object exists in the visual field of the distance measuring unit 10. In addition, since the three-dimensional information is used, the shape of the background object is not limited to the planar shape, and can be applied to a stepped object or a curved object.

First, in step S110, the exposure condition control unit 14 of the distance measuring unit 10 reads the first exposure condition suitable for the background object from the exposure condition storage unit 15 and the like, and sets it in each unit of the distance measuring unit 10. An object used as a background object is generally a dedicated screen, a whiteboard, or the like, and the projection surface 3 of the dedicated screen is also used as a background object in this embodiment. Therefore, it is relatively easy to predict suitable exposure conditions. Therefore, in this step, the suitable first exposure condition stored in advance in the exposure condition storage unit 15 (or set each time) is read out and set, thereby realizing the distance measurement suitable for the background object. be able to.

Next, in step S111, the distance measuring unit 10 acquires the depth map using the first exposure condition set in step S110 in the state where only the background object exists in the visual field. The acquired depth map is stored in the depth map storage unit 21 as the first depth map (step S112). Then, it progresses to step S120.

The above steps S110 to S112 do not need to be repeated if the relative position condition between the state determination device 30 and the projection surface 3 does not change. Therefore, when the state determination process is started again after the first state determination process is completed, the processes of steps S110 to S112 may be skipped and the process may be performed from step S120.

Steps S120 to S121 are steps by which the distance measuring unit 10 acquires the depth map under the second exposure condition suitable for the input operation unit 4 which is the target object. By the way, it is general that the projection surface 3 serving as a background object and the input operation unit 4 serving as a target object have different reflectances with respect to the measurement light. When a dedicated screen or whiteboard is used as the projection surface 3 as in this embodiment, the dedicated screen or whiteboard is made of a material having a higher reflectance than the operator's finger which is the input operation unit 4. Often.

In this way, if the measurement target objects have different reflectances with respect to the measurement light, and the distance measuring unit 10 performs the depth map acquisition under the same exposure conditions, the exposure conditions are not suitable for the respective measurement target objects. That is, when an object with low reflectance is imaged under an exposure condition suitable for an object with high reflectance, the luminance value of the object with low reflectance becomes small in the luminance image obtained in the process of acquiring the depth map. As a result, the distance measurement error (distance measurement error) becomes large. As a result, the accuracy of detecting the state of the input operation unit 4 decreases.

FIG. 7 is a graph showing the relationship between the brightness value and the distance measurement error. As shown in FIG. 7, generally, the smaller the brightness value, the larger the distance measurement error by the distance measuring unit 10. In the distance measuring unit 10 that also has such characteristics, when depth maps are acquired under the same exposure conditions for a plurality of measurement target objects having different reflectances with respect to measurement light, an object with a relatively high reflectance is obtained. Since the luminance value becomes large, it is possible to perform highly accurate distance measurement with a small error. However, distance measurement with a large error is performed for an object having a low reflectance. That is, it is not possible to perform high-precision distance measurement compatible with both the background object and the target object.

Therefore, in the first embodiment according to the present application, in consideration of the difference in reflectance, the distance measuring unit 10 sets the first exposure condition suitable for the background object and the second exposure condition suitable for the target object. Highly accurate distance measurement is possible by using them properly. Therefore, in step S120, the exposure condition control unit 14 reads the second exposure condition different from the first exposure condition from the exposure condition storage unit 15 and the like, and sets it in each unit of the distance measuring unit 10. Thereby, the depth map can be acquired under the exposure condition suitable for the object of interest.

The input operation unit 4 used as the target object in the present embodiment is, for example, the operator's finger or hand. When the human body is the object to be measured by the distance measuring unit 10, it is relatively easy to predict suitable exposure conditions. Therefore, in this step, the suitable second exposure condition stored in advance in the exposure condition storage unit 15 (or set each time) is read and set, thereby realizing the distance measurement suitable for the object of interest. be able to. In the present embodiment, the projection surface 3 of the dedicated screen serving as the background object has higher reflectance than the operator's finger serving as the input operation unit 4. In order to make the exposure conditions suitable for each, the second exposure condition at the time of distance measurement of the input operation unit 4 is The brightness of the pattern to be projected, the brightness of the illumination light emitted by the illumination unit 16, the sensitivity of the imaging unit 12, the sensitivity of the light receiving unit 17, and the like are set.

Next, in step S121, the distance measuring unit 10 acquires the depth map in which the object of interest exists in the visual field using the second exposure condition set in step S120. Since real-time processing is required thereafter, the depth map is acquired at any time, for example, at a predetermined frame rate of 30 fps or higher.

Here, considering the operation speed (detection speed) of the interactive image projection device 1, when the initialization is performed to acquire the distance information within the visual field when only the background object exists, the operation at a relatively low frame rate is performed. Permissible. On the other hand, when the state of the target object is detected, that is, when the distance information in the visual field is acquired in the state where the target object exists, the operation at a relatively high frame rate is required. In consideration of such a constraint on the system operation speed, the second exposure condition suitable for the object of interest is better than the frame rate for acquiring the distance information within the field of view under the first exposure condition suitable for the background object. It is preferable to increase the frame rate when acquiring the distance information within the visual field. However, if the frame rate is increased, the exposure time that can be set is shortened. Therefore, in order to obtain an appropriate value for the brightness value of the brightness image obtained in the process of acquiring the depth map and to obtain high distance measurement accuracy, the projection unit 11 It is necessary to make settings such as increasing the brightness of the pattern projected by, the brightness of the illumination light emitted by the illuminating unit 16, increasing the sensitivity of the imaging unit 12 and the sensitivity of the light receiving unit 17, and the like. Therefore, under the second exposure condition at the time of state detection, the frame rate is set relatively high, and correspondingly, the brightness of the pattern projected by the projection unit 11 and the brightness of the illumination light emitted by the illumination unit 16 are increased. By setting the sensitivity of the imaging unit 12 and the sensitivity of the light receiving unit 17 to be high, the state detection speed is improved while maintaining the exposure condition suitable for the object of interest. Further, under the first exposure condition at the time of initialization, the frame rate is relatively low, and the brightness of the pattern projected by the projection unit 11 and the brightness of the illumination light emitted by the illumination unit 16 are correspondingly low. By setting the sensitivity of the imaging unit 12 and the sensitivity of the light receiving unit 17 to be low, noise can be reduced while maintaining the exposure condition suitable for the background object. Further, since the system operation speed has a margin, it is possible to obtain highly accurate distance measurement value data by averaging a plurality of distance measurement signals.

Next, steps S122 to S123 are steps of determining the reliability of the second depth map and complementing the reliability according to the determination result. In step S122, the reliability determination unit 22 determines whether or not there is distance measurement value data with low reliability in the second depth map acquired under the second exposure condition suitable for the object of interest. It is performed based on the brightness value of each pixel of the data.

As described above with reference to FIG. 7, the so-called S/N ratio decreases in the region where the luminance value is small, and the distance measurement error increases. On the other hand, in a region where the brightness value is larger than necessary, the brightness is likely to be saturated, and if the brightness is saturated, there is a possibility that the distance measurement value data with low accuracy may be output. Areas in which these phenomena occur have low reliability of distance measurement value data, and it is desirable to avoid them in order to realize highly accurate distance measurement.

FIG. 8 is a graph showing the relationship between the brightness value and the data reliability of the measured value data during distance measurement. It is desirable that the luminance value at the time of distance measurement be within a certain predetermined range in which the data reliability of the distance measurement value data is a predetermined value or more. In the example of FIG. 8, it is defined that distance measurement is possible when the luminance value is in the range of Ia to Ib. When the brightness value is in a range other than Ia to Ib, it is determined that the data reliability is low.

In this way, the reliability determination unit 22 determines the data reliability based on the brightness value in the output signal of the image sensor such as the CCD image sensor, so that the data can be easily obtained without the need for additional components. The reliability can be determined. The determination of the data reliability of the distance measurement value data is not limited to the method shown here. For example, the output distance measurement data can be used to determine the degree of coincidence with the distance measurement value data of the surrounding pixels, or the distance measurement value range determined in advance according to the distance measurement range and the object to be measured. Can also be determined by whether or not it is included in.

When it is determined in step S122 that the distance measurement value data with low reliability does not exist in the second depth map (No), the process proceeds to step S130. On the other hand, when it is determined that the distance measurement value data with low reliability exists (Yes), the process proceeds to step S123.

In step S123, the depth map generation unit 23 complements the low-reliability distance measurement value data of the second depth map with the distance measurement value data of the first depth map to newly generate a state determination depth map. .. Specific processing will be described with reference to FIG. Here, as described above, since the reflectance of the projection surface 3 is higher than the reflectance of the input operation unit 4, the second exposure condition when measuring the distance of the input operation unit 4 is that when the distance of the projection surface 3 is measured. For the first exposure condition, the brightness of the pattern projected by the projection unit 11 and the brightness of the illumination light irradiated by the illumination unit 16 are increased, the sensitivity of the imaging unit 12 and the sensitivity of the light receiving unit 17 are increased, and the like. Is set.

(1-1) shown in FIG. 9 shows how the depth map is acquired under the first exposure condition, and (1-2) shows a part of the luminance image 60a in the visual field of the distance measuring unit 10 at that time. And (1-3) shows the first depth map 70a. Only the projection surface 3 that is a background object exists in the field of view, and distance measurement is performed under the first exposure condition suitable for the projection surface 3. Therefore, the luminance value of each pixel in the luminance image 60a is uniformly the luminance value of the measurable range Ia to Ib (measurable area 61a). Further, the distance measurement value of each pixel (region 71a) in the first depth map 70a indicates that uniform distance measurement value data is obtained.

(2-1) shown in FIG. 9 shows a state in which the depth map is acquired under the second exposure condition, (2-2) shows a part of the luminance image 60b in the visual field at that time, and (2) -3) shows the acquired second depth map 70b. In the field of view, the projection surface 3 as a background object and the operator's finger as the input operation unit 4 are present. In this case, since the distance measurement is performed under the second exposure condition suitable for the input operation unit 4, the brightness value of the portion corresponding to the input operation unit 4 in the brightness image 60b is the brightness value (Ia ~ Ib) (measurable region 61b). On the other hand, the brightness value of the portion corresponding to the projection plane 3 may be the brightness value (>Ib) of the saturated region (saturated region 61b'). In the second depth map 70b as described above, the portion corresponding to the input operation unit 4 (area 71b; a meshed portion in a grid pattern) is appropriately distance-measured, while the portion corresponding to the projection surface 3 is formed. (Area 71b'; black-painted portion) is distance measurement value data with low data reliability.

When the state determination is performed using the second depth map 70b, the state determination is likely to be erroneously recognized in the saturated region 71b'. For example, when the projection plane 3 is not output as correct distance measurement data and the distance measurement data before the state detection boundary M is output, that portion is determined to be the touch portion. Therefore, a touch operation at a timing or a touch position not intended by the operator is recognized.

On the other hand, in the first embodiment according to the present application, the measurement value data of the region 71b′ of the second depth map 70b having the low data reliability is measured under the first exposure condition to measure the first depth map 70a. By complementing with the value data to form the complementing area 71c, the state determining depth map 70b' as shown in (3) of FIG. 9 is generated. By performing such processing, the depth maps of both the background object and the target object can be generated with the values measured under suitable exposure conditions, and the state detection with high recognition accuracy can be realized.

In the present embodiment, the case where the projection surface 3 serving as the background object has a higher reflectance than the operator's finger serving as the input operating unit 4 has been described. However, the present application is not limited to this, and even when the reflectance of the projection surface 3 is low in the input operation unit 4, the reflectance is high by generating the depth map based on the same idea. It is possible to obtain the same effect as that of the case. When the state determination depth map generation is completed, the process proceeds to step S130.

In step S130, the state detection of the target object with respect to the background object is performed using the complemented state determination depth map in the procedure described with reference to FIGS. 5A and 5B. More specifically, the position and movement of the finger that performs an input operation on the projected image is acquired, and the state determination unit 24 analyzes information regarding the input operation.

Hereinafter, a specific example of the operation of detecting the state of the target object by the state determination unit 24 will be described with reference to the flowchart of FIG. 6B. First, the state determination unit 24 acquires the first depth map regarding the background object from the depth map storage unit 21 (step S131). Next, the state determination unit 24 acquires the state detection depth map, which is acquired at any time in real time in a state where the input operation unit 4 (operator's finger) is present in the visual field, and is complemented, and the first depth. The difference from the map is calculated (step S132). That is, since it is the difference between the depth maps when there is a finger in the field of view of the distance measuring unit 10 and when there is no finger, the distance information (reference distance) of the projection surface 3 serving as the background object is canceled out, and the background object is used as the reference. Only the distance information (target distance) of the finger portion is calculated. This is called a differential depth map.

The state determination unit 24 extracts the finger shape from the difference depth map by appropriate image processing and estimates the fingertip position at which the input operation is performed (step S133). It is determined whether or not the fingertip position is in contact with the projection surface 3 by obtaining the estimated distance information of the fingertip position (distance information when the projection surface 3 is the reference) from the difference depth map (step S134). ). If the distance information obtained here is zero, it means that the fingertip position is in contact with the projection surface 3. Note that it is desirable to consider the distance measurement error of the distance measurement unit 10, the thickness of the finger, and the like, for example, if the difference distance information is 20 mm or less, the condition determination such as contact is made. Further, as a result, even if the projection surface 3 is not in contact with the projection surface 3, it can be considered that the projection surface 3 is in contact with the projection surface 3, which is practical.

If it is determined that the fingertip is in contact (Yes), information regarding the desired input operation is detected (step S135). The input operation is, for example, an input operation of clicking an icon according to the instruction of the projected image projected on the fingertip position, or an input operation of writing a character on the projected image while the fingertip is moving while touching. It is a movement. If it is determined that the fingertip is not in contact (No), the process of step S135 is skipped. With the above, the state detection process of the object of interest is completed, the process returns to the state determination process of the flowchart of FIG. 6A, and proceeds to step S140.

In step S140, the input operation information is transmitted (instructed) to the image management device 2 that manages the image information in order to execute the input operation information detected by detecting the state of the target object. The image management device 2 transmits the input operation information to the projection unit 40, so that the input operation information is reflected in the projected image. Thereby, an interactive operation function can be realized. Then, it progresses to step S150. When the input operation information is not detected in the detection of the state of the target object in step S130, the process of step S140 is not performed and the process proceeds to step S150.

In step S150, it is determined whether or not the interactive image projection process is finished. If it is finished (Yes), the state determination process is finished, and if the interactive process is still being performed (No), the process returns to step S121. , The state determination of the input operation unit 4 is continued.

As described above, in the first embodiment, by the state determination device 30, the first depth map acquired by the distance measuring unit 10 under the first exposure condition suitable for the projection plane 3 and the second depth map suitable for the input operation unit 4 are used. The state of the input operation unit 4 with respect to the projection surface 3 can be determined with high accuracy based on the second depth map acquired by the distance measuring unit 10 under the exposure condition. Therefore, in the image projection system 100 including the image projection device 1 having such a state determination device 30, the state of the input operation by the input operation unit 4 on the projection image projected on the projection surface 3 can be detected with high accuracy. It is possible to properly reflect the input operation information on the projection image. Thereby, an excellent interactive operation function can be realized.

(Second embodiment)
Next, an electronic blackboard device including the state determination device according to the second embodiment of the present application will be described with reference to FIG. As shown in FIG. 10, an electronic blackboard device 200 according to the second embodiment stores a projection panel 201 on which various menus and command execution results are displayed, a panel unit 202 that houses a coordinate input unit, a controller and a projector unit. And a panel unit 202 and a stand that supports the storage unit at a predetermined height, and a device storage unit 203 that stores a computer, a scanner, a printer, a video player, and the like. 2002-278700 gazette). In addition to such a basic configuration, the electronic blackboard device 200 according to the second embodiment of the present application includes a state determination device 30A for detecting an input operation of an operator (user) on an image displayed on the projection panel 201. I have it. The state determination device 30A is stored in the device storage unit 203 or the like, and the screen of the projection panel 201 (hereinafter referred to as “electronic blackboard screen”) is the projection surface 3. From below the projection plane 3, the operator's finger or pen as the input operation unit 4 is detected. As the state determination device 30A, for example, a device having the same configuration as the state determination device 30 of the above-described first embodiment can be used.

When the electronic whiteboard device 200 is compatible with gesture recognition, the state determination device 30A recognizes the position and movement of the input operation unit 4 to detect operations such as enlargement, reduction, and page feed of various images. Input operation on the electronic blackboard screen. Further, when the electronic blackboard device 200 has a function of touching the electronic blackboard screen, the behavior of the input operation unit 4 is tracked by the state determination device 30A and whether the input operation unit 4 is in contact with the electronic blackboard screen or not. Input to the electronic blackboard screen is detected by detecting whether it is a non-contact state. Further, by outputting the input information to a computer that manages an image, the input operation information is reflected in the output image on the electronic blackboard screen.

Also in the electronic blackboard device 200 of the second embodiment, the state determination device 30 is the first depth map acquired under the first exposure condition suitable for the electronic blackboard screen that is the background object, and the input operation unit 4 that is the target object. The second depth map acquired using the second exposure condition suitable for is complemented. Accordingly, the input operation of the input operation unit 4 can be detected with high accuracy, and the excellent interactive operation function of the electronic blackboard device 200 can be realized.

(Third Embodiment)
Next, a digital signage device including the state determination device according to the third embodiment of the present application will be described with reference to FIG. As shown in FIG. 11, the digital signage device 300 according to the third embodiment of the present application is configured to include a display 301 on which advertisement images and the like are displayed, a projector body 302, and a state determination device 30B.

In this embodiment, the glass surface of the display 301 is the projection surface 3. The image is rear-projected by the projector body 302 from the rear of the projection surface 3. The state determination device 30B is installed in front of and below the projection surface 3, and detects the operator's finger or the like as the input operation unit 4 from below the screen of the display 301. As the state determination device 30B, for example, a device having the same configuration as the state determination device 30 of the above-described first embodiment can be used.

When the digital signage device 300 is compatible with gesture recognition, by recognizing the position and movement of the input operation unit 4 by the state determination device 30B, operations such as enlargement, reduction, and page feed of various images are detected, Input operation on the screen. Further, when the digital signage device 300 has a screen touch function, the behavior of the input operation unit 4 is tracked by the state determination device 30B, and the input operation unit 4 is in a contact state or a non-contact state with the glass surface. By detecting that, the input to the digital signage device 300 is detected. Further, the input operation information is output to the computer that manages the image, so that the input operation information is reflected in the output image on the screen.

Also in the digital signage device 300 of the third embodiment, the state determination device 30B is the first depth map acquired under the first exposure condition suitable for the projection surface 3 that is the background object, and the input operation unit 4 that is the object of interest. The second depth map acquired using the second exposure condition suitable for is complemented. Accordingly, the input operation of the input operation unit 4 can be detected with high accuracy, and the excellent interactive operation function of the digital signage device 300 can be realized.

In each embodiment, as the object detection device, the state determination device that determines the state of the target object, or the example applied to the input operation detection device that detects the input operation by the input operation unit has been described. It is not limited to these. For example, the present invention can be applied to an object extracting device that extracts an object in a predetermined imaging area, an intruding object into a predetermined monitoring area, or the like. Moreover, the state determination device and the input operation detection device are not limited to the above-described embodiments. Further, as an image display device including such an object detection device, an example in which the image display device is applied to a projector device, an electronic blackboard device, and a digital signage device has been described, but the image display device is not limited to these. It can be applied to an appropriate device having an interactive function.

Although the object detection device of the present invention and the image display device including the object detection device have been described based on the respective embodiments, the specific configuration is not limited to the respective embodiments, and the gist of the present invention is not limited. As long as it does not deviate, design changes and additions are allowed. Further, the number, position, shape, etc. of the constituent members are not limited to each embodiment, and the number, position, shape, etc. suitable for carrying out the present invention can be adopted.

1 image projection device (image display device) 3 projection surface (reference object)
4 Input operation unit (target object) 10, 10A, 10B, 10C Distance measuring unit 30, 30A, 30B State determination device (object detection device)
200 Electronic blackboard device (image display device)
300 Digital signage device (image display device)

JP, 2013-61552, A JP, 2007-122508, A

Claims (10)

An object detection device having a distance measuring unit and acquiring distance information from the distance measuring unit within a predetermined visual field,
The distance measuring unit includes a control unit that controls a condition when acquiring distance information about the field of view,
The control unit obtains first distance information about the visual field under the first condition suitable for the reference object existing in the visual field, and under the second condition suitable for the target object existing in the visual field. Controlling the distance measuring unit to obtain second distance information about the field of view,
Determining the reliability of the value for the reference object in the second distance information,
When it is determined that the reliability is high, a reference distance from the reference position in the distance measuring unit to the reference object and a target distance from the reference position in the distance measuring unit to the target object are obtained from the second distance information. ,
When it is determined that the reliability is low, a reference distance from the reference position in the distance measuring unit to the reference object is obtained based on the first distance information, and the reference distance from the reference position in the distance measuring unit to the target object is calculated. Obtain the target distance from the second distance information,
An object detection apparatus, which detects a state of the target object with respect to the reference object based on the reference distance and the target distance. The object detection apparatus according to claim 1 , wherein the reliability is determined based on a light receiving amount when the distance measuring unit acquires the second distance information. The first condition and the second condition are exposure amounts in the distance measuring unit,
Wherein the control unit according to claim 1 or 2, characterized in that said first condition is set based on the reflectance of the reference object is set based on the second condition to the reflectivity of the target object The object detection device according to. The control unit may set the frame rate of the distance measuring unit when acquiring the second distance information to be higher than the frame rate of the distance measuring unit when acquiring the first distance information. object detection apparatus according to any one of claims 1 to 3, characterized. The distance measuring unit projects the predetermined reference pattern onto an object to be measured, and acquires the distance information using a method based on a pattern projection method that obtains three-dimensional coordinates based on a captured image. The object detection device according to any one of claims 1 to 4 . The distance measuring unit acquires the distance information by using a method based on a time-of-flight method for measuring a distance from a flight time of light emitted from a light source to an object to be measured. 4. The object detection device according to claim 4. The distance measuring unit, according to claim 1-4, characterized by obtaining the distance information by using a method based on the stereo method to obtain a three-dimensional coordinates based on the parallax of two or more captured images with different viewpoints The object detection device according to any one of claims. The object detection device is a state determination device that determines a state of the target object with respect to the reference object, an input operation detection device that detects a state of the target object as an input operation unit that performs an input operation with respect to the reference object, and the object detection apparatus according to any one of claims 1 to 7, characterized in that the reference object is either the object extracting apparatus for extracting the target object existing in a predetermined range. The image display apparatus comprising the object detection apparatus according to any one of claims 1-8. The image display device according to claim 9 , wherein the image display device is any one of an image projection device, an electronic blackboard device, and a digital signage device.