JP7540307B2 - Image processing device and image processing program - Google Patents

Description

The present invention relates to an image processing device that processes images captured by an imaging unit, and an image processing program.

As of 2020, COVID-19 (hereafter referred to as the novel coronavirus) infections are spreading worldwide. Measures to prevent infection with the novel coronavirus include wearing masks, washing hands, and maintaining social distance. In addition, to prevent infection through objects, it is recommended that objects that people have touched be frequently disinfected.

Conference rooms and convention halls can attract large numbers of people, posing the risk of cluster infection. After use, conference rooms and convention halls must be thoroughly cleaned and disinfected.

JP 2011-257849 A

However, when cleaning and disinfection are required in a short period of time, it is difficult to thoroughly clean and disinfect the entire room. When cleaning and disinfecting a room, if it is possible to identify areas where the virus may be present, those areas can be disinfected as a priority, making the disinfection process more efficient.

This embodiment was made in consideration of these circumstances, and its purpose is to provide technology that assists in the efficient disinfection of spaces where viruses may be present.

In order to solve the above problem, an image processing device according to one aspect of the present embodiment includes an image acquisition unit that acquires an image of a space in which a person exists, captured by an imaging unit, an image analysis unit that analyzes the acquired image to detect an action area in the space that is a combination of a detection area of the person and an associated area set around the person, and an image generation unit that generates a display image that can recognize all action areas associated with the movement of the person from the start of imaging by the imaging unit. The image generation unit generates a display image that displays the associated area set around a person who has spoken in a more noticeable display manner than the associated area associated with the detection area of a person who has not spoken.

In addition, any combination of the above components and expressions of this embodiment converted between devices, methods, systems, recording media, computer programs, etc. are also valid aspects of this embodiment.

This embodiment can assist in the efficient disinfection of spaces where viruses may be present.

FIG. 1 is a diagram illustrating an example of installation of an imaging device in a vehicle. 1 is a diagram showing a configuration example of a person monitoring system according to a first embodiment; 4 is a diagram showing an example of a frame image captured while a vehicle is traveling in the first embodiment; FIG. FIG. 11 is a diagram showing an example of a contact location image according to the first embodiment; FIG. 11 is a diagram showing another example of a contact place image according to the first embodiment. FIG. 13 is a diagram showing a first example of a stay place image according to the second embodiment. FIG. 13 is a diagram showing a second example of a stay place image according to the second embodiment. 8A to 8C are diagrams showing a third example of a stay place image according to the second embodiment. FIG. 13 is a diagram showing a fourth example of a stay place image according to the second embodiment. FIG. 13 is a diagram showing a fifth example of a stay place image according to the second embodiment. FIG. 13 is a diagram illustrating a configuration example of a person monitoring system according to a third embodiment.

The first embodiment of the present invention relates to a person monitoring system that captures images of occupants using a camera installed inside the vehicle cabin and monitors and records the places that the occupants touch with their hands.

Figure 1 is a diagram showing an example of installation of an imaging device 10 inside a vehicle C1. In Figure 1, the imaging device 10 is attached to the rearview mirror. The imaging device 10 may also be installed on the center visor or center console. The imaging device 10 may also be installed on the ceiling inside the vehicle cabin so as to overlook the interior of the vehicle cabin from above. For example, if a circular fisheye lens is used, the entire hemispherical circumference of the vehicle cabin can be captured in the angle of view.

As the imaging device 10, a dedicated camera may be installed, or a camera from a drive recorder or an in-vehicle monitoring system may be used. The in-vehicle monitoring system is a system that monitors the entire vehicle interior, including the passenger seat and rear seats, and can detect not only whether the driver is dozing or looking away, but also the number of passengers sitting in the passenger seat or rear seats, and whether all passengers, including the driver, are wearing seat belts.

The imaging device 10 includes a lens, a solid-state imaging element, and a signal processing circuit. For example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Devices) image sensor can be used as the solid-state imaging element. The solid-state imaging element converts light incident through the lens into an electrical video signal and outputs it to the signal processing circuit. The signal processing circuit performs signal processing such as A/D conversion and noise removal on the video signal input from the solid-state imaging element, and outputs it to the image processing device 20.

The imaging device 10 may be equipped with a depth sensor for generating a distance image. For example, a 3D LiDAR (Light Detection and Ranging) can be used as the depth sensor. LiDAR emits laser light in the shooting direction, measures the time it takes to receive the reflected light, and detects the distance to an object in the shooting direction. The imaging device 10 may also be configured with a two-lens stereo camera. In this case, a distance image can be generated based on the parallax of the two images, even without providing a depth sensor.

Figure 2 is a diagram showing an example of the configuration of a person monitoring system 1 according to the first embodiment. The person monitoring system 1 includes an image processing device 20, a display device 30, a recording medium 40, and an audio output device 50. The image processing device 20 includes a video acquisition unit 21, a video analysis unit 22, an image storage unit 23, an image generation unit 24, and a warning control unit 25. These components can be realized by a combination of hardware resources and software resources, or by hardware resources alone. As hardware resources, a CPU, ROM, RAM, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ISP (Image Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and other LSIs can be used. As software resources, programs such as firmware can be used.

The image processing device 20 may be implemented on a dedicated IC board, or may be integrated with a car navigation system or a drive recorder system. The image processing device 20 may also be configured as one function of a general-purpose information terminal device such as a smartphone or tablet. The imaging device 10 and the image processing device 20 are connected by wire or wirelessly. When the image processing device 20 is implemented on a dedicated IC board, the image processing device 20 and the imaging device 10 may be installed in the same housing or in separate housings.

The video acquisition unit 21 acquires the video captured by the imaging device 10. The video analysis unit 22 analyzes the video acquired by the video acquisition unit 21. In the first embodiment, the video analysis unit 22 detects the location in the vehicle cabin that the occupant touched with his/her hand.

The video analysis unit 22 includes an object recognition unit 22a, an object tracking unit 22b, and a three-dimensional space recognition unit 22c. The object recognition unit 22a searches for an object within a frame of the video acquired by the video acquisition unit 21. The object recognition unit 22a has a classifier for a specific object generated by learning a large number of images in which the specific object appears as dictionary data. In the first embodiment, a classifier for a person's hand is prepared. The classifier for the person's hand includes at least a classifier for the back of the hand in an open state. Note that it is preferable to prepare a plurality of classifiers for the hand, which view hands in various states from various angles.

In addition to the hand identifier, identifiers for various equipment present in the vehicle cabin may also be prepared. For example, identifiers for the steering wheel, seat belt, shift lever, etc. may be prepared.

The object recognition unit 22a searches within the video frame using a classifier for each object. For example, HOG (Histograms of Oriented Gradients) features can be used to recognize objects. Haar-like features and LBP (Local Binary Patterns) features may also be used. If an object is present within the frame, the object recognition unit 22a captures the object within a rectangular detection frame.

The object tracking unit 22b tracks the object recognized by the object recognition unit 22a in the subsequent frames. For example, a particle filter or a mean shift method can be used to track the object. In the first embodiment, the object to be tracked is the hand of a person recognized by the object recognition unit 22a.

The three-dimensional space recognition unit 22c maps the object detected by the object recognition unit 22a in the three-dimensional space specified by the distance image acquired from the imaging device 10. In the first embodiment, the three-dimensional space recognition unit 22c detects the location in the three-dimensional space touched by the occupant's hand. If equipment present in the vehicle cabin is also recognized as an object, the equipment touched by the occupant's hand can also be identified.

When the three-dimensional space recognition unit 22c detects a frame image in which the occupant's hand is touching an equipment item, it temporarily stores the detected frame image in the image storage unit 23. The image storage unit 23 is a volatile memory for temporarily storing frame images.

Based on the frame images stored in the image storage unit 23, the image generation unit 24 generates an image for displaying contact locations (hereinafter referred to as a contact location image) that allows the user to recognize all contact locations touched by the occupant in the vehicle cabin from the start of imaging by the imaging device 10. The image generation unit 24 may generate a heat map image as the contact location image, in which the more likely the location is to have been touched by the occupant's hand, the more noticeable the color. For example, a heat map image classified into three levels based on the possibility of contact may be generated. For example, a heat map image may be generated in which areas touched by the occupant's hand are displayed in red, areas that may have been touched in yellow, and areas that have not been touched in green. Note that areas that have not been touched may be colorless.

The image generating unit 24 may generate a contact location image in which the area touched by the occupant's hand is displayed the brightest, the area that may have been touched is displayed the next brightest, and the area that has not been touched is displayed the darkest. The image generating unit 24 may also generate a contact location image in which the area touched by the occupant's hand is flashing.

The area in which the display is highlighted may be on an equipment unit basis, or on an area that is actually touched. When highlighting on an equipment unit basis, for example, even if only part of a seat belt is touched, the entire seat belt will be highlighted.

The image generating unit 24 may generate a contact location image in a display manner that is more noticeable for contact locations that have been touched by the occupant's hands more frequently. For example, the saturation of a contact location may be increased for contact locations that have been touched more frequently.

The image generating unit 24 may connect multiple frame images stored in the image holding unit 23, each showing a state in which the occupant's hand is touching something, to generate a digest video for displaying the contact location. The image generating unit 24 may also superimpose multiple frame images, each showing a state in which the occupant's hand is touching something, to generate a single still image.

The image generating unit 24 may generate an image that simply lists, in text information, the equipment that the occupant has touched in the vehicle cabin. In this case, it is not necessary to store in the image storage unit 23 a frame image of the occupant's hand touching the equipment, and it is sufficient to store management information for identifying the equipment that the occupant's hand has touched.

When a user who is about to clean and disinfect the interior of the vehicle issues a command to display an image of a contacted place on an operation unit (not shown), the image generation unit 24 generates an image of a contacted place and displays the generated image of a contacted place on the display device 30.

The display device 30 may be a display (e.g., a liquid crystal display, an organic EL display) of a car navigation system, a display audio system, or a drive recorder system installed in the vehicle C1. The display device 30 may also be a display of a smartphone or a tablet linked to the car navigation system, the display audio system, or the drive recorder system. The display device 30 may also be a display in a meter panel installed in the vehicle. The display device 30 may also be a display dedicated to the person monitoring system 1 according to this embodiment.

The image generating unit 24 may generate a contact location image in real time while the vehicle C1 is traveling and display it on the display device 30. In this case, when an occupant touches something inside the vehicle cabin, the contact location image is updated. The image generating unit 24 may display, on the display device 30, an AR image in which a focus marker indicating the contact location is superimposed on the video captured by the imaging device 10 as the contact location image.

The recording medium 40 is a non-volatile recording medium for recording the video captured by the imaging device 10. For example, a semiconductor memory card (e.g., an SD card) or an optical disk can be used. The contact location image generated by the image generation unit 24 can also be recorded on the recording medium 40.

The above explanation is based on the premise that the video analysis unit 22 can acquire distance images from the imaging device 10. Providing a depth sensor in the imaging device 10 or configuring the imaging device 10 to have a twin-lens configuration increases costs. In the following explanation, a processing example will be described in which the video analysis unit 22 cannot acquire distance images from the imaging device 10. In this processing example, the three-dimensional space recognition unit 22c is not provided, and the location in the vehicle cabin that the occupant has touched with his or her hand is detected from only the two-dimensional image.

In this processing example, for each piece of equipment in the vehicle cabin, a large number of images of occupants touching the equipment in various positions are learned, and multiple classifiers for recognizing when an occupant is touching the equipment are prepared as dictionary data. A set of classifiers for recognizing when an occupant is touching an equipment item may be prepared for each vehicle model. In this case, the position of each piece of equipment in the captured video can be identified with high accuracy, and it can be determined with high accuracy whether or not an occupant is touching the equipment item.

Furthermore, as dictionary data, a classifier for recognizing the faces of people wearing masks may be prepared. Also, a classifier for recognizing when a passenger is coughing may be prepared.

When the object recognition unit 22a detects an occupant not wearing a mask in the frame image, it outputs a detection signal of not wearing a mask to the warning control unit 25. When the object recognition unit 22a detects an occupant coughing in the frame image, it outputs a cough detection signal to the warning control unit 25. When the distance between two occupants sitting in the back seat in the frame image is equal to or less than a set value, the object recognition unit 22a outputs a detection signal of approaching to the warning control unit 25. When the warning control unit 25 receives a detection signal of not wearing a mask, coughing, or approaching from the object recognition unit 22a, it causes the audio output device 50 to output a warning sound or warning message.

The audio output device 50 may be a speaker of a car navigation system or a display audio system installed in the vehicle C1. The audio output device 50 may also be a speaker of a smartphone or a tablet connected to a car navigation system, a display audio system, or a drive recorder system. The audio output device 50 may also be a speaker dedicated to the person monitoring system 1 according to this embodiment.

When a detection signal indicating that a mask is not being worn is input from the video analysis unit 22, the warning control unit 25 outputs a warning message such as "Please wear a mask" to the audio output device 50. When a cough detection signal is input, the warning control unit 25 outputs a warning message such as "Please ventilate the room," "Please open a window," "Please set the air conditioner to outside air introduction mode," or "Please operate the negative ion generator" to the audio output device 50. The person monitoring system 1 according to this embodiment may be linked to an air conditioning system in the vehicle C1. In that case, when a cough detection signal is input, the warning control unit 25 can automatically switch the air conditioning system to outside air introduction mode.

When a detection signal of approaching distance is input from the video analysis unit 22, the warning control unit 25 outputs a warning message such as "Please move a little more space between seats" to the audio output device 50. If the video analysis unit 22 detects three occupants in the back seat but no occupant in the passenger seat, it may output a seat change instruction signal to the warning control unit 25. When a seat change instruction signal is input from the video analysis unit 22, the warning control unit 25 outputs a warning message such as "One person, please move to the passenger seat" to the audio output device 50.

By registering an image of a specific person, the specific person can be excluded from the monitoring process of places touched by the hand. Specifically, the face of the person to be excluded from the monitoring process is captured by the imaging device 10 and registered as dictionary data in the object recognition unit 22a. Note that a still image showing the face of the person may be input to the image processing device 20 from outside.

When the object recognition unit 22a recognizes the face of a person in the frame image, it determines whether or not the face of the person is the face of a person registered as an excluded person. If the face is the face of a person to be excluded, the object recognition unit 22a or the three-dimensional space recognition unit 22c excludes the hand of the person from the monitoring target. If the face of a person recognized in the frame image is not the face of a person to be excluded, the above-mentioned process of detecting and recording the location where the hand touched is applied.

The owner of vehicle C1 and his/her family members may be registered as people to be excluded. In addition, people who are determined to be unlikely to be infected based on the results of a PCR test, antigen test, or antibody test may be registered. For example, by registering the owner of vehicle C1 and his/her family members as people to be excluded, the load on the image processing device 20 can be reduced.

Figure 3 is a diagram showing an example of a frame image F1 captured while a vehicle C1 is traveling according to the first embodiment. In this frame image F1, the following people are detected: a driver P1, a passenger P2 in the front passenger seat, and two passengers P3 and P4 in the rear seats. The steering wheel O1 and a bottle holder O2a on the center console O2 are detected as equipment in the vehicle cabin. In addition, the hand Dh1 of the driver P1 is tracked as a monitoring target. Although not shown, the hands of passengers P2, P3, and P4 are also monitored.

Figure 4 is a diagram showing an example of a contact place image Ft1 according to embodiment 1. When a user who is about to clean and disinfect the vehicle interior issues an instruction to display the contact place image Ft1, the contact place image Ft1 is displayed on the display device 30. Figure 4 shows an example in which the driver P1 touches the steering wheel O1 and the bottle holder O2a while driving. In Figure 4, the steering wheel O1 and the entire area of the center console O2, including the bottle holder O2a, are marked in red.

Figure 5 is a diagram showing another example of a contact location image Ft2 according to embodiment 1. In Figure 5, the steering wheel O1 and the bottle holder O2a are marked in red, and the entire area of the center console O2 where the bottle holder O2a is installed (excluding the area of the bottle holder O2a) is marked in yellow.

The contact location image Ft may be an image of the interior of the vehicle cabin that has a warning marker superimposed on it, indicating areas where the virus may be present, or a simplified schematic diagram of the interior of the vehicle cabin that has a warning marker superimposed on it. For equipment that passengers will always touch with their hands (e.g., steering wheel, shift lever, turn signal lever, door handles, etc.), warning markers may be superimposed by default. The contact location image Ft may be generated using a heat map image.

As described above, according to the first embodiment, a contact location image showing areas that may have been touched by occupants is generated and displayed, allowing a user who is about to clean and disinfect to intuitively understand which areas should be disinfected as a priority. This is particularly useful for people who were not in the vehicle C1, such as cleaning staff at rental car companies or gas station attendants, who need to clean and disinfect the interior of the vehicle efficiently in a short amount of time. Of course, it is also difficult for people who were in the vehicle C1 to understand and remember the areas touched by all occupants, so it is useful for them to clean and disinfect while looking at the contact location image.

Next, a person monitoring system 1 according to embodiment 2 will be described. In embodiment 1, a person monitoring system 1 was described that captures images of occupants with a camera installed in the vehicle cabin, detects the occupants' hands, and monitors and records the places the occupants touch with their hands. In contrast, embodiment 2 relates to a person monitoring system 1 that detects people in a room in a building with a camera installed in the room, and monitors and records the places where the people are staying.

The person monitoring system 1 according to the second embodiment is assumed to be used to monitor a large indoor space such as a conference room or convention hall. Indoor surveillance cameras are generally installed on the ceiling of a room. When monitoring a large room with a single surveillance camera, wide-angle shooting is required, which reduces the size of people in the frame image. Also, there are cases where a large number of people are included in the frame image. In contrast, in the first embodiment, the narrow vehicle interior space is shot, so the size of people in the frame image never falls below a certain size. Also, for a standard vehicle, the maximum number of people included in the frame image is five.

As such, it is difficult to detect the hands of each person by image recognition from frame images captured by the imaging device 10 installed in a conference room, convention hall, etc., and to track the detected hands. Therefore, in the second embodiment, the head or entire body of a person is detected, and the detected head or entire body is tracked.

In the following explanation, we will assume an example in which the imaging device 10 is installed in the center of the ceiling of a room and captures an image from above of the entire room. In this case, the monitoring target is a person's head as viewed from above. Note that the imaging device 10 may also be installed at the corner of the ceiling and a wall and capture the entire room from an angle looking down. In that case, the monitoring target is a person's head or entire body as viewed from an angle.

The image processing device 20 is configured as one function of a general-purpose information terminal device such as a PC, tablet, or smartphone. The image processing device 20 and the imaging device 10 are connected by wire (e.g., a LAN cable or a USB cable) or wirelessly (e.g., wireless LAN), and the image acquisition unit 21 acquires images from the imaging device 10.

In the second embodiment, the video analysis unit 22 detects the location in the room where the person has stayed. The object recognition unit 22a has a classifier for the person's head or whole body as dictionary data. In addition to the classifier for the person's head or whole body, classifiers for various furniture present in the room may also be prepared. For example, classifiers for desks, chairs, etc. may be prepared.

The object recognition unit 22a may set an associated region around the detection region of the person in the frame image. For example, an area in the image equivalent to a circle with a radius of about 1 m to 2 m centered on the person may be set as the associated region. Hereinafter, the area consisting of the detection region of the person and the associated region set around the person will be referred to as the behavior region. Note that the conversion ratio between the distance in real space and the distance in the image, and the setting range of the associated region can be changed by the user.

The object tracking unit 22b tracks the head or whole body of a person recognized by the object recognition unit 22a. If multiple people are recognized in a frame image, each person is tracked separately.

The three-dimensional space recognition unit 22c detects the location where the person stayed within the three-dimensional space identified by the distance image acquired from the imaging device 10. If furniture present in the room is also recognized as an object, it can also identify the furniture that is presumed to have been touched by the person. For example, if there is an overlap between the person's activity area and the detection area of the furniture, it is presumed that the person touched the furniture.

When the three-dimensional space recognition unit 22c detects a frame image in which it is estimated that a person has touched furniture, it temporarily stores the detected frame image in the image storage unit 23. Note that in the second embodiment, all frame images in which a person is detected may be temporarily stored in the image storage unit 23.

The image generating unit 24 generates, based on the frame images stored in the image holding unit 23, images for displaying stay locations (hereinafter referred to as stay location images) that enable the user to recognize all the stay locations where the person has stayed in the room from the start of imaging by the imaging device 10. The image generating unit 24 may generate a stay location image that enables the user to recognize the movement trajectory of the person in the room. The image generating unit 24 may also generate a stay location image that enables the user to recognize the trajectory of the activity area accompanying the movement of the person in the room.

The image generating unit 24 may generate a heat map image as the stay location image in which areas closer to the places where the person stayed are displayed in more prominent colors. For example, a heat map image may be generated in which the areas where the person stayed are displayed in red, associated areas in yellow, and other areas in green (or colorless). Alternatively, a heat map image may be generated in which the areas where the person stayed are displayed in red, and the color gradually changes from red to yellow to green (or colorless) as the area moves away from the area where the person stayed.

The image generating unit 24 may also generate a stay place image in which the brightness of the area where the person stayed is the brightest, and the brightness becomes darker as the person moves away from the area where the person stayed. Also, a stay place image may be generated in which the area where the person stayed blinks.

When an overlap occurs between a person's activity area and a fixture detection area, the image generation unit 24 may display the entire fixture in a prominent manner, or may display only the overlapping area in a prominent manner. When the height of the fixture from the floor can be estimated, the image generation unit 24 may display the fixture in a prominent manner the closer the height of the fixture from the floor is to the height of the person's hands when seated or standing.

The image generating unit 24 may display an area in a more noticeable manner, the greater the overlap between the detection areas of multiple people. For example, the greater the overlap between the detection areas, the higher the saturation may be. The image generating unit 24 may display an area in a more noticeable manner, the greater the overlap between the activity areas of multiple people.

The image generating unit 24 may generate a digest video for displaying the place of stay by stitching together a plurality of frame images stored in the image holding unit 23, in which there are overlapping areas between the person's activity area and the equipment detection area. The image generating unit 24 may also superimpose a plurality of frame images, in which there are overlapping areas between the person's activity area and the equipment detection area, to generate a single still image.

The image generating unit 24 may generate an image that simply lists, in text information, the furniture in the room that is presumed to have been touched by the person. In this case, it is not necessary to store the frame image in the image holding unit 23, and it is sufficient to hold management information for identifying the furniture that is presumed to have been touched by the person.

When a user who is about to clean and disinfect the room issues an instruction to display an image of the place of stay via an operation unit (not shown), the image generation unit 24 generates an image of the place of stay and displays the generated image of the place of stay on the display device 30.

The display device 30 may be a display of a general-purpose information terminal device such as a PC, tablet, or smartphone. The display device 30 may also be a projector connected to the information terminal device. In this case, the projector projects the image of the place of stay onto a screen. The display device 30 may also be a head-mounted display connected to the information terminal device. In this case, the head-mounted display can display the image of the place of stay as a VR image. Note that if the display image for the contact location is generated as a stereoscopic image, the head-mounted display can display the stereoscopic VR image.

The image generating unit 24 may generate a stay location image in real time while the room is in use (e.g., during a meeting) and display it on the display device 30 (e.g., a projector). In this case, the stay location image is updated when a person enters or leaves the room, or when a person moves within the room. The image generating unit 24 may display, on the display device 30, an AR image in which an attention marker indicating the person's stay location is superimposed on the video captured by the imaging device 10 as the stay location image.

Also in the second embodiment, the object recognition unit 22a can output a distance approach detection signal to the warning control unit 25 when the distance between people in the frame image is equal to or less than a set value. When the warning control unit 25 receives a distance approach detection signal from the object recognition unit 22a, it causes the audio output device 50 to output a warning sound or warning message. Note that, even if a person not wearing a mask or a person coughing is detected, the warning control unit 25 can cause the audio output device 50 to output a warning sound or warning message.

Also in the second embodiment, by registering an image of a specific person, the registered person can be excluded from the monitoring process of the place of stay.

Figure 6 is a diagram showing a first example of a stay place image Fs1 according to embodiment 2. When a user who is about to clean and disinfect the room issues an instruction to display the stay place image Fs1, the stay place image Fs1 is displayed on the display device 30. Figure 6 shows a stay place image Fs1 in which three people P1, P2, and P3 are using the first desk D1 on the left side while the room is in use. The second desk D2 on the right side is not in use. In Figure 6, the first desk D1 on the left side is marked in red as an attention area.

The object recognition unit 22a determines whether the detection area of each person P1, P2, P3 overlaps with the detection area of the first desk D1 in each frame image, and whether the detection area of each person P1, P2, P3 overlaps with the detection area of the second desk D2. The object recognition unit 22a determines that the desk with the overlap is the desk that has been used (the first desk D1 in FIG. 6).

Fig. 7 is a diagram showing a second example of a stay location image Fs2 according to the second embodiment. Fig. 7 shows a stay location image Fs2 displaying the movement trajectory of person P1 inside a room. In the image shown in Fig. 7, the entrance to the room is on the right side, and person P1 moves left under the second desk D2 and first desk D1, moves up the left side of the first desk D1, and moves right above the first desk D1 to reach the current location. Note that the movement trajectory of the action area with an associated area added around person P1 may also be displayed.

Figures 8(a)-(c) are diagrams showing a third example of a stay place image Fs3 according to embodiment 2. The third example is an example of a room where the positions of desks and chairs are fixed, such as an exam hall or a large classroom. In the stay place image Fs3a shown in Figure 8(a), a first person P1 is sitting in the first chair S1 on the left. The second chair S2 on the right is vacant. In this case, the first chair S1 on the left and the first desk D1 are marked in red as attention areas.

In the stay location image Fs3b shown in FIG. 8(b), a first person P1 is sitting in a first chair S1 on the left side, and a second person P2 is sitting in a second chair S2 on the right side. In this case, both the first chair S1 and the first desk D1 on the left side, and the second chair S2 and the second desk D2 on the right side are marked in red as attention areas.

The stay location image Fs3c shown in FIG. 8(c) shows the state after the second person P2, who was sitting in the second chair S2 on the right side, has left. Because the second person P2 was staying at the position of the second chair S2 on the right side and the second desk D2, the second chair S2 on the right side and the second desk D2 are marked in red as attention areas even after the second person P2 has left.

Figure 9 is a diagram showing a fourth example of a stay location image Fs4 according to the second embodiment. In the examples shown in Figures 6 to 8(a)-(c), the entire area of the desk that is presumed to have been touched by a person is marked in red. In contrast, in the fourth example shown in Figure 9, only the area of the entire desk that is likely to have been touched by a person is marked in red as an attention area. In Figure 9, a first range R1 at a predetermined distance from the position of a first person P1, and a second range R2 at a predetermined distance from the position of a second person P2 are marked in red as attention areas. This marked area expands as the persons P1 and P2 move.

The specified distance is set to, for example, 2 m, taking social distancing into consideration. This specified distance may be switched depending on whether or not a mask is being worn. In that case, the specified distance for a person not wearing a mask is set longer than the specified distance for a person wearing a mask.

In addition, if the object recognition unit 22a can detect the movement of a person's hand, the area within the entire desk where the hand moved may be marked in red as an attention area. In this case, the area marked within the entire desk may be an area where the actual hand movement was detected plus a predetermined buffer area. In addition, the area marked within the entire desk may be an area that overlaps with the activity areas of persons P1 and P2.

The overlapping range R3 of the first range R1 and the second range R2 may be marked in a relatively conspicuous color as a higher level attention area. For example, the ranges of the first range R1 and the second range R2 where they do not overlap may be marked in green, and the overlapping range R3 between them may be marked in red. Also, the range of a specified distance for one person may be marked in green, the overlapping range of a specified distance for two people may be marked in yellow, and the overlapping range of a specified distance for three or more people may be marked in red.

Figure 10 is a diagram showing a fifth example of a stay place image Fs5 according to the second embodiment. In the stay place image Fs5 shown in Figure 10, an area R5 of a passageway in a room where people move around a lot is marked in red as an attention area. Note that the color may be changed depending on the number of times a person has passed through the passageway. For example, 1 to 5 times may be marked in green, 6 to 10 times in yellow, and 11 or more times in red. Note that although the entire passageway area R5 is marked as an attention area in Figure 10, only the actual movement trajectory of the person, i.e., only the part where the person and the passageway overlap, may be marked.

As described above, according to the second embodiment, by generating and displaying a place-of-stay image showing the area where a person has stayed, a user who is about to clean and disinfect can intuitively understand which areas should be disinfected as a priority. This is particularly useful for people who did not use the room (e.g., did not participate in a meeting), such as external cleaning companies, and who need to clean and disinfect the room efficiently in a short amount of time. Of course, it is also difficult for people who used the room to keep track of and remember the movements of everyone who was in the room, so cleaning and disinfecting while looking at the place-of-stay image is useful.

Next, a person monitoring system 1 according to a third embodiment will be described. In the second embodiment, an example was described in which the image generating unit 24 generates a stay location image that allows the movement trajectory of a person in a room to be recognized, and displays the image on the display device 30. In the third embodiment, further consideration is given to whether or not a person in the room is speaking. When a person without a mask speaks indoors, droplets may be flying, increasing the possibility that viruses are present in the area surrounding the person.

Figure 11 is a diagram showing an example configuration of a person monitoring system 1 according to embodiment 3. The person monitoring system 1 according to embodiment 3 includes an image processing device 20, a display device 30, a recording medium 40, an audio output device 50, and a sound collection device 60. The image processing device 20 includes a video acquisition unit 21, a video analysis unit 22, an image storage unit 23, an image generation unit 24, a warning control unit 25, an audio acquisition unit 26, and an audio analysis unit 27. The differences from embodiment 2 are described below.

The sound collection device 60 includes a microphone for collecting sounds within the space being monitored by the imaging device 10. The sound collection device 60 may be built into the imaging device 10. The sound collection device 60 converts the collected air vibrations into an electrical audio signal and outputs the converted audio signal to the image processing device 20.

The voice acquisition unit 26 acquires a voice signal from the sound collection device 60. The voice analysis unit 27 analyzes the voice signal acquired by the voice acquisition unit 26. The voice analysis unit 27 analyzes the spectrum of the voice signal to detect the presence or absence of human speech sounds. If human speech sounds are detected, the voice analysis unit 27 detects the number of speakers. Furthermore, the voice analysis unit 27 detects and retains at least one of the number of times each speaker has spoken, the cumulative speaking time, and the volume of the voice. Note that in this embodiment, there is no need to recognize the content of the speech, and therefore no analysis using an acoustic model or a language model is required.

As described above, the video analysis unit 22 can recognize whether or not a person is wearing a mask. In the third embodiment, the video analysis unit 22 further has a classifier for various states of a person's mouth. When a person not wearing a mask is detected, the video analysis unit 22 recognizes the shape of the person's mouth for each frame image and determines the movement of the mouth in a series of frame images. The video analysis unit 22 determines whether or not the person is speaking from the movement of the person's mouth. The video analysis unit 22 detects the number of speakers. Furthermore, the video analysis unit 22 detects the degree to which each speaker's mouth is open.

If the number of speakers identified by the video analysis unit 22 is less than the number of speakers detected by the audio analysis unit 27, it is possible that the speaker is not within the field of view of the imaging device 10, that the speaker is not facing the imaging device 10, or that the speaker is wearing a mask. In this case, the video analysis unit 22 may consider all of the speakers who are not facing the imaging device 10 and all of the speakers who are wearing masks.

The image generation unit 24 determines the display mode of the associated area associated with the detection area of each person in the staying place image by referring to at least one of the number of times each person speaks, the duration of the speech, the volume of the voice, and the degree of mouth opening of each person among the people imaged by the imaging device 10.

The image generating unit 24 may set the area of the associated area associated with the detection area of a person who has spoken to be larger than the associated area associated with the detection area of a person who has not spoken. In this case, the image generating unit 24 may set the area of the associated area associated with the detection area of the person to be larger the more times the person speaks, the longer the cumulative speaking time of the person, the louder the person's voice, or the wider the opening of the person's mouth. Note that the associated area to be increased in area may only be the area in the direction in which the person's face is facing (for example, the 180-degree area).

The image generating unit 24 may set the area of the associated region associated with the detection region of a person who is not speaking to be smaller than the area of the associated region associated with the detection region of a person who is speaking.

The image generating unit 24 may mark the associated area associated with the detection area of a person who has spoken in a color (e.g., red or purple) that stands out more than the color of the associated area associated with the detection area of a person who has not spoken. In this case, the image generating unit 24 may darken the color of the associated area associated with the detection area of the person the more times the person speaks, the longer the cumulative speaking time of the person, the louder the person's voice, or the wider the opening of the person's mouth. Note that the associated areas that are darkened may only be areas in the direction in which the person's face is facing.

For example, in FIG. 6 above, if person P2 turns his face toward the second desk D2 and speaks or coughs, the image generation unit 24 marks the second desk D2 in red. Similarly, in FIG. 7 above, if person P1 turns his face toward the second desk D2 and speaks or coughs while moving, the image generation unit 24 marks the second desk D2 in red.

The image generating unit 24 may make the color of an area darker the more overlapping the associated area associated with the detection area of the person who has spoken is. The image generating unit 24 may mark the associated area associated with the detection area of the person who has not spoken in a color (e.g., green or colorless) that is less noticeable than the color of the associated area associated with the detection area of the person who has spoken.

In FIG. 9 above, a first range R1 at a predetermined distance from the position of a first person P1, and a second range R2 at a predetermined distance from the position of a second person P2 are marked in red as attention areas (attached areas). The predetermined distance may be set to 1 m, for example, when a person is not speaking, and set to 2 m when a person is speaking. The attention area of a person who is not speaking is set to 1 m, taking into account the range that the person's hands can reach. The attention area of a person who is speaking is set to 2 m, taking into account the range that the person's droplets can reach.

The image generating unit 24 may mark the overlapping range R3 between the first range R1 and the second range R2 in red when the first person P1 and the second person P2 are not talking, and may mark the overlapping range R3 in purple when the first person P1 and the second person P2 are talking. Here, purple is considered to be a more cautionary color than red.

As described above, according to the third embodiment, when a stay location image showing the area where a person has stayed is generated and displayed, areas where droplets may have spread are prominently displayed, thereby making it possible to appropriately present to the user areas that should be cleaned and disinfected as a priority.

The present invention has been described above based on an embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present invention.

In the first embodiment, for example, while driving, the driver basically holds the steering wheel with his/her hands. On the other hand, when the driver is making other movements (for example, touching a bottle holder to drink water), the image is significantly different from the image of the normal state in which the driver is holding the steering wheel. The object recognition unit 22a may store all frame images in which the driver is not holding the steering wheel with his/her hands in the image storage unit 23. The image generation unit 24 may connect multiple frame images stored in the image storage unit 23 to generate a digest video for displaying the contact location. The image generation unit 24 may also superimpose multiple frame images stored in the image storage unit 23 to generate a single still image. In this case, the accuracy is reduced, but the contact location image can be generated simply.

In the second embodiment, the object recognition unit 22a may store all frame images in which a person is detected in the frame image in the image storage unit 23. The image generation unit 24 may connect multiple frame images stored in the image storage unit 23 to generate a digest video for displaying the stay location. The image generation unit 24 may also superimpose multiple frame images stored in the image storage unit 23 to generate a single still image. In this case, although accuracy decreases, it is possible to simply generate a stay location image.

In the first embodiment, an example of monitoring the interior of a vehicle is described, with a person's hands as the monitoring target. In this regard, the monitoring is not limited to the example of monitoring the interior of a vehicle, as long as it is a small space where a person's hands are easily detected. Even if it is a room in a building, the person monitoring system 1 according to the first embodiment functions effectively if it is a small room that only a few people can enter.

In the second embodiment, an example was described in which the person itself is the monitoring target, and the inside of a room in a building is monitored. In this regard, the person monitoring system 1 according to the second embodiment can also be applied to monitoring indoor spaces inside moving objects such as trains, buses, airplanes, and ships. It can also be applied to monitoring outdoor spaces within a certain range.

The new coronavirus also dies in about seven days at the longest, depending on the material it is attached to and the environmental conditions. Therefore, if more than the set time has passed since the vehicle or room was last used, there is no need to display a warning area in the image.

C1 vehicle, 1 person monitoring system, 10 imaging device, 20 image processing device, 21 video acquisition section, 22 video analysis section, 22a object recognition section, 22b object tracking section, 22c three-dimensional space recognition section, 23 image storage section, 24 image generation section, 25 warning control section, 26 audio acquisition section, 27 audio analysis section, 30 display device, 40 recording medium, 50 audio output device, 60 audio collection device.

Claims (6)

an image acquisition unit that acquires an image of a space in which a person is present, the image being captured by the imaging unit;
A video analysis unit that analyzes the acquired video and detects a behavior area that is a combination of a detection area of the person and an associated area set around the person in the space;
an image generating unit that generates a display image that allows recognition of an entire action area accompanying the movement of the person from the start of imaging by the imaging unit;
Equipped with
The image generating unit generates a display image in which an associated region set around a person who has spoken is displayed in a more prominent display manner than an associated region set around a person who has not spoken.
Image processing device. the image generation unit generates a display image in which a trajectory of the person's activity area can be recognized;
The image processing device according to claim 1 . the image generation unit generates a display image in such a manner that an area having a greater overlap of the person's action areas is displayed in a more conspicuous manner;
3. The image processing device according to claim 1 or 2. the image generation unit determines a display mode of an associated area associated with a detection area of each person in the display image by referring to at least one of the number of times each person speaks and the degree of opening of the mouth of each person among the people imaged by the imaging unit;
The image processing device according to claim 1 . The imaging unit is installed in a building or a moving body and is an imaging unit for imaging a certain space in the building or the moving body.
The image processing device according to claim 1 . A process of acquiring an image of a space in which a person is present, the image being captured by an imaging unit;
A process of analyzing the acquired video to detect a behavior area in the space that is a combination of a detection area of the person and an associated area set around the person ;
A process of generating a display image in which all behavioral areas associated with the movement of the person can be recognized from the start of imaging by the imaging unit;
on the computer ,
The process of generating the display image generates a display image in which an auxiliary region set around a person who has spoken is displayed in a more prominent display manner than an auxiliary region set around a person who has not spoken.
Image processing program.

Images