JP5707561B1 - MONITORING DEVICE, MONITORING SYSTEM, AND MONITORING METHOD - Google Patents

Abstract

An object of the present invention is to enable a user to efficiently perform a task of grasping a behavior tendency of a person by date and time. A monitoring device that generates and outputs an output video in which an image area of a person detected from a captured video in a monitoring area is changed to a mask image is output in accordance with a user input operation for specifying a shooting date and time. A date and time setting unit 71 that sets the shooting date and time, a person image analysis unit 32 that has a person detection unit that detects a person from the shot video and acquires area information for each person, and a mask image that sets a mask image for each shooting date and time A video output control unit 34 that generates and outputs an output video in which the image area of the person is changed to a mask image for each shooting date and time based on the area information for each person output from the human image analysis unit; Shall be provided. [Selection] Figure 7

Description

  The present invention relates to a monitoring apparatus, a monitoring system, and a monitoring method for generating and outputting an output video in which an image area of a person detected from a captured video in a monitoring area is changed to a mask image.

  In stores such as convenience stores, a monitoring system that installs a camera that captures the inside of the store and monitors the person in the store with the image of the camera is widely used. When using captured images for the purpose of improving services, it is necessary to protect the privacy of customers.

  Under such circumstances, the whole person area in each image of the video shot by the camera is displayed on the image converted into an image filled with a predetermined color, and the feature point is displayed to determine how the person moves. A technique for grasping is known (see Patent Document 1). In addition, a background image without a foreground image of a person is generated, and the foreground image is subjected to mask processing or filter processing that is partially or wholly transparent so that the face or figure of the person cannot be specified. There is also known a technique for generating (see Patent Document 2).

JP 2013-186838 A Japanese Patent No. 5159381

  Now, in order to grasp the action of a person from the video, the person is seen while distinguishing the person from the background. However, as in the conventional technique, by replacing the image area of the person with a specific image, And a person can be easily identified, and it becomes easy to grasp the behavior of the person.

  On the other hand, there is a request for grasping the tendency of a person's behavior according to date and time (day of the week, time zone, etc.), but the conventional technology does not give any consideration to such a request. There was a problem that it was not possible to efficiently perform the work of grasping the behavior tendency of another person. In particular, in order to grasp the behavioral behavior of a person by date and time, it is necessary to see the behavior of the person while comparing videos with different dates and times. A technique that can efficiently perform the work of grasping the behavior tendency of a person is desired.

  The present invention has been devised to solve such problems of the prior art, and a monitoring device and a monitoring device that enable a user to efficiently perform a task of grasping a behavior tendency of a person by date and time. The main purpose is to provide a system and monitoring method.

A monitoring device according to the present invention is a monitoring device that generates and outputs an output video in which a person's image area detected from a video shot in a monitoring area is changed to a mask image, and is used for a user's input operation for specifying a shooting date and time. Accordingly, a date and time setting unit for setting a plurality of shooting dates and times, and a person image analysis unit for acquiring region information for each person from the captured images at a plurality of shooting dates and times set by the date and time setting unit for each shooting date and time A mask image setting unit for setting the mask image according to the shooting date and time, and a person in the shot video of each shooting date and time based on the area information for each person acquired by the person image analysis unit for each shooting date and time change the image area on the imaging date and time-of the mask image, out to generate said corresponding output image superimposed on a position on the background image these mask images A video output control unit that is configured to include a.

In addition, the monitoring system of the present invention is a monitoring system that generates and outputs an output video in which a person's image area detected from a video shot in the monitoring area is changed to a mask image, and a camera that shoots the monitoring area; includes a plurality of information processing apparatuses, and any of the plurality of information processing apparatuses, in accordance with the input operation of the user to specify the shooting date and time, the date and time setting unit that sets a plurality of photographing date, the time setting A person image analysis unit that acquires region information for each person from the captured video at a plurality of shooting dates and times set by the shooting unit for each shooting date and time, a mask image setting unit that sets the mask image for each shooting date and time, the person on the basis of the area information of each person obtained for each of the shooting date and time by the image analysis unit, the shooting date and time the image area of the person in the photographed image of the photographing date and time The change in the mask image, and a video output controller for generating and outputting a corresponding said output image superimposed on a position on the background image these mask images, and configurations with the.

The monitoring method of the present invention is a monitoring method for causing an information processing apparatus to perform processing for generating and outputting an output video in which an image area of a person detected from a captured video in a monitoring area is changed to a mask image, in response to an input operation of the user to specify the shooting date and time, a first step of setting a plurality of shooting date and time, the area information of each person from the captured image of the plurality of shooting date and time set by the first step In the second step of acquiring for each shooting date and time, the third step of setting the mask image for each shooting date and time, and the region information for each person acquired for each shooting date and time by the second step. based, change the image area of the person in the photographed image of the shooting date and time on the photographing date and time-of the mask image, the corresponding position on the background image these mask images A fourth step of the output image generated by an output superimposed to be configured to have.

  According to the present invention, since mask images are displayed for a plurality of shooting dates and times, it is possible to easily grasp the shooting date and time of a person appearing in a video, thereby grasping the behavior tendency of a person by date and time. Can be performed efficiently.

Overall configuration diagram of the monitoring system according to the first embodiment Floor plan of the store explaining the layout of the store and the installation status of the camera 1 Functional block diagram showing schematic configuration of PC3 Explanatory drawing which shows the monitoring screen displayed on the monitor 4 Explanatory drawing which shows the mask image setting screen displayed on the monitor 4 Explanatory drawing which shows the principal part of the image | video displayed on the video display part 48 of a monitoring screen. Functional block diagram which shows schematic structure of PC3 in the monitoring system which concerns on 2nd Embodiment. Explanatory drawing which shows the monitoring screen displayed on the monitor 4 Explanatory drawing which shows the mask image setting screen displayed on the monitor 4 Explanatory drawing showing images of two shooting dates and times Explanatory drawing which shows the principal part of the video display part 48 of a monitoring screen. Functional block diagram which shows schematic structure of PC3 in the monitoring system which concerns on 3rd Embodiment. Explanatory drawing which shows the monitoring screen displayed on the monitor 4 Explanatory drawing which shows the mask image setting screen displayed on the monitor 4 Explanatory drawing which shows the principal part of the video display part 48 of a monitoring screen. Explanatory drawing which shows the principal part of the video display part 48 of a monitoring screen. Explanatory drawing which shows another example of the monitoring screen displayed on the monitor 4

A first invention made to solve the above-described problem is a monitoring device that generates and outputs an output video in which an image area of a person detected from a captured video in a monitoring area is changed to a mask image, A date and time setting unit for setting a plurality of shooting dates and times according to a user's input operation, and region information for each person from the shot images of the plurality of shooting dates and times set by the date and time setting unit for each shooting date and time A person image analysis unit to be acquired, a mask image setting unit for setting the mask image for each shooting date and time , and each shooting based on the region information for each person acquired by the person image analysis unit for each shooting date and time. change the image area of the person within the date and time of shooting image on the imaging date and time-of the mask image, the output of these mask image superimposed on the corresponding position on the background image A configuration with a video output controller for generating and outputting an image.

  According to this, since mask images are displayed for multiple shooting dates and times, it is possible to easily grasp the shooting date and time of the person appearing in the video, thereby making it possible to efficiently grasp the behavior tendency of the person by date and time. Can be done automatically.

  In the second invention, the person image analysis unit includes an attribute determination unit that determines an attribute for each person detected from the captured video, and the mask image setting unit includes the shooting date and time and the attribute. In addition, according to the input operation of the user who selects the mask image, the mask image according to the shooting date and the attribute is set.

  According to this, since the user can freely set the mask image, the convenience for the user can be improved. In addition, since mask images are displayed for each attribute, it is possible to easily grasp the attributes of the person appearing in the video, thereby efficiently performing the task of grasping the person's behavior tendency for each attribute of the person. Can do.

  In addition, according to a third aspect of the present invention, when the mask image for each of a plurality of persons overlaps with each other, the video output control unit displays the overlapping portion so as to be distinguishable from other portions in the mask image. .

  According to this, since the overlapping part of the mask image is displayed in an easy-to-understand manner, it is possible to grasp in detail the state of the person in an area where a large number of persons stay at a plurality of shooting dates and times.

  According to a fourth aspect of the present invention, the date and time setting unit assigns priorities to a plurality of shooting dates and times, and the video output control unit determines the overlapping portions when the mask images for a plurality of persons overlap each other. The mask image with the shooting date and time with the higher priority is displayed so as to cover the mask image with the shooting date and time with the lower priority.

  According to this, by setting the priority of the shooting date and time of interest to a high priority, the mask image of the shooting date and time of interest is displayed in an easy-to-understand manner, so it becomes easier to grasp the behavior tendency of the person at the shooting date and time of interest. .

  According to a fifth aspect of the present invention, the mask image can be identified by changing the display element of at least one of color, shading, pattern, and contour line according to the shooting date and time. The configuration.

  According to this, persons with different shooting dates and times can be easily identified due to the difference in mask images.

Further, the sixth invention, the provided from captured image further background image processing unit for generating the background image, the video output control unit, the output image obtained by superimposing the mask image of permeability on the background image Is generated.

  According to this, since the mask image is transmissive, the background image can be seen through, so that the behavior of the person can be observed while observing the situation of the monitoring area that appears in the background image.

The seventh invention is a monitoring system that generates and outputs an output video in which a person's image area detected from a video shot in the monitoring area is changed to a mask image, and a camera that shoots the monitoring area; A date and time setting unit for setting a plurality of shooting dates and times according to a user input operation for specifying the shooting date and time, and the date and time setting unit. and the mask image setting unit that sets a person image analysis unit for acquiring region information for each person for each of the shooting date and time from the captured image of the plurality of shooting date and time is set, the shooting date and time-of the mask image by the based on the area information of each person obtained for each of the shooting date and time by the person image analysis unit, the mask image an image area of a person in the shooting image by the shooting date and time of each shooting date Change to a video output control unit that generates and outputs the output image superimposed on the corresponding position on the background image these mask images, and configurations with.

  According to this, similarly to the first invention, the user can efficiently perform the work of grasping the behavior tendency of the person by date and time.

The eighth invention is a monitoring method for causing an information processing device to perform processing for generating and outputting an output video in which a person's image area detected from a captured video in a monitoring area is changed to a mask image. in response to an input operation of a user to specify a date, the a first step of setting a plurality of shooting date and time, the area information of each person from the captured image of the plurality of shooting date and time set by the first step Based on the second step acquired for each shooting date and time, the third step for setting the mask image for each shooting date and time, and the area information for each person acquired for each shooting date and time by the second step. changes the image area of the person in the photographed image of the shooting date and time on the photographing date and time-of the mask image, by superimposing these mask image at the corresponding position on the background image A structure having a fourth step of generating and outputting the output image.

  According to this, similarly to the first invention, the user can efficiently perform the work of grasping the behavior tendency of the person by date and time.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is an overall configuration diagram of a monitoring system according to the first embodiment. This monitoring system is constructed for a retail chain store such as a convenience store. A camera (imaging device) 1, a recorder (image recording device) 2, a PC (monitoring) provided for each of a plurality of stores. Device) 3 and a monitor (display device) 4, and a PC 11 and a monitor 12 provided in a headquarters for generalizing a plurality of stores.

  The camera 1 is installed at an appropriate place in the store, the inside of the store is imaged by the camera 1, and image information obtained thereby is recorded in the recorder 2. With the PC 3 provided in the store and the PC 11 provided in the headquarters, the store image captured by the camera 1 can be viewed in real time, and the past store image recorded in the recorder 2 can be viewed. It is possible to check the situation in the store at the store or the headquarters.

The PC 3 installed in the store is configured as a monitoring device that monitors the state of persons and products in the store, and the monitoring information generated by the PC 3 can be browsed by the PC 3 itself, and further installed in the headquarters. Sent to the PC 11 and can be browsed by the PC 11 , and the PCs 3 and 11 are configured as browsing devices for browsing the monitoring information.

  Next, the store layout and the installation status of the camera 1 will be described using a convenience store as an example. FIG. 2 is a plan view of the store explaining the store layout and the installation status of the camera 1.

  The store is provided with an entrance, a display shelf (display area), and a cashier counter. Display shelves are divided into different types of products such as bento boxes, plastic bottles, and rice balls. Next to the checkout counter is a fast food display shelf.

  The customer enters the store from the entrance, moves through the passage between the display shelves, finds the desired product, and goes to the checkout counter with the product and pays at the checkout counter (payment). After exiting, exit from the entrance.

  In the store, a plurality of cameras 1 that capture images of the store (monitoring area) are installed. In particular, in the example shown in FIG. 2, an omnidirectional camera having a 360-degree shooting range using a fish-eye lens is adopted as the camera 1, and the camera 1 can image products displayed on the display shelf, It is also possible to take an image of a person staying in front of a display shelf or a cashier counter.

  Next, processing performed by the PC 3 shown in FIG. 1 will be described based on a functional block diagram. FIG. 3 is a functional block diagram showing a schematic configuration of the PC 3.

  The PC 3 includes a monitoring unit 31. The monitoring unit 31 causes the PC 3 to function as a monitoring system for monitoring the inside of the store. The operation of the camera 1 and the recorder 2 is controlled by the monitoring unit 31, and the inside of the store imaged by the camera 1 is controlled. The image can be browsed in real time, and the in-store image recorded in the recorder 2 can be browsed.

  The PC 3 includes a person image analysis unit 32, a background image processing unit 33, a video output control unit 34, and a mask image setting unit 35.

  The person image analysis unit 32 analyzes information captured by the camera 1 and acquires information about a person, and includes a person detection unit 37 and an attribute determination unit 38.

  The person detection unit 37 performs processing for detecting a person from the captured video and acquiring area information for each person (position information of the person's image area). The attribute determination unit 38 performs processing for determining attributes (such as gender and age) for each person detected from the captured video. The processing performed by the person detection unit 37 and the attribute determination unit 38 may use a known image recognition technique. The person image analysis unit 32 desirably performs person tracking processing for associating a person detection result for each frame of the captured video, thereby enabling highly accurate attribute determination.

  The mask image setting unit 35 performs processing for setting a mask image for each attribute. In particular, in the present embodiment, processing for setting a mask image for each attribute is performed in accordance with an input operation of a user who selects a mask image for each attribute of a person. A mask image for each attribute may be set in advance.

  In the background image processing unit 33, processing for generating a background image from the captured video is performed. This background image is obtained by removing a person image (foreground image) from a photographed video, removes a person image from each of the most recently photographed videos, and synthesizes the background image from the plurality of images. You can do that. Moreover, you may make it prepare previously as a background image the picked-up image when a person does not exist, for example, the picked-up image before a store opening.

  The video output control unit 34 generates and outputs an output video in which the person's image area is changed to a mask image based on the area information and attributes for each person output from the person image analysis unit 32. In the present embodiment, a mask image having an outline corresponding to an image area of a person in a captured video is generated, and a process of superimposing the mask image on the background image generated by the background image processing unit 33 is performed.

  The mask image is one in which attributes can be identified by changing display elements of at least one of color, shading, pattern (pattern), and outline according to the attributes. It is assumed that an outline of a person image area is generated from area information for each person (position information of the person image area), and the inside of the outline is filled with a transparent image. Note that the contour line may not be displayed, and the contour line may be displayed so as to be distinguishable from the filled region by changing the color, shading, or the like.

  Next, the monitoring screen displayed on the monitor 4 shown in FIG. 1 will be described. FIG. 4 is an explanatory diagram showing a monitoring screen displayed on the monitor 4.

  This monitoring screen is used by the user to check the situation in the store for the purpose of marketing analysis and service improvement. The monitoring screen includes a store selection unit 41, a start button 42, a setting button 43, and an operation. A mode selection unit 44, a date display unit 45, a date setting button 46, a legend display unit 47, a video display unit 48, and a display time operation unit 49 are provided.

  In the store selection unit 41, the user selects a store from a pull-down menu. The start button 42 is used to start the monitoring process on the PC. The setting button 43 is used to set various conditions in the monitoring process. In this embodiment, when the setting button 43 is operated, a mask image setting screen (see FIG. 5) is displayed in a pop-up. The operation mode selection unit 44 selects an operation mode, and operation modes such as sales floor monitoring, product monitoring, display alert, and shortage alert are prepared. Here, sales floor monitoring is selected.

  The date and time display unit 45 displays the date and time. The date and time setting button 46 is for setting the date and time. When the date / time setting button 46 is operated, a date / time setting screen (not shown) is displayed. By selecting a date / time here, the selected date / time is displayed on the date / time display unit 45, and a video of the selected date / time is displayed on the video display unit 48. Is displayed.

  The legend display unit 47 is for explaining a mask image displayed on the video display unit 48. In the present embodiment, a mask image set for each person attribute (such as gender and age) is displayed on the legend display unit 47. In the example shown in FIG. 4, the ages are all generations. However, when a mask image is set for each generation, for example, teens to 60s, the legend display unit 47 displays a mask image of each generation.

  The video display unit 48 displays store videos (moving images). In particular, in the example shown in FIG. 4, two images obtained by panoramic development of an image captured by the camera 1 that is an omnidirectional camera are displayed. The entire store is reflected in these two images, and the status of the entire store can be confirmed. On the video display unit 48, the video captured by the camera 1 is displayed as it is in the normal playback mode, and the video in which the image area of the person is changed to a mask image is displayed in the mask playback mode. The video displayed on the video display unit 48 will be described in detail later.

  The display time operation unit 49 adjusts the time of the video displayed on the video display unit 48. By operating the display time operation unit 49, the video can be switched to a video at a desired time. Specifically, when the slider 50 is shifted using the input device 6 such as a mouse, an image at the time indicated by the slider 50 is displayed on the video display unit 48. Thereby, the situation of the store at the required time can be confirmed on the video of the video display unit 48. In addition, when the video on the video display unit 48 is played, the operation button 51 is used to enable fast-forward playback and rewind playback in addition to normal playback. In this case, even when a video recorded for a long time is played back in the mask playback mode, the video can be confirmed efficiently.

  Next, an operation for setting a mask image for each attribute will be described. FIG. 5 is an explanatory diagram showing a mask image setting screen displayed on the monitor 4.

  In this mask image setting screen, a user specifies a mask image for each attribute of a person. In this mask image setting screen, a person attribute selecting unit 61, a mask image selecting unit 62, a setting button 63, A setting content display section 64 is provided.

  In the person attribute selection unit 61, the user selects a person attribute (gender and age) from a pull-down menu. The person attribute selection unit 61 can select, for example, each generation of teens to 60s and all generations. It should be noted that the attributes of the person can be adults and children. In the mask image selection unit 72, the user selects a mask image corresponding to the attribute selected by the person attribute selection unit 61 from the tile menu.

  When the person attribute selection unit 61 and the mask image selection unit 62 select a person attribute and the corresponding mask image and operate the setting button 63, the setting content display unit 64 displays the setting content, that is, the mask for each attribute. An image is displayed. In the example shown in FIG. 5, mask images of men and women are set for all generations.

  Next, the video displayed on the video display unit 48 of the monitoring screen shown in FIG. 4 will be described. FIG. 6 is an explanatory diagram showing the main part of the video displayed on the video display unit 48 of the monitoring screen shown in FIG.

  As shown in FIG. 6A, in the normal playback mode, a video image taken by the camera 1 is displayed as it is. On the other hand, in the mask reproduction mode after setting the mask image, as shown in FIG. 6B, an image in which the person's image area is changed to a mask image is displayed. In this video, the image area of the person in the captured video of FIG. 6A is replaced with a mask image for each attribute (here, male and female). Further, since the mask image is transmissive, the background image can be seen through.

  When playing back a captured video, a playback mode identification icon may be superimposed on the video so that the user can identify which playback mode is being played back. Further, the playback mode may be identified by displaying the video display frame in the mask playback mode in a different color from the color of the video display frame in the normal playback mode.

  As described above, in the first embodiment, since different mask images are displayed for each attribute, for example, men and women, it is possible to easily grasp the attributes of the person appearing in the video. It is possible to efficiently perform a task of grasping a person's behavior tendency for each attribute.

  In addition, although sex and age were demonstrated as an example of a person's attribute, various directions, such as a person's direction, a race, an adult / child, a type (OL and office worker), etc. can be acquired as an attribute obtained using a well-known person recognition technique. It is possible to set the attribute. In particular, if the attribute of the direction of the person is used, it is possible to grasp the preference for the product for each person even in the mask reproduction mode.

(Second Embodiment)
Next, a second embodiment will be described. The points not particularly mentioned here are the same as in the first embodiment.

  FIG. 7 is a functional block diagram showing a schematic configuration of the PC 3 in the monitoring system according to the second embodiment. In the second embodiment, similarly to the first embodiment, the PC 3 includes a monitoring unit 31, a person image analysis unit 32, a background image processing unit 33, a video output control unit 34, a mask image setting unit 35, and the like. Further, in the second embodiment, the PC 3 includes a date / time setting unit 71.

  In the second embodiment, similarly to the first embodiment, it is possible to display mask images by person attributes (such as sex and age), but it is also possible to display mask images by date and time. Note that it is possible to display only mask images by date and time without displaying mask images by person attributes.

  The date and time setting unit 71 performs a process of setting a plurality of dates and times in accordance with a user input operation that specifies the shooting date and time. The mask image setting unit 35 performs a process of setting a mask image for each shooting date and time and for each person attribute according to a user input operation for selecting a mask image for each shooting date and time and a person attribute. The person image analysis unit 32 acquires captured images of a plurality of shooting dates and times set by the date and time setting unit 71 and acquires person area information (position information of person image areas) for each person for each shooting date and time. Is done.

  In the video output control unit 34, based on the region information for each person output from the person image analysis unit 32, the person image region for each of a plurality of shooting dates and times is changed to a mask image for each shooting date and time and for each person attribute. Processing to generate and output an output video is performed, and the output video is a mask image for each person at a plurality of dates and times displayed on one background image so that the shooting date and time and the attributes of the people can be identified. .

  In the second embodiment, the mask image can identify the attribute and the shooting date / time by changing the display element of at least one of color, shading, pattern, and contour line according to the attribute and the shooting date / time. In particular, in this embodiment, it is assumed that the inside of the outline of the person's image area is filled with a transparent image.

  Next, the monitoring screen displayed on the monitor 4 will be described. FIG. 8 is an explanatory diagram showing a monitoring screen displayed on the monitor 4.

  Also in the second embodiment, a monitoring screen that is substantially the same as that of the first embodiment (see FIG. 4) is displayed on the monitor 4. In this monitoring screen, the legend display unit 47 displays a plurality of shooting dates and times and individual attributes. The set mask image is displayed.

  Next, an operation for setting a mask image for each attribute and shooting date / time will be described. FIG. 9 is an explanatory diagram showing a mask image setting screen displayed on the monitor 4.

  In this mask image setting screen, the user designates a mask image for each person attribute and shooting date / time. This mask image setting screen includes a person attribute selection unit 81, a date / time selection unit 82, and a mask image selection. A section 83, a setting button 84, and a setting content display section 85 are provided.

  In the person attribute selection unit 81, the user selects a person attribute (gender and age) from a pull-down menu. In the date and time selection unit 82, the user selects the date and time (year, month, day and time zone) from the pull-down menu. In the date and time selection unit 82, the user selects a priority order from a pull-down menu. In the mask image selection unit 83, the user selects a mask image corresponding to the attribute selected by the person attribute selection unit 81 and the shooting date / time selected by the date / time selection unit 82 from the tile menu.

  When the person attribute selection unit 81, the date and time selection unit 82, and the mask image selection unit 83 select a person attribute and shooting date and time and a corresponding mask image and operate the setting button 84, the setting content display unit 85 is displayed. The setting contents, that is, mask images for each attribute and shooting date / time are displayed. This setting operation is performed at least twice, and a plurality of mask images with shooting dates and times are set. In the example shown in FIG. 9, the same time zone of different dates is set as the two shooting dates and times.

  Next, an outline of processing performed in the video output control unit 34 and video displayed on the video display unit 48 of the monitoring screen will be described. FIG. 10 is an explanatory diagram showing images of two shooting dates and times. FIG. 11 is an explanatory diagram showing a main part of the video display unit 48 of the monitoring screen shown in FIG.

  FIG. 10A shows a photographed image at the first photographing date and time. FIG. 10B shows a photographed image at the second photographing date and time. The video output control unit 34 generates a mask image having a contour corresponding to the image area of the person in the captured image at the first shooting date and time, and the image area of the person in the captured image at the second shooting date and time. A mask image having an outline corresponding to is generated, and a process of superimposing the mask image on a corresponding position on the background image generated by the background image processing unit 33 is performed. As a result, as shown in FIG. 11, the mask image is displayed on the video display section 48 of the monitoring screen.

  Here, in this second embodiment, mask images with different shooting dates and times are displayed at the same time. Mask images with different shooting dates and times may overlap each other. As a method for displaying mask images with different shooting dates and times, In the second embodiment, as shown in FIG. 11A, a normal display mode in which display based on priority is not performed, and a priority display mode in which display based on priority is performed as shown in FIG. 11B. And can be selected.

  As shown in FIG. 11A, in the normal display mode in which display based on the priority order is not performed, an overlapping portion of mask images having different dates and times is a portion that does not overlap another portion of the mask image, that is, another person's mask image. And identifiable. In the example shown in FIG. 11A, the overlapping portion of the mask image is displayed as an image obtained by combining the mask images. For example, the overlapping portion of the red mask image and the blue mask image is displayed in purple, and the overlapping portion of the mask image is displayed with a transmittance obtained by adding the transmittance of each mask image.

  As shown in FIG. 11B, in the priority display mode in which the display based on the priority order is performed, the mask image with the higher priority date and the mask image with the lower priority date and time are the parts where the mask images with different date and time overlap. Is displayed so as to cover up.

  As described above, in the second embodiment, since different mask images are displayed for a plurality of shooting dates and times, for example, today and yesterday, it is possible to easily grasp the shooting date and time of the person appearing in the video. Thus, it is possible to efficiently perform the work of grasping the behavior tendency of the person by date and time. Here, if a plurality of shooting dates and times are set to the same time on different days, it is possible to compare a person's behavior tendency for each day, and if a plurality of shooting dates and times are set to morning and evening time zones on the same day, It is possible to compare a person's behavior tendency in the morning and evening time zones.

(Third embodiment)
Next, a third embodiment will be described. The points not particularly mentioned here are the same as in the first embodiment.

  FIG. 12 is a functional block diagram showing a schematic configuration of the PC 3 in the monitoring system according to the third embodiment. In the third embodiment, the PC 3 is similar to the first embodiment in that the monitoring unit 31, the person image analysis unit 32, the background image processing unit 33, the video output control unit 34, the mask image setting unit 35, and the like. Further, in the third embodiment, the PC 3 includes a background image analysis unit 91.

  The background image analysis unit 91 performs a process of acquiring article change information related to the disappearance and appearance of the product in the background image area in the captured video. In this background image analysis processing, a change state (disappearance or appearance) of a product is detected by comparing a plurality of background images obtained by removing a person image (foreground image) from a captured video, and region information (product Position information of the image area).

  The video output control unit 34 generates and outputs an output video in which the change (disappearance or appearance) of the product in the background image area is identifiable based on the object change information output from the background image analysis unit 91. Is done. Here, when the product disappears, the image area where the product existed in the video before disappearing may be changed to a transparent mask image. When the product appears, the image area of the product Should be highlighted (highlighted).

  For highlighting when a product appears, various appropriate methods may be used. For example, the outline of the image area of the product may be displayed, and the product is small and difficult to understand. Alternatively, an animation such as blinking may be displayed. Further, when a product appears, the image area of the product may be changed to a transmissive mask image in the same manner as when the product disappears. In addition, an appropriate mark image, for example, an image of a symbol such as ○ or × may be displayed at the position of the product that has changed (disappeared or appeared).

  In the third embodiment, the change state (disappearance or appearance) of the product can be displayed in an identifiable manner, and a mask image of a person can be displayed for each attribute, as in the first embodiment. In the unit 35, a process of setting a mask image is performed in response to a user's input operation for selecting a mask image for each attribute of the person. In the attribute determination unit 38, the attribute (gender) of each person detected from the captured video is processed. In addition, the video output control unit 34 generates and outputs an output video in which the person's image area is changed to a mask image for each attribute.

  Next, the monitoring screen displayed on the monitor 4 will be described. FIG. 13 is an explanatory diagram showing a monitoring screen displayed on the monitor 4.

  Also in the third embodiment, a monitoring screen that is substantially the same as that of the first embodiment (see FIG. 4) is displayed on the monitor 4. In this monitoring screen, the mask image set for each attribute of the person on the legend display unit 47. Then, a mask image representing the lost product is displayed.

  Next, an operation for setting a mask image for each person attribute and a mask image for a change state (disappearance and appearance) of a product will be described. FIG. 14 is an explanatory diagram showing a mask image setting screen displayed on the monitor 4.

  This mask image setting screen sets a mask image for each attribute of a person and sets a mask image of a change state (disappearance and appearance) of a product. The mask image setting screen includes a person setting unit 101 and a mask image setting screen. A product setting unit 102 and a setting content display unit 103 are provided.

  As in the first embodiment (see FIG. 5), the person setting unit 101 is provided with a person attribute selection unit 104, a mask image selection unit 105, and a setting button 106, and the person attribute selection unit 104 is provided. When the person's attributes (gender and age) are selected with the button, the mask image corresponding to the person's attribute is selected by the mask image selection unit 105 and the setting button 106 is operated, the setting content display unit 103 displays the setting contents, A mask image for each attribute is displayed.

  The product setting unit 102 includes a change state selection unit 107, a mask image selection unit 108, and a setting button 109. In the change state selection unit 107, the user selects a change state (disappearance and appearance) from a pull-down menu. In the mask image selection unit 108, the user selects a mask image corresponding to the change state selected by the change state selection unit 107 from the tile menu. When the change state and the mask image selection unit 108 select the change state and the corresponding mask image and operate the setting button 109, the setting content display unit 103 displays the setting content, that is, the change state of the product. A mask image is displayed.

  Next, an outline of processing performed in the video output control unit 34 and video displayed on the video display unit 48 of the monitoring screen illustrated in FIG. 13 will be described. FIGS. 15 and 16 are explanatory diagrams showing the main part of the video display unit 48 of the monitoring screen shown in FIG.

  In the third embodiment, similarly to the first embodiment (see FIG. 6A), in the normal playback mode, the video image taken by the camera 1 is displayed as it is on the video display unit 48, and the mask image is set. In the mask reproduction mode, as shown in FIGS. 15A and 15B, an output video in which a person's image area is changed to a mask image is displayed.

  Here, when the customer in front of the display shelf takes away the product as shown in FIG. 16A-1 and the product disappears from the display shelf as shown in FIG. 16A-2, background image analysis is performed. The unit 91 detects the disappearance of the product, and the video output control unit 34 performs a process of changing the image area of the disappeared product on the background image to a mask image. Thereby, in the mask reproduction mode, as shown in FIG. 16B, the mask image is displayed at the position where the product is present.

  It should be noted that the display of the mask image and the highlight display indicating the change state (disappearance and appearance) of the product may be terminated when a predetermined time has elapsed from the time of the change (disappearance and appearance) of the product. Similarly to the mask image of a person, the product mask image generates an outline of the product image area from the product area information (position information of the product image area), and the inside of the outline is transparent. It is assumed that the image is filled with

  As described above, in the third embodiment, since the mask image is displayed for each attribute, it is possible to easily grasp the attribute of the person appearing in the video, thereby grasping the behavior tendency of the person for each attribute of the person. Work can be performed efficiently. And since the change (at least one of disappearance and appearance) of the article is displayed in an identifiable manner, the user can identify the changing article, thereby grasping the change status of the article together with the behavior of the person. Can be performed efficiently by the user.

  In the third embodiment, both the disappearance and appearance of the product are detected, and the output video is displayed so that both the disappearance and appearance can be identified. It is good also considering only one side.

  FIG. 17 is an explanatory diagram illustrating another example of the monitoring screen displayed on the monitor 4. In the above example, the panoramic image of the image captured by the camera 1 that is an omnidirectional camera is displayed on the image display unit 48. In the example shown in FIG. 17, the image is captured by the camera 1 that is an omnidirectional camera. The video is displayed as it is.

  As mentioned above, although this invention was demonstrated based on specific embodiment, these embodiment is an illustration to the last, Comprising: This invention is not limited by these embodiment. In addition, each component of the monitoring device, the monitoring system, and the monitoring method according to the present invention described in the above embodiment is not necessarily essential, and can be appropriately selected as long as it does not depart from the scope of the present invention. It is.

  For example, in each of the first to third embodiments, an example of a retail store such as a convenience store has been described. However, the present invention is not limited to such a retail store, and is applied to a store having a business form other than a retail store. It is also possible. Furthermore, it is possible to use for monitoring areas other than stores.

  In each of the first to third embodiments, as shown in FIG. 2, the camera 1 is an omnidirectional camera having a 360-degree shooting range using a fisheye lens, but a camera having a predetermined angle of view. A so-called box camera is also possible.

  In each of the first to third embodiments, the output video is generated by superimposing the mask image for each person on the background image. However, only the mask image is displayed and the background image is not displayed. It is.

  In the first to third embodiments, necessary setting items are input by selecting a pull-down menu or a tile menu. However, the present invention is not limited to this, and an appropriate input unit is provided on the screen. What is necessary is just to be able to select with a radio button etc., or to be able to input a numerical value directly.

  Moreover, in each of the first to third embodiments, various configurations having characteristics are shown, but these configurations are not limited to the combinations shown in the first to third embodiments. It is also possible to appropriately combine the configurations shown separately in the first to third embodiments.

  Further, in each of the first to third embodiments, the processing necessary for monitoring is performed by the PC 3 provided in the store, but this necessary processing is performed in the headquarters as shown in FIG. You may make it carry out to the cloud computer 21 which comprises provided PC11 and a cloud computing system. In addition, necessary processing may be shared by a plurality of information processing apparatuses, and information may be transferred between the plurality of information processing apparatuses via a communication medium such as an IP network or a LAN. In this case, the monitoring system is composed of a plurality of information processing apparatuses that share necessary processing.

  In such a configuration, it is preferable to cause the PC 3 provided in the store to perform at least a process with a large calculation amount among processes necessary for monitoring, for example, a person detection process. With this configuration, the amount of information required for the remaining processing can be reduced, so that the remaining processing is performed by an information processing apparatus installed at a location different from the store, for example, the PC 11 installed at the headquarters. Even if it does in this way, since communication load can be reduced, operation of the system by a wide area network connection form becomes easy.

  In addition, among the processes necessary for monitoring, at least a process with a large calculation amount, for example, a person detection process may be performed by the cloud computer 21. If comprised in this way, since the amount of calculations of the remaining processes may be small, a high-speed information processing apparatus will become unnecessary for users, such as a shop, and the cost which a user will bear can be reduced.

  In addition, the cloud computer 21 may perform all the necessary processes, or at least the video output control process among the necessary processes may be shared by the cloud computer 21. With this configuration, the store or the head office In addition to the PCs 3 and 11 provided in the mobile phone, the mobile terminal such as the smartphone 22 or the tablet terminal 23 can display the video. You can check the situation in the store.

  In the present embodiment, the PC 3 installed in the store performs processing necessary for monitoring, and the monitoring screen and mask image setting screen are displayed on the monitor 4 of the PC 3 to perform necessary input and output on the PC 3. However, the necessary input and output are performed on an information processing apparatus different from the information processing apparatus that performs processing necessary for monitoring, for example, a portable terminal such as the PC 11 or the tablet terminal 23 installed in the headquarters. Also good.

  In recent years, 4K televisions and the like that have pursued high image quality have been developed as monitors, and by employing a camera that supports such 4K televisions, the ability to identify people can be improved, and there are many people. Even in such an environment, the user can easily grasp the person's behavior tendency from the mask reproduction video.

  The monitoring device, the monitoring system, and the monitoring method according to the present invention have the effect that the user can efficiently perform the task of grasping the behavior tendency of the person according to the date and time, and the person detected from the captured video in the monitoring area It is useful as a monitoring device, a monitoring system, a monitoring method, and the like that generate and output an output video in which the image area is changed to a mask image.

1 Camera 2 Recorder 3 PC
4 monitor 6 input device 12 monitor 21 cloud computer 22 smartphone 23 tablet terminal 32 human image analysis unit 33 background image processing unit 34 video output control unit 35 mask image setting unit 37 human detection unit 38 attribute determination unit 71 date and time setting unit 91 background image Analysis part

Claims (8)

A monitoring device that generates and outputs an output video in which an image area of a person detected from a captured video in a monitoring area is changed to a mask image,
A date and time setting unit for setting a plurality of shooting dates and times according to a user input operation for specifying shooting dates and times,
A person image analysis unit that acquires area information for each person from the captured video at a plurality of shooting dates and times set by the date and time setting unit for each shooting date and time ;
A mask image setting unit for setting the mask image for each shooting date and time;
Based on the area information for each person acquired by the person image analysis unit for each shooting date and time, the image area of the person in the shooting video for each shooting date and time is changed to the mask image for each shooting date and time , A video output control unit that generates and outputs the output video in which the mask image is superimposed on a corresponding position on the background image; and
A monitoring device comprising: The person image analysis unit includes an attribute determination unit that determines an attribute for each person detected from the captured video,
The mask image setting unit sets the mask image according to the shooting date and time and the attribute according to an input operation of a user who selects the mask image for each of the shooting date and time and the attribute. The monitoring apparatus according to 1.   2. The video output control unit according to claim 1, wherein when the mask images for a plurality of persons overlap each other, the overlapping portion is displayed so as to be distinguishable from other portions in the mask image. 2. The monitoring device according to 2. The date and time setting unit gives priority to a plurality of shooting dates and times,
When the mask images for a plurality of persons overlap each other, the video output control unit is configured such that the mask image at the shooting date with the higher priority is the mask image at the shooting date with the lower priority at the overlapping portion. The monitoring device according to claim 1, wherein the monitoring device is displayed so as to cover the screen.   2. The mask image can be identified by changing display elements of at least one of color, shading, pattern, and contour line according to the shooting date and time. The monitoring device according to claim 4. Further comprising a background image processing unit configured to generate the background image from the captured image,
6. The monitoring apparatus according to claim 1, wherein the video output control unit generates the output video in which the transparent mask image is superimposed on the background image. A monitoring system that generates and outputs an output video in which an image area of a person detected from a captured video in a monitoring area is changed to a mask image,
A camera that captures the surveillance area;
A plurality of information processing devices;
Have
Any of the plurality of information processing devices
A date and time setting unit for setting a plurality of shooting dates and times according to a user input operation for specifying shooting dates and times,
A person image analysis unit that acquires area information for each person from the captured video at a plurality of shooting dates and times set by the date and time setting unit for each shooting date and time ;
A mask image setting unit for setting the mask image for each shooting date and time;
Based on the area information for each person acquired by the person image analysis unit for each shooting date and time, the image area of the person in the shooting video for each shooting date and time is changed to the mask image for each shooting date and time , A video output control unit that generates and outputs the output video in which the mask image is superimposed on a corresponding position on the background image; and
A monitoring system characterized by comprising A monitoring method for causing an information processing device to perform processing for generating and outputting an output video in which an image area of a person detected from a captured video in a monitoring area is changed to a mask image,
A first step of setting a plurality of shooting dates and times in accordance with a user input operation for specifying shooting dates and times;
A second step of acquiring region information for each person from each of the plurality of shooting dates and times set by the first step for each shooting date and time ;
A third step of setting the mask image for each photographing date and time;
Based on the area information for each person acquired for each shooting date and time in the second step, the image area of the person in the shooting video for each shooting date and time is changed to the mask image for each shooting date and time , A fourth step of generating and outputting the output video in which the mask image is superimposed at a corresponding position on the background image;
The monitoring method characterized by having.

Images