JP7581558B1 - Information processing system, information processing method, and program - Google Patents

Abstract

[Problem] As the number of locations where cameras are installed increases, there is a growing need to be able to search for specific people from images captured by the cameras.
According to one aspect of the present disclosure, there is provided an information processing system for displaying video captured by a camera on an information terminal, which searches for video containing a facial image from among videos, displays a plurality of videos as a result of the search, selects a video from the plurality of videos in response to a user's instruction, displays the facial image contained in the video together with the selected video, and registers the facial image in response to the user's instruction.
[Selected figure] Figure 5

Description

The present invention relates to an information processing system, an information processing method, and a program.

Patent document 1 discloses a program that causes one or more computers to execute the following processes: displaying multiple camera icons on a display device; receiving an input to surround at least two or more camera icons from among the multiple camera icons displayed on the display device; and searching for a person based on images captured by cameras corresponding to each of the at least two or more camera icons in response to receiving the input to surround at least two or more camera icons.

JP 2023-129429 A

As the number of locations where cameras are installed increases, there is a growing need to search for specific people using footage captured by the cameras.

An information processing system is provided that displays video captured by a camera on an information terminal. The information processing system searches for video containing a facial image from among the videos, displays multiple videos as a result of the search, selects a video from the multiple videos in response to a user's instruction, displays the facial image contained in the video together with the selected video, and registers the facial image in response to the user's instruction.

FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system. FIG. 2 is a diagram illustrating an example of a hardware configuration of the server device. FIG. 3 is a diagram illustrating an example of a hardware configuration of the client device. FIG. 4 is a diagram illustrating an example of a hardware configuration of a surveillance camera. FIG. 5 is a flowchart showing an example of information processing related to registration of a face image in the information processing system. FIG. 6 is a diagram showing an example of the list display screen. FIG. 7 is a diagram showing an example of a calendar screen being superimposed and displayed. FIG. 8 is a diagram showing an example of the video detail screen (part 1). FIG. 9 is a diagram (part 2) showing one example of the video detail screen. FIG. 10 is a diagram showing an example of the person registration screen. FIG. 11 is a diagram showing an example of the editing screen. FIG. 12 is a diagram showing an example of a movie clip creation screen. FIG. 13 is a flowchart showing an example of information processing related to a face image search in the information processing system. FIG. 14 is a diagram showing an example of the person search screen. FIG. 15 is a diagram showing an example of the list display screen. FIG. 16 is a diagram showing an example of the video details screen. FIG. 17 is a diagram showing an example of the notification setting screen. FIG. 18 is a diagram showing an example of the notification setting screen.

The following describes embodiments of the present invention with reference to the drawings. The various features shown in the following embodiments can be combined with each other.

<Embodiment 1>
1. System Configuration Diagram FIG. 1 is a diagram showing an example of a system configuration of an information processing system 1000. As shown in FIG. 1, the information processing system 1000 includes a server device 100, a client device 110, a client device 120, and a plurality of surveillance cameras 160 as a system configuration. The server device 100, the client device 110, the client device 120, and the surveillance cameras 160 are communicatively connected via a network 150. The network 150 may include either or both of a WAN (Wide Area Network) and the Internet. The network 150 is configured to enable communication between a plurality of devices connected to the network 150 via wired and wireless communication.

The information processing system 1000 is a system that provides video search services.

The server device 100 is a device that provides the main functions of the information processing system 1000, and executes the main processing of the embodiment described below. The server device 100 is a device that outputs (displays) images captured by multiple surveillance cameras 160 on the client device 110 or the client device 120.

The surveillance cameras 160 are cameras installed for the purpose of monitoring and/or recording. In the specification, the surveillance cameras 160 are installed in multiple locations in a specific store. For example, multiple locations may be multiple entrances to the store (e.g., an east entrance, a west entrance, a dedicated exit, etc.). Note that these are examples and do not limit the locations where the surveillance cameras 160 are installed. For simplicity, only three surveillance cameras 160 are shown in FIG. 1, but there may be two, four or more. It is sufficient that multiple surveillance cameras 160 are included in the information processing system 1000. In the specification, the surveillance cameras 160 are installed in one store, but multiple stores may each have multiple cameras.

The client device 110 is a terminal device operated by the owner or manager (hereinafter simply referred to as the owner) of the surveillance camera 160. The client device 110 displays screens such as those shown in Figs. 17 and 18, which will be described later.

The client device 120 is a terminal device operated by a manager or the like of the store in which the surveillance camera 160 is installed. The client device 120 displays screens such as those shown in Figures 6 to 12 and Figures 14 to 16, which will be described later.

Here, the information processing system described in the claims may be composed of multiple devices, or may be composed of one device. When the information processing system described in the claims is composed of one device, an example of that device is the server device 100. When the information processing system described in the claims is composed of multiple devices, an example of the multiple devices is the server device 100 and at least one of the multiple surveillance cameras 160 and the client device 110 or 120, or a cloud server composed of multiple server devices that provide the functions of the server device 100.

2. Hardware Configuration (1) Hardware Configuration of Server Apparatus 100 FIG. 2 is a diagram showing an example of the hardware configuration of the server apparatus 100. As shown in FIG.
2, the server device 100 includes, as its hardware configuration, a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240. The control unit 210, the storage unit 220, and the communication unit 230 are electrically connected via the internal bus 240.

The control unit 210 is a CPU (Central Processing Unit) or the like, and controls the entire server device 100.

The storage unit 220 is any one of a hard disk drive (HDD), a read only memory (ROM), a random access memory (RAM), a solid state drive (SSD), etc., or any combination of these, and stores programs, data used by the control unit 210 when executing processing based on the programs, etc. The storage unit 220 is an example of a storage medium. Examples of data used by the control unit 210 when executing processing based on the programs include video data sent from the surveillance camera 160, facial images of the detection target, data related to the detection target, data related to the transmission destination set in Figures 17 and 18, etc., described below, etc.

In the specification, the data used by the control unit 210 when executing processing based on the program is described as being stored in the storage unit 220, but it may also be stored in a storage unit of another device that can communicate with the server device 100. The data may be stored in the storage unit of any device as long as the control unit 210 can refer to or obtain the data. The control unit 210 executes processing based on the program stored in the storage unit 220, thereby realizing the functions of the server device 100 and the processing of the flowcharts shown in Figures 5 and 13 described below. Note that, although this processing is described as being mainly executed by the server device 100, it may instead be executed by either the client device 110 or client device 120, or one of the multiple surveillance cameras 160.

The communication unit 230 connects the server device 100 to the network 150 and manages communication with other devices.

Note that the hardware configuration of each of the control unit 210, the storage unit 220, and the communication unit 230 is not limited to one. For example, multiple control units may be included in the server device 100. The same applies to the client device 110 and the client device 120 described below.

(2) Hardware Configuration of the Client Device 110/120 FIG. 3 is a diagram showing an example of the hardware configuration of the client device 110/120.
3, the client device 110 includes, as its hardware configuration, a control unit 310, a storage unit 320, an input unit 330, an output unit 340, a communication unit 350, and an internal bus 360. The control unit 310, the storage unit 320, the input unit 330, the output unit 340, and the communication unit 350 are electrically connected via the internal bus 360.

The control unit 310 is a CPU or the like, and controls the entire client device 110.

The storage unit 320 is an HDD, ROM, RAM, SSD, etc., or any combination of these, and stores programs, data used by the control unit 310 when executing processing based on the programs, etc. The storage unit 320 is an example of a storage medium.

In the specification, the data used by the control unit 310 when executing processing based on a program is described as being stored in the memory unit 320, but the data may also be stored in a memory unit of another device that can communicate with the client device 110. The data may be stored in the memory unit of any device as long as the control unit 310 can refer to or obtain the data. The functions of the client device 110 are realized by the control unit 310 executing processing based on the program stored in the memory unit 320.

The input unit 330 is a device that inputs information to the client device 110 in response to an operation by an operator. The input unit 330 accepts an operation input made by a user. The operation input is transferred as a command signal to the control unit 310 via the internal bus 360. The control unit 310 may execute a predetermined control and/or calculation based on the transferred command signal as necessary. The input unit 330 may be included in the housing of the client device 110 or may be externally attached. For example, the input unit 330 may be implemented as a touch panel integrated with the output unit 340. When the input unit 330 is implemented as a touch panel, the user can input a tap operation, a swipe operation, or the like to the input unit 330. Instead of a touch panel, a switch button, a mouse, a track pad, a keyboard, or the like can be adopted as the input unit 330.

The output unit 340 is, for example, a display unit such as a display, and is a device that outputs (displays) information as a screen of a graphical user interface (GUI) that can be operated by a user. The output unit 340 may be included in the housing of the client device 110, or may be attached externally. More specifically, the output unit 340 may be implemented as a display device such as a liquid crystal display, an organic EL (Electron-Luminescence) display, or a plasma display. It is preferable that these display devices are implemented by using different devices depending on the type of client device 110.

The communication unit 350 connects the client device 110 to the network 150 and manages communication with other devices.

The hardware configuration of client device 120 is similar to that of client device 110.

In the specification, the client device 110 and the client device 120 are described using a PC (Personal Computer) as an example. However, the client device 110 and the client device 120 may be a smartphone, a tablet computer, or the like. The client device may be any device that can display a screen as described below, accept user operations via the screen, or the like, and transmit information to the server device 100.

(3) Hardware Configuration of Surveillance Camera 160 FIG. 4 is a diagram showing an example of the hardware configuration of the surveillance camera 160. As shown in FIG.
4, surveillance camera 160 includes, as its hardware configuration, a control unit 410, a storage unit 420, an image capturing unit 430, a communication unit 440, and an internal bus 450. Control unit 410, storage unit 420, image capturing unit 430, and communication unit 440 are electrically connected via internal bus 450.

The control unit 410 is a CPU or the like, and controls the entire surveillance camera 160.

The storage unit 420 is an HDD, ROM, RAM, SSD, or any combination of these, and stores programs, data used by the control unit 410 when executing processing based on the programs, and the like. The storage unit 420 is an example of a storage medium.

In the specification, the data used by the control unit 410 when executing processing based on a program is described as being stored in the memory unit 420, but the data may also be stored in a memory unit of another device that can communicate with the surveillance camera 160. The data may be stored in the memory unit of any device as long as the control unit 310 can refer to or obtain the data. The functions of the surveillance camera 160 are realized by the control unit 410 executing processing based on the program stored in the memory unit 420.

The photographing unit 430 is a camera that photographs a subject. The camera includes, for example, an image sensor, a lens, and an IR cut filter.

The communication unit 440 connects the surveillance camera 160 to the network 150 and handles communication with other devices.

3. Information Processing The information processing of the first embodiment will now be described.

(1) Overview of Processing (1-1) Facial Image Registration The control unit 210 searches for images containing facial images from images captured by the surveillance camera 160 included in the information processing system 1000, and outputs (displays) multiple images containing facial images as search results. The control unit 210 selects an image from multiple images included in the search results in response to a user's selection operation, and outputs (displays) the facial image of a person included in the image together with the selected image. The control unit 210 registers the facial image in response to a user's instruction.

By performing this type of processing, you can easily find people from the video and register facial images of people of interest in the information processing system 1000.

(1-2) Facial Image Search The control unit 210 outputs (displays) the multiple registered facial images. The control unit 210 selects a facial image from the multiple facial images in response to a user's selection operation, searches for a video including the selected facial image, and outputs (displays) the multiple videos including the selected facial image as a search result. The control unit 210 selects a video from the multiple videos included in the search result in response to the user's selection operation, and outputs (displays) the selected video.

By performing this type of processing, it is possible to detect and output an image of the person whose face image is selected from the registered face images.

(2) Processing Details (2-0) Pre-processing First, as pre-settings, the angle to be determined as the front angle of the face, the angle difference to be determined as being sufficiently different from the front angle, the number of reliable image pixels, the designated time for obtaining the best shot, etc. These settings are set, for example, according to the user's instructions via the client device 120, and are stored in each surveillance camera 160 (or the server device 100).

In this system, each security camera 160 basically always captures video and transmits it to the server device 100 for storage, but at the same time, a best shot extraction process is also performed. Here, the best shot is the optimal face image (or whole body image) of the same person captured from different specified angles, and corresponds to the face images (740, 750, 760, 770, 910, 920, 930, 940) in FIG. 8 and/or FIG. 10 described below. This process is mainly performed on the security camera 160 side, but may also be performed on the server device 100 side. The advantage of performing the process on the camera side is that the processing load and usage fee on the server side are reduced accordingly, but on the other hand, the camera side is required to have specifications to perform the corresponding process.

The details of the best shot extraction process are as follows. First, the surveillance camera 160 tracks the whole body image of a person from the camera image. Next, a face is detected from the data of the whole body image and tracked. Then, an angle is detected from the tracked face, and an image that falls within the range of the front angle of the face is set as the 1st best shot, and the angle of the face is recorded. The 1st best shot may be the one with the highest score, for example, using an AI model that detects a front image of a face. Furthermore, the angle is continuously detected from the tracked face, and if there is a face whose angle difference with the angle of the face of the 1st best shot is equal to or greater than the front angle, it is saved as the 2nd best shot. The 2nd best shot may be the one with the highest score, for example, using an AI model that detects an image of a face facing left (or right) relative to the camera. Furthermore, an image that has an angle difference with both the 1st and 2nd best shots is saved as the 3rd best shot. The 3rd best shot may be the one with the highest score, for example, using an AI model that detects an image of a face facing right (or left) relative to the camera. Furthermore, for a person corresponding to at least one of the 1st, 2nd, and 3rd best shots, a full-body image (or a bust-up image, etc.) of that person may be saved as the 4th best shot. A bust-up image is an image in which the upper half of a person's body is shown. The 4th best shot may be, for example, the one with the highest score using an AI model that detects a full-body image (or a bust-up image, etc.). If the tracked face is continuously detected and an image that is within the front angle range and has a number of images that exceeds the reliable image pixel count is found, the 1st best shot may be replaced. Similarly, the 2nd, 3rd, and 4th best shots may also be replaced. In addition, if the designated time for obtaining the best shot is exceeded, the extraction of the best shot may be stopped. In addition, although the 1st, 2nd, 3rd, and 4th best shots are selected from the highest scores from each perspective, multiple images including those with the next highest scores from each perspective may be selected. Then, when tracking is completed, the first, second, third, and fourth best shots are sent to the server device 100 together with metadata indicating the camera and date and time when each image was taken. After receiving the first, second, third, and fourth best shots and the metadata indicating the camera and date and time when each image was taken, the surveillance camera 160 stores the images in association with each other/with each image and the metadata. At this stage, the face images may be averaged based on the first, second, and third best shots, and the facial features may be calculated and stored in association with the images and metadata. Hereinafter, this set of associated information (first, second, third, and fourth best shots/metadata indicating the camera and date and time when each image was taken/facial features) is referred to as best shot information. The best shot information is stored as multiple best shot information that are different from each other for each person/scene.

5 is a flowchart showing an example of information processing related to registration of a face image in the information processing system 1000. Note that, although this processing will be described as being mainly executed by the server device 100, it may instead be executed by any of the client device 110 or 120, or the multiple surveillance cameras 160.

In step S510, the control unit 210 searches for footage containing a facial image from footage captured by the surveillance camera 160 included in the information processing system 1000. Specifically, first, the control unit 210 reads out a number of pieces of best shot information stored in the pre-processing described above. Then, for each piece of best shot information, the image is identified based on metadata indicating the camera and date and time that captured the first (or fourth) best shot.

Note that step S510 may be realized as follows. Videos captured by the multiple surveillance cameras 160 included in the information processing system 1000 are transmitted to the server device 100 via the network 150 and stored in a predetermined storage area such as the storage unit 220. The video is stored in a predetermined storage area such as the storage unit 220 in a manner that allows identification of which surveillance camera 160 the video was captured by. As a manner that allows identification of which surveillance camera 160 the video was captured by, a different folder (directory) may be created in the storage unit 220 for each surveillance camera and the video may be stored in the folder, identification information that identifies the surveillance camera may be written in the file name of the video, or identification information that identifies the surveillance camera may be written in the metadata of the video file. In addition, the video is stored in a predetermined storage area such as the storage unit 220 in a manner that allows identification of when the video was captured. As a mode for identifying when the video was shot, a different folder (directory) may be created in the storage unit 220 or the like for each date, and the video may be stored in the folder, or the date and time information when the video was shot may be described in the file name of the video, or the date and time information when the video was shot may be described in the metadata of the video file. The control unit 210 searches for videos that contain facial images from the videos stored in a specified storage area such as the storage unit 220, and obtains the search results.

For example, the control unit 210 inputs a video to the trained model. This trained model is trained using the video as input data, whether or not the video contains a face image, and, if the video contains a face image, time information from when a person in the face image appears to when it disappears as output data. Furthermore, if the same video contains multiple face images, the trained model is trained using time information from when a person in each of the multiple face images appears to when it disappears as output data.
The control unit 210 obtains, from the learned model, information as to whether or not the video contains a facial image, and, if the video contains a facial image, time information from when a person in the facial image appears to when the person in the facial image disappears as a search result.

In step S520, the control unit 210 generates a list display screen including a reduced image (hereinafter also simply referred to as a thumbnail) of the timing at which the facial image appears in the video including the facial image, based on the search result. The control unit 210 transmits the generated list display screen to the client device 120, and controls the output unit of the client device 120 to display it.

Figure 6 is a diagram showing an example of a list display screen 600. The list display screen 600 shown in Figure 6 includes a person video tag 610 and a person video area 670. The person video tag 610 is included in a side menu of the screen. In addition, in a screen that includes a side menu, the entire screen including the side menu may be referred to as the screen, or the portion not including the side menu may be referred to as the screen. The same applies to the following screens. When the person video tag 610 is selected, the processes of steps S510 and S520 described above are executed, and thumbnails of the search results are displayed in the person video area 670. The example of Figure 6 shows that the person video tag 610 is selected on the list display screen 600.

The device section 620 of the person video area 670 displays a list of thumbnails of people detected from the video captured by the surveillance camera 160 at the east entrance. Each thumbnail may display the person's tracking start time (not shown). Although three thumbnails are displayed in FIG. 6 as a list of thumbnails of people detected from the video captured by the surveillance camera 160, this is not limited to this. The number of thumbnails displayed on the screen may be changed depending on the size of the screen. Furthermore, when the "View more button" is selected, the control unit 210 controls the display to display further thumbnails of people detected from the video captured by the surveillance camera 160.

The device section 630 of the person image area 670 displays a list of thumbnails of people detected in the image captured by the security camera 160 at the west entrance of the store.
The device section 640 of the person image area 670 displays a list of thumbnails of people detected in the image captured by the surveillance camera 160 at the dedicated exit entrance.
The device sections included in the screen are arranged in order from the top (the top of the screen) of the list display screen 600 in order of devices with thumbnails (surveillance cameras 160) and the tracking start times displayed in the thumbnails. The same applies to the list display screen 1300 in Fig. 15 described later.

On the list display screen 600, a plurality of videos output as search results are output (displayed) in the form of reduced images for each imaging device that captured the plurality of videos.
The process of step S520 is an example of a process of outputting (displaying) reduced images of each of a plurality of videos including a face image as a search result.

The list display screen 600 includes a keyword input area 650 and a period input area 660. The keyword input area 650 is an area where a keyword can be input. When a keyword is input in the keyword input area 650, the control unit 210 searches for the device name of the surveillance camera 160, the serial number of the surveillance camera 160 that can identify the surveillance camera 160, the device tag set in the surveillance camera 160, and the like, based on the keyword. The control unit 210 then searches for facial images from the video of the corresponding surveillance camera 160, and displays the search results in the person video area 670. This process is an example of a process of searching for video containing facial images based on search conditions input via the screen.

The period input area 660 is an area in which a period can be input. When the period input area 660 is selected, the control unit 210 causes the list display screen 600 to be displayed with a calendar screen superimposed thereon.
FIG. 7 is a diagram showing an example of a calendar screen 1510 being displayed in an overlapping manner.
The calendar screen 1510 is a screen that displays dates and days of the week in a table format, and is configured to allow the start time and end time of the video to be searched to be selected. When a period (start time and end time) is input on the calendar screen 1510, the control unit 210 searches for the video shot during the input period as the search target, and displays the search results in the person video area 670. This process is an example of a process for searching for a video containing a face image based on the search conditions input via the screen. In the example of FIG. 7, the period is selected from March 29, 2024 to April 16, 2024. Note that the period input area 660 may be configured to allow the period to be specified as after March 29, 2024, etc. That is, the period may be configured to allow the period to be specified as after a certain point in time. Similarly, the period input area 660 may be configured to allow the period to be specified as before April 16, 2024, etc. That is, the period may be configured to allow the period to be specified as up to a certain point in time.

As described above, the list display screen 600 is configured to allow at least one of the time period and keywords to be input as a search criterion. The list display screen 600 may also be configured to allow the imaging device that captured the video to be input as a search criterion.

When one of the thumbnails is selected in the person image area 670 in FIG. 6, thumbnail selection information including identification information that identifies the selected thumbnail is transmitted from the client device 120 to the server device 100.

In step S530 of FIG. 5, the control unit 210 determines whether or not thumbnail selection information has been received. If thumbnail selection information has been received, the control unit 210 proceeds to step S540, and if thumbnail selection information has not been received, the control unit 210 returns to step S520.

In step S540, control unit 210 selects the image of the thumbnail identified by the thumbnail selection information.
In step S550, the control unit 210 generates a video detail screen including a timeline of the selected video and facial images of people included in the video. More specifically, the control unit 210 identifies the corresponding best shots 1st, 2nd, 3rd, and 4th for the video selected in S540 based on the best shot information, and generates a video detail screen for displaying the best shots together with the timeline of the video. The control unit 210 transmits the generated video detail screen to the client device 120 and controls the output unit of the client device 120 to display the video detail screen.

Figure 8 is a diagram (part 1) showing an example of a video detail screen 700. The video detail screen 700 includes a display area 790. The video captured by the corresponding surveillance camera 160 is displayed in the display area 790. The video includes a person 735. The person 735 is a person included in the video selected by the thumbnail. The video also includes a background 736, which is omitted in Figure 8 for simplification. The background 736 corresponds to a streetscape, an interior of a room, a natural landscape, etc. captured by the corresponding surveillance camera 160. By including the person 735 and the background 736 in the display area 790, the operator can understand what the person is doing and in what situation.

The video detail screen 700 includes a timeline 710. The timeline 710 indicates the time of the video displayed in the display area 790. More specifically, the timeline displays a time axis in a strip shape from the start point of the video to the end point of the video. The timeline of this embodiment displays the elapsed time from the start point as a scale. However, the timeline may display the time or date as a scale. The playback position display object 720 is an object (GUI part) that indicates the shooting date and time of the video displayed in the display area 790. When the playback position display object 720 is at the leftmost position, it is the start position of the video playback, and when it is at the rightmost position, it is the end position of the video playback. The playback position display object 720 indicates the position on the timeline 710 of the video being played in the display area 790. The operator of the client device 120 (such as the manager of the store where the surveillance camera 160 is installed) can also operate the playback position display object 720 to specify the shooting date and time of the video displayed in the display area 790. Flag 730 is a flag that indicates a period during which a person appears in the video, and is displayed on timeline 710. Flag 730 indicates that a person appeared at the leftmost position at the time, and indicates that the person disappeared at the rightmost position at the time.
The control unit 210 outputs a timeline relating to the selected video. The timeline is output in a manner that allows distinction between a portion showing the person of the registered face image and a portion not showing the person of the face image.

In the face image display area 795, face images of people appearing in the video displayed in the display area 790 are displayed. In the example of Fig. 8, the face image display area 795 includes a face image 740, a face image 750, a face image 760, and a face image 770.
Facial image 740 is a facial image facing forward with respect to surveillance camera 160. This is the best shot 1st corresponding to the displayed video. A facial image showing both eyes may be said to be a facial image facing forward with respect to surveillance camera 160. A facial image in which the face is oriented toward surveillance camera 160 may also be said to be a facial image facing forward. A facial image with the viewpoint in the direction of surveillance camera 160 may also be said to be a facial image facing forward.
Facial image 750 is a facial image facing left with respect to surveillance camera 160. This is the second best shot corresponding to the displayed video. A facial image in which only the right eye is shown instead of both eyes may be said to be a facial image facing left with respect to surveillance camera 160. Also, an image in which the right side of the face is captured but the left side of the face is not captured may be said to be a facial image facing left with respect to surveillance camera 160.
Facial image 760 is a facial image facing right with respect to surveillance camera 160. This is the third best shot corresponding to the displayed video. A facial image in which only the left eye is shown instead of both eyes may be said to be a facial image facing right with respect to surveillance camera 160. Also, an image in which the left side of the face is captured but not the right side of the face may be said to be a facial image facing right with respect to surveillance camera 160.
In addition, the face direction may be other than forward/left/right, and other directions such as diagonally left/right/backward may be used additionally/alternatively.
Facial image 770 is a facial image including the entire body of a person. This is the fourth best shot corresponding to the displayed video. The entire body includes not only the facial image but also the torso, hands, legs, etc. of the person.
In addition to the whole body, for example, a bust-up shot may be used in addition/alternatively.

In the face image display area 795, the multiple face images include at least one of a face image facing forward, a face image facing right, a face image facing left, and a face image including the entire body.
It can also be said that the face image display area 795 contains a plurality of face images of the same person taken from different angles.

The control unit 210 outputs (displays) images of people included in the selected video in which the person's face is identifiable, together with the video.

The control unit 210 may display a face image in the face image display area 795 at the time of the leftmost position of the flag 730, and may hide (or delete) the face image in the face image display area 795 at the time of the rightmost position of the flag 730. The control unit 210 may display the face image of a person in the face image display area 795 while the person appears in the video displayed in the display area 790. By executing such processing, the control unit 210 can display the face image of the person in a manner that makes it clear that the person is included in the selected video.
Examples of manners in which it can be recognized that the person is included in the selected video include when display area 790 (or person 735) and facial image display area 795 are displayed on the same screen, or when facial image display area 795 is displayed superimposed on display area 790.

The process of step S550 is an example of a process of outputting (displaying) a screen including the video selected in person video area 670 and a timeline of the video.
The process of step S550 is an example of a process of outputting (displaying) a plurality of facial images of the same person as the person included in the image of the thumbnail selected in person image area 670 together with the image.

The facial image display area 795 further includes a "Register Person" button 775. If the person appearing in the video displayed in the display area 790 is not yet registered in the information processing system 1000, the control unit 210 outputs the "Register Person" button 775 in addition to the facial image display area 795. This process can also be said to be that if the person appearing in the video displayed in the display area 790 is not yet registered in the information processing system 1000, the control unit 210 activates the "Register Person" button 775 and displays it in the facial image display area 795. Activating a button means making the button selectable.
If the face image of the person included in the video has not been registered, the control unit 210 outputs (displays) a registration button for registering the face image in a user-selectable manner.

If the person appearing in the image displayed in the display area 790 is a person who has already been registered in the information processing system 1000, the control unit 210 outputs the face image display area 795 including a “Registered” button 810.
9 is a diagram (part 2) showing one example of the video detail screen 700. The "registered" button 810 is a button indicating that the person whose face image is included in the face image display area 795 has already been registered as a person to be searched in the information processing system 1000. Note that this is not limited to a button and may be any GUI component, image, or the like, as long as it can indicate that the person whose face image is included in the face image display area 795 has already been registered in the information processing system 1000.
The facial image of a person included in the video is output (displayed) in a manner that makes it possible to identify whether or not the person has already been registered.

When the "Register Person" button 775 in FIG. 8 is selected, the control unit 210 generates a person registration screen. The control unit 210 transmits the generated person registration screen to the client device 120 and controls the output unit of the client device 120 to display it.

FIG. 10 is a diagram showing an example of a person registration screen 900. As shown in FIG. 10, the person registration screen 900 includes face images 910, 920, 930, and 940 that are the same as the face image displayed in the face image display area 795. The control unit 210 can replace the positions of the face images (face image 910, face image 920, face image 930, face image 940, etc.) based on a drag operation by the operator of the client device 120. The image at the position of the face image 910 is used as a cover photo. The cover photo is used on a screen or the like as a face image of a saved person shown in, for example, FIG. 14, FIG. 15, etc. described later.
In addition, the control unit 210 can delete each face image (face image 910, face image 920, face image 930, face image 940, etc.) in response to the operator of the client device 120 clicking the "x" button in the upper right corner of the face image. This is to prevent a decrease in the accuracy of the search in step S1140 described later when a face image other than that of a desired person is mistakenly included in the face images. Note that, in order to improve the accuracy of the search, instead of simply deleting the face image, it is also possible to automatically set a face image with the same orientation as the next candidate, or to display a list of several face images with the same orientation as the next candidate and set the one selected by the user.

The person registration screen 900 also includes a GUI component 950 indicating a plus sign. When the GUI component 950 is selected, the control unit 210 generates a screen for adding a face image. The control unit 210 transmits the generated screen to the client device 120 and controls it to be displayed on the client device 120. The control unit 210 displays, on the screen for adding a face image, another face image of the same person as the face image displayed in face image 910, face image 920, face image 930, and face image 940 in a selectable manner. The operator of the client device 120 can select another face image of the same person via this screen and add the face image. Note that as another method of adding a face image, for example, a face image separately prepared in the client device 120 may be uploaded.

Furthermore, when any one of the facial images 910, 920, 930, and 940 is selected and a predetermined operation is performed, the control unit 210 generates a screen for changing the facial image. The control unit 210 transmits the generated screen to the client device 120 and controls it to be displayed on the client device 120. The control unit 210 displays, on the screen for changing the facial image, another facial image of the same person as the selected facial image so that the selected facial image can be selected. The operator of the client device 120 can change the facial image to another facial image of the same person via this screen.
These processes are examples of processes for changing a registered face image or adding/deleting a face image based on a predetermined operation by the user.

The operator of the client device 120 enters, for example, a person's name, a person tag, a memo, etc., on a person registration screen 900 as shown in FIG. 10, and performs an operation to register a face image of the person as a person to be searched. When the "Save with this content" button is selected on the person registration screen 900, the registration information included in the person registration screen 900, including the face image, etc., is transmitted from the client device 120 to the server device 100.

In step S560 of FIG. 5, the control unit 210 determines whether or not registration information has been received. If registration information has been received, the control unit 210 proceeds to step S570, and if thumbnail selection information has not been received, the control unit 210 repeats the process of step S550.

In step S570, the control unit 210 registers the face image included in the registration information as a person to be searched for (also called a saved person). Specifically, among the multiple best shot information, the one that corresponds to the registration information received in S560 is made identifiable by setting a flag, etc.

When the operator of the client device 120 performs a predetermined operation on the person registration screen 900 (for example, right-clicking on a face image and selecting an edit button from the displayed menu), the control unit 210 generates an editing screen on which the face image can be edited. The control unit 210 transmits the generated editing screen to the client device 120 and controls it so that it is displayed on the output unit of the client device 120.

Figure 11 is a diagram showing an example of an editing screen 1010. The editing screen 1010 includes the face image selected on the person registration screen 900. The editing screen 1010 is configured to enable cropping of the face image, rotation and tilt correction, and adjustment of brightness and contrast.

The video detail screen 700 in FIG. 8 includes a "Create Movie Clip" button 780. When the "Create Movie Clip" button 780 is selected, the control unit 210 creates a movie clip creation screen 1700. The control unit 210 then transmits the movie clip creation screen 1700 to the client device 120, and controls it so that it is displayed on the output unit of the client device 120.

Figure 12 is a diagram showing an example of a movie clip creation screen 1700. The movie clip creation screen 1700 is configured to enable the creation of a movie clip. A movie clip is a function that cuts out a video by specifying the time that you want to keep from the video, and stores the video in the storage unit 220 or the like. Note that a movie clip may include a video to be saved, which is the specified video, and a time lapse, which is a fast-forwarded video of the specified video.

Movie clip creation screen 1700 displays a video timeline 1710. The operator can specify the start and end times of the video to be cut out by operating anchors 1740 and 1750 on timeline 1710. Anchor 1740 is a GUI component that specifies the start time of the video. Anchor 1750 is a GUI component that specifies the end time of the video. A still image of the video at the time specified by anchor 1740 is displayed in display area 1720. A still image of the video at the time specified by anchor 1750 is displayed in display area 1730.

The control unit 210 can create movie clips of video based on operations by the operator of the client device 120 and store them in a specified memory area such as the memory unit 220.

The control unit 210 outputs a timeline related to the selected video, and outputs a screen for creating a movie clip related to the video based on a specified operation via the screen including the timeline.

13 is a flowchart showing an example of information processing related to face image search in the information processing system 1000. Note that, although this processing will be described as being mainly executed by the server device 100, it may instead be executed by any of the client device 110 or 120, or the multiple surveillance cameras 160.
In step S1110, the control unit 210 generates a screen (saved person display screen) including a plurality of registered face images in response to a request from the client device 120. Specifically, from among the plurality of best shot information, those flagged in step S570 are extracted, and the best shot 1 for each is identified, and a saved person display screen for displaying them is generated. The control unit 210 transmits the generated saved person display screen to the client device 120.

Figure 14 is a diagram showing an example of a person search screen 1200. As shown in Figure 14, the person search screen 1200 includes a saved person tab 1210 and a saved person display area 1250. When the saved person tab 1210 is selected, a display request for the saved object display screen is sent from the client device 120 to the server device 100. Then, in response to the display request, a list of face images of people whose face images were registered in step S570 is displayed in the saved person display area 1250. Figure 14 shows that the saved person tab 1210 has been selected. As an example, the saved person display area 1250 in Figure 14 displays face image 1220, face image 1230, and face image 1240 as a list of saved people.

The operator of the client device 120 (such as the owner of the surveillance camera 160) selects the face image of the person to be searched from the face images displayed in the saved person display area 1250. When the operator of the client device 120 selects a face image, selection information including identification information for identifying the selected face image is transmitted from the client device 120 to the server device 100.

In step S1120, the control unit 210 determines whether or not selection information has been received from the client device 120. If selection information has been received, the control unit 210 proceeds to step S1130, and if selection information has not been received, the control unit 210 repeats the process of step S1120.

In step S1130, the control unit 210 selects a face image based on the identification information included in the selection information.
In step S1140, the control unit 210 searches for video captured by the surveillance camera 160 that includes the selected face image. Specifically, first, best shot information of the face image selected in step S1130 is referenced, and a video in which the person is included is identified using the corresponding face feature amount by an AI model or the like. Note that if the face feature amount was not calculated in the pre-processing described above, the face feature amount may be calculated in a similar manner at this point.
Note that step S1140 may be realized as follows. For example, the control unit 210 inputs a face image and multiple videos stored in the storage unit 220 or the like as input data to the trained model. This trained model is a trained model trained with the face image and multiple videos as input data, and whether or not the video includes a person identical to the person in the face image, and, if the video includes the person identical to the person in the face image, time information from when the person in the face image appears to when it disappears as output data. Furthermore, if multiple face images are included in the same video, the trained model is a trained model trained with time information from when each of the multiple face images appears to when it disappears as output data.

The control unit 210 obtains, from the trained model, information on whether the video contains a person identical to the person in the facial image, and, if the video contains a person identical to the person in the facial image, time information from when this person appeared to when he or she disappeared, as search results.

In step S1150, the control unit 210 generates a list display screen including thumbnails of the timing when a facial image of a video including the same person as the person in the selected facial image appears based on the search results. The control unit 210 transmits the generated list display screen to the client device 120 and controls the output unit of the client device 120 to display it.

Figure 15 is a diagram showing an example of a list display screen 1300. The list display screen 1300 shown in Figure 15 includes a saved person display area 1320. The saved person display area 1320 displays the face image 1310 selected on the person search screen 1200 and a list of thumbnails of videos that include the same person as the person in the face image 1310. The list of thumbnails is displayed for each device. For simplicity's sake, in Figure 15, only a list of thumbnails of videos captured by the surveillance camera 160 at the east entrance of the store is displayed as the device section 1340, but this is not limited to this. As shown in the list display screen 600 of Figure 6, a list of thumbnails is displayed for each device section.

When one of the thumbnails is selected on the list display screen 1300 in FIG. 15, thumbnail selection information including identification information for identifying the selected thumbnail is sent from the client device 120 to the server device 100.

In step S1160 of FIG. 13, the control unit 210 determines whether or not thumbnail selection information has been received. If thumbnail selection information has been received, the control unit 210 proceeds to step S1170, and if thumbnail selection information has not been received, the control unit 210 repeats the processes of steps S1150 and S1160.

In step S1170, control unit 210 selects the image of the thumbnail identified by the thumbnail selection information.
In step S1180, the control unit 210 generates a video detail screen including a timeline of the selected video and facial images of people included in the video. The control unit 210 transmits the generated video detail screen to the client device 120 and controls the output unit of the client device 120 to display the video detail screen.

Figure 16 is a diagram showing an example of a video detail screen 1400. The video detail screen 1400 is almost the same as the video detail screen 700 shown in Figure 8 and the video detail screen 700 shown in Figure 9. However, the face image display area 1410 included in the video detail screen 1400 does not include the "Register Person" button 775 included in the video detail screen 700 in Figure 8 and the "Already Registered" button 810 in Figure 9.

(2-3) Destination Setting In response to a request from the client device 110, the control unit 210 generates a notification setting (also called destination setting) screen and transmits it to the client device 110.

Fig. 17 is a diagram showing an example of a notification setting screen 1800. As shown in Fig. 17, the notification setting screen 1800 includes a notification setting tag 1810 and a notification setting area 1820. When the notification setting tag 1810 is selected, a request for generating a notification setting screen is transmitted from the client device 110 to the server device 100. Then, in response to the generation request, a GUI part 1830 capable of setting a destination for each device is displayed in the notification setting area 1820. The example of Fig. 17 shows that the notification setting tag 1810 has been selected.
This process is an example of a process that outputs a notification setting screen 1800 for a destination to which information indicating that a person in a registered facial image has been detected has been sent when a person in the facial image has been detected in video.

The GUI component 1830 is configured to allow the destination of email and Webhook integration to be set for each device name. The device name may be capable of uniquely identifying the device, or may be capable of identifying a tag (device tag) attached to the device. A device tag can be attached to a group of surveillance cameras 160. The notification setting screen 1800 is configured to allow the destination to be set for each of multiple cameras or for each group of cameras.

When a device name (e.g., "Store Entrance East Entrance" 1820) is selected in the GUI component 1830, the control unit 210 generates a notification setting screen 1900 and controls it to be displayed on the output unit 340 of the client device 110.

18 is a diagram showing an example of a notification setting screen 1900. A destination of a notification email can be set in an item 1910. A destination of a Webhook link can be set in an item 1920. A plurality of contacts can be set as a destination of a Webhook link.
In FIG. 18, the notification setting screen 1900 is configured so that a user can be set as a destination.

As described above, according to the first embodiment, it is possible to easily find a person from a video and register a facial image of the person of interest in the information processing system 1000. Furthermore, according to the first embodiment, it is possible to detect and output a video of a person whose facial image is selected from the registered facial images.

<Additional Notes>
The present invention may be provided in the following manner.
(Appendix 1)
An information processing system for displaying an image captured by a camera on an information terminal,
Searching the video for video containing a face image;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
displaying a facial image included in the selected image together with the selected image;
registering the face image in response to a user's instruction;
Information processing system.
(Appendix 2)
An information processing system for displaying an image captured by a camera on an information terminal,
Displaying multiple registered face images,
selecting a face image from the plurality of face images in response to a user instruction;
Searching for an image including a person corresponding to the selected face image;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
Displaying the selected image;
Information processing system.
(Appendix 3)
2. The information processing system according to claim 1,
The facial images to be displayed are multiple facial images of the same person.
Information processing system.
(Appendix 4)
4. The information processing system according to claim 3,
The plurality of facial images include facial images of the same person photographed from different angles.
Information processing system.
(Appendix 5)
5. The information processing system according to claim 4,
The plurality of face images include at least one of a face image facing forward, a face image facing right, a face image facing left, and a face image including the entire body.
Information processing system.
(Appendix 6)
2. The information processing system according to claim 1,
displaying, together with the face image, whether or not the face image has been registered in a manner that makes it possible to identify the face image;
Information processing system.
(Appendix 7)
2. The information processing system according to claim 1,
displaying a button for registering the face image together with the face image;
Information processing system.
(Appendix 8)
10. The information processing system according to claim 1,
Displaying a timeline relating to the video together with the video;
Information processing system.
(Appendix 9)
9. The information processing system according to claim 8,
the timeline displays a portion showing a registered face image in a identifiable manner;
Information processing system.
(Appendix 10)
9. The information processing system according to claim 8,
displaying a screen for creating a movie clip related to the video based on a predetermined operation via a screen including the timeline;
Information processing system.
(Appendix 11)
10. The information processing system according to claim 1,
The search for the video containing the face image is performed based on search conditions input by a user;
The search criteria include at least one of a period and a keyword.
Information processing system.
(Appendix 12)
10. The information processing system according to claim 1,
The plurality of images are displayed in the form of thumbnail images for each camera that captured the plurality of images.
Information processing system.
(Appendix 13)
10. The information processing system according to claim 1,
notifying the user when the registered face image is detected in the video;
Information processing system.
(Appendix 14)
14. The information processing system according to claim 13,
The notification destination when the registered face image is detected from the video can be set for each of the plurality of cameras or for each group of cameras.
Information processing system.
(Appendix 15)
10. The information processing system according to claim 1,
The registered face image can be changed or a face image can be added based on the user's instructions.
Information processing system.
(Appendix 16)
10. The information processing system according to claim 1,
The face image is displayed in such a manner that it is clear that the face image is a person included in the selected video.
Information processing system.
(Appendix 17)
10. The information processing system according to claim 1,
displaying an image of a person whose face is identifiable among the people included in the selected video together with the video;
Information processing system.
(Appendix 18)
An information processing method executed by an information processing system that displays an image captured by a camera on an information terminal, comprising:
Searching the video for images containing facial images;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
displaying a facial image included in the selected image together with the selected image;
registering the face image in response to a user's instruction;
Information processing methods.
(Appendix 19)
An information processing method executed by an information processing system that displays an image captured by a camera on an information terminal, comprising:
Displaying multiple registered face images,
selecting a face image from the plurality of face images in response to a user instruction;
Searching for an image including a person corresponding to the selected face image;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
Displaying the selected image;
Information processing methods.
(Appendix 20)
A computer that displays the images captured by the camera on an information terminal.
Searching the video for videos containing facial images;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
displaying a facial image included in the selected image together with the selected image;
registering the face image in response to a user's instruction;
A program for executing a process.
(Appendix 21)
A computer that displays the images captured by the camera on an information terminal.
Displaying multiple registered face images,
selecting a face image from the plurality of face images in response to a user instruction;
Searching for an image including a person corresponding to the selected face image;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
Displaying the selected image;
A program for executing a process.

Although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. New embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The embodiments and modifications of the embodiments are included within the scope and gist of the invention, and are included in the scope of the invention and its equivalents as set forth in the claims.

For example, part of the processing of the server device 100 described above may be executed by the client device 110 and/or the client device 120. Also, part of the processing of the server device 100 described above may be executed by the surveillance camera 160.

100: Server device 110: Client device 150: Network 160: Surveillance camera 210: Control unit 220: Storage unit 230: Communication unit 1000: Information processing system

Claims (21)

An information processing system for displaying an image captured by a camera on an information terminal,
Searching the video for video containing a face image;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
displaying a facial image included in the selected image together with the selected image;
registering the face image in response to a user's instruction;
Information processing system. An information processing system for displaying an image captured by a camera on an information terminal,
Displaying multiple registered face images,
selecting a face image from the plurality of face images in response to a user instruction;
Searching for an image including a person corresponding to the selected face image;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
Displaying the selected image;
Information processing system. 2. The information processing system according to claim 1,
The facial images to be displayed are multiple facial images of the same person.
Information processing system. 4. The information processing system according to claim 3,
The plurality of facial images include facial images of the same person photographed from different angles.
Information processing system. 5. The information processing system according to claim 4,
The plurality of face images include at least one of a face image facing forward, a face image facing right, a face image facing left, and a face image including the entire body.
Information processing system. 2. The information processing system according to claim 1,
displaying, together with the face image, whether or not the face image has been registered in a manner that makes it possible to identify the face image;
Information processing system. 2. The information processing system according to claim 1,
displaying a button for registering the face image together with the face image;
Information processing system. 3. The information processing system according to claim 1,
Displaying a timeline relating to the video together with the video;
Information processing system. 9. The information processing system according to claim 8,
the timeline displays a portion showing a registered face image in a identifiable manner;
Information processing system. 9. The information processing system according to claim 8,
displaying a screen for creating a movie clip related to the video based on a predetermined operation via a screen including the timeline;
Information processing system. 3. The information processing system according to claim 1,
The search for the video containing the face image is performed based on search conditions input by a user;
The search criteria include at least one of a period and a keyword.
Information processing system. 3. The information processing system according to claim 1,
The plurality of images are displayed in the form of thumbnail images for each camera that captured the plurality of images.
Information processing system. 3. The information processing system according to claim 1,
notifying the user when the registered face image is detected in the video;
Information processing system. The information processing system according to claim 13,
The notification destination when the registered face image is detected from the video can be set for each of the plurality of cameras or for each group of cameras.
Information processing system. 3. The information processing system according to claim 1,
The registered face image can be changed or a face image can be added based on the user's instructions.
Information processing system. 3. The information processing system according to claim 1,
The face image is displayed in such a manner that it is clear that the face image is a person included in the selected video.
Information processing system. 3. The information processing system according to claim 1,
displaying an image of a person whose face is identifiable among the people included in the selected video together with the video;
Information processing system. An information processing method executed by an information processing system that displays an image captured by a camera on an information terminal, comprising:
Searching the video for video containing a face image;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
displaying a facial image included in the selected image together with the selected image;
registering the face image in response to a user's instruction;
Information processing methods. An information processing method executed by an information processing system that displays an image captured by a camera on an information terminal, comprising:
Displaying multiple registered face images,
selecting a face image from the plurality of face images in response to a user instruction;
Searching for an image including a person corresponding to the selected face image;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
Displaying the selected image;
Information processing methods. A computer that displays the images captured by the camera on an information terminal.
Searching the video for video containing a face image;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
displaying a facial image included in the selected image together with the selected image;
registering the face image in response to a user's instruction;
A program for executing a process. A computer that displays the images captured by the camera on an information terminal.
Displaying multiple registered face images,
selecting a face image from the plurality of face images in response to a user instruction;
Searching for an image including a person corresponding to the selected face image;
displaying a plurality of images as a result of said search;
Selecting an image from the plurality of images in response to a user instruction;
Displaying the selected image;
A program for executing a process.

Images