JP5376403B2 - Video display device and program - Google Patents

Description

  The present invention relates to a video display device that displays a video of a person cut out from a video shot at an angle of view where a plurality of people enter with a video camera in a main display area of the screen, and more specifically, displays an individual human video. The present invention relates to a video display device having a function of switching between a screen and a screen that displays the video of all the members according to the user's intention, and a program used for the computer of the video display device.

Conventionally, for example, when displaying a participant's video in a video conference or the like, a portion where a person is extracted from the video captured by the camera, and the extracted person is zoomed and displayed. A technique for creating a screen necessary for the progress of a conference by cutting and displaying is known.
Patent document 1 can be illustrated as a prior art which performs creation and display control of the above display screens applied to a video conference. In Patent Literature 1, a specific one selected by a user is cut out using a mouse pointer or the like on a screen that displays an image of a scene including a plurality of persons (all members), and the cut out specific person is displayed. It is described that control is performed.

However, the video display of Patent Document 1 does not indicate what kind of user operation is used to switch the display screen after a specific person is displayed. Therefore, there arises a problem that it is impossible to respond to a user's request to return the screen to everyone's video or to directly switch to another person's video.
The present invention has the above-mentioned problems in the conventional video display device that detects a person in real time by using a wide-angle video camera capable of shooting at an angle of view where a plurality of persons enter, and cuts out and displays the detected person's video. The purpose was to quickly switch from a screen displaying personal images to a screen displaying images of other members or everyone selected according to the user's intention, so that it can be displayed. It is in.

  The video display device according to the present invention includes a person detection unit that detects a human candidate from video captured by a video camera at an angle of view where a plurality of persons enter, and a human candidate detected by the human detection unit from a time-series captured video. Display target determining means for determining a person to be displayed based on a detection result, area classifying means for classifying an area on a captured video in which the person determined by the display target determining means is contained, and the area classifying means Video cutout means for cutting out the video of the divided area from the shot video, image display means for displaying an image on the screen based on the video, and all of the video cut out by the video cutout means as the image display means Processing for selecting a thumbnail to be used for the selection screen, and processing for converting the video clipped by the video cropping means to an image to be used for the main screen of the image display means Display data processing means, area selection means for selecting a part of the area by an operation on the thumbnail of the selection screen of the image display means, and the clipped video of the area selected by the area selection means It has a display control means for controlling the display on the image display means using the data processed by the display data processing means.

  According to the present invention, in a video display in which a person is detected in real time by using a wide-angle video camera capable of shooting at an angle of view where a plurality of persons enter, and an image of the detected person is cut out and displayed, individual person images are displayed. It is possible to quickly switch the display to the screen selected according to the user's intention between the screen displaying the video and the screen displaying the video of all members.

It is a figure which shows one structural example of the screen for selection of the video area displayed on a screen in the video display apparatus which concerns on embodiment of this invention. It is a figure explaining the transition of the screen by the selection operation which a user performs via the screen for selection (FIG. 1). It is a figure which shows the other structural example of the screen for selection of the video area | region displayed on a screen in the video display apparatus which concerns on embodiment of this invention. It is a functional block diagram which shows schematic structure of the image processing system of the video display apparatus concerning this invention. It is explanatory drawing (A) of the target image and detection concept of face detection and motion detection, and explanatory drawing (B), (C) of the tracking method of face detection and motion detection. It is a flowchart which shows the procedure of the process which produces the detection list of a person candidate based on a face or a motion detection result. It is a flowchart which shows the procedure of the process which produces a detection list. It is a flowchart which shows the procedure of the seek process performed by step S201 of the flow (FIG. 7) of the process which produces a detection list. It is a flowchart which shows the procedure of the reverse seek process performed by step S202 of the flow (FIG. 7) of the process which produces a detection list. It is a flowchart which shows the procedure of tracking performed by step S302, S306 of the flow (FIG. 8) which shows the procedure of a seek process. It is a conceptual diagram explaining the relationship between the person (person candidate) on the picked-up image at the time of creating a detection list, a detection list, and a detection list. It is a flowchart which shows the procedure of the production | generation process of the display data.

Embodiments according to the video display apparatus of the present invention will be described below.
The present invention relates to a video display device that displays on a main screen of a display device a video of a person cut out from a video shot at an angle of view where a plurality of people enter with a video camera, and displays a video of a participant in a video conference or the like It becomes an effective means when doing. In the following, an example that is adapted to applications such as video conferencing is shown as an embodiment.
In the present embodiment, an area to be cut out as a captured video in which a person is accommodated is an area in which individuals, a plurality of people (two people, three people, etc.) and all are accommodated, and is further diversified to include a full-size captured video area. A feature is that the user can select a video image to be displayed as a display target and display the screen.

"Display video selection screen"
In order for the user to select a desired video area from among various video areas to be displayed, the video display device indicates the video area to be displayed to the user via the display screen and instructs the selected video area. Accept user operations.
As a method for showing the video area to be displayed to the user through the display screen and instructing the video area to be selected, here, corresponding embodiments are shown by two different methods.
One is to create a thumbnail for the screen of the video area to be displayed as a selection screen and display this selection screen separately from the main screen (screen for displaying the screen of the selected video area, etc.). (Hereinafter, a function for selecting a video area to be displayed by this method is referred to as a “first selection function” or a “thumbnail selection function”).
The other is to create a selection screen by embedding a display element for selecting a video region to be displayed in a captured video (image) displayed on the main screen, and display this selection screen. (Hereinafter, a function for selecting a video area to be displayed by this method is referred to as a “second selection function” or an “embedding element selection function”).

FIG. 1 is a diagram showing a configuration example of a screen for selecting a video area by the first selection function (thumbnail selection function).
In FIG. 1, the screen of the image display unit (see FIG. 4 described later) of the video display device includes a main screen 110 and a selection screen 120 for selecting a screen to be displayed on the main screen 110. The main screen 110 in FIG. 6 is a screen that displays, for example, a video captured by a wide-angle video camera in full size, and displays an image of three persons on the screen.
In addition, the selection screen 120 arranges all the plurality of selection target screens as thumbnails, and in the embodiment of FIG. 1, the selection screen 120 is combined with the main screen 110 and always displayed.
In the example of FIG. 1, the thumbnails arranged on the selection screen 120 are a full-size screen as the screen [1], a screen where everyone is stored as the screen [2], and a screen where only “Mr. A” is stored as the screen [3]. The screen [4] is a screen that contains only “Mr. B”, and the screen [5] is a screen that contains only “Mr. C”. “Mr. A”, “Mr. B”, and “Mr. C” are displayed in the order of arrangement on the captured video of each person (from left to right).
In the example of FIG. 1, the selection screen 120 constitutes the same screen as the main screen 110, but is temporarily displayed on a part of the main screen 110 by an external operation (refer to the operation unit 210 in FIG. 2 described later). Or may be switched to the main screen to display only the selection screen.

The transition of the screen by the selection operation performed by the user via the screen for selecting the video area displayed on the screen of the image display unit will be described with reference to the operation example shown in FIG. FIG. 2 shows an example of the video area selection screen shown in FIG.
In each of the display screens of FIGS. 2A, 2 </ b> B, and 2 </ b> C, a screen selected by a user operation is displayed on the main screen 110. In addition, on the selection screen 120 of the display screens (A), (B), and (C), thumbnails of all the selection target screens are always displayed.
In the operation example of FIG. 2, when the video display device is activated, the full-size screen [1] shown in FIG. 2A is displayed as the current main screen as the initial screen. In this display state, when an operation for selecting the screen [3] (specific individual) from the thumbnails shown as selection targets on the selection screen 120 using the operation means 210 such as a remote controller or a mouse is performed. The video display device accepts the instruction and changes the main screen 110 from the screen [1] to the screen [3]. The selection operation of the operation means 210 is performed by a method of designating thumbnails on the selection screen 120 with a pointer, a method of designating thumbnails arranged on the selection screen 120 with an identification number, or an arrangement for each operation. This can be implemented by adopting a method of selecting thumbnails in order according to the above.
Since the selection screen 120 always displays thumbnails of all screens to be selected, the screen [3] (specific individual) to the screen [2] (fits all) or screen [1] (full size) It is possible to immediately transition to a screen where any thumbnail can be selected like transition to.

FIG. 3 is a diagram showing a configuration example of a screen for selecting a video area by the second selection function (embedding element selection function).
In FIG. 3, a selection screen is configured by embedding a display element for selecting a video region to be displayed in a captured video (image) being displayed on the screen of the image display unit of the video display device. Then, switching to the selected screen is performed by an operation on the selection display element of the selection screen. A screen 110 illustrated in FIG. 3 is a screen that displays, for example, a video captured by a wide-angle video camera in full size, and displays an image of three persons on the screen. A selection screen is configured by representing a selectable area using a frame indicating each area of [1] to [5] as a display element for selecting.
In addition, the method of presenting the selection screen to the user is to always display the display element indicating the area on the screen being displayed with an inconspicuous line, or to display the area according to the user's operation when necessary. The display can be switched to a selection screen having

The transition of the screen by the selection operation performed by the user via the selection screen shown in FIG. 3 displayed on the screen of the image display unit indicates the selection area using the operation means 210 such as a remote controller or a mouse. A method that is basically similar to the first selection function described with reference to the operation example shown in FIG. 2 can be adopted in that the above video is displayed on the main screen.
However, in the second selection function for embedding a display element for selecting a video area, as shown in the example of FIG. 2 of the first selection function, a full-size thumbnail is always displayed on the selection screen 120, Since it is not possible to select a full-size screen, the full-size display screen shown in FIG. 3 can be selected according to the user's operation even if the displayed screen is a screen other than the full-size screen such as an area where an individual fits. In order to always perform the selection operation to the full-size display screen, it is possible to implement by adding another key input means or the like.

"Image processing system configuration"
Next, the configuration of an image processing system that processes display data (data for displaying the screen described in the “display video selection screen”) used in the image display unit of the video display device from the captured image will be described. To do.
FIG. 4 is a functional block diagram showing a schematic configuration of the image processing system according to the present embodiment.
The image processing system according to the present embodiment illustrated in FIG. 4 operates under the control of a control unit (not shown) of the video display device, and includes a face detection unit 201, a motion detection unit 202, a merge unit 203, and a detection list creation unit 204. An image cutout unit 205, an image display unit 206, and a screen selection instruction unit 207.

  The control unit (not shown) of the video display device controls the entire image processing device, captures video captured by the video camera, processes the captured video as display data, and displays the processed display data. As means for controlling the means (display), a CPU (Central Processing Unit), a RAM (Random Access Memory) for temporarily storing programs and data necessary for CPU processing, and the CPU are operated for calculation. And a computer including a storage unit such as a ROM (Read Only Memory) storing a program for executing processing and the like. In this embodiment, a program for causing a computer constituting the control unit of the video display device to function as each processing unit of the image processing system shown in FIG. 4 is stored (recorded) in the storage unit such as the ROM, and the CPU By using this as a work memory and driving this program, the image processing system can be implemented by a computer of the control unit. Of course, this image processing system may be constituted by a dedicated image processing IC.

Each functional unit included in the image processing system (FIG. 4) has the following processing functions.
The face detection unit 201 and the motion detection unit 202 each detect a face and a motion, which are one of feature amounts regarded as a person, from the captured video (details will be described later with reference to FIG. 5).
The merge unit 203 merges the detection results of the face detection unit 201 and the motion detection unit 202 in accordance with predetermined processing conditions, collects images that can be regarded as persons (hereinafter referred to as “person candidates”) in the form of a detection list, and sets the detection results. (Details will be described later with reference to FIG. 6).
The detection list creation unit 204 is a processing unit that determines a person to be displayed according to a predetermined determination condition based on the detection list received from the merge unit 203, and summarizes the determination results in the form of a detection list (for details, see FIG. 7). 10 to 10).
The image cutout unit 205 cuts out a display target area in which a person detected based on the detection list received from the detection list creation unit 204 is stored from the captured video, and passes it to the image display unit 206.
The image display unit 206 selects an image used for the main screen from the captured video (full-size video and clipped video) in the display target area, and selects an image used for the selection screen (a thumbnail creation process or a video area). Display data processing such as processing for embedding display elements to be performed), and display data for the screen selected in accordance with an instruction from the screen selection instruction unit 207 is output as a display control signal for the display.
The screen selection instruction unit 207 receives a user's screen selection operation and instructs the image display unit 206 to display the selected screen.

"Data processing by image processing system"
Next, details of the data processing of the captured video performed by the image processing system (FIG. 4) will be described. This process creates a selection screen shown to the user in order to select the video to be displayed on the main screen from the full-size captured video and the human area video clipped from the captured video to be displayed. This is data processing of a captured video that displays the selected video on the main screen.
In this process, first, a person candidate is detected from a photographed video. This person candidate detection is a process of capturing a person appearing in a captured image and selected as a display target for the purpose of displaying a screen centering on the person, and it is necessary to correctly capture individual persons with changes over time. .

“Face and motion detection”
In order to correctly capture each person, the face and movement, which are the feature amounts of the person appearing in the captured video, are detected. This detection is performed on the captured video (image) input by the face detection unit 201 and the motion detection unit 202 of the image processing system (FIG. 4).
Face detection and motion detection performed by the face detection unit 201 and the motion detection unit 202 will be described with reference to FIG.
When a full-size photographed image as shown on the screen 110 in FIG. 5A is targeted, three persons appearing in this image are targeted for detection.
In the face detection, as shown in FIGS. 5A and 5B, a face area surrounded by a square is a detection target, and a face image included in a photographed image is detected by pattern recognition. This detection by pattern recognition is, for example, a method using a classifier (www.2ken.no-ip.com/), which is known as a pattern matching method, which combines feature value detection using Haar-Like features and Adaboost learning. This can be done by adopting publication / 2008 / P-06026.pdf.
In addition, as shown in FIGS. 5A and 5C, the motion detection is the detection target of the motion schematically indicated by the white arrow, and the motion of the moving body included in the captured image is optically flowed (between frames). The motion information on the screen is expressed as a vector). Motion detection using this optical flow is, for example, creating a motion model and color model of a tracked person, and probabilistically integrating them to determine the pixels (person candidate areas) belonging to the tracked person and determining the person area Can be implemented by adopting the existing method (see www-cv.mech.eng.osaka-u.ac.jp/~kubo/research/index-j.html).

“Create Discovery List”
The face and motion detected by the face detection unit 201 and the motion detection unit 202 are merged under a predetermined processing condition by the merge unit 203 to which the detection result is passed every time these detections are performed, and the detected person Put candidates together in a detection list. The person candidate detection list created here uses at least entries (1) to (3) exemplified below for each person candidate.
Detection list:
Entry (1) Detection position (coordinates of the vertex of the circumscribed rectangle of the face or the center point of movement)
Entry (2) Detected size (vertical and horizontal size of circumscribed rectangle of face or inscribed rectangle of motion)
Entry (3) Detection type (type of detection target face, movement, etc.)
The data in the detection list is data that detects a human candidate appearing in the current captured image, that is, an image that is regarded as a person, and may not be a person. Therefore, the processing result of the merge unit 203 is once in the form of the detection list until it is determined that the person candidate is a person to be displayed in the subsequent process and is registered in the detection list (described in detail later). to manage.
In addition, since the processing condition at the time of merging captures the feature regarded as a person with two features of face and motion, if the same person's face and motion are detected at the same time, the detection results may overlap, This is a condition for eliminating this duplication (see the description of S101 in the processing flow of FIG. 6 described later).

The detection list creation process performed by the merge unit 203 will be described with reference to the flowchart shown in FIG.
When the merge unit 203 receives the detection results of the face and the motion from the face detection unit 201 and the motion detection unit 202, respectively, the merge unit 203 eliminates duplication of detection results considered to be the same person detection result. Here, if the detection position (coordinate of the center point of the motion) obtained by motion detection is within a certain range around the detection position (coordinate of the vertex of the circumscribed rectangle of the face) obtained by face detection, The result is discarded (step S101). In other words, since the relationship between the face of the same person and the detection position of the motion is a fixed relationship, the above range of deduplication is determined from the positional relationship obtained by verifying this condition in advance, and the motions within the defined range are determined. Discard detection results and leave face detection results.
Next, the face detection result and the motion detection result that has not been discarded are merged (step S102). This process is a process for putting together human candidates as detection results obtained in each detection unit by attaching identifiers to the faces and movements detected as human candidates and labeling them.
Thereafter, the detection result merged for each detection unit is registered in the detection list (step S103), and the detection list creation processing is terminated. The creation of the detection list is a process of registering data in the detection list entries (1) to (3) for each person candidate labeled.

“Create detection list”
The data registered in the detection list created by the merging unit 203 is detection data of a candidate for a person (what is regarded as a person) and may not be a person.
Therefore, by finding the candidate person who is judged to be likely to be a person based on the relationship between the results obtained in person detection conducted so far in time series and the detection results obtained this time (currently) as the current detection results, Guarantees detection accuracy and obtains stable detection results.
As a technique, it is determined that the person seems to be a person including newly detected person candidates from the detected person candidates in consideration of the history obtained up to the previous time in the same determination process as the current attempt. Only the person candidate to be selected is selected as the detection result of this time (details will be described later with reference to FIGS. 7 to 10).

The detection list creation unit 204 is a processing unit that receives the current detection list (person candidate detection result) created by the merge unit 203 and performs the above-described person detection. The person obtained as a detection result by each processing is , In the form of a detection list. The detection list created here uses an entry (1) to (7) exemplified below for each person.
Detection list:
Entry (1) Detection position (coordinates of the vertex of the circumscribed rectangle of the face or the center point of movement, etc.)
Entry (2) Detection size (vertical and horizontal size of circumscribed rectangle of face or inscribed rectangle of movement)
Entry (3) Tracking target range (tracking range predetermined as a target with respect to the origin, see description of FIG. 10)
Entry (4) Histogram for tracking (Histogram elements, target area for which histogram is obtained, etc., see description of FIG. 10)
Entry (5) Detection counter (the count value is used as a factor for the image cutout unit 205 to determine the cutout range)
Entry (6) Detection end counter (used to delete entries whose count value is 0 or less)
Entry (7) Detection type (type of detection target face, movement, etc.)
This detection list data is used to determine the person candidate as a person to be displayed and to determine the result and the image area of the person to be cut out as a display image. And is managed so that it can be used to create the next detection list.

"Detection list creation process flow"
The detection list creation processing performed by the detection list creation unit 204 will be described in detail with reference to the flowcharts shown in FIGS.
In the following description, reference is made to the conceptual diagram of FIG. 11 for explaining the relationship between a person (person candidate) on a captured image, a detection list, and a detection list when a detection list is created. In FIG. 11, a screen 110 on the left corresponds to the detection list obtained this time, and each registered person candidate and each of the detection lists 306, 301 to 303 are connected by arrows. The right screen 110 corresponds to the previous detection list creation, and each registered person and each detection list 401 to 403, 405 are connected by arrows.

According to the processing flow of FIG. 7, when the detection list creation unit 204 receives the current detection list (person candidate detection result) created by the merge unit 203, first, the person candidate registered in the detection list and the current candidate are registered. Correlation with a person in the detection list that is managed (created last time) is performed based on the detection result (detection result) (step S201). In the process of step S201, the seek process of FIG. 8 showing a subsequence of this step is performed. This seek process is a process of searching for detection data of a person (person candidate) corresponding to the detection list side from the detection list side. Details of the seek process in FIG. 8 will be described later.
Next, contrary to the process of step S201, the detection list side is searched for detection data of the person corresponding to the detection list side, and additional registration is performed on a list of person candidates that are not currently in the detection list (step S202). . The process of step S202 performs the reverse seek process of FIG. 9 showing the subsequence of this step. Details of the reverse seek process in FIG. 9 will be described later.
Thereafter, as a result of performing the processing of steps S201 and S201, data relating to a person whose detection end counter value, which is an entry of the detection list to be created, becomes “0 or less” is deleted from the detection list (step S203). It should be noted that the data of the person associated with the seek process in step S201 is continuously registered in the detection list.
After performing the above processing, the creation of the detection list is terminated.

Here, the seek process in step S201 in the process flow of FIG. 7 will be described in detail with reference to FIG. 8 showing a subsequence of this step.
This seek process is a process of searching for detection data of a person (person candidate) corresponding to the detection list side from the detection list side. Referring to the example of FIG. 11, this is processing for confirming that each of the detection lists 401 to 403 and 405 has a predetermined correspondence with any of the detection lists 306 and 301 to 303.
According to the flow of FIG. 8, first, whether or not a candidate person in the detection list exists within a predetermined range from the detection position of the person to be processed in the detection list is registered in each list. Based on the detected position, the number of the detected positions is obtained, and the number of the obtained positions is determined. The processing is branched depending on the obtained number of “0”, “1”, “2 or more” (step S301).

If there is no person candidate in the detection list (step S301-0), the person to be processed in the detection list is tracked to obtain a tracking result (step S302). The tracking in step S302 is based on the flow of FIG. 10 showing the subsequence of this step (details of tracking in FIG. 10 will be described later).
After obtaining the tracking result, the end counter managed in the detection list is counted down by a predetermined value (−d0) (step S303), and the processing flow is ended.
Referring to the example of FIG. 11, the processing in steps S302 and S303 corresponds to the processing for the detection list 405. In this case, since the candidate person in the detection list does not correspond, the end counter is decreased by −1 (: −d0). Counting. The count value of the end counter is deleted from the entry in the detection list and is not displayed on the screen when the count value becomes 0 or less, as described above as the processing of step S203 in FIG. Accordingly, the predetermined value (-d0) set in the end counter is appropriately weighted in consideration of the detection failure of the person and the erroneous detection.

When there is only one person candidate in the detection list (step S301-1), the detection position registered in the detection list is set as the detection position registered in the newly created detection list (step S304).
Thereafter, the detection counter managed in the detection list is up-counted by a predetermined value (+ d1) (step S305), and the processing flow ends.
Referring to the example of FIG. 11, the processes in steps S304 and S305 correspond to the processes for the detection lists 402 and 403, and in this case, the person candidates managed by the data of the detection lists 302 and 303 correspond respectively. The detection counter is incremented by +2 (: + d1).

When there are two or more person candidates in the detection list (step S301-2 or more), the person to be processed in the detection list is tracked, and the tracking position for each of the two or more person candidates in the detection list is set. The most likely position (the closest position) obtained is obtained as a tracking result (step S306). The tracking in step S306 is based on the flow of FIG. 10 showing the subsequence of this step (details of tracking in FIG. 10 will be described later).
Thereafter, the detection counter managed in the detection list is up-counted by a predetermined value (+ d2) (step S307), and the processing flow ends.
Referring to the example of FIG. 11, the processing of steps S306 and S307 corresponds to the processing for the detection list 401, and in this case, two person candidates managed by the data of the detection list 301 and 302 correspond. Since the tracking position of the detection list 301 among them is close, the detection list 301 is used as a detection result, and the detection counter is incremented by +1 (: + d2).

Next, tracking in step S302 or S306 in the processing flow of FIG. 8 will be described in detail with reference to FIG. 10 showing a subsequence of this step.
The tracking area differs depending on whether the detection type is face detection or motion detection. In the case of face detection, as shown in FIG. 5B, an area having the same size as the face below the face is set as the tracking area. In the case of motion detection, as shown in FIG. 5C, an inscribed rectangle including the center of detected motion (difference between frames, etc.) is set as a tracking region.
In both cases of face detection and motion detection, a histogram in the tracking area determined as described above is obtained. Here, the obtained histogram is obtained by detecting the color of each pixel in the region and obtaining the distribution of the number of pixels with respect to the color.
According to the flow of FIG. 10, since the process branches depending on the detection type, first, it is confirmed whether face detection or motion detection (step S501).

If the detection type is a face, an area having the same size as the face below the face is set as a tracking area (step S502). On the other hand, if the detection type is movement, an inscribed rectangle including the center of movement is set. A tracking area is set (step S503), an image of each set area is taken out from a captured image based on the detection list, and a histogram of pixel values in this area (in this embodiment, the pixel value is a color value). Therefore, the distribution of the number of pixels with respect to the color) is obtained (step S504).
Next, tracking is performed on the captured image based on the detection list (step S505). This tracking uses a fixed range around the position on the image when the histogram is obtained in step S504 as a tracking target range, and is based on the detection list in the same manner as when the histogram is obtained in step S504. Process the image and determine the histogram.
In step S505, the tracking target range is scanned to obtain a histogram. This tracking is repeatedly executed for each frame for a predetermined number of frames (steps S505 to S507).
As a result of tracking, a histogram closest to the histogram of the detection list obtained in step S504 is obtained from the histograms of the detection list obtained in steps S505 to S507, and obtained as an area based on the histogram (step S506). ).
As described above, the area obtained as a result of tracking is used as the tracking position in steps S302 and S306 of the seek process (FIG. 8).

Here, the reverse seek process in step S202 in the process flow of FIG. 7 will be described in detail with reference to FIG. 9 showing a subsequence of this step.
This reverse seek process is a process of searching for detection data of a person corresponding to the detection list side from the detection list side. Referring to the example of FIG. 11, this is processing for confirming that each of the detection lists 306, 301 to 303 has a predetermined correspondence with any one of the detection lists 401 to 403 and 405. However, in this embodiment, as described with reference to FIG. 8, since the seek process has already been performed, a candidate person who is not currently in the detection list is searched and newly added to the list.

According to the flow of FIG. 9, since the person candidates not in the detection list are targeted, first, the person candidates in the detection list that do not correspond to the detection list are searched (step S401).
Next, it is confirmed whether or not the detection position of the human candidate obtained in step S401 is outside the tracking target range registered in the entire detection list (step S402). In this embodiment, it is premised that if it is within the tracking target range, it becomes a target of seek processing.
If it is outside the tracking target range in step S402 (step S402-YES), the person candidate is regarded as a person who is not in any detection list, and the detection counter managed in the detection list is increased by a predetermined value (+ d3). In addition to counting, the end counter is incremented by a predetermined value (+ d4) (step S403).
Referring to the example of FIG. 11, the process of step S403 corresponds to the process for the detection list 306. In this case, since there is no person corresponding to the detection list, a detection list 406 is newly added and registered in the detection list. The detection counter is incremented by +1 (: + d3), and the end counter is incremented by +3 (: + d4).
On the other hand, if it is within the tracking target range in step S402 (step S402-NO), it is out of the reverse seek target, so the process of step S403 is passed.
Steps S401 to S403 are repeated one by one until all the human candidates in the entire detection list that do not correspond to the entire detection list are completed.
After confirming that the reverse seek process has been completed for all person candidates in the all detection list, this process flow is terminated.

“Processing data for display”
The image cutout unit 205 cuts out a display target area in which a person detected based on the detection list received from the detection list creation unit 204 is contained from the captured video.
The new detection list received from the detection list creation unit 204 continues from the previous detection list as a result of the seek process (see FIG. 8), as described with reference to FIG. A person to be captured and a person newly captured as a result of the reverse seek process (see FIG. 9) are placed, and these are displayed.
When the area where each person on the newly created detection list is stored is cut out from the captured video, the count value of the detection counter in the entry of each person on the detection list is greater than or equal to a predetermined value (threshold). The image of the area is cut out as a display target on the condition that there is.
The count value of the detection counter is a predetermined value (d1) that is added according to the detection state of each person as described in steps S305 and S307 of the seek process (FIG. 8) and S403 of the reverse seek process (FIG. 9). -D3) can be changed. Accordingly, the predetermined values (d1 to d3) set in the detection counter that determines whether or not the display target is to be displayed are appropriately weighted in consideration of the detection failure of the person and the erroneous detection. In addition, from the same viewpoint, it is necessary to set an appropriate value for the threshold value.

Display data creation processing performed by the image cutout unit 205 and the image display unit 206 will be described in detail with reference to the flowchart shown in FIG.
According to the processing flow of FIG. 12, first, the cutout ranges of individual persons in the newly created detection list are respectively determined (step S601). This cut-out range is calculated by taking out each person's detection list having an entry whose count value of the detection counter is equal to or greater than a predetermined value from the persons in the detection list, and adding them to each person's detection list entry. A region where the person is contained around a certain detection position is obtained as a cutout range.
Next, based on the detection list, a cutout range of an area in which all persons are accommodated is obtained (step S602). The method for obtaining the cut-out range is to take out a detection list of all persons whose entries in the detection list have a count value of a detection counter equal to or greater than a predetermined value, and have the detection list entry for each person. An area in which the person is contained around the detected position is obtained, and an area in which a certain margin is provided in a minimum rectangle including each obtained area is obtained as a cutout range.
In this processing flow, the cut-out range of individuals and all members is obtained, but a cut-out range of any number of two or more people may be obtained by the same method as in step S602.

Next, an image is cut out based on the cutout range of the photographed image area obtained in steps S601 and S602, and screen display is performed (step S603).
In the image cutout performed here, the region cutout range is adjusted so that at least the image of the region including the cutout range obtained in steps S601 and S602 is stored as an appropriate image in relation to the size of the screen to be displayed.
In addition, the image display unit 206 performs processing for converting a captured image (a full-size video and a clipped image) in the display target area into an image to be used for the display screen. In this embodiment, an image is displayed in accordance with the screen used for the main screen 110 that displays the screen selected according to the instruction of the screen selection instruction unit 207 or the screen used for the image thumbnail used for the selection screen 120 in FIG. Processing such as magnification conversion is performed. Further, the processing for embedding display elements used for screen selection described with reference to FIG. 3 is performed, and the display data processing for creating the selection screen shown in FIG. 3 is performed.

  110 ··· Main screen, 120 ··· Screen for selection, 201 ·· Face detection unit, 202 ·· Motion detection unit, 203 · · Merge unit, 204 · · Detection list creation unit, 205 · · Image cropping unit, 206 · · Image display unit, 207... Screen selection instruction unit, 210.

JP-A-10-51755

Claims (10)

A person detecting means for detecting a person candidate from a video shot by a video camera at an angle of view where a plurality of persons enter;
Display target determination means for determining a person to be displayed based on a detection result of a person candidate detected by the person detection means from a time-series captured video;
Area classifying means for classifying an area on the captured video in which the person determined by the display target determining means is stored;
Video cutout means for cutting out the video of the area divided by the area dividing means from the shot video;
Image display means for displaying an image on the screen based on the video;
Processing that uses all of the video clipped by the video cutout means as thumbnails for use in the selection screen of the image display means, and an image using the video cut out by the video cutout means as the main screen of the image display means Display data processing means for performing the processing
Area selection means for selecting a part of the area by an operation on the thumbnail of the selection screen of the image display means;
Video having display control means for controlling display on the image display means using data processed by the display data processing means based on a cut-out video of the area selected by the area selection means Display device. The video display device according to claim 1 ,
The image display device according to claim 1, wherein the area classifying unit classifies an area in which all persons are accommodated and an area in which individual persons are accommodated. In the video display device according to claim 1 or 2 ,
The display data processing means further adds all captured images as data used for the display area for screen selection, and processes the all captured images as data used for the main screen. The video display device according to any one of claims 1 to 3 ,
The video display device, wherein the person detecting means detects a face and a motion from a captured video and detects a human candidate based on the detection result. The video display device according to claim 4 ,
The person detection means discards the motion detection result when the detected motion is within a certain range around the detected face, and collects the motion detection result and the face detection result that have not been discarded, together with a human candidate in a captured video unit An image display device provided as a detection result of The video display device according to any one of claims 1 to 5 ,
The display target determining means determines whether or not the candidate person obtained as the current detection result and the previously determined display target person can be regarded as the same person according to a predetermined determination condition based on both detection results. A video display device characterized by: The video display device according to claim 6 ,
The video display apparatus according to claim 1, wherein the display target determination unit uses a predetermined determination condition that both are within a predetermined range. In the video display device according to any one of claims 1 to 7 ,
The display target determination unit uses the detection data of the person determined as the display target last time, the detection data of the human candidate detected by the person detection unit from the current captured video, and the determination reference data, respectively. A video display device comprising means for managing data common to both in a form in which registration is performed with the same entry.   The computer is caused to function as each means of the person detection means, display target determination means, area segmentation means, video segmentation means, display data processing means, area selection means, and display control means in the video display device according to claim 1. Program for. A computer-readable recording medium on which the program according to claim 9 is recorded.

Images