JPH10234004A - Time-varying image display device and display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To confirm a person and to retrieve it quickly by extracting a time- varying image as a plurality of still images, classifying the images for each shot, discriminating whether or not a face is in existence in each shot, counting the number of times of appearance and generating a video appearance list. SOLUTION: A still image extract means 1 extracts a received time-varying image as a plurality of still images, an image consecutive classification means 2 classifies the extracted consecutive still images for each shot in the unit of a prescribed number. A face video image discrimination means 3 discriminates whether or not a human face is picked up for each shot classified by the image consecutive classification means 2, and in the case that the presence of the face is discriminated, a face existence discrimination means 4 discriminates whether or not the face has already been picked up in the time-varying image for a prescribed length, and a face classification count means 5 counts the number of times of appearance of each face. Based on the result, a characters list generating means 6 generates a characters list of the time-varying image based on the result and provides an output of it as characters data and the characters list is displayed on a display means 7.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image display device and a display method, and more particularly to a method for dividing a moving image having a predetermined length into a plurality of scenes according to scene changes and the like. The present invention relates to a moving image display device and a display method capable of classifying a predetermined video together with the number of appearances.

[0002]

2. Description of the Related Art With the spread of satellite broadcasting and cable television broadcasting in recent years, the number of video channels that can be viewed in ordinary households is steadily increasing. Also,
DVD (Digital Versatile Disc) that can record and reproduce video and audio information as digital data in addition to conventional video recording media such as video tapes and laser disks
It was commercialized in 1996, and it can be said that the environment in which general users can access a large amount of video media is improving.

[0003] On the other hand, a technique for efficiently accessing a desired image desired by a user from such a large amount of images has not been provided. For example, a television viewer may select a video (program) that the user wants to refer to in a program column published in a newspaper or the like, and a consumer who wants to purchase a video software package may select a title, performer, Select the desired video by referring to the outline etc. These are text descriptions of video information in characters, but not all of the video is described, and such a description requires a lot of trouble and time. .

[0004] Further, since the requirement of which part of the video the user wants to see must naturally differ for each user, it is impossible to create a text description applicable to all users. However, even if it can be created, it causes an increase in the amount of additional information, so that it is difficult to put it into practical use mainly in a source such as a broadcast medium which has a limited amount of information that can be supplied. For this reason, even if a TV viewer wants to watch only a specific topic in a news program, he / she keeps the TV switched on from the beginning of the program and waits until the topic he / she wants to watch is broadcast. It has to be had, and the topic watched on another news program has been repeatedly seen, which is a significant waste of time.

[0005] As described above, if the video access environment remains poor even if the information media has advanced dramatically, in the near future,
Even when a situation in which a user can access a large amount of video information is realized, it cannot be said that the information amount can be enjoyed smoothly, and there is a concern that the number of users who are confused by the large amount of information increases.

[0006]

SUMMARY OF THE INVENTION An object of the present invention is to make it easy for a user to grasp the whole image of a video being viewed by structuring and displaying video data when the user watches a digital video. And

In addition, the contents of a video represented in a moving image are analyzed, and face areas in the video are compared and determined by a dictionary.
By classifying and counting the number of appearances for each character, it is possible to confirm the person appearing in the moving image, and to perform a quick search.

[0008]

According to a first aspect of the present invention, there is provided a moving image display device for extracting a moving image as a plurality of still images, comprising: Image segmenting means for segmenting the still image into shots each consisting of one or an arbitrary number of images; and predetermined video judging means for judging whether or not a face has been shot for each shot segmented by the image segmenting means. A predetermined image existing determination means for determining whether a face has already been captured in the moving image when it is determined that a face exists in each of the shots, and the predetermined English layer for each different image A predetermined video classification and counting means for classifying and counting the number of appearances of each face in the moving image, and a video appearance list of the moving image based on the classified faces and the counted number of appearances of each face. And the video appeared list creating means for forming,
It is characterized by having.

According to a second aspect of the present invention, in the moving image display device according to the first aspect, when the character list creating means as the video appearance list creating means performs the character list display, It is characterized by further comprising display means for rearranging and outputting in the order of the number of appearances, and displaying a list of characters which have been rearranged and output.

According to a third aspect of the present invention, there is provided the moving image display device according to the first aspect, wherein a predetermined unit of video scene divided by scene division using a scene dividing unit as the screen dividing means. In displaying a list of characters as the above-mentioned video appearance list, a representative frame selection unit for selecting one still image from this scene is associated with a representative frame of the video scene in which the character appeared, and displayed in a list. And a display unit that simultaneously displays the information on a screen.

According to a fourth aspect of the present invention, in the moving image display device according to the third aspect, a user operates and instructs the representative frame on the list display screen displayed on the display unit. Accordingly, an input operation unit capable of reproducing a moving image directly from a predetermined unit of scene represented by the representative frame is provided.

According to a fifth aspect of the present invention, in the moving image display method, a moving image input with a predetermined time length is extracted as a continuous still image, and the continuous still images are extracted on a time axis. A step of dividing the unit into units; a step of determining whether or not the classified predetermined unit of still image includes a human face; and determining that the face is included in the predetermined unit of still image. If the face is the first one, the face is classified as one item. If the face included in the still image of the predetermined unit is already classified and registered, the classification is performed. And a step of performing a face extraction operation on all the still image groups of a predetermined unit to create a list of characters in the moving image.

According to a sixth aspect of the present invention, in the moving image display method according to the fifth aspect, the step of dividing the continuous still images into a predetermined unit on a time axis is performed by expressing the predetermined unit. The method further includes extracting and creating a representative image of a scene as a representative frame.

According to a seventh aspect of the present invention, in the moving image display method according to the fifth aspect, it is determined whether or not processing for all predetermined units has been performed before the step of creating the characters. The method is characterized in that a step of determining is provided.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION A video display device according to the present invention
It is assumed that an additional user interface is provided by cooperating with a video recording / reproducing device capable of random access. Hereinafter, a moving image display device according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

First, a moving image display device according to a first embodiment of the present invention will be described with reference to FIG. In FIG. 1, a moving image display device includes a still image extracting unit 1 for extracting a moving image of a predetermined length that has been input as a plurality of still images temporally continuous and extracting these still images. Continuous image dividing means 2 for dividing the continuous still images into shots of a predetermined unit, and determining whether or not a human face is photographed for each shot divided by the continuous image dividing means 2 A face image judging means 3, and a face appearance judging means 4 for judging whether or not this face has already been photographed in a moving image of this predetermined length when it is judged that a face exists in each of the shots; A face classification / counting unit 5 for classifying each face and counting the number of appearances of each face in the moving image having the predetermined length; and, based on each of the classified faces and the counted number of appearances of each face, Characters of the moving image The characters list generation means 6 to create a list, provided with a display means 7 for display to accept the character data output from the character list creating means 6 may be provided.

Next, a moving image display device according to a second embodiment showing a more detailed configuration of the moving image display device according to the first embodiment will be described with reference to FIG. In FIG. 2, the same components as those of the moving image display device according to the first embodiment of FIG. 1 are indicated by the same reference numerals, and duplicate explanations are omitted. The moving image display device according to the second embodiment comprises a still image extraction unit 1, a scene division unit 11 as a screen continuation division unit 2, and the screen continuation division unit 2 and is divided by the scene division unit 11. A representative frame selecting unit 12 for selecting a representative frame from a group of scenes, a face searching unit 13 and a face determining unit 14 constituting the face video determining unit 3, and a video determined to be a face is A face appearance determination unit 4 for determining whether or not a face has already been detected in a moving image; Face cluster creation unit 1 for creating a cluster
5, a cluster classification storage unit 16 for storing the classification of the created face cluster, a count-up unit 5 for counting up the face specified by the face determination unit 4, and a face read from the cluster classification storage unit. And a display screen 17 displaying the classification of the face and the number of appearances of the face in the order of appearance.

The face cluster creation unit 15 and the cluster classification storage unit 16 constitute a character list creation unit 6, and the display screen 17 constitutes a display unit 7. FIG. 3 is a conceptual diagram showing a display example of the moving image display device according to the second embodiment. In the screen 17, the persons who appeared in the video are arranged in order of the number of appearances. In this screen, a representative frame 23 of a shot in which the person appears is listed together with the rank 21 of the number of appearances for the person 22. Conventionally, a representative frame detects a transition of a scene by detecting a cut, and a section divided for each cut is taken as a shot, and one still image for each shot (for example, the first frame of the shot) is used as a representative frame. Although a general method, the present invention does not limit the shot division method and representative frame selection.

As the simplest method, the video may be mechanically divided every 10 seconds, and the video at the time when 5 seconds have elapsed in each of the divided shots may be used as a representative frame.
By pointing the representative frame 23 with a pointing device such as a mouse, the user can view the main part of the video from any shot in which the target person 22 appears.

Next, a method for displaying a moving image using the moving image display device having the above configuration will be described with reference to third and fourth embodiments. The moving image display method according to the third embodiment corresponds to the processing operation of the moving image display device according to the second embodiment shown in FIG. 2, and the details are shown in FIG. In the following description, a processing procedure for performing the above display by the moving image display device according to the second embodiment of the present invention will be described with reference to FIG. FIG. 4 is a flowchart showing a processing procedure of the video display device of the present invention. For moving images, first, a scene division step ST
It is divided into shots as processing units by 1. The generation of the shot does not necessarily have to be based on the scene in the video as described above.
It may be a mechanical unit of a predetermined time unit such as seconds.
The following processing is performed for each shot.

All frames in the shot are searched for a face area for each pixel area (eg, “20 × 20” pixels) in the frame (step ST).
2). For example, by scanning the inside of a frame with a search area including a range of “20 × 20” pixels, and by scanning the search area itself with a plurality of sizes, it is determined whether a face exists in the frame. Explore.

Here, an example of a method for determining the presence or absence of a face will be described. FIG. 5 is a conceptual diagram illustrating a face determination method. Hereinafter, a case where “20 × 20” pixels are set as the search area will be described. Since this area includes 400 pixels, if an image of “20 × 20” pixels is placed in a 400-dimensional superspace in which each pixel has coordinate values, it is expressed as one point. When a large amount of “20 × 20” pixel images that are known to be faces in advance are prepared and plotted in this superspace, a distribution having a certain bias is obtained. When the center position (average value) and the covariance matrix of the distribution are calculated, the feature amount of the distribution can be expressed by using the eigenvector and the eigenvalue.

When the eigenvectors are appropriately sorted in descending order of corresponding eigenvalues, a subspace (hyperplane) spanned by the selected eigenvectors
Can be said to be a hyperplane that represents the "face-like" spread. This is called an eigenface space, and is shown as 34 in FIG. When an image 31 of unknown “20 × 20” pixels is input, a distance 32 to the unique face space 34 in this hyperspace is observed, and when the distance 32 is equal to or less than a set threshold value. The area of “20 × 20” pixels may be determined to be a face.

On the other hand, also in the unique face space 34, feature points generated from the same person are distributed in a close range and can be clustered. This is shown as a cluster 35 in FIG. Already a personal cluster 35
Is generated, a point 37 at which the above-described unknown image 31 is projected onto the unique face space 34 and a center point 3 of the known face cluster
6 and the distance 33 are calculated, and when the distance 33 is equal to or smaller than the set threshold value, it is determined that the area of “20 × 20” pixels is the same person as the known face including 306.

The above distance calculation may be simply the Euclidean distance taking the square root of the sum of squares of the difference between the coordinate values, or the Mahalanobis distance taking the covariance matrix of the known face cluster 35 into consideration. The effects of the moving image display device and the display method according to the present invention are not limited by such a distance calculation method.

Now, returning to FIG. 4, the description of the processing procedure will be continued. If it is determined in step ST2 that a face exists in a frame within a shot, it is determined in the next step.
In ST3, it is determined whether or not the face is included in the frame in the shot using the above-described method or the like.
If it is determined in step ST3 that the face is included in the frame, it is determined in step ST4 whether the face is a face that has already appeared in the video of interest.

In step ST4, if this face is the first appearance, in step ST5, a plurality of area images of the face in the shot are picked up (or all of the images in the shot), and the cluster of the face is picked up. Create Thus, as in step ST6, the number of appearances is added for all the faces in the shot.

When this series of processing is completed for all shots, the number of appearances has been counted for each face. That is, in step ST7, it is determined whether or not a person's face is included in other shots. If not, it is determined in step ST8 whether or not the processing for the last shot has been completed. If it is determined that the processing has been performed, the processing routine ends. If it is determined in step ST7 that another face is included, the process returns to step ST4 and returns to step ST4.
The processing operations up to 6 are repeated. If it is determined in step ST8 that the processing for the last shot has not been completed, the process returns to step ST3 and repeats the operations up to step ST7.

Next, a moving picture diagram display method according to a fourth embodiment of the present invention will be described with reference to FIG. The display method according to the fourth embodiment corresponds to the moving image display device according to the first embodiment shown in FIG. FIG.
In step ST10, a still image such as a frame, which constitutes the moving image, is extracted from the input moving image. A moving image can be considered as a sequence of these extracted still images in a predetermined time unit.

In step ST11, a continuous still image is divided into predetermined units. For how to do this,
A scene in which a certain same scene continues may be classified as one shot, or a certain time, for example, a still image every 5 seconds or a still image every 10 seconds may be randomly set as one predetermined unit. Next, in step ST12, it is determined whether or not the face of the person is included in the divided still images of the predetermined unit. If it is determined that a person's face is not included, the process proceeds to the next shot.

In step ST12, when it is determined that the image of the predetermined unit includes the face of the person,
In step ST13, it is determined whether or not the face is a person's face that has appeared in a previously processed shot. If this is your first face that has not appeared in the previous shot,
In step ST15, the face is set as one of the classification items.
It is classified as having appeared once, and the process for the next shot is performed via step ST14. If it is determined in step ST13 that the face is already classified and registered by the previous process, in step ST16, the number of appearances of the classified and registered face is counted up once and the next The process proceeds to step ST17.

In step ST17, it is determined whether or not the shot is the last predetermined unit, that is, a shot. If it is determined that the shot is the last shot, in step ST18, a moving image having the predetermined length is included in the moving image. A list of appearing characters is created along with the number of times, and displayed on the display screen 17, for example. If it is determined in step ST17 that the shot is not the last shot, the process returns to step ST14, and the subsequent processing operations are repeated.

In the above description, clusters are created only when the face of a person appears for the first time. However, the present invention is not limited to this, and the face of a person may appear after the second appearance. If so, recreate the cluster,
The number of appearances may be counted up and displayed in a constantly updated new cluster. By doing so, the face displayed on the display screen 17 is the face that appears last and has a unique effect that it is convenient when performing a reverse search.

[0034]

As described above, according to the video display apparatus of the present invention, the user can select only a desired video, especially regarding characters, while grasping the entire structure of a moving image of a predetermined length. This makes it possible to significantly reduce the time required for video search.

Also, even for a large amount of moving image data, a scene in which a desired person appears can be viewed one by one without repeatedly viewing the same image without wasting time for searching. There is also an effect that search time can be saved and a duplicated viewing time can be saved and a quick response can be made to an evolving society.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a basic configuration of a moving image display device according to a first embodiment of the present invention.

FIG. 2 is a block diagram showing a detailed configuration of a moving image display device according to a second embodiment of the present invention.

FIG. 3 is a conceptual diagram illustrating a display example on a display screen of a moving image display device according to a second embodiment of the present invention.

FIG. 4 is a flowchart illustrating a processing procedure in a moving image display method according to a third embodiment of the present invention.

FIG. 5 is a conceptual diagram for describing a processing procedure in a moving image display method according to a third embodiment of the present invention.

FIG. 6 is a flowchart illustrating a processing procedure of a moving image display method according to a fourth embodiment of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Still image extraction means 2 Image continuation division means 3 Face image judgment means 4 Face transfer judgment means 5 Face classification counting means 6 Character list creation means 7 Display means 11 Scene division unit 12 Representative frame selection unit 13 Face search unit 14 Face judgment Unit 15 face cluster creation unit 16 cluster classification recording unit 17 display screen 21 appearance order of characters 22 characters 23 representative frames of appearance shots of characters 34 unique face space 35 cluster

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Susumu Kubota 1 Toshiba R & D Center, Komukai Toshiba-cho, Saiwai-ku, Kawasaki-shi, Kanagawa

Claims (7)

[Claims] 1. A still image extracting means for extracting an input moving image as a plurality of still images, and an image dividing means for dividing the extracted still images into shots each consisting of one or an arbitrary number of images. A predetermined video determining unit that determines whether or not a predetermined video is shot for each shot divided by the image partitioning unit; and when it is determined that the predetermined video exists in each of the shots, A predetermined-video-presence determining means for determining whether or not the predetermined video has already been captured in any of the moving images; and classifying the predetermined video into different videos, and A predetermined video classification and counting means for counting the number of appearances of the predetermined video; and a video appearance list of the moving image based on the classified predetermined video and the counted appearance of each predetermined video. Moving image display device characterized by comprising a video appeared list creating means for creating. 2. A character list creating means as the video appearance list creating means, when displaying a character list, sorts and outputs the order in the number of appearances, and outputs the sorted and output characters. The moving image display device according to claim 1, further comprising a display unit that displays a person list. 3. A representative frame selecting section for selecting one still image from a predetermined unit of video scene divided by scene division using a scene dividing section as the screen dividing means, and A display unit that, when displaying the character list as the character list, associates a representative frame of a video scene in which the character has appeared with the display unit and simultaneously displays the representative frame on the list display screen. The moving image display device according to the above. 4. A user operates and instructs the representative frame on the list display screen displayed on the display unit, thereby reproducing a moving image directly from a scene of a predetermined unit represented by the representative frame. The moving image display device according to claim 3, further comprising an input operation unit capable of performing the operation. 5. A step of extracting a moving image input with a predetermined time length as a continuous still image, and dividing the continuous still images into predetermined units on a time axis; Determining whether or not the face of the person is included in the obtained predetermined unit of still image. If it is determined that the face is included in the predetermined unit of still image, the face 1 when things are
Classifying as a single item, adding one to the number of appearances of the classified face when the face included in the predetermined unit of still image is already classified and registered, Performing a face extraction operation on a group of still images in a predetermined unit to create a list of characters in the moving image, and a method for displaying a moving image. 6. In the step of dividing the continuous still images into predetermined units on a time axis, a representative image of a scene represented by the predetermined unit is extracted as a representative frame.
The method of claim 5, further comprising the step of creating.
3. The moving image display method according to 1. 7. The method according to claim 1, wherein before the step of creating the characters,
The moving image display method according to claim 5, further comprising a step of determining whether or not processing has been performed for all predetermined units.