JP2003018540A - Image summarizing method and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image summarizing method which summarizes effectively image information comprising a plurality of image segments constituted by adding contents describing information for expressing image contents to every partial image, and a control program of the method. SOLUTION: About one combination of the contents describing information added en bloc to the image segment from younger one, the degree of similarity is calculated for each item, thereby setting the average value as the degree of similarity. After the calculation, combination whose degree of similarity is the highest is searched (S100). The contents describing information is merged simply by setting the younger segment to be precedent. Contents describing information after being merged is added to the partial image of an image segment of identity character number high order of the contents describing information. Low order segments are deleted, and this process is performed until a desired number of image segments is obtained (S104). A front frame of each summarized image segment and its contents describing information are displayed in a chart format on a monitor, or print output is performed, and the process is finished.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video summarization method and a control program therefor capable of grasping the outline of a video without playing or watching the video which generally requires a lot of time.

[0002]

2. Description of the Related Art In order to grasp the contents of video information, it is necessary to reproduce and watch the video. However, this work generally requires about the same time as the video time, which is extremely troublesome when it is desired to simply grasp the outline of the video information. Focusing on such a problem, the inventions disclosed in Japanese Patent Application Laid-Open No. 2000-308008 and Japanese Patent Application Laid-Open No. 10-112835 are examples of measures for improving work efficiency.

The method of determining the segment importance of a video and the method of packing a frame set into a limited area, which are disclosed in Japanese Patent Laid-Open No. 2000-308008, determine the importance of each shot of the video and determine the importance. Based on this, it is intended to generate a video summary, resize the representative frame based on its importance, and pack it into a limited area.

As a solution, the measured value of importance is
Calculated on the segmented part of the video.
For the measure of importance, select the most important segment,
It can be used to generate a representative frame for the selected segment. A thresholding process is applied to the importance score to provide a predetermined number of shots or segments to be represented by the frame, or an appropriate number generated during the run. The representative frames are then packed into the video summary. The size of the frames to be packed is predetermined by their measure of importance and is adjusted according to the available space.

A video summarization method and a video display method disclosed in Japanese Patent Laid-Open No. 10-112835 efficiently display a video summarization device and summary information corresponding to a variety of video contents and a variety of user preferences. It is an object of the present invention to provide a video display device for doing so.

As a means for solving the problem, the video summarization system of the present invention comprises a plurality of video summarization means for extracting video summarization information from a plurality of scenes formed by dividing a captured video based on a predetermined standard. Video summarizing device including
The video display device capable of selecting the summary information is provided. With this configuration, compared to a conventional video summarizing device that uniformly selects a representative image based on only a certain video summarization standard and a conventional video display device that selects a representative image regardless of the video length, It is said that it can cope with the variety and the variety of user's preference.

At the present time, the International Organization for Standardization and the International Electrotechnical Commission are working on the standardization of "multimedia content description interface", that is, MPEG-7, as a method for searching digital contents by characteristics. Here, an example of a segment and content description information for a video is shown in FIG. As shown in FIG. 5, it is assumed that there is an image of “report of conference presentation”. This "report of the conference presentation" is mainly about the presentation report at the conference, that is, "practice before departure", "distance to the conference site", "inside the venue" and "presentation". , First, it consists of a large section, and further, for example, in the case of "the way to the meeting place", the scene is "outside the city",
It has a tree structure such as "waiting for a signal", "station square", etc., and is generally used as video information. Note that the shots, scenes, and partial videos that have a semantic unity that form a tree structure in this manner are hereinafter referred to as "video segments."

[0008] The content description information is text information that describes the content of the video. For example, in a simple case, the characters, the shooting place name, the date and time, the outline, etc. are described. If you want to include more information,
It is also possible to describe information such as the background related to the scene and the subjective impression of the content description person, and as shown in FIG. 5, content description information can be added to each scene. This content description information, as described above,
Currently, standardization is being promoted in MPEG-7 and the like. Generally, the content description information is used as an index for searching a segment from a video database having a huge amount of video data.

[0009]

On the other hand, in the prior art cited as the conventional technology, both of them use the color information of the image or the like to automatically divide the video into shots in order to make the above work efficient.・ Reduce the number of shots by selecting a representative frame and merging or deleting similar shots based on the similarity between the representative frames of each shot, and explain the representative frame and shots of each shot like a paper medium. By mapping on a two-dimensional display medium, we are efficiently grasping video information. These are called video information summaries.

However, these methods depend on the characteristics of the image, and when the shots are completely different in image (visual), one shot is merged even if the semantic contents of the shots are similar. -It will not be deleted.
That is, although the video summary generated from these methods is a summary of the images in the video, it cannot be said to be a summary of the content of the video.

In order to solve the above problem, the present invention improves the efficiency of grasping the outline of the video information by utilizing the content description information added to each video segment and summarizing the contents of the video. A video summarization method and a control program therefor are provided.

[0012]

As described in the prior art cited as the prior art, the video summarization is to repeat segment merging. Therefore, the inventor of the present invention has found that video summary information that can efficiently grasp the content of the video is generated by using the text similarity and the segment length of the content description information instead of the image information. , Came to invent.

That is, in order to achieve the above object, the invention of claim 1 is such that content description information consisting of character information representing the video content of each partial video such as scene or cut is added to the partial video. A video summarization method for summarizing video information in which a plurality of video segments are arranged in a tree shape, the calculation step of calculating a similarity between the content description information, and the calculation step Of the plurality of video segments having a high degree of similarity, the partial video that constitutes one representative video segment is left and deleted, and the content description information is added to the partial video to obtain a plurality of the videos having a high degree of similarity. A first merging step of merging segments with each other, and selecting partial images to such an extent that the contents of the video information can be grasped in the calculation step and the first merging step. In the video summary and wherein the creating the content video summary description information is added.

According to a second aspect of the present invention, in the first aspect, the content description information is composed of a plurality of items, and the calculation step determines the degree of similarity only for a part of designated items of the content description information. It is a video summarization method characterized by calculation.

According to a third aspect of the present invention, in the first aspect, the content description information is composed of a plurality of items, and each item is given a weight that is a degree of executing a video summary, and the calculation step is performed by The similarity is calculated only for the weighted items, and the first merging step determines the priority order of merging according to the degree of the weight, and the priority order is determined until the desired number of video segments is reached. A video summarization method characterized by merging video segments based on each other.

According to a fourth aspect of the present invention, in any one of the first to third aspects, in the first merging step, in the case where there are a plurality of sets of the video segments having a high degree of similarity, the partial video of each set is divided. The video summarization method is characterized in that the group having the smallest total time length is preferentially merged.

According to a fifth aspect of the present invention, in any one of the first to third aspects, in the first merging step, when there are a plurality of sets of video segments having a high degree of similarity, the layer depth of the tree structure is A video summarization method is characterized in that a combination of deep video segments is preferentially merged.

According to a sixth aspect of the present invention, in any one of the first to fifth aspects, the first merging step merges the content description information of a representative video segment among the plurality of merging target video segments. The video summarization method is characterized in that the content description information of the video segment is used.

According to a seventh aspect of the present invention, in the sixth aspect, the representative video segment is a higher-order video segment having an inclusive relationship among a plurality of high-similarity video segments calculated in the calculation step. There is a video summarization method characterized in that there is.

According to an eighth aspect of the present invention, in any one of the first to fifth aspects, the first merging step merges respective content description information of a plurality of video segments with high similarity calculated in the calculation step. Then, the merged content description information is used as the content description information of the merged video segment.

In a ninth aspect of the present invention, in any one of the first to eighth aspects, a set of video segments having a shortest sum of video times different for each video segment is searched for and merged, and the video time is set in the set. A second merging step of using the content description information added to the longer video segment as the content description information of the merged video segment, and appropriately combining together with the first merging step so as to obtain an optimum video summary. Alternatively, the video summarization method is characterized by being selectively configured.

According to a tenth aspect of the present invention, in any one of the first to ninth aspects, a weight, which is a degree to execute the video summarization, is given to each layer layer of the tree-shaped video information, and the weight is determined according to the weight. A video summarization method is characterized in that a change step for changing the priority of video segments to be merged is added.

The invention of claim 11 is the method according to any one of claims 1 to 10, wherein a step of merging the video segments according to the image features is added, and the merging of the video segments according to the content description information and the video segment according to the image features are performed. A video summarization method is characterized in that a video summarization is created by appropriately combining or selectively configuring so as to obtain an optimum video summarization.

According to the twelfth aspect of the present invention, in any one of the first to eleventh aspects, a video summary is created and output by the still frame of each video segment forming the summarized video information and the content description information for the still frame. In the video summarization method, a summarization output step is added.

According to the thirteenth aspect of the invention, for each partial video such as a scene or a cut, content description information consisting of character information expressing the video content is added to the partial video to form a video segment, and the video segment is A control program in video summarization for summarizing video information formed by connecting a plurality of tree-like ones, a calculation means for calculating the similarity between the content description information, and a plurality of high similarity calculated by the calculation means. A first merging means for merging the image segments with each other, and causing a computer to function as the calculating means and the first merging means, and selecting partial images to the extent that the contents of the video information can be grasped. The control program in the video summary is characterized by creating a video summary to which content description information is added.

According to a fourteenth aspect of the present invention, in the thirteenth aspect, the content description information is composed of a plurality of items,
In the control program in the video summary, the calculating means calculates the degree of similarity only with respect to some specified items of the content description information.

According to a fifteenth aspect of the present invention, in the thirteenth aspect, the content description information is composed of a plurality of items,
Each item is given a weight that is a degree to execute the video summarization, the calculating means calculates the similarity only to the weighted items, and the first merging means determines the degree of the weight. The control program in the video summary is characterized in that the priority order for merging is determined, and the video segments are merged based on the priority order until the desired number of video segments is reached.

The invention of claim 16 is based on claims 13 to 15.
In any one of the above, in the case where there are a plurality of sets of video segments with high similarity,
The control program in video summarization is characterized in that a group having the smallest total time length of partial videos in each group is preferentially merged.

The invention of claim 17 is based on claims 13 to 15.
In any one of the above, in the case where there are a plurality of sets of video segments with high similarity,
A control program in video summarization characterized by preferentially merging a set of video segments having a deep tree structure layer depth.

The invention of claim 18 is based on claims 13 to 17.
In any one of the above, the first merging means uses the content description information of a representative video segment among the plurality of video segments to be merged as the content description information of the merged video segment. It is in the control program in the summary.

According to a nineteenth aspect of the present invention, in the eighteenth aspect, the representative video segment is a plurality of video segments having a high degree of similarity calculated by the calculation means.
A control program in video summarization characterized by being a higher-order video segment having an inclusive relationship.

The invention of claim 20 is based on claims 13 to 17.
In any one of the above, the first merging unit merges the content description information of each of the plurality of video segments with high similarity calculated by the calculation unit, and the merged content description information is merged into the video segment. In the control program in the video summary, which is characterized by the content description information.

The invention of claim 21 is from claim 13 to claim 20.
In any of the above, the set of video segments having the shortest sum of the video times different for each video segment is searched and merged, and the content description information added to the video segment with the longer video time in the set is merged. Second merging means for making the content description information of the generated video segment added, and causing the computer to function as the second merging means, and appropriately combining or selectively with the first merging means so as to obtain an optimum video summary. It is in a control program in video summarization characterized by being configured.

The invention of claim 22 is based on claims 13 to 21.
In any one of the above, adding a changing means for giving a weight that is a degree of executing video summarization to each layer layer of the tree-shaped video information, and changing the priority of the video segment to be merged according to the weight. The control program in the video summary is characterized by causing a computer to function as the changing means.

The invention of claim 23 is from claim 13 to claim 22.
In any one of the above, a means for merging video segments according to image features is added, causing a computer to function as means for merging video segments according to image features, and merging video segments according to the content description information and merging video segments according to the image features. A control program for video summarization, characterized in that the video summarization is created by appropriately combining or selectively constructing an optimum video summarization.

The invention of claim 24 is based on claims 13 to 23.
In any one of the above, a summarization output means for creating and outputting a video summarization by the still frame of each video segment constituting the summarized video information and the content description information for the still frame is added, and the summary output means for the computer. It is in the control program for video summarization that is characterized by functioning as.

[0037]

DETAILED DESCRIPTION OF THE INVENTION Referring to the accompanying drawings,
Embodiments of the present invention will be described in detail. First, before explaining the control program in the video summary in the present embodiment, the video information to which the content description information is added will be described with reference to FIG.
Will be explained. This video information exemplifies the video of the "report of academic conference presentation" as described in the section of the related art in FIG. In other words, this "report of academic conference presentation" is mainly before and after the presentation report at the academic conference, that is,
"Practice before departure", "Journey to the meeting place", "Inside the meeting place", and "Presentation", etc. First, there are big sections. For example, "Journey to the meeting place" The scene is "outside the city", "waiting for the signal",
It is composed of partial images with a tree structure (hierarchical structure) such as "Station Square".

Each layer having different video time
Each partial video is Segmen so that each scene can be identified.
tID is assigned to this SegmentID, Who, When,
Where, WhatAction, WhatObject, Why, FreeText 7
Content description information (text) consisting of one item is added to each to form a video segment, which facilitates retrieval, management, and editing. It should be noted that this content description information is not limited to the seven items shown in the drawing, and is not limited to such an item division mode, and any other format may be used as long as it is associated with a video segment.

Here, in order to simplify the explanation, a part of the image in FIG. 5, that is, the path Segm to the conference site is shown.
Of the lower layers of entID 1.2, SegmentID 1.2.3
Then, the description will proceed by focusing on SegmentID 1.2.4 and the content description information corresponding to each video segment of SegmentID 1.2.5. The contents description information is as follows.

(1) Information describing the contents of SegmentID 1.2.3 Who = "Nozomi Takahashi" When = "October 16, 1999" Where = "Capitole city, Toulouse, France" WhatAction = "Walk while explaining to the station square "WhatObject =" People passing each other "Why =" Heading to the IDMS'99 presentation site "FreeText =" A beautiful city where various people come and go. Stop at a red light.

(2) Information describing the contents of SegmentID 1.2.4 Who = "Nozomu Takahashi" When = "October 16, 1999" Where = "Capitole city, Toulouse, France" WhatAction = "Waiting for signal" WhatObject = "Signal signal""Why="I'm heading to the IDMS'99 presentation site "FreeText =" If you think something like an alarm is ringing, it looks like a signal. It's blue. "

(3) Information describing the contents of SegmentID 1.2.5 Who = "Nozomi Takahashi" When = "October 16, 1999" Where = "Metro Capitole Station Station Square, Toulous
e, France "WhatAction =" Down the stairs to the station yard "WhatObject =" Metro entrance and signboard "Why =" Towards the IDMS'99 presentation site "FreeText ="It's beautiful like a park. Metro sign. "

As shown in FIG. 1, the control program for video summarization in the present embodiment is a content description information parsing means 1, an image information acquisition means 2, and a video summarization output means 3.
And is configured. Content description information parsing means 1
Is for merging the video information and the content description information. The first merging means 11 for merging each video segment based on the content description information and each video segment based on the time length of each video segment. Second merging means 12
And segment merging means 13 by layers.

The first merging means 11 is provided with a calculating means for calculating the degree of similarity of the content description information added to each video segment, and gradually calculates the content description information of a plurality of video segments forming the video information. The similarity is calculated, and video segments with high similarity are extracted and merged. The merging operation of the first merging means 11 will be described. First, the similarity is calculated for all combinations of the content description information added to the video segments of all leaf nodes in the tree.

As a simple example of the similarity calculation method, there is a method of calculating the similarity for each item (Who, Where, ...) With the following formula and using the average as the similarity. In this case, the value range is 0.0 to 1.0, and the higher the value, the higher the similarity.

[0046]

[Equation 1]

In the present invention, the calculation method is not particularly limited as long as the similarity between texts can be calculated. However, a text comparison engine that can absorb the fluctuation of words (for example, "Good morning" and "Good morning") is more preferable. Also,
In the example of the above calculation method, by weighting each item instead of the average, it is possible to calculate the similarity by paying attention to any item (a plurality of items may be included) of the content description information.

[0048]

[Equation 2]

Here, the weight (n) is a priority degree when the similarity judgment is set for each of the seven items, and the total sum is 1. For example, the degree for Where and WhatAction items is set to "0.3" respectively, and WhatObject and Wh
The degree for each item of y is set to "0.2", and the remaining
The degree to the items of Who, When and FreeText respectively ”
When it is set to 0 ”, first the similarity is calculated for the items of Where and WhatAction, and then the similarity is calculated for the items of WhatObject and Why. At this time, the remaining Who and
Even if the When and FreeText items are the same or very similar, the similarity is not calculated. In this way, the similarity may be calculated only for a specific item, and the similarity may be determined for each item instead of the average value. In that case, the degree different for each item represents the priority at the time of determination. It will be. It is also possible to simply calculate the degree of similarity only for a specific item without applying such a weight and use the average value as the degree of similarity.

In this way, the similarity is gradually calculated for all combinations of the content description information added to the video segments of all leaf nodes in the tree. And
Among the similarities of all the combinations of the content description information, the set of the video segments having the highest similarity is determined as the merge target of the video information and the content description information and merged. Then, the similarity is gradually calculated and merged for all combinations of the merged video segment and other video segments. This series of operations is repeated until the number of video segments designated in advance is reached.

After the similarity of the content description information added to each video segment is calculated by the calculating means in this way, the sets having a high similarity are merged. The merging of the content description information in these two video segments is performed, for example, in the following 3
There are several types. Consider the content description information in the two video segments to be merged as C and D, and the content description information in the merged video segment as E.

When there is an inclusive relation between the content description information of C and D, only the content description information of the video segment on the inclusion side is the content description information of E. That is, it is as follows. When C text ⊃ D text holds, E content description information = C content description information C text ⊂ D text holds, E content description information = D content description information C text = D When the text is valid, the content description information of E may be either C or D.

When new content description information is determined in this way, among the video segments to be merged,
New content description information is added to the partial video in the video segment with the higher inclusion relation, and the video segment with the lower inclusion relation is truncated. By doing so, a plurality of video segments having a high degree of similarity are put together. Note that this merging is performed for all combinations of all video segments regardless of each layer layer, limited combination for each layer layer, or from the lowest layer layer to the upper layer layer. The form of merging video segments is not particularly limited, such as gradual grading toward each other or vice versa, or merging with a "weight" that is the priority of merging for each layer layer.

As another method of determining the above-mentioned new content description information, the content description information of C and D is merged,
The result may be used as the content description information of E. The method of merging is to simply connect words for each item and omit the overlapping parts.For the FreeText part, the summarization method from multiple similar documents, which is the existing technology (Inagaki Hiroto et al. Proposal of Transfer-type Electronic Document Summarization Method by Integration of Information, Proceedings of 56th National Convention of Information Processing Society of Japan,
Separate volume 2, pp.255-256, 1998) may be used. If you simply connect words (sentences), SegmentI
It is preferable that the younger one of D comes first because it often becomes a reasonable word (sentence). As the former example, the result of merging the content description information of SegmentID 1.2.3 and SegmentID 1.2.4 is shown below.

(1) SegmentID 1.2.3 and SegmentID 1.
Description information after merging 2.4 2.4 Who = "Nozomi Takahashi" When = "October 16, 1999" Where = "Capitole city, Toulouse, France" WhatAction = "Walk while explaining to the station square" WhatObject = "Passing traffic lights" Why = "Towards the presentation site of IDMS'99" FreeText = "A beautiful city where various people come and go.
Stop at the red light. If you think something like an alarm is ringing, it looks like a signal. Over the blue hour. "

In this way, SegmentID 1.2.3 and SegmentID
Comparing the pair with 1.2.4, the same number of "signals" were found in the FreeText part, and the Who part, When part, Where part and Why part were
The sections and the sections are completely matched to be merged, and the younger one of the SegmentID is merged at the beginning, and the merged content description information is added to the partial video of the younger SegmentID. Then delete SegmentID 1.2.4.

When there are a plurality of groups having a high degree of similarity, the merge process is performed on the group having the smallest total time length of the two video segments. In this case, the partial videos having a short time length may be discarded, or the partial videos may be simply connected. In addition, when there are a plurality of groups having a high degree of similarity, it is preferable to preferentially perform the merging process for the group of video segments having a deep tree structure layer depth.

Next, the second merging means 12 based on the segment time length will be described. The second merging means 12 simply finds out a group having the shortest sum of the video time of each video segment without judging the similarity of the content description information in each video segment, and merges the group. The merging of the content description information in each of the two video segments is as follows.

The time lengths of C and D are compared, and only the content description information of the longer video segment is used as the content description information of E. That is, it is as follows. When C time length> D time length holds, E content description information = C content description information C time length <D time length holds, E content description information = D content description information C When the time length = D is satisfied, the content description information of E may be either C or D.

When new content description information is determined in this way, of the video segments to be merged,
Truncate the video segment with the shorter video time,
Alternatively, the partial videos are connected to each other and new content descriptive information is added to combine them.

Segment segment merging means 13
Is a means for changing the priority of video segments to be merged according to the weight, which is a degree of executing the video summarization for each layer of the tree-shaped video information. This is added to each of the merging means and the second merging means.

For example, in FIG. 5, the layer layer of the image information "Presentation by the academic society" is "3", the weight of the middle layer layer is "0.6", and the weight of the bottom layer layer is "
In the case of 0, 4 ", first, the first merging unit or the second merging unit described above is used to search for any video segment that can be merged in the middle layer layer. Then, the first merging unit or the second merging unit is used to search for a video segment that can be merged by moving to the lowermost layer layer.
Then, this "weight" is set as the priority when merging,
The merging process is performed on the middle layer layers, and then on the lowest layer layers until the desired number of video segments is reached. In this case, if the desired number of video segments is reached only by the merging by the middle layer layers, the processing is ended there.

Separately from this, when the "weight" is taken as a simple ratio, the number of video segments to be reduced when the original video information composed of a plurality of video segments is reconstructed with a desired number of video segments. Is replaced with the ratio of each layer, and the first merging unit 11 or the second merging unit 1 is used.
The search and merge may be performed using 2.

The image information acquisition means 2 comprises a segment merging means 21 based on image information, and this technical means is a known technique utilizing the image feature of each partial video, and the details thereof are not the gist of the present application, so that the description thereof will be omitted. To do.

The video summary output means 3 includes the first merging means 11, the second merging means 12, the layer merging means 13 and the image information segment merging means 2 described above.
One of the two is selected, the original video information composed of a plurality of video segments is merged until a desired number of video segments is obtained, and the reconstructed video information is output. As its mode, simply, the summarized video information is displayed on the display as a "moving image", or the front frame of each summarized video segment is extracted, and the front frame and its content description information are listed. It can be displayed on the display or printed out.

The output example before the summary output in the latter list format is shown in FIG. 2 (SegmentID 1.2.3 to 1.2.5 is illustrated) and the output example after the summary is illustrated in FIG. 3 (SegmentID 1.2.3 to 1.2.6 is illustrated. ). In this drawing, a control program for video summarization according to the present embodiment is installed in a computer in which an OS that realizes a GUI environment is installed, and is operated and output. The summarizing means used in this output example uses the first merging means 11 that merges the content description information with each other and uses the result as new content description information.

As shown in FIG. 3, Segment ID 1.2.3 and Segmen
Comparing the pair with tID1.2.4, the “Signal” matches in the FreeText part, and the Who part, When part, Where part, and Why part completely match each other, and are to be merged. Then, as shown in FIG. 4, the content description information is simply merged with the younger one of the video segment IDs at the beginning, and the younger one of the video segment IDs is left, and the video segment of SegmentID 1.2.4 is deleted and displayed on the display. Display in list format. In this example, according to the present invention, the viewer can reduce the download amount by three images and can view four video segments from one screen that could only view three video segments. There is. Note that this summarized list can be printed.

The functions of the respective means of the video summarization control program according to the present embodiment have been described above, but as shown in FIG. 1, the respective merging means are appropriately switched to output the video summarization or combined to perform the image summarization. . Here, an example of a series of procedures for specifying the function of each means and performing video summarization will be described with reference to the flowchart of FIG.

First, the number of video segments after summarization is determined. If the video information composed of 300 video segments is to be 100 video segments, 200 video segments will be deleted. Once the number of video segments after summarization is determined, the similarity is calculated for each item (Who, Where, ...) for one set of content description information added to the video segments in a brute force manner from the youngest segment ID. Then, the average value is used as the similarity. When the calculation is completed, the set with the highest similarity is searched (step S100).

At this time, when there are a plurality of groups having a high degree of similarity, the group of video segments having a deep layer depth of the tree structure is given a higher priority (merge order) for merging (step). S101). In addition, a group having the smallest total time length of two video segments is also prioritized (step S102). In addition, the priority balance of both is
It is assigned by a setting value that takes into consideration the total time length and the layer depth.

When the merging target is determined in this way, the video segments of the highest priority order (merging order) are merged. In this case, the content description information is simply merged with the younger segment ID first, and the merged content description information is added to the partial video of the video segment whose matching character count of the content description information is higher, The video segment of that is deleted (step S103).

Then, it is judged whether or not the desired number of video segments after summarization has been reached (step S104). If it is determined that the number of times has not been reached, the process returns to step S100 to perform the merge process again until the merge process is performed (step S10).
4: NO). When the desired number of video segments after summarization is reached (step S104: YES), image features of each partial video are captured and image segments are merged (step S105) to obtain summarized image information. The image information obtained in this way is processed by extracting the front frame of each video segment summarized and displaying the front frame and its content description information in a list format on the display or by printing out. finish.

It is also possible to add a limit by giving a threshold to the degree of similarity set by the user. In this case, the process is repeated until the similarity of all the content description information sets falls below the threshold value.

On the contrary, it is also possible to reduce the number of similar video segments by first using the summarizing method based on the image features and then using the present method, or by alternately using the methods. This is not limited to the summarization using the content description information and the image feature, which is the present method,
It is possible to combine other methods, for example, a video segment time length and a summary using audio information. Of course, only the summarizing means using the content description information of this method may be used.

[0075]

As described above, the present invention has the following advantageous effects. According to the present invention, the similarity described in the content description information added to each video segment is calculated, and the video segments are merged based on the similarity. You can certainly summarize. Therefore, when grasping the outline of the video information, the contents of the video can be grasped in a short time. For example, the video summary is made by the still frame of each video segment forming the summarized video information and the content description information for the still frame. When the list is output, it is not necessary to reproduce and watch the video, and the time cost for this work can be reduced. In addition, when an overview of a video is released over a network such as the Internet, it is not necessary to download the representative frames and content description information of all video segments, and time costs and network bandwidth usage costs can be reduced. A very suitable image summarization method and control program can be provided.

[Brief description of drawings]

FIG. 1 is an explanatory diagram showing a configuration of a control program in image summarization according to the present invention.

FIG. 2 is an explanatory diagram showing an output example before a summary output in a list format.

FIG. 3 is an explanatory diagram showing an output example after summary output in a list format.

FIG. 4 is a flowchart showing an example of a series of procedures for performing video summarization.

FIG. 5 is a conceptual diagram showing an example of video information having a tree structure.

[Explanation of symbols]

1 Content description information parsing means 11 First merging means 12 Second merging means Segment merging means by 13 layers 2 Image information acquisition means 3 Video summary output means

Front page continuation (51) Int.Cl. ⁷ Identification code FI theme code (reference) // H04N 7/24 H04N 7/13 Z

Claims (24)

[Claims] 1. Content description information consisting of character information expressing the video content of each partial video such as a scene or a cut is added to the partial video to form a video segment, and the video segments are arranged in a tree form. A video summarizing method for summarizing video information composed of: a calculating step of calculating a similarity between the content description information, and a representative of a plurality of video segments having a high similarity calculated in the calculating step. A first merging step of merging a plurality of video segments having a high degree of similarity with each other, while deleting the partial video constituting one video segment while leaving the partial video, and adding the content description information to the partial video. In the calculation step and the first merging step, the video summaries to which the content description information is added are created by selecting partial videos to the extent that the content of the video information can be grasped. Characteristic video summarization method. 2. The content description information is made up of a plurality of items, and the calculation step calculates the similarity only for a part of designated items of the content description information. Video summary method described. 3. The content description information is made up of a plurality of items, each item is given a weight which is a degree of executing a video summary, and the calculation step only gives the similarity to the weighted items. And the first merging step determines the priority order of merging according to the degree of the weight, and merges the video segments based on the priority order until the desired number of video segments is reached. The video summarization method according to claim 1, characterized in that 4. In the first merging step, when there are a plurality of sets of video segments having a high degree of similarity, the group having the smallest total time length of partial videos in each set is preferentially merged. The video summarizing method according to claim 1, wherein 5. The first merging step prioritizes merging of image segment sets having a deep tree structure layer depth when there are a plurality of image segment sets with high similarity. The video summarization method according to any one of claims 1 to 3. 6. The first merging step uses the content description information of a representative video segment among the plurality of merging target video segments as the content description information of the merged video segment. Any one of items 1 to 5
Video summarization method described in section. 7. The representative video segment is a higher-order video segment having an inclusive relationship among a plurality of video segments having a high degree of similarity calculated in the calculation step. Video summarization method. 8. The video segment in which the first merging step merges the content description information of each of the plurality of video segments having a high degree of similarity calculated in the calculation step, and the merged content description information is merged. 6. The video summarizing method according to any one of claims 1 to 5, wherein the content descriptive information is the content description information. 9. A set of video segments having a shortest sum of video times different for each video segment is searched and merged, and the content description information added to the video segment having the longer video time in the set is merged. 7. A second merging step for making the content description information of the video segment created is added and appropriately combined or selectively configured so as to obtain an optimum video summary together with the first merging step. 9. The video summarizing method according to any one of 1 to 8. 10. A changing step of changing a priority of a video segment to be merged according to a weight, which is a degree of executing video summarization, for each layer layer of the tree-shaped video information is added. Claim 1 characterized by the above.
10. The video summarizing method according to any one of 1 to 9. 11. A step of merging video segments according to image features is added, and the merging of the video segments according to the content description information and the merging of the video segments according to the image features are appropriately combined or selected so as to obtain an optimum video summary. The video summarization method according to claim 1, wherein the video summarization is configured to be configured as a physical image. 12. The method according to claim 1, further comprising a summary output step of creating and outputting a video summary from still frames of each video segment forming the summarized video information and content description information for the still frames. 11. The video summarizing method according to any one of 11 above. 13. A video segment is formed by adding content description information consisting of character information expressing the video content of each partial video such as a scene or a cut to the partial video, and a plurality of the video segments are arranged in a tree shape. A control program in a video summary that summarizes video information composed of: a calculation means for calculating the similarity between the content description information, and a plurality of video segments having a high similarity calculated by the calculation means. A first merging unit for merging, and causing the computer to function as the calculating unit and the first merging unit, and selecting the partial video to such an extent that the content of the video information can be grasped, and the content description information is displayed. A control program for video summarization, characterized by creating an added video summarization. 14. The content descriptive information is composed of a plurality of items, and the calculating means calculates the degree of similarity only for a part of designated items of the content descriptive information. The control program in the video summary described. 15. The content description information is made up of a plurality of items, each item is given a weight that is a degree of executing a video summary, and the calculation means gives a similarity only to the items to which the weight is attached. And the first merging means determines the priority order for merging according to the degree of the weight, and merges the video segments based on the priority order until the desired number of video segments is reached. 14. The control program in the video summary according to claim 13. 16. The first merging means, when there are a plurality of sets of the video segments having a high degree of similarity, preferentially performs the merging in which the total time length of the partial videos in each set is the smallest. The control program for video summarization according to any one of claims 13 to 15. 17. The first merging unit preferentially merges a set of video segments having a deep tree structure layer depth when there are a plurality of sets of video segments having a high degree of similarity. The control program for video summarization according to any one of claims 13 to 15. 18. The first merging means uses the content description information of a representative video segment among the plurality of merged video segments as the content description information of the merged video segment. Item 18. A control program for video summarization according to any one of items 13 to 17. 19. The representative video segment is a higher-order video segment having an inclusive relationship among a plurality of video segments having a high degree of similarity calculated by the calculating means. Control program for the video summarization. 20. The first merging means merges respective content description information of a plurality of video segments having a high degree of similarity calculated by the calculating means, and the merged content description information is merged into a video segment. 18. The control program for video summarization according to any one of claims 13 to 17, wherein the control program is content description information. 21. Searching and merging a set of video segments having a shortest sum of different video times for each video segment, and merging the content description information added to the video segment having the longer video time in the set. A second merging unit that is used as the content description information of the created video segment, and causes a computer to function as the second merging unit,
21. The control program for video summarization according to any one of claims 13 to 20, which is appropriately combined or selectively configured with the first merging unit so as to obtain an optimum video summarization. 22. Change means is provided for giving a weight that is a degree of performing video summarization to each layer layer of the tree-shaped video information, and changing the priority of the video segments to be merged according to the weight. 22. The control program for video summarization according to claim 13, wherein the control program causes a computer to function as the changing unit. 23. A means for merging video segments according to image features is added to allow a computer to function as means for merging video segments according to image features, wherein merging of video segments according to the content description information and merging of video segments according to the image features. The video summarization control program according to any one of claims 13 to 22, wherein the video summarization is created by appropriately combining or selectively configuring so as to obtain an optimum video summarization. 24. A summary output means for creating and outputting a video summary from still frames of each video segment forming the summarized video information and content description information for the still frame is added, and the summary output means to a computer. The control program for video summarization according to any one of claims 13 to 23, characterized in that the control program functions as a control program.