KR20040041127A - An intelligent agent system for providing viewer-customized video skims in digital TV broadcasting - Google Patents

Abstract

PURPOSE: A system for providing the televiewer-customized video summary of digital TV broadcasting is provided to generate the video summary of televiewer-customized broadcasting contents in a service provider without the instruction or action of a televiewer in digital broadcasting environment and offer the generated video summary to the televiewer through a televiewer terminal. CONSTITUTION: A televiewer terminal has an interface unit(10), a televiewer preference update unit(20) and a televiewer terminal agent unit(50). The interface unit(10) acquires an action pattern associated with the televiewing of TV, and outputs the acquired pattern. The televiewer preference update unit(20) generates televiewer preference from the action pattern outputted from the interface unit(10). The televiewer terminal agent unit(50) transmits the televiewer preference generated in the televiewer preference update unit(20). A service provider has a semantic feature extracting unit(90), a semantic feature storing unit(80), a video summary generating unit(70) and a service provider agent unit(60). The semantic feature extracting unit(90) extracts video features of contents, and segments the contents into semantic features. The semantic feature storing unit(80) stores the semantic features extracted in the semantic feature extracting unit(90). The video summary generating unit(70) retrieves the semantic feature storing unit(80), and generates a video summary including semantic features suitable for the televiewer preference transmitted from the televiewer terminal agent unit(50). The service provider agent unit(60) transmits the video summary generated in the video summary generating unit(70) to the televiewer terminal agent unit(50).

Description

An intelligent agent system for providing viewer-customized video skims in digital TV broadcasting}

The present invention relates to a digital TV broadcasting system, and more particularly, to a viewer-based video summary providing system that autonomously generates, stores, and recommends a video summary based on individual viewers' preferences.

The digitization of the TV environment currently in progress seeks to provide a two-way broadcast service and an enhanced additional information service for a broadcast service provided unidirectionally in a conventional broadcasting environment. However, broadcast services provided in the digital TV environment so far are limited to unilateral additional information service based on EPG (Electric Program Guide) and broadcast content reservation storage service based on PVR (Personal Video Recorder). This shows a limitation in the digital TV broadcasting environment pursuing interactiveness. In addition, due to such services, viewer instruction required by the viewer is gradually increasing, causing inconvenience to the viewer. In particular, in a digital broadcasting environment having multiple channels, the necessity of summarizing the broadcasting contents is increasing in order for the viewer to select the contents. However, not only does the viewer watch all of the summaries of the broadcast contents of a large number of channels and selects the broadcast contents that he / she wants, but it also wastes time for the user, and the summaries of the broadcast contents that are uniformly made by the service providers' standards are not limited. As a one-way broadcast service provided regardless of interests or preferences for broadcast content, scenes that a viewer does not want to watch may be inserted or scenes that may not be inserted.

Therefore, in order to provide a summary of broadcast contents centering on individual viewers, it is necessary to generate taste or preference information of individual viewers and provide content summary information generated according to the information. In addition, for a viewer-centered video summary broadcasting service, a video summarization system capable of minimizing viewer's instruction or behavior and inferring the viewer's taste is needed. Intelligent agents for summarizing video are required in broadcast systems.

Disclosure of Invention The present invention has been made to solve this problem, and in a digital broadcasting environment, a service provider generates a video summary of broadcast content that is tailored to the viewer without the user's instruction or action, and then provides and recommends to the viewer through the viewer terminal. An object of the present invention is to provide a TV broadcasting system.

1 is a block diagram of a system for providing viewer customized video summaries in a digital broadcasting environment according to the present invention;

2 is a flow diagram of a process of generating and recommending a video summary in accordance with the present invention; and

3 is a flowchart illustrating a process of segmenting broadcast content into video clips (hereinafter, referred to as semantic features) for a specific event or a specific scene in a service provider according to the present invention.

-Explanation of symbols for the main parts of the drawings-

10: viewer interface unit 20: viewer preference update unit

30: viewer preference DB unit 40: video summary DB unit

50: viewer terminal agent 60: service provider agent

70: video summary generator 80: semantic feature DB unit

90: semantic feature extraction unit 100: content DB unit

In order to achieve the above object, the present invention provides a viewer terminal unit having a viewer preference generator for acquiring a viewer's viewing behavior pattern to generate viewer preferences, and a viewer terminal agent for transmitting viewer preferences generated by the viewer preference generator, and contents. A semantic feature extractor for segmenting content into semantic features by extracting the video features of the semantic feature; A video summary generator for generating a video summary including semantic features that best match the preferences, and a broadcast service provider having a service provider agent for transmitting the video summary generated in the video summary generator to the viewer terminal agent; It provides a customized viewer video summary available systems.

The terms used in the video summary providing system of the present invention are described as follows.

'Service provider' refers to a system for providing broadcast content or additional information to a viewer. The 'viewer terminal' refers to a digital TV reception environment having a storage space in which a viewer can watch broadcast content and store desired broadcast content and additional information. 'Agent' is a software component that acts on behalf of viewers based on knowledge-based reasoning, and is an intelligent agent used in artificial intelligence. 'Viewer preference' refers to the viewer's propensity to the viewer's broadcast content predicted according to the behavior pattern and behavior pattern performed by the viewer's terminal, and 'meaning feature' refers to the higher level meanings of the broadcast content, namely human Refers to a specific event and scene video clip carried by broadcast content classified by recognition.

Hereinafter, the video summary providing system of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a system for providing viewer customized video summaries under a digital broadcasting environment according to the present invention.

As shown in FIG. 1, the viewer terminal includes an interface unit 10, a viewer preference updating unit 20, a viewer preference DB unit 30, a video summary DB unit 40, and a viewer terminal agent unit 50. Include. Meanwhile, the service provider includes a service provider agent unit 60, a video summary generator 70, a semantic feature DB unit 80, a semantic feature extractor 90, and a broadcast content DB unit 100.

The interface unit 10 includes a TV viewing screen as a block in which a viewer can watch broadcast content or input viewer intention. In addition, the viewer terminal agent 50 provides the viewer's input or the behavior pattern performed by the viewer's terminal. Here, the behavior pattern means a channel selected by the viewer, a content title, a viewing time, a main scene repeatedly watched, a main character, and the like. It is also responsible for viewer authentication input to the viewer terminal.

The viewer preference updater 20 is a block for predicting and determining viewer preferences for content using statistical means of viewer behavior patterns obtained from the viewer terminal agent unit 50 to generate and update viewer preferences.

The viewer preference DB unit 30 is a block for storing viewer preferences for viewers registered in the viewer terminal. It is generated and updated by the viewer preference updating unit 20.

The video summary DB unit 40 is a block for storing video summaries obtained through negotiation between the viewer terminal agent unit 50 and the service provider agent unit 60.

The viewer terminal agent unit 50 is a core component in which intelligent agents of the viewer terminal exist. The viewer terminal agent unit 50 transmits a viewer behavior pattern obtained from the interface unit 10 to the viewer preference update unit 20, or obtains a viewer-specific video summary. In charge of negotiation with the service provider agent unit 60. In addition, the video summary delivered from the service providers is stored in the video summary DB unit 40. The communication between the viewer terminal agent unit 50 and the service provider agent unit 60 uses an agent communication message, and the connected communication channel plays a role of a return channel in the digital TV environment.

The service provider agent unit 60 is a block in which the intelligent agents of the service provider exist. The service provider agent unit 60 analyzes the input viewer preferences and searches whether there is broadcast content corresponding to the preferences. It is responsible for controlling the video summary generator to generate a video summary for the corresponding broadcast content. It also negotiates the delivery of the video summary with the viewer terminal agent unit 50.

The video summary generator 70 extracts and combines semantic features that match the viewer preferences delivered from the viewer terminal agent unit 50 in the semantic feature DB 80 to generate a new video summary corresponding to the viewer preferences.

The semantic feature DB unit 80 stores semantic features of contents pre-segmented by the semantic feature extractor 90.

The semantic feature extractor 90 segments and extracts a semantic feature of the broadcast content, that is, an event or a scene, using feature values of audio and visual lower levels. In addition, the description content and keyword information of the extracted event or scene are included. For example, the content of the sports genre may include a description of a shot scene, a goal shot, a penalty kick scene, and the like of a soccer game, and a description of a title of each scene, a name of a participating player, and the like.

The broadcast content DB unit 100 stores broadcast contents provided by a service provider.

2 is a flow diagram of a process of generating and recommending a video summary in accordance with the present invention. As shown in FIG. 2, when the viewer terminal agent unit 50 confirms the absence of the viewer or the system logout (step 200), the viewer terminal agent unit 50 registers with the viewer terminal from the viewer preference DB 30. The obtained viewer's preferences are obtained (step 201). Agents exchange queries and responses for the generation and transmission of video summaries via agent communication messages.

The viewer preference P A with respect to the viewer _A is an order of a list of broadcast contents having a high number of viewing times, and is expressed by the following formula (1).

[Equation 1]

P _A = Sort {R _i }

R _i = W ₁ * N _w + W ₂ * N _m

If R _i > = R _th : Include in list to be sorted

R _i <R _th : Excluded from the list to be sorted by viewer preference below the threshold

Here, R _i : Viewing ratio for the broadcast content C _i

Sort {X}: Sort descending order by size of X

W ₁ and W ₂ are weights that satisfy the condition W ₁ > W ₂

N _w : Number of times you watched broadcast content within a week

N _m : Number of times watched broadcasting content within a month

R _th : Threshold for the minimum percentage of viewers of content that can be included in the viewer preference list.

[Equation 1] represents the viewer's preference for analyzing the viewer's propensity to view broadcast content recently by setting the viewer's viewing rate higher than the viewing rate for one month.

The content information M _i stored in the viewer preferences while the viewer watches the broadcast content C _i is composed of information related to the broadcast content included in the following [Formula 2], and the information is provided by the service providers. Information parsed and stored from metadata of contents. The agent infers the viewer's favorite scenes from M _i .

[Equation 2]

M _i = [{T, C, t, A (+), S (+)} (+)]

T: title, C: channel, t: airtime, A: main character, S: main scene, (+) means that it can have more than one value.

The viewer terminal agent unit 50 transmits the preferences to the agent units 60 of the subscribed service providers in order to find broadcast content that matches the viewer preferences. The transmitted channel uses an agent communication channel, and the communication range is limited to the agent units 60 of the service providers subscribed by the viewer for the protection of the viewer information (step 202). Subscribed service providers may provide basic profiles such as the viewer's permission, the viewer's age, gender, and the like. In addition, the viewer terminal agent unit 50 must have permission from the viewers to transmit the registered viewer preferences. Here, when the agent of the service providers not registered in the viewer terminal requests viewer preferences, the acquisition of viewer information may be restricted by excluding the viewer profile and detailed preferences.

The service provider agent unit 60 that obtains the viewer preferences determines whether there is content in the content DB unit 100 that matches the preference information (step 203). Items determined through the preference information are the content genre, title, channel, time, main character, and main scene.

If there is no broadcast content that matches the preference information in the content DB unit 100, the communication connection between the agent units is terminated (step 204). If there is broadcast content that matches the preference information in the content DB unit 100, the service provider agent unit 60 analyzes the viewer preferences for the content in detail (step 205). The preference information analyzed is events or scenes desired by the viewer for the selected content.

The service provider agent unit 60 may generate a video summary generator 70 based on corresponding events or scenes when there are events or scenes for broadcast content required by the preference information analyzed in the semantic feature DB unit 80. Generate a video summary (step 207). Meaning Feature If there are no events or scenes for the broadcast content required by the preference information analyzed in the DB unit 80, select the general event or scenes provided by the service providers, and select the video summary generator 70. Generate a video summary (step 208). The semantic feature extraction method stored in the semantic feature DB unit 80 of the service provider will be described in detail with reference to FIG. 3.

The generated video summary is transmitted to the viewer terminal through agent communication (step 209), and the viewer terminal agent unit 50 stores the video summary DB unit 40 (step 210).

When the viewer logs in to the system, the stored video summary is recommended according to the corresponding viewer (step 211).

3 illustrates a process of extracting semantic features from a service provider. As shown in Fig. 3, service providers extract semantic features of respective contents and store them in a DB in order to generate video summaries that meet the needs of viewers for the broadcast contents. First, when broadcast content is input, a shot is detected for all content (step 301). The detection of the shot is expressed by the following expression (3).

[Equation 3]

ST = True if FD _i ＞ Th _frame and ｜ FD _i -FD _i-1 ｜ ＞ Th _diff , False Otherwise

FD _i = colorDiff _i + edgeDiff _i + textureDiff _i

colorDiff _i = (colorFrame _i-10 -colorFrame _i ) ² / colorDiffMAX

edgeDiff _i = (edgeFrame _i-10 -edgeFrame _i ) ² / edgeDiffMAX

textureDiff _i = (textureFrame _i-10 -textureFrame _i ) ² / textureDiffMAX

ST: Shot Variation

Th _frame : Threshold for frame difference

Th _diff : Threshold value when comparing frame difference between adjacent frames

FD _i : Characteristic value combined in the i-th frame

colorDiff _i : Feature value of the color system of the HSV in the i th frame.

edgeDiff _i : Feature value of the edge component in the i th frame

textureDiff _i : feature value of the homogeneous component in the i-th frame

colorDiffMAX: largest colorDiff _i value in the content

edgeDiffMAX: largest edgeDiff _i value in the content

textureDiffMAX: The largest textureDiff _i value in the content

As described in [Equation 3], it is determined that the shot transition has occurred when the combined feature value FD _i in the current frame is more than a certain level and the difference between the feature values between two adjacent frames is more than a certain level. At this time, in order to prevent excessive detection, the difference between the feature values between the frame 10 frames before and the current frame is used. If ST has True, it can be seen that a shot has been detected.

In order to classify the scene or event in units of detected shots, audio and visual features of the lower levels of the shots are extracted (step 302). The extracted audio features are fundamental frequency, audio power, audio spectrum deviation, harmonic spectrum centroid, and harmonic spectrum deviation. The visual features to be extracted are motion intensity, HSV color information, homogeneous texture, and camera motion values.

Based on the audio feature values, shots having similar audio feature values among successive shots are merged into one shot (step 303). Equation 4 below expresses concatenating adjacent shots into one shot.

[Equation 4]

If | AF _i -AF _i-1 | <AF _th , Merge {S _i-1 , S _i }

AF _i = (W ₁ * FF _i + W ₂ * AP _i + W ₃ * AS _i ) / 3

Merge {A, B}: Merge two consecutive shots A and B into B

AF _i : Audio feature value of the i th shot

FF _i : Normalized frequency value at i-th shot interval

AP _i : Normalized audio power value in the i th shot section

AS _i : Normalized audio spectral deviation value in the i th shot section

W ₁ , W ₂ , W ₃ : weight

[Equation 4] represents a process of viewing and merging one shot when the difference in audio feature values between adjacent shots is equal to or less than a predetermined value. In the shot extracted through Equation 3, normalized audio feature values, that is, normalized fundamental frequency value, normalized audio power value, and normalized audio spectrum deviation value are calculated and defined as the audio feature value of the shot. If the difference in audio feature values between two adjacent shots is equal to or less than a predetermined value, it is determined that the audio information has similar audio information between the two shots, and the two shots are defined as one event or scene and merged into one shot.

The genre of the broadcast content, length information of the extracted shots, and audio and visual features of a lower level are inputted, and an event or scene corresponding to each shot is detected through a content-based classifier (step 304). The content-based classifier detects different events or scenes for each genre. Each classifier has feature values for preset events or scenes and is detected as the closest event or scene through Euclidean distance measurement with the input feature values.

Acquired events or scenes are verified by the producer (step 306).

Through the above process, the semantic features of one broadcast content are output and stored in the semantic feature DB unit 80.

As described above, according to the video summary providing system according to the present invention, it is possible to provide each viewer with a video summary reflecting the preferences of the viewers in the digital TV broadcasting environment, thereby developing the digital TV broadcasting environment centered on the viewer. have.

Claims (4)

A system for automatically providing a viewer-tailored video summary in a digital broadcast environment, (a) an interface unit for acquiring and outputting a behavior pattern related to viewing of a channel selected by a viewer, a title of selected content, a viewing time, a repeated viewing scene, a main character, and the like; (b) a viewer preference generator for generating viewer preferences from behavior patterns output from the interface unit; And (c) a viewer terminal unit having a viewer terminal agent unit configured to transmit viewer preferences generated by the viewer preference generator. And (A) a semantic feature extractor for extracting video features of the content and segmenting the content into semantic features; (B) a semantic feature storage unit for storing the semantic feature extracted by the semantic feature extractor; (C) a video summary generator for searching the semantic feature storage unit to generate a video summary including semantic features that best match the viewer's preferences transmitted from the viewer terminal agent; And (d) a broadcast service provider having a service provider agent for transmitting the video summary generated by the video summary generator to the viewer terminal agent. The method of claim 1, wherein the viewer terminal unit And a video summary database unit for storing the video summaries transmitted from the service provider agent unit to the viewer terminal agent unit to recommend video summaries that match the viewer's preferences. The method of claim 1, wherein the viewer terminal unit A viewer preference updating unit which updates the viewer's preference according to a change in viewer behavior patterns of the interface unit; And And a viewer preference database unit for storing viewer preferences. The method of claim 3, wherein the semantic feature extraction unit And extracting video features such as motion intensity, HSV color information, homogeneous texture, and camera motion from the content, and segmenting the content into events or scenes using the video features.