JP2008252667A - System for detecting event in moving image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a robust system for detecting events in moving images independent of a recording system. <P>SOLUTION: The system detects a highlighted scene or its start in moving images and is composed of a feature quantity extraction part, a feature quantity observation part, a model storing part, a category classification part and a determination part. The feature quantity observation part observes a feature quantity evaluated by the feature quantity extraction part, and if the result does not match a specified feature quantity level, a system for properly detecting events can be constituted by changing the feature quantity in the extraction part. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a moving image event detection apparatus that analyzes moving images and detects important scenes for the purpose of efficiently handling a large amount of moving images.

  In recent years, with the spread of digital video equipment such as DVD recorders and personal computers capable of TV recording with a TV tuner, daily events and TV programs can be stored in large quantities on hard disks (HDD) or optical disks as digital moving images. Holding is becoming common. In the future, it is believed that individuals will be able to further accelerate the possession of a large amount of video content by increasing the capacity of HDDs and recording media, increasing the number of recording devices, and improving the compression efficiency of video. Yes.

  For such situations, detect important scenes such as highlights of recorded content and scenes that users want to see, and use these results during playback to efficiently watch a large amount of videos in a short time. Has been proposed. As a conventional detection technique, in order to extract highlights, a method for determining the type or determination of audio using a threshold value for audio power has been proposed (see, for example, Patent Document 2 and Patent Document 3). . These methods are not suitable for appropriately classifying and classifying a non-linear state, for example, a state where voice and cheer overlap.

On the other hand, as a method of appropriately separating nonlinear states, a feature model of a scene given in advance as training data is learned to construct a probability model, and using this probability model, the feature amount of the input video is highlighted. A method for determining whether or not a scene is included is known. In order to learn the model in advance, this method depends on the training data given in advance, and the detection performance depends on the input video with unknown features that could not be reflected during training when incorporated in an actual system. In some cases, the determination result may be unexpected. To solve such a problem, for example, training is performed by artificially superimposing noise on training data so that the output result does not deviate from the correct answer for an unknown input video, that is, robustness Has been proposed (see, for example, Patent Document 1).
Japanese Patent Laid-Open No. 10-63789 (FIG. 1) JP 2004-260734 A (FIG. 1) JP 2003-101939 A (FIG. 1)

Although the above-mentioned conventional method can enhance the robustness against momentary disturbance of the input video, the expected detection result for the input video that is significantly different from the training data on a regular or temporary basis Was not obtained. Therefore, it was necessary to train and tune the model for each recording system. In addition, there are increasing opportunities to view videos produced in various places due to the diversification of recording environment due to the recent spread of recording devices and TV personal computers, the spread of broadband and the spread of digital broadcasting. For this reason, it is not always necessary to efficiently view only videos recorded by a fixed recording device, and it is necessary to maintain the same detection accuracy without bias for input videos stored in various environments as described above. .

The present invention solves the above-described conventional problems, and provides a robust detection device that does not depend on a system for recording a moving image by detecting a moving image based on machine learning.

  In order to solve the above-described conventional problem, the moving image event detection apparatus of the present invention inputs moving image data, changes the specified feature amount to be changed and the input data according to the change level, and the feature amount. The feature amount extraction unit that calculates and outputs the feature amount, and the feature amount output from the feature amount extraction unit is input, the feature amount to be observed is observed, and if the feature amount level is not the input, the feature amount extraction is performed A feature amount observing unit that conveys a feature amount to be changed and a change level thereof, a model storage unit that stores learning model data, and learning model data corresponding to given genre information is read from the model storage unit, A category that receives the feature amount output from the feature amount extraction unit, calculates which classification of the classification types the input video data is determined in advance, and outputs a close classification result A classification unit, enter the result and genre information classified by the category classification unit, the start of important scenes or has a determination unit for determining a critical section.

  In addition, the moving image event detection apparatus of the present invention includes a model storage unit that stores feature quantities and feature levels to be observed corresponding to a learning model, and a model storage unit that observes feature quantities corresponding to a learning model corresponding to genre information. And a feature amount level, and a feature amount observing unit for inputting the observed feature amount.

  As described above, according to the video event device of the present invention, the feature amount observing unit observes whether or not the characteristic is deviated from the training data at the time of learning the feature amount of the input moving image. There is an effect that an appropriate detection result can be obtained by correcting the input moving image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(Embodiment 1)
Details of the moving image event detection apparatus according to the first embodiment of the present invention will be described with reference to FIGS.

FIG. 1 is a block diagram showing a configuration of a moving image event detection apparatus according to Embodiment 1 of the present invention. In FIG. 1, 11 is a feature amount extraction unit for calculating the feature amount of an input moving image, 12 is a feature amount to be observed designated from the outside and its reference level, and observes the feature amount of the input moving image to obtain a reference level. If the reference level is not reached, a feature amount observing unit that informs the correction level to 11, 14 is a category classification unit that categorizes the input video in segment units, and 13 is a model of 14 models. A model storage unit 15 that stores parameters, and a determination unit 15 that detects an event part based on 14 classification results and genre information of analysis content.
In this embodiment, a case where an event section of a TV program is extracted will be described. It is assumed that a desired event is extracted for recorded content or content being received. For example, it is assumed that a lively part such as a hit, a home run, a goal scene in a sports program baseball or soccer is extracted. In this embodiment, in sports programs, if audio feature values are used, prominence can be detected prominently from characteristics such as audience cheers and commentator's emphasized speech, and the processing amount of detection processing can be suppressed. The feature amount of is used.

  The feature quantity observing unit 12 receives an externally designated feature quantity to be observed and a range restriction of the feature quantity, and determines whether or not the feature quantity from the feature quantity extracting unit 11 is within the designated range. Check. If it is not within the specified range, the feature amount extraction unit 11 is notified of which feature amount is to be corrected. Here, the feature amount and feature level to be observed specified from the outside specify the level determined from the feature amount obtained by analyzing the data used as the teacher data when the learning model parameters are generated in advance. . In the present embodiment, it is assumed that the upper limit Pw_max of the audio power average value and the reference value of the audio power average are designated as Pw_ref from the outside. First, the feature amount observation unit 12 calculates and holds the average value of the audio power for the first 20 seconds for each sample. Next, if it exceeds 20 seconds, after calculating the average value of the audio power for 20 seconds, it is determined whether or not the obtained value exceeds Pw_max. When no gain is applied to the audio signal and Pw_max is exceeded, for example, the gain α to be changed is obtained by the following formula 1. Here, it is assumed that the average value of the currently obtained audio power is Pw_realmean.

Next, the feature quantity extraction unit 11 is informed of the type name of gain α and audio power feature quantity applied to the audio signal input to the feature quantity extraction unit 11. When the gain α is set to a value other than 1, and the average value of the audio power does not exceed Pw_max, the gain α is returned to 1. The feature amount observation unit 12 continues the above-described processing until there is no input signal.

  FIG. 2 is a configuration example of the feature amount extraction unit 11. In FIG. 2, 21 is a power calculation unit that calculates power for each sample, 22 is a feature value calculation unit that calculates feature values such as short-time power average, cepstrum, MFCC, and fundamental frequency, and 23 is an input signal. It is a gain setting unit for applying gain.

  The feature quantity extraction unit 11 takes uncompressed audio as input and calculates the feature quantity of this input. As the feature amount, many feature amounts indicating the acoustic properties of sound such as short-time power average, cepstrum, MFCC, fundamental frequency, and the like can be calculated and used for later processing. Here, the feature quantity used for the subsequent processing is calculated, and the feature quantity designated by the feature quantity observation unit 12 is output to the feature quantity observation unit 12.

  The model storage unit 13 stores parameters of the learning model. The parameters of the model here are generated by learning in advance in another step. Here, parameter data of the learning model is stored corresponding to the genre of the program.

  The category classification unit 14 receives the feature amount output from the feature amount extraction unit 11 as input, and whether the input audio data is speech, cheer, music, or any two of speech, cheer, and music Or three types of data, or other data. The model learned by defining the classification type in this way can be classified when the cheer and music overlap, or when the speech and cheer overlap. In this embodiment, GMM (Gaussian Mixture Model) is used. This model is a generally known model in which each output probability is calculated, and from which cumulative likelihood, it is possible to determine which kind of speech is closest to the input. The model here determines the model by taking out the parameters of the learning model corresponding to the program information input from the outside from the model storage unit 13. The input moving image data is divided into segments at regular intervals, and the classification type close to each segment is output and used as the classification result. For example, which classification type is close every second is output.

  TV program information can be obtained using EPG (Electric Program Guide) data. This data may be superimposed on a television broadcast wave or acquired from the Internet. If the input moving image to be analyzed is not being broadcast, but data already recorded on the HDD or recording medium is used, it is necessary to record the program information in association with the recorded data.

  The determination unit 15 determines and outputs a highlight section and a scene switching point based on a rule that defines the classification result output from the category classification unit 14 according to the genre of the TV program input from the outside. Specifically, the noise of the classification result is taken, that is, the section (start time and end time) including the cheer is included as the detection result except for the change in which the result changes within a predetermined short time. Output.

  According to such a configuration, the feature amount observing unit 12 observes the designated feature amount, and designates the gain to be applied to the input audio to the feature amount extraction unit 11 whose feature amount is designated from the outside, whereby the input amount of the input audio is specified. Since the feature amount can be limited to the specified range, it is possible to avoid the problem that the category classification unit 14 causes the learning model to be in an unexpected state and makes an inappropriate classification determination.

  In the present embodiment, the case where the average value of the audio power exceeds the specified maximum value is set as a condition for applying the gain. I do not care.

  In the present embodiment, the model storage unit 13 stores learning model parameters in accordance with TV program genres, but stores parameters by associating one learning model with a plurality of genres. Alternatively, the parameters may be stored by associating one learning model for each TV program, or may be stored by associating one learning model for each sub-genre.

  In the present embodiment, the category classification unit 14 classifies combinations of two or three of speech, cheer, music, speech and cheer, and music. Any classification can be used as long as it is determined. If the genre, sub-genre, or part of the program to be detected has different characteristics, it is better to define the type of classification to suit each case and learn the model.

  In the present embodiment, the category classification unit 14 outputs only the classification result, but the classification result and its likelihood may be output. In that case, it is possible to consider the likelihood at the time of determination in the determination unit 15 at the subsequent stage.

(Embodiment 2)
The moving image event detection apparatus according to the second embodiment of the present invention will be described in detail with reference to FIG. FIG. 3 is a block diagram showing a configuration of the moving image event detection apparatus according to the first embodiment of the present invention. In the present embodiment, it is assumed that the event section of the TV program is extracted using the audio feature amount as in the first embodiment.

  In FIG. 3, the feature quantity extraction unit 11, the category classification unit 14, and the determination unit 15 operate in the same manner as in the first embodiment. The second feature amount observing unit 32 is the feature described in the first embodiment except for the operation of acquiring the observed feature amount and the feature amount level corresponding to the learning model parameter corresponding to the input genre information. The same operation as that of the quantity observation unit 12 is performed. The second model storage unit 33 stores learning model parameters. Furthermore, the observed feature quantity and the feature quantity level obtained from the feature quantity of the training data used when generating the learning parameter are stored in correspondence with the learning model parameter.

  FIG. 4 is an image diagram of data stored in the second model storage unit 33. The learning model parameter is defined for each TV genre, and the type of feature quantity to be observed and the average value and maximum value of the feature quantity are stored corresponding to one set of learning model parameters. The average value and the maximum value of the observed feature amount stored here are the average value and the maximum value of the feature amount of the training data used when generating the learning model parameter. In the present embodiment, as described above The average value and maximum value of audio power of training data are saved. According to this configuration, when the second feature quantity observation unit 32 observes the designated feature quantity and the feature quantity level does not fall within the value read from the second model storage unit 33, the feature quantity extraction unit By calculating the gain multiplied by the input audio for 11 and specifying this, the feature amount of the input audio can be suppressed to the specified range. Therefore, it is possible to avoid the problem that the category classification unit 14 causes the learning model to be in an unexpected state and makes an inappropriate classification determination.

  In the present embodiment, the model storage unit 13 stores learning model parameters in association with TV program genres. However, the model storage unit 13 stores parameters in association with one learning model for a plurality of genres. The parameters may be stored by storing one TV model for each learning program, or by storing one parameter for each sub-genre.

  In the present embodiment, the learning model parameter, the type of feature quantity to be observed, and the average value and maximum value of the feature quantity are images stored in the same file, but the type and average of the feature quantity to be observed As long as the data set of the value and the maximum value is associated with the learning model parameter set, any configuration may be used.

(Embodiment 3)
The moving image event detection apparatus according to the third embodiment of the present invention will be described in detail with reference to FIG. FIG. 5 is a block diagram showing a configuration of the moving image event detection apparatus according to the first embodiment of the present invention. In the present embodiment, it is assumed that the event section of the TV program is extracted using the audio feature amount as in the first embodiment.

  In FIG. 5, 51 is a second feature quantity extraction unit, 52 is a third feature quantity observation unit, 53 is a third model storage unit, 54 is a second category classification unit, and 55 is a second determination unit. is there.

  The second feature quantity extraction unit 51 extracts the same feature quantity as in the first embodiment. Then, the extracted feature amount is output to the third feature amount observation unit 52 and the second category classification unit 54.

  The third feature quantity observation unit 52 calculates the maximum value and the average value of each feature quantity obtained from the second feature quantity extraction unit, and sends it to the second category classification unit 54.

  The third model storage unit 53 stores the learning model parameter set in the same manner as in the first embodiment. In addition, the third model storage unit 53 stores the maximum value and the average value of the training data when learning the model for each classification. Keep it. For example, in the present embodiment, the combination of speech, cheer, and music is classified into two or three of speech, cheer, and music. The maximum and average feature values are stored.

  In addition to the same operations as those in the first embodiment, the second category classification unit 54 calculates the maximum value and the average value of each feature value of the training data corresponding to each classification read from the third model storage unit 53. Among them, the maximum value and the average value of each feature amount of the classification corresponding to the classification result are compared with the maximum value and the average value of each feature amount received from the third feature amount observation unit 52, and not greatly deviated. Confirm. If they differ greatly, a classification result is output and a small value weight (1 or less) is output. If there is no significant difference, 1 is output.

  The second determination unit 55 determines the highlight section and the scene switching point based on the rule according to the genre of the TV program inputted from the outside, based on the classification result output by the second category classification unit 54, and outputs it. To do. Specifically, the noise of the classification result is taken and output in consideration of the weight output from the second category classification unit 54 when determining the section (start time and end time) in which cheers are included. .

  Here, it is assumed that the average value and the maximum value of the audio power of the training data corresponding to each classification are the values shown in FIG. The maximum value of the audio power input to the second category classification unit 54 at time t is InPW_max (t), the average value is InPW_mean (t), and the likelihood calculated by the second category classification unit 54 Is the value shown in FIG. When the average value of the audio power of the training data of the classification is 1.5 times or more, the likelihood of the corresponding classification is expressed as likelihood × [average of audio power of training data / (InPW_mean (t))]. The likelihood of all classifications is obtained, and the classification with the highest likelihood is taken as the classification result at that time.

  According to this configuration, the third feature amount observing unit 52 observes the feature amount obtained by the second feature amount extracting unit, and the maximum value and the average value of each feature amount are sent to the second category classification unit 54. The second category classification unit 54 outputs the classification result obtained by comparing the classification result obtained from the input feature vector with the maximum value and the average value of the training data of the model corresponding to the classification result. Output reliability as weight. As a result, it is possible to avoid the problem of improper classification determination for input patterns that are not well learned during learning.

  In the present embodiment, the maximum value and the average value of each feature amount are compared. However, it is also possible to compare the maximum value and the average value focusing only on the feature amount that contributes dominantly to the learning model. In this comparison, any value may be used as long as it is not a maximum value and an average value but a value that characterizes data such as variance.

  In the moving image event detection apparatus according to the present invention, the feature amount observation unit observes whether or not the characteristic is deviated from the training data at the time of learning the feature amount of the input moving image. In addition, since an appropriate detection result can be obtained without preparing learning data for each recording system, the present invention can also be applied to a purpose of automatically detecting a desired section, for example, a rising section from a moving image.

1 is a block diagram showing a configuration of a moving image event detection apparatus according to Embodiment 1 of the present invention. Configuration diagram of the feature quantity extraction unit 11 of the video event detection device according to the first embodiment of the present invention. The block diagram which shows the structure of the moving image event detection apparatus in Embodiment 2 of this invention. The image figure of the data which the 2nd model preservation | save part in Embodiment 2 of this invention preserve | saves The block diagram which shows the structure of the moving image event detection apparatus in Embodiment 3 of this invention. The figure which represented the maximum value and average value of each feature-value of the training data matched with each classification | category which the 3rd model preservation | save part in Embodiment 3 of this invention preserve | saved The figure showing the likelihood with respect to each classification calculated in the 2nd category classification part in Embodiment 3 of the present invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Feature-value extraction part 12 Feature-value observation part 13 Model preservation | save part 14 Category classification | category part 15 Judgment part 21 Power calculation part 22 Feature-value calculation part 23 Gain setting part 32 2nd feature-value observation part 33 2nd model preservation | save part 51 Second feature quantity extraction unit 52 Third feature quantity observation unit 53 Third model storage unit 54 Second category classification unit 55 Second determination unit

Claims (5)

In a device that analyzes video and detects events and important parts, input video data, change the input data according to the specified feature quantity to be changed and its change level, calculate the feature quantity and output it And the feature quantity output from the feature quantity extraction section is input, and the observed feature quantity is observed, and if it is not the feature quantity level designated as input, the feature quantity extraction section is changed. A feature amount observing unit for conveying a feature amount and its change level, a model storage unit for storing learning model data, and learning model data corresponding to given genre information are read from the model storage unit, and the feature amount extraction is performed. A category classification unit that takes the feature value output from the input as input, calculates which classification is close among the classification types for which the input video data is determined in advance, and outputs a close classification result Inputs the results and genre information the classified by category classification section, the beginning of the important scenes or video event detection device, characterized in that it comprises a determining unit for determining a critical section. The model storage unit stores an observed feature amount and feature level corresponding to a learning model, and the feature amount observation unit receives an observation feature associated with a learning model corresponding to genre information from the model storage unit. The moving image event detection apparatus according to claim 1, wherein an amount and a feature amount level are input, and a feature amount to be observed from the feature amount extraction unit is input. In an apparatus that detects an event or an important part by analyzing a moving image, a moving image data is input, a feature amount extraction unit that calculates and outputs a feature amount, and a feature amount output from the feature amount extraction unit is input. A feature amount observing unit that calculates and outputs the range and distribution of the feature amount to be observed, and a model storage unit that stores the learning model data and the feature amount range and distribution corresponding to the training data of the learning model data. The learning model data corresponding to the genre information is read from the model storage unit, the feature amount output from the feature amount extraction unit is input, and the input data is close to any of the predetermined classification types , And output a close classification result, and the range and distribution of the observed feature value output from the feature value observation unit and the training read from the model storage unit The category classification that outputs the weights for the classification results with a lighter weight when the ranges or distributions of the feature quantities corresponding to the classification results are compared. A moving image event detection apparatus comprising: a determination unit configured to input a classification result classified by the category classification unit, a weight, and genre information, and to determine an important scene start or an important section. The feature amount observation unit receives the audio power feature amount output from the feature amount extraction unit and the average value and maximum value of the audio power feature amount to be referenced, and calculates the average value and maximum value of the audio power feature amount. And confirming that the average value and the maximum value of the audio power feature values are the feature value level, and if it is not the specified feature value level, the change level applied to the audio power feature value by the feature value extraction unit The moving image event detection apparatus according to claim 1, wherein: The feature amount observation unit receives the audio power feature amount output from the feature amount extraction unit and the average value and maximum value of the audio power feature amount to be referenced, and calculates the average value and maximum value of the audio power feature amount. And calculating the change level applied to the audio input when the average value of the audio power feature value exceeds the average value of the audio power feature value to be referred to for a certain period, or to refer to the audio power for a certain period. The moving image event detection apparatus according to claim 1, wherein, when a maximum value of a feature amount is exceeded, the change level is transmitted to a feature amount extraction unit.

Images