JP5467346B2 - Motion estimation method, motion estimation device, and motion estimation program - Google Patents

Description

  The present invention relates to the fields of motion analysis of moving objects in moving images and compression coding of moving images, and in particular, a motion estimation method, a motion estimation device, and the like for estimating the motion of one or more moving objects in a moving image, And a motion estimation program.

  In motion picture compression coding, a motion prediction technique for predicting a motion amount of a moving object from a reference frame is an extremely important technique for greatly reducing the description amount of moving picture data.

  As a method widely used for motion prediction, MPEG-4 AVC / H. A method standardized as H.264 is known. MPEG-4 AVC / H. In the motion prediction technology used in the H.264 video encoding technology, motion prediction is performed by analyzing data obtained from a video in the time domain or the spatial domain. In other words, this motion prediction technique is an analysis method for performing orthogonal transform with integer precision on a two-dimensional pixel block, and the analysis uses frames before and after configuring moving image data, This is performed on the prediction error image predicted from the pixel information of the frame.

  However, since such motion prediction is based on pattern recognition in units of pixel blocks, it can handle multiple objects that move in different directions and objects that move while changing their shape in a complex manner. However, another method is required.

  Another method is to capture a moving image as a three-dimensional signal consisting of height, width, and time, analyze it three-dimensionally, and use the obtained spatio-temporal spectrum to move the moving object in the moving image. There is a method to predict.

  In general, when attention is paid to spectra having high energy among spatio-temporal spectra obtained by three-dimensional analysis of moving images including moving objects, these spectra are distributed in a plane in a three-dimensional frequency domain space. It becomes a plane group. The configuration of the plane group is determined by the motion of the object in the moving image, and the motion can be predicted by obtaining the inclination of the plane group. Therefore, in the case of a moving image including a plurality of moving objects that move differently, the number of moving objects is equal to the number of planes. It becomes possible to predict.

However, the Fast Fourier Transform (Fast
In an analysis method having an equal frequency resolution width represented by Fourier Transform (FFT) and Discrete Fourier Transform (DFT), the resolution depends on the frame length, and the resolution is sufficient for analysis of moving images. Therefore, in order to cope with a motion that cannot be analyzed due to insufficient resolution, it is necessary to combine another method.

  For example, in Non-Patent Document 1, a linear group filter is used as a method for determining a slope of a plane group represented by a spatiotemporal spectrum after obtaining a spatiotemporal spectrum by three-dimensionally analyzing a moving image using FFT. A technique is disclosed.

  In the technique described in Non-Patent Document 2, a moving image is three-dimensionally analyzed by obtaining a spatio-temporal frequency spectrum using an FFT algorithm based on plane estimation, and fuzzy inference is used for plane estimation. Thus, the accuracy of analysis is improved by correcting the error of the spectrum data.

  On the other hand, Patent Document 1 discloses Non-Harmonic Analysis (NHA), which is an aperiodic signal analysis method, as a frequency analysis method devised by some of the inventors of the present application. This NHA has a frequency f ′ at which the sum of squares of the difference between the signal to be analyzed and the sine wave model signal represented by the phase using the frequency f ′ and the initial phase φ ′ and the amplitude A ′ becomes a minimum value. , Amplitude A ′, and initial phase φ ′ are calculated as parameters of the Fourier transform equation of the aperiodic signal.

International Publication No. 2009/038056

Akira KOJIMA and Jun-ichiHISHIGAMI, “Motion Detection using 3D-FFT Spectrum”, ITEC’92, 1992 C. E. Erdem, G. Karabulut, E. Yanmaz and E. Anarim, “Motion Estimation in thefrequency domain using fuzzy c-planes clustering”, vol.10, pp. 1873-1879, December 2001

  However, the technique described in Non-Patent Document 1 does not improve the accuracy of the obtained space-time spectrum, and there is a limit to the prediction accuracy of moving objects. Further, the technique described in Non-Patent Document 2 has a problem that a complicated process is required for fuzzy inference and the calculation cost is increased.

  The present invention has been made in view of such circumstances, and a moving object in a moving image using a spatio-temporal spectrum obtained by three-dimensional analysis of a moving image with high accuracy and a small amount of calculation. An object of the present invention is to provide a motion estimation method, a motion estimation apparatus, and a motion estimation program that can estimate the motion of the image.

In the motion estimation method of the present invention, the difference between a moving image data to be analyzed is a three-dimensional signal of the moving image data and a sine wave model signal represented by a phase and an amplitude using a frequency and an initial phase. The frequency, the amplitude and the initial phase at which the sum of squares becomes the minimum value are obtained as parameters of the Fourier transform equation of the aperiodic signal, the spatiotemporal spectrum is extracted, and the extracted spatiotemporal spectrum distribution is expressed as a plane. The one or a plurality of plane groups are estimated by dividing each of the spectrum groups, and the motion of one or a plurality of moving objects in the moving image is estimated by obtaining the estimated inclination of the plane group. A motion estimation method comprising:
Clustering using the least squares method is performed on the spatiotemporal spectrum, and when estimating one or a plurality of plane groups, the existence probability variable when the spatiotemporal spectrum belongs to a predetermined plane is set to 1 or It is characterized in that it is set to a decimal number and clustering using the least square method is performed .

The motion estimation apparatus of the present invention is represented by a moving image input means for inputting moving image data to be analyzed, a three-dimensional signal of the input moving image data, and a phase and amplitude using a frequency and an initial phase. Space-time spectrum in which the frequency, the amplitude, and the initial phase such that the sum of squares of the difference from the sine wave model signal is minimized are obtained as parameters of the Fourier transform equation of the non-periodic signal, and the space-time spectrum is extracted A spectrum extraction unit, a plane estimation unit that estimates one or a plurality of plane groups by dividing the extracted spatiotemporal spectrum distribution for each spectrum group forming a plane, and obtains an inclination of the estimated plane group A motion estimation device for estimating the motion of one or more moving objects in the moving image,
Clustering using the least squares method is performed on the spatiotemporal spectrum, and when estimating one or a plurality of plane groups, the existence probability variable when the spatiotemporal spectrum belongs to a predetermined plane is set to 1 or It is characterized in that it is set to a decimal number and clustering is performed using the least square method .

The motion estimation program of the present invention is represented by moving image input means for inputting moving image data to be analyzed, a three-dimensional signal of the input moving image data, and a phase and amplitude using a frequency and an initial phase. Space-time spectrum in which the frequency, the amplitude, and the initial phase such that the sum of squares of the difference from the sine wave model signal is minimized are obtained as parameters of the Fourier transform equation of the non-periodic signal, and the space-time spectrum is extracted Spectrum extraction means, plane estimation means for estimating one or a plurality of plane groups by dividing the spatio-temporal spectrum distribution for each spectrum group forming a plane, and inclination calculation means for obtaining an inclination of the estimated plane group A computer-executable motion estimation program for causing a computer to function as a computer and estimating the motion of one or more moving objects in the moving image,
Clustering using the least squares method is performed on the spatiotemporal spectrum, and when estimating one or a plurality of plane groups, the existence probability variable when the spatiotemporal spectrum belongs to a predetermined plane is set to 1 or A motion estimation program that is set to a decimal number and performs clustering using the least squares method .

  In such a motion estimation method, motion estimation device, and device in which a motion estimation program according to the present invention is implemented, a frequency analysis method whose frequency resolution does not depend on the analysis window length is used for three-dimensional signal analysis of moving images. As a result, the spatio-temporal spectrum can be extracted with higher accuracy than in the case of using the conventional frequency analysis method, and the movement of the moving object in the moving image is accurately reflected in the frequency domain space. Therefore, in the motion estimation method, the motion estimation device, and the device in which the motion estimation program according to the present invention is implemented, the error from the theoretical plane of the spectrum having high energy among the obtained spatiotemporal spectrum is extremely reduced. be able to.

  In the present invention, since the spatio-temporal spectrum distribution obtained by frequency analysis can be obtained with high accuracy, the process of estimating the plane of motion does not require complicated processing, and the moving image can be performed with high accuracy and with a small amount of calculation. The motion of the moving object in the image can be estimated.

It is a block diagram which shows the structure of the motion estimation apparatus shown as embodiment of this invention. It is a figure for demonstrating the difference between this frequency analysis method, DFT, and GHA, and is a figure which shows the result of having calculated | required the error of each method. 5 is a flowchart showing a series of processing when estimating the motion of one or more moving objects in a moving image in the motion estimation device shown as an embodiment of the present invention. It is a figure which shows the moving image used in 1st verification. When the moving speed V _x of the object in the first verification is 1.0 pixels / frame, it is a diagram showing a spatial spectrum distribution when determined using a 3-dimensional FFT. When the first 3.3 pixels / frame the moving speed V _x of the object in the verification is a diagram showing a spatial spectrum distribution when determined using a 3-dimensional FFT. When the first 3.3 pixels / frame the moving speed V _x of the object in the verification is a diagram showing a spatial spectrum distribution when determined using the frequency analysis technique. It is a figure which shows the moving image used by 2nd verification. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using 3D FFT in the 2nd verification. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using this frequency analysis method in 2nd verification. It is a figure which shows the moving image used in 3rd verification. It is a figure which shows the spatio-temporal spectrum distribution calculated | required using 3D FFT in the 3rd verification. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using this frequency analysis method in the 3rd verification. It is a figure which shows a mode that the plane was drawn changing the viewpoint of the spatiotemporal spectrum distribution of FIG. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using the three-dimensional FFT about the moving image containing a several similar motion. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using this frequency analysis method about the moving image containing a several similar motion. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using the three-dimensional FFT about the moving image containing the several motion different from the case of FIG. It is a figure which shows the spatio-temporal spectrum distribution calculated | required using the three-dimensional FFT about the moving image containing the several motion different from the case of FIG. It is a figure which shows the moving image used in the 4th verification. It is a figure which shows a mode that the plane estimated based on 100 spatiotemporal spectra calculated | required using this frequency analysis method in 4th verification was drawn. It is a figure which shows a mode that the plane estimated based on 125 spatiotemporal spectra calculated | required using this frequency analysis method in 4th verification was drawn. It is a figure which shows a mode that the plane estimated based on 160 spatiotemporal spectra calculated | required using this frequency analysis method in 4th verification was drawn. It is a figure which shows a mode that the plane estimated based on 180 spatiotemporal spectra calculated | required using this frequency analysis method in the 4th verification was drawn. It is a figure which shows a mode that the plane estimated based on 200 spatio-temporal spectra calculated | required using this frequency analysis method in 4th verification was drawn. It is a figure which shows a mode that the plane estimated based on 225 spatiotemporal spectra calculated | required using this frequency analysis method in 4th verification was drawn. It is a figure which shows a mode that the plane estimated based on 250 spatiotemporal spectra calculated | required using this frequency analysis method in 4th verification was drawn. It is a figure which shows the moving image used in 5th verification. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using 3D FFT in 5th verification. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using this frequency analysis method in 5th verification. It is a figure which shows the moving image used in 6th verification. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using three-dimensional FFT in the 6th verification. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using this frequency analysis method in 6th verification. It is a figure which shows the moving image used in 7th verification. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using three-dimensional FFT in the 7th verification. It is a figure which shows the spatiotemporal spectrum distribution calculated | required using this frequency analysis method in 7th verification. It is a figure which shows a mode that the plane estimated based on the spatiotemporal spectrum shown in FIG. 34 was drawn. It is a figure which shows a mode that the plane estimated based on the spatiotemporal spectrum shown in FIG. 35 was drawn.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.

  This embodiment is a motion estimation device that estimates the motion of one or more moving objects in a moving image. In particular, this motion estimation device performs a three-dimensional signal analysis of a moving image using a new frequency analysis method in which the frequency resolution does not depend on the analysis window length by estimating a Fourier coefficient by solving a nonlinear equation. is there.

[Configuration of motion estimation device]
The motion estimation device is composed of, for example, a computer or the like, and as shown in FIG. 1, a CPU (Central Processing Unit) 11 for overall control of each unit and a read-only ROM (Read Read) for storing various information including various programs. Only Memory 12 and RAM (Random) that functions as a work area
(Access Memory) 13, a storage unit 14 that stores various information in a readable and / or writable manner, an input operation control unit 15 that performs processing and control of an input operation via a predetermined operation device (not shown) as a user interface, And a display unit 16 for displaying various information.

  The CPU 11 executes various programs including various application programs stored in the storage unit 14 and the like, and comprehensively controls each unit.

  The ROM 12 stores various information including various programs. Information stored in the ROM 12 is read under the control of the CPU 11.

  The RAM 13 functions as a work area when the CPU 11 executes various programs, and temporarily stores various information and reads the stored various information under the control of the CPU 11.

  The storage unit 14 stores various information including moving image data to be analyzed, in addition to application programs such as a motion estimation program according to the present invention. For example, a hard disk or a non-volatile memory can be used as the storage unit 14. The storage unit 14 also includes a drive device that reads and / or writes various types of information on a storage medium such as a flexible disk or a memory card that can be attached to and detached from the main body. Various types of information stored in the storage unit 14 are read out under the control of the CPU 11.

  The input operation control unit 15 accepts an input operation via a predetermined operation device (not shown) as a user interface such as a keyboard, a mouse, a keypad, an infrared remote controller, a stick key, or a push button, for example, and indicates operation contents. A control signal is supplied to the CPU 11.

The display unit 16 is, for example, a liquid crystal display (Liquid Crystal).
Various display devices such as a display (LCD), a plasma display panel (PDP), an organic electroluminescence (Organic ElectroLuminescent) display, or a CRT (Cathode Ray Tube). Display information. For example, when the motion estimation program is activated by the CPU 11, the display unit 16 displays the screen and displays the input moving image data as the analysis target, the motion estimation result, and the like.

  When a motion estimation program is executed under the control of the CPU 11, the motion estimation device including each unit performs a frequency analysis of input moving image data under the control of the CPU 11, thereby performing a spatio-temporal spectrum. Is extracted, and the obtained spatio-temporal spectrum is subjected to clustering using the least square method, so that the spatio-temporal spectrum distribution is divided for each spectrum group forming a plane to estimate one or a plurality of plane groups. Note that a signal to be subjected to frequency analysis, that is, moving image data to be analyzed, is input to the CPU 11 via a moving image input unit (not shown). For example, when the motion estimation apparatus compresses and encodes moving image data obtained by recording a television video, the motion estimation apparatus performs analysis via a predetermined interface that connects the recorder including the motion estimation apparatus and the television. Input moving image data as a target. That is, the moving image input unit is a part having a function of causing the CPU 11 to input moving image data to be analyzed. Needless to say, the moving image input unit also has a function of performing A / D conversion and converting to a digital signal when an analog signal is input. At this time, the moving image input unit may be an A / D converter including an anti-aliasing filter as necessary. Under the control of the CPU 11, the motion estimation device performs motion estimation by performing frequency analysis of the moving image data as the analysis target input in this way, and outputs compression-encoded data or the like (not shown) The data is stored in the storage unit 14 via the unit or output to other devices.

[Frequency analysis algorithm]
First, prior to description of a series of motion estimation algorithms in the motion estimation apparatus, a frequency analysis algorithm used when performing motion estimation will be described in detail. Needless to say, the moving image data is a three-dimensional signal, but here, for convenience of explanation, a frequency analysis algorithm for a one-dimensional analysis target signal will be described.

  In the frequency analysis method (hereinafter referred to as this frequency analysis method) applied to the motion estimation device, the problem of obtaining the frequency parameter of the Fourier transform equation of the aperiodic signal shown in the following equation (1) is obtained as an optimal solution of the nonlinear equation. Replaced with a problem.

Specifically, in this frequency analysis method, as shown in the following equation (2), as an optimal solution of the nonlinear equation represented by the sum of squares of the difference between the analysis target signal x (n) and the sine wave model signal: Then, the frequency f ′, the amplitude A ′, and the initial phase φ ′ are obtained so that the right side of the nonlinear equation becomes the minimum value. In the following equation (2), L is a frame length (analysis window length), and f _s is a sampling frequency [Hz]. In this frequency analysis method, the effect of the analysis window and aliasing are eliminated by reducing the problem of finding the optimal solution of the nonlinear equation by the least square method, and the analysis window length is less than one cycle. In addition, it is not necessary to be an integral multiple of the period, and furthermore, it is possible to realize flexible frequency analysis processing such as unequal intervals.

  In order to actually obtain the optimum solution of the nonlinear equation shown in the above equation (2), the following method can be taken.

  In this frequency analysis method, appropriate initial values are obtained for each of the amplitude A ′, the frequency f ′, and the initial phase φ ′, and converged to an optimal solution from these initial values using a solution of a nonlinear equation. In this nonlinear problem, the above equation (2) is a minimization problem with a cost function. Appropriate initial values are arbitrary frequency transformations such as Discrete Fourier Transform (DFT) and wavelet transformation, and approximate existing values by filtering, etc. Can be obtained by applying.

First, in this frequency analysis method, the so-called steepest descent method is applied to the frequency parameters f ′ and φ ′ constituting the phase of the sine wave model signal in the above equation (2), and the frequency parameters f _m ′ and φ _m ′ are applied. Is obtained by the following equations (3) and (4).

In the above formulas (3) and (4), the following formula (5) is abbreviated. Further, mu _m, a weighting factor based on the so-called reduction method, to a cost function determined by the recursion formula monotonically decreasing sequence takes a value of timely 0-1.

If the frequency parameters f _m ′ and φ _m ′ can be obtained, the frequency parameter A ′ as a coefficient of the sine wave model signal in the above equation (2) can be uniquely obtained. The frequency parameter A _m ′ is converged by equation (6).

  In this frequency analysis method, it is possible to converge the amplitude A ′, the frequency f ′, and the initial phase φ ′ with high accuracy by repeatedly performing these series of calculations. In particular, in this frequency analysis method, the calculation is performed by separately obtaining the frequency parameters f ′ and φ ′ constituting the phase of the sine wave model signal in the above equation (2) and the frequency parameter A ′ as a coefficient. It can be performed simply.

  However, although the steepest descent method converges from a relatively wide range, the accuracy is low in one iteration, and it takes time to converge.

Therefore, in this frequency analysis method, it is desirable to converge the frequency parameters f _m ′ and φ _m ′ to some extent by applying the steepest descent method, and then to converge with high accuracy by applying the so-called Newton method. . Specifically, in this frequency analysis method, the frequency parameters f _m ′ and φ _m ′ are obtained by the recurrence formulas shown in the following equations (7) and (8) as the Newton method.

However, in the above formulas (7) and (8), J is the following formula (9) and is abbreviated as the following formula (10). Also, [nu _m is also a weighting coefficient based on the reduction method in the same manner as mu _m, taking the value of timely 0-1.

In the present frequency analysis technique, the above equation (7) and the equation (8) frequency parameter f _m by ', phi _m' sought after, like the steepest descent method, frequency parameter by the above equation (6) A _m 'Is converged and this series of calculations is repeated further.

  As described above, in this frequency analysis method, the frequency parameters A ′, f ′, and φ ′ are estimated at high speed and with high accuracy by using a hybrid method combining the steepest descent method and the Newton method. Can do.

  Further, in this frequency analysis method, even if the analysis target signal x (n) is a composite sine wave, the spectral parameter can be approximately derived by performing successive subtraction processing. Here, it is assumed that the analysis target signal x (n) is the sum of a plurality of sine waves and is expressed as the following equation (11).

According to the Parseval theorem, if the frequency _fk of the signal to be analyzed x (n) and the frequency parameter f ′ of the sine wave model signal do not coincide at all, that is, if The nonlinear equation shown in the equation (2) becomes the following equation (13). When the set of frequency parameters f ′ and φ ′ matches either of the set of the frequency f _k and the initial phase φ _k , the nonlinear equation shown in the above equation (2) becomes the following equation (14). . Furthermore, when the amplitude A _j matches the frequency parameter A ′, the frequency component related to the estimated spectrum can be completely eliminated from the analysis target signal. Therefore, the problem of obtaining the optimum solution is independent of the frequency, and can be applied to signals represented by a plurality of sine waves if individually estimated from the analysis target signal.

  That is, in this frequency analysis method, even if the analysis target signal x (n) is a composite sine wave, it is possible to extract a plurality of sine waves by performing the same process on the residual signal successively. .

  In order to express a signal such as an audio signal or an acoustic signal with a composite sine wave, a large number of spectra (the number of sine waves) has been required so far. Even if it exists, it can express with few errors and without an error. That is, being able to express a signal with a smaller number of spectra indicates that it is effective for information compression.

[Effectiveness of this frequency analysis method]
Hereinafter, the effectiveness of this frequency analysis method will be specifically described.

  In this frequency analysis method, the frequency f ′, the amplitude A ′, and the initial phase φ ′ of the sine wave model signal can be obtained at high speed and with high accuracy by obtaining an optimal solution of the nonlinear equation. In order to verify the specific accuracy, the inventor of the present application verified accuracy by comparing DFT and GHA (Generalized Harmonic Analysis), which is said to have the highest analysis accuracy among the developed types of DFT. .

  Note that DFT and GHA apparently have a plurality of window lengths in one analysis window length, so the frequency resolution depends on the analysis window length, but the resolution frequency is finite, and the signal to be analyzed If the analysis signal is different from the frequency that can be analyzed accurately, the analysis signal cannot be analyzed when the frequency is other than the decomposition frequency. A frequency (sideband component) appears, and a plurality of frequencies appear.

  Whether or not such a phenomenon occurs also in the present frequency analysis method, that is, in order to verify the frequency resolution of the present frequency analysis method, the analysis window length is one second (1024 samples) and is very short. A single sine wave was analyzed, one sine wave was extracted by each method, and the square error from the original signal was examined. The result is shown in FIG.

  As shown in FIG. 2, in the DFT, the analysis accuracy deteriorated at frequencies other than an integral multiple of the fundamental frequency. In addition, in GHA, an accuracy improvement of about 2 to 5 digits was observed compared with DFT at a frequency of 1 Hz or higher. On the other hand, in this frequency analysis method, an accuracy improvement of 10 digits or more compared to DFT and 5 digits or more compared to GHA was observed at a frequency of 1 Hz or more. That is, this frequency analysis method showed an improvement in accuracy of 100,000 to 10 billion times or more compared with the existing frequency analysis method. In particular, the fact that a frequency of 1 Hz or less can be accurately estimated indicates that even a long periodic signal exceeding the analysis window length can be analyzed.

  As described above, this frequency analysis method can perform analysis with surprisingly high accuracy even compared to GHA, which is said to have the highest analysis accuracy. The motion estimation apparatus estimates the motion of one or more moving objects in the input moving image by performing a series of processes as shown in FIG. 3 using such a frequency analysis method.

[Extract area from video]
First, as shown in FIG. 3, the motion estimation apparatus receives an original moving image that is input via a moving image input unit (not shown) and stored in a memory such as the RAM 13 in step S <b> 1 under the control of the CPU 11. A region including a moving object is cut out from the image data. Here, the number of pixels in the horizontal direction, the number of pixels in the vertical direction, and the number of frames of the cut-out area are N _x , N _y , and N _z , and the elements are i, j, and k, respectively. Note that i, j, and k are indexes for reproducing an image and are real numbers. Also, the moving image data composed of the cut out three-dimensional signal is assumed to be ξ _obj (i, j, k). The extracted moving image data is stored in the RAM 13 or the like.

[Extraction of spatiotemporal spectrum]
Subsequently, in step S2, under the control of the CPU 11, the motion estimation device uses the above-described frequency analysis method to obtain moving image data ξ _obj (3D signals) using the frequency analysis method described above. Extract L spatiotemporal spectra from i, j, k). In the following equation (15), f _xs , f _ys , and f _zs are sampling frequencies [Hz] in the horizontal axis direction, the vertical axis direction, and the time axis direction of the moving image, respectively, and A ′, f _x ′, f _y ′, f _z ′, and φ ′ are the amplitude of the spectrum actually extracted, the frequency corresponding to each axis, and the initial phase, respectively. The motion estimation device represents moving image data, which is a three-dimensional signal, using a sine wave model function represented by the following equation (15), and an actual signal and a sine wave model signal represented by the following equation (15): Each parameter is obtained by changing the parameter using the following equation (16) so that the difference is minimized.

  Since the moving image data can be expressed by the synthesis of the spatio-temporal spectrum as shown in the following equation (17), the motion estimator is configured to add L lines of moving image data under the control of the CPU 11. Extract the spatiotemporal spectrum. At this time, the l-th spatio-temporal spectrum is expressed as shown in the following equation (18).

[Spatio-temporal clustering]
In step S3, the motion estimation apparatus clusters the spatiotemporal spectrum under the control of the CPU 11. Here, assuming that there are I planes, let P _i (l) be the existence probability variable when the l-th spatio-temporal spectrum belongs to the i-th plane. Here, it is assumed that the random variable P has a value of 0 or 1, and is initially initialized with 0.

First, the motion estimation apparatus selects two spatiotemporal spectra S _u and S _v from the spatiotemporal spectrum S ₁ under the control of the CPU 11. Note that u and v are included in the set L as shown in the following equation (19) and have different values.

Subsequently, under the control of the CPU 11, the motion estimation device, as shown in the following equation (20), is a plane normal vector n _u composed of the selected spatiotemporal spectra S _u and S _v and the origin. _{, V.}

Subsequently, under the control of the CPU 11, the motion estimation device assumes a plane equation using the obtained normal vectors n _{u, v} as shown in the following equation (21).

Subsequently, under the control of the CPU 11, the motion estimation device, as shown in the following equation (22), the space obtained by the above equation (21) and the spatiotemporal spectrum excluding the spatiotemporal spectra S _u and S _v. The distances D _{u, v} (m) from S _m are respectively determined. Note that m is included in the set L and is different from u and v.

Then, under the control of the CPU 11, the motion estimation device searches for a spatiotemporal spectrum in which the obtained distance D _{u, v} (m) is equal to or less than a predetermined threshold ε, as shown in the following equation (23): The random variable P _i (m) is set to 1.

  Here, the calculation is performed assuming that the random variable is set to 1 so that it completely belongs to the plane. However, in the motion estimation apparatus, a decimal random variable is set, and a threshold value that is regarded as a point close to the plane is set. It is also possible to perform calculation using this threshold value, and this threshold value may be one of the calculation parameters for the next calculation.

The motion estimation apparatus, under the control of the CPU 11, applying the least square method to the spatial spectrum S _m when satisfying the above equation (23), calculates the equation of the plane shown in the following equation (24) The horizontal and vertical velocities V _x and V _y of the moving object are estimated from the estimated plane inclination. Incidentally, d in the following equation (4) is a section of _{f z} axis.

[Plane estimation from spatiotemporal spectrum]
In step S4, the motion estimation device estimates the plane from the spatiotemporal spectrum as described above under the control of the CPU 11, and if there is a spatiotemporal spectrum that does not belong to the plane i, the above formula ( Using the spatio-temporal spectrum that does not satisfy 23), the clustering of the spatio-temporal spectrum in step S3 is repeated.

  The motion estimation apparatus can estimate the motion of one or a plurality of moving objects in the input moving image by performing such a series of processes. In particular, the motion estimation apparatus can estimate a plane with less error from the theoretical plane by using the above-described frequency analysis method for extracting a spatiotemporal spectrum. Therefore, the motion estimation device can apply a clustering method using a least square method, which is a simple method of plane estimation, instead of a clustering method using a linear group filter or fuzzy inference, and with high accuracy and Motion estimation can be performed with a small amount of calculation.

[Effectiveness of motion estimation device]
Hereinafter, the effectiveness of the motion estimation apparatus will be specifically described.
[First verification]
First, as shown in FIG. 4, the number of pixels in the horizontal direction, the number of pixels in the vertical direction, and the number of frames are N _x , N _y , and N _z , respectively, and a black vertical bar-like object at the left end of the initial frame is Consider a moving image that moves to the right as time progresses. The moving speed V _x is defined as the number of pixels that the object moves between successive frames, and is positive when the object moves to the right or above the frame. If the velocity is the number of pixels per frame and N _z > (N _x / V _x ), the object exists up to the nth frame. Since a vertical bar-like object is used, the vertical frequency f _y is zero. Therefore, the moving image will be characterized by the frequency _f x, _{f z.} The inventor of the present application obtained a spectral structure on a three-dimensional plane f _x -f _z and observed changes in amplitude and frequencies f _x and f _z . In this case, space-time spectra are distributed linearly on a plane f _x -f _z, it is defined as shown in the following equation (25). Note that in the following equation (25), d is the intercept of _{f z-axis,} the speed _{V x} is a coefficient of a frequency _{f x} in an equation of the theoretical movement. This straight line is called a theoretical straight line.

Here, when the frequency analysis by the three-dimensional FFT is executed, the velocity of the moving object and the analysis window affect whether the spectrum distribution takes the form of a theoretical straight line. The resolution of the three-dimensional FFT also depends on the analysis window. When N _z = N _x / V _x is constant, the moving object is present in all the frames constituting the moving image, and the moving image data is substantially periodic. Since the resolution and the number of frames are integers, the speed is also an integer. Therefore, in this case, the spectrum having high energy in the three-dimensional FFT is on the theoretical line. However, when N _z <N _x / V _x or N _z > N _x / V _x , the analysis data is equal to data whose value is zero in the horizontal direction x and the time axis direction z. In such a case, since the data is not periodic in the analysis window, the spectrum is raised at a plurality of locations on the theoretical line.

Actually, in the case of the moving image shown in FIG. 4, when the size is (N _x × N _y × N _z ) = (64 × 64 × 64) and the size of the object is 64 pixels × 8 pixels, and when the moving velocity V _x 1.0 pixels / frame, to determine the distribution of the spatial spectrum when the the case of a 3.3 pixel / frame. Note that the maximum value of the spectrum amplitude was set to 40 dB so that all the spectra were larger than 0 dB. Further, the initial phase to 1, the frequency _{f x,} the normalized f y, the _{f z,} respectively 64 Hz.

Figure 5 shows a spatial spectrum when the plane _f x -f _z of the three-dimensional FFT in a case where the moving speed _{V x} of the object was 1.0 pixels / frame. The broken line in FIG. 5 is a theoretical straight line, and if this straight line can be obtained accurately, an accurate moving speed can be obtained. In FIG. 5, since the resolution of the three-dimensional FFT is 1, which matches the moving speed of the object, the spectrum having high energy forms a straight line. However, when the moving speed V _{x of the} object is 3.3 pixels / frame which is not equal to the resolution of the three-dimensional FFT, the spectrum does not exist only on the theoretical line as shown in FIG. It also exists around it. Therefore, it is important to consider the energy of the spectrum that is not on the theoretical line in order to obtain an accurate speed.

On the other hand, when the moving speed V _{x of the} object is 3.3 pixels / frame, the spatiotemporal spectrum obtained using the frequency analysis method described above is on a theoretical line as shown in FIG. It existed, but it did not exist around it. Although not shown, even when the moving speed V _x of the object is 1.0 pixels / frame, likewise, distributed space-time spectrum is present theoretical straight line is obtained.

In order to quantitatively evaluate these results, the inventor of the present application determined the variance between the spatiotemporal spectrum and the theoretical line for each of the three-dimensional FFT and the present frequency analysis method. The variance σ ² was obtained by the following equation (26). Note that in the following equation _{(26), f x ',} f z' are each a spectrum parameter of a three-dimensional FFT or the frequency analysis technique, L is the number of spectrum.

The higher the variance sigma ² is small, the error becomes smaller with respect to the theoretical straight line. In the case of the three-dimensional FFT, the variance σ ² is 560.0, and in the case of this frequency analysis method, the variance is as small as 0.0268. The inventor of the present application converges the dispersion σ ² to about 570.0 in the case of a three-dimensional FFT for a spectrum having an amplitude of about 40 dB to 60 dB, while the dispersion σ ² is about 0 in the case of this frequency analysis method. It is confirmed that it converges to .03.

  Therefore, it can be seen that the spectrum obtained by the present frequency analysis method is almost on the theoretical line. If the period of the moving object is equal to the length of the analysis window, the spectrum by the three-dimensional FFT forms a straight line. Conversely, otherwise, the spectrum does not take the form of a straight line. It can be seen that the motion estimation apparatus to which this frequency analysis method is applied can always obtain an accurate spectrum regardless of the size of the analysis window, and can perform motion estimation with high accuracy.

[Second verification]
In the first verification, the case where the object moves only in one direction has been shown. As an example of the case where the object moves in two directions, as shown in FIG. 8, a black square object at the lower left of the initial frame is shown. We considered a moving image that moves to the upper right at a uniform speed in the vertical and horizontal directions as time progresses. Such movement of the object is expressed by superposition of horizontal bar-shaped objects that move upward from below the frame in the movement of the object as shown in FIG. When the horizontal moving speed is V _x and the vertical moving speed is V _y , the spatiotemporal spectrum having high energy in the three-dimensional FFT is excluding the conjugate spectrum on the plane in the three-dimensional frequency domain space. It will be on this plane. This plane is defined as a theoretical plane such as the following equation (27). Therefore, the two velocity _V x, _{V y} is the frequency _f x in theory plane equation is determined by calculating the coefficients of _{f z.}

Here, when N _z is equal to the least common multiple of N _x / V _x , N _y / V _y , the moving image data is substantially periodic, and most of the spectrum by the three-dimensional FFT is on the theoretical plane. It will exist nearby. However, when N _z is not equal to the least common multiple of N _x / V _x , N _y / V _y , the spatiotemporal spectrum is considered to exist everywhere, not only in the theoretical plane. Therefore, it is presumed that the spatiotemporal spectrum by the three-dimensional FFT is affected by the analysis window. The inventor of the present application tried to solve this problem by using the frequency analysis method described above and setting the speed to an integer in order to test the resolution of the frequency analysis method.

Actually, as the moving image shown in FIG. 8, when the size is (N _x × N _y × N _z ) = (64 × 64 × 64) and the size of the object is 8 pixels × 8 pixels, The spatio-temporal spectrum distribution was obtained for the case where the moving speed of V _x = V _y = 3.3 pixels / frame. In order to clarify the observation of the spectrum distribution in the three-dimensional frequency region, the maximum value of the spectrum amplitude was set to 20 dB so that all spectra were larger than 0 dB.

9 and 10 show spatiotemporal spectra obtained using a three-dimensional FFT and the present frequency analysis method in the three-dimensional frequency domain, respectively. 9 and 10, the plane formed by the three broken lines is a theoretical plane, and the parameters V _x and V _y in the above equation (27), that is, the moving speed of the object is 3.3. 9 and 10 show the relationship between the obtained spatiotemporal spectrum and the theoretical plane.

A part of the spatio-temporal spectrum by the three-dimensional FFT exists on the theoretical plane, but most of the spectrum exists near the theoretical plane. On the other hand, most of the spatio-temporal spectrum obtained by this frequency analysis method exists on the theoretical plane. The variance σ ² can be obtained by the following equation (28).

In the case of the three-dimensional FFT, the variance σ ² is 11.66, and in the case of this frequency analysis method, it is an extremely small value of 0.01. Although not particularly shown, when the size of the object is as large as 32 pixels × 32 pixels, the result that the variance σ ² increases logarithmically is obtained.

  Thus, it can be seen that the motion estimation apparatus to which the present frequency analysis method is applied is also useful for obtaining an accurate spatio-temporal spectrum for the movement of an object having a velocity in two directions.

[Third verification]
Next, as an example of a moving image including two objects moving at the same time, as shown in FIG. 11, two black square objects a and b at the lower left and lower right of the initial frame are respectively timed. Along with this, we considered moving images that move to the upper right and upper left at different speeds that do not match the resolution. Note that the size of the moving image is (N _x × N _y × N _z ) = (64 × 64 × 64), and the size of the object is 8 pixels × 8 pixels.

  12 and 13 show the spatio-temporal spectra obtained using the three-dimensional FFT and the present frequency analysis method in the three-dimensional frequency domain, respectively.

  From FIG. 12, the spatio-temporal spectrum obtained by the three-dimensional FFT has insufficient resolution, and cannot be extracted accurately. Therefore, in the case of a moving image including an object moving at a speed that does not match the resolution. It can be seen that the spectrum energy leaks around the ideal location, and the spectrum distribution is like a layer having a thickness in which the planes are stacked. When such a layered spectrum distribution is obtained, it is difficult to estimate the plane by separating these spectra. That is, in an analysis method having an equally spaced frequency resolution represented by DFT and FFT, an accurate spectrum can be obtained as long as the frequency component matches the equally spaced frequency. Spectrum cannot be obtained, and a spectrum swarmed at frequencies around the theoretical plane appears, resulting in a layered spectrum distribution.

On the other hand, the spatio-temporal spectrum obtained by this frequency analysis method is distributed in two planes corresponding to two objects as shown in FIG. 13, and there is almost no variation. Such two planes can be easily estimated using the least square method as described above. FIG. 14 shows a state where the plane is drawn by changing the viewpoint so that the two planes can be observed well. From this figure, it can be seen that the two planes are estimated with high accuracy. In the motion estimation device, the horizontal and vertical velocities of the object can be estimated by obtaining the inclinations of the estimated plane _fx axis and _fy axis, respectively.

  Note that the above-described layered spectrum distribution in the spatiotemporal spectrum by the three-dimensional FFT is particularly noticeable in the case of a moving image including a plurality of similar motions. The inventor of the present application has also conducted an experiment for confirming the effectiveness of the present frequency analysis method in a moving image including a plurality of similar motions.

When the size of the moving image is (N _x × N _y × N _z ) = (64 × 64 × 64) and the size of the object is 8 pixels × 8 pixels, as shown in FIG. A moving image in which two black square-shaped objects at the lower left in the initial frame move to the upper right at a uniform speed in the vertical and horizontal directions as time progresses was targeted. Assuming that two objects are a and b, the moving speed of the object a is V _x = 1.0 pixel / frame, V _y = 1.7 pixels / frame, and the moving speed of the object b is V _x = 1. 0.0 pixels / frame, V _y = 2.3 pixels / frame. That is, these two objects a and b start from the same place and move gradually away from each other in the height direction while moving in the upper right direction as time progresses. The maximum amplitude of the spectrum was set to 28 dB so that all the spectra were larger than 0 dB.

In both the three-dimensional FFT and the present frequency analysis method, distributions were obtained using 200 spectra, and the results shown in FIGS. 15 and 16 were obtained, respectively. Then, when the inclination of the plane estimated from this result, that is, the moving speeds of the objects a and b are obtained, in the case of the three-dimensional FFT,
For object a, V _x = 1.0, V _y = 1.8545
For object b, V _x = 1.0, V _y = 1.5484
In the case of this frequency analysis method,
For object a, V _x = 0.99999, V _y = 1.7211
For object b, V _x = 0.98712, V _y = 2.3108
was gotten. That is, it can be seen that when the three-dimensional FFT is used, the accuracy of the moving speed V _{y in} the vertical direction is poor, and when the frequency analysis method is used, the motion is estimated with extremely high accuracy.

Similarly, the moving speed of the object a is changed to V _x = 1.0 pixel / frame and V _y = 2.3 pixels / frame, and the moving speed of the object b is changed to V _x = 1.7 pixel / frame, The spectrum distribution was obtained in the same manner while changing to V _y = 3.2 pixels / frame. Again, the maximum value of the spectrum amplitude was set to 28 dB so that all the spectra were larger than 0 dB.

When the distribution was obtained using 300 spectra in the case of the three-dimensional FFT and 200 spectra in the case of this frequency analysis method, the results shown in FIGS. 17 and 18 were obtained, respectively. Then, when the inclination of the plane estimated from this result, that is, the moving speeds of the objects a and b are obtained, in the case of the three-dimensional FFT,
For object a, V _x = 1.0034, V _y = 2.1525
For object b, V _x = 1.8053, V _y = 3.5693
In the case of this frequency analysis method,
For object a, V _x = 1.0106, V _y = 2.3116
For object b, V _x = 1.7502, V _y = 3.2304
In this case as well, a better result was obtained when this frequency analysis method was used than when a three-dimensional FFT was used.

  As described above, the motion estimation apparatus to which the present frequency analysis method is applied is extremely effective when performing motion estimation in a moving image including a plurality of similar motions.

[Fourth verification]
Next, as an example of a moving image including three objects that move at the same time, as shown in FIG. 19, three black square objects a, b, and c on the upper right, lower left, and lower right of the initial frame are respectively We considered moving images that move to the lower left, upper right, and upper left at different speeds as time progresses. Note that the size of the moving image is (N _x × N _y × N _z ) = (64 × 64 × 64), and the size of the object is 8 pixels × 8 pixels. The moving speed of the object a is V _x = −1.0 pixel / frame and V _y = −1.0 pixel / frame, and the moving speed of the object b is V _x = 1.3 pixel / frame, V _y = 2.7 pixels / frame, and the moving speed of the object c is V _x = −3.3 pixels / frame and V _y = 2.5 pixels / frame. For such a moving image, the spatio-temporal spectrum was obtained using this frequency analysis method, and the plane was estimated while changing the number of spectra to be used.

First, when 100 spectra are used, a result as shown in FIG. 20 is obtained, and the estimated inclinations of the three planes, that is, the moving speeds of the objects a, b, and c are as follows.
For object a, V _x = -1.0038, V _y = -1.0016
For object b, V _x = 1.2353, V _y = 2.6933
However, the moving speed of the object c cannot be obtained because the number of spectra is small.

Further, when 125 spectra are used, a result as shown in FIG. 21 is obtained, and the moving speeds of the objects a, b, and c are as follows.
For object a, V _x = -1.0038, V _y = -1.0024
For object b, V _x = 1.2593, V _y = 2.7166
For object c, V _x = −3.2332, V _y = 2.452
was gotten.

Furthermore, when 160 spectra are used, a result as shown in FIG. 22 is obtained, and the moving speeds of the objects a, b, and c are as follows.
For object a, V _x = −1.0053, V _y = −1.0015
For object b, V _x = 1.2948, V _y = 2.7012.
For object c, V _x = −3.2897, V _y = 2.4594
was gotten.

Furthermore, when 180 spectra are used, the result shown in FIG. 23 is obtained, and the moving speeds of the objects a, b, and c are as follows.
For object a, V _x = −1.003, V _y = −1.0013
For object b, V _x = 1.3085, V _y = 2.6974
For object c, V _x = −3.2906, V _y = 2.4561
was gotten.

In addition, when 200 spectra are used, the result shown in FIG. 24 is obtained, and the moving speeds of the objects a, b, and c are as follows.
For object a, V _x = -1.0024, V _y = -1.0008
For object b, V _x = 1.3104, V _y = 2.7007
For object c, V _x = −3.2874, V _y = 2.4595
was gotten.

Furthermore, when 225 spectra are used, the result shown in FIG. 25 is obtained, and the moving speeds of the objects a, b, and c are as follows.
For object a, V _x = −1.0023, V _y = −1.0011
For object b, V _x = 1.3093, V _y = 2.6994
For object c, V _x = −3.2998, V _y = 2.4494
was gotten.

When 250 spectra are used, the result shown in FIG. 26 is obtained, and the moving speeds of the objects a, b, and c are as follows.
For object a, V _x = −1.0019, V _y = −1.001
For object b, V _x = 1.3091, V _y = 2.683
For object c, V _x = −3.2756, V _y = 2.4747
was gotten.

  That is, the accuracy of the motion estimation apparatus to which this frequency analysis method is applied depends on the number of spatiotemporal spectra, and in the case of the moving image shown in FIG. 19, it is sufficient to use about 180 spatiotemporal spectra. It can be seen that high accuracy can be realized. Note that the number of spatio-temporal spectra that can achieve this sufficient accuracy differs depending on the content of the moving image, such as the speed and moving direction of the moving object. In other words, in the motion estimation device, it is possible to perform motion estimation with high accuracy while reducing the amount of computation by using an appropriate spatiotemporal spectrum according to the moving image.

[Fifth verification]
Next, as an example of the case where an object as a foreground of a moving image with a background that does not move moves, as shown in FIG. 27, a black square object at the lower left of the initial frame moves to the upper right as time progresses. I thought about moving images. Note that the size of the moving image is (N _x × N _y × N _z ) = (64 × 64 × 64), the size of the object is 8 pixels × 8 pixels, and the moving speed of the object is V _x = _Vy = 3.3 pixels / frame. The background is composed of a predetermined repeating pattern. Specifically, the background is a pattern of amplitude A = 128 (0-255), phase φ = 0, frequency f _x = 8, f _y = 0, f _z = 1, and amplitude A = 128 (0-255). ), A pattern of phase φ = 0, frequency f _x = 0, f _y = 8, and f _z = 1. For such a moving image, a space-time spectrum was obtained using each of the three-dimensional FFT and the present frequency analysis method, and the plane was estimated. Note that the maximum value of the spectrum amplitude was set to 25.07 dB in the case of the three-dimensional FFT, and 22.75 dB in the case of this frequency analysis method, so that all the spectra were larger than 0 dB.

First, when the distribution of 101 spatio-temporal spectra by three-dimensional FFT was obtained, an intermittent distribution was obtained under the influence of the background pattern as shown in FIG. From this result, when the moving speed of the object is obtained while changing the threshold value ε described above,
When the threshold ε = 0.1, V _x = 3.4697, V _y = 3.4697
When the threshold ε = 0.2, V _x = 3.3824, V _y = 3.3824
When the threshold ε = 0.3 to 0.5, V _x = 3.2857, V _y = 3.2857
As described above, a result with a large error was obtained regardless of the threshold value ε.

On the other hand, when the distribution of 100 spatio-temporal spectra obtained by this frequency analysis method was obtained, a continuous distribution was obtained regardless of the background pattern as shown in FIG. From this result, when the moving speed of the object is obtained while changing the threshold value ε described above,
When the threshold ε = 0.1, V _x = 3.3093 and V _y = 3.3033
When the threshold ε = 0.2, V _x = 3.306, V _y = 3.3053
When the threshold ε = 0.3 to 0.5, V _x = 3.3037, V _y = 3.3303
As described above, a result with extremely small error was obtained.

  Thus, the motion estimation apparatus to which the present frequency analysis method is applied is not easily affected by the background, and can perform motion estimation with high accuracy.

[Sixth verification]
Next, as an example of the case where the foreground moves in a color moving image, a moving image in which a person moves from left to right as shown in FIG. 30 was considered. Note that the size of the moving image is (N _x × N _y × N _z ) = (64 × 64 × 64), and the moving speed of the person is V _x = 1.641 pixels / frame. The moving image is composed of a thermal image captured in an environment having a predetermined temperature distribution. For such a moving image, a space-time spectrum was obtained using each of the three-dimensional FFT and the present frequency analysis method, and the plane was estimated. Note that the maximum value of the spectrum amplitude was set to 19.4 dB in the case of the three-dimensional FFT, and 22.58 dB in the case of the present frequency analysis method, so that all the spectra were larger than 0 dB.

First, when the distribution of 101 spatio-temporal spectra by three-dimensional FFT is obtained, a result as shown in FIG. 31 is obtained. From this result, the movement speed of the person is obtained while changing the threshold value ε described above.
When the threshold ε = 0.4, V _x = 1.4838
When the threshold ε = 0.5, V _x = 1.5732
As described above, a result with a large error was obtained regardless of the threshold value ε.

On the other hand, when the distribution of 100 spatio-temporal spectra obtained by this frequency analysis method is obtained, the result shown in FIG. 32 is obtained. From this result, when the moving speed of the person is obtained while changing the threshold value ε,
When the threshold ε = 0.4, V _x = 1.6452
When the threshold ε = 0.5, V _x = 1.07027
As described above, a result with extremely small error was obtained.

  As described above, the motion estimation apparatus to which the frequency analysis method is applied can perform motion estimation with high accuracy even in the case of a color moving image. In particular, in a motion estimation apparatus, the accuracy of motion estimation can be improved by using an appropriate threshold value ε according to a moving image.

[Seventh verification]
Next, as an example when applied to a natural moving image, as shown in FIG.
A standard video called “Guard” was used. In this moving image, the boat which is the foreground is almost stationary, but is observed so that the background moves to the left by camera work. In this experiment, the amount of boat movement between 32 frames from the 130th frame to the 161st frame was estimated.

First, when obtaining the distribution of the spatial spectrum when by 3D FFT, as shown in FIG. 34, variation seems errors were observed conspicuously low frequency region of the frequency f _x, f _y. On the other hand, when the distribution of 100 spatio-temporal spectra obtained by the present frequency analysis method was obtained, as shown in FIG. From these results, when the plane was estimated by the clustering method using the least square method, the results shown in FIGS. 36 and 37 were obtained for the three-dimensional FFT and the present frequency analysis method, respectively. When the speed was obtained based on the obtained inclination of the plane, the results shown in the following table 1 were obtained.

  In the case of 3D FFT, the background could be estimated, but there was an error in the estimated speed result, and the boat could not be estimated, and the wrong speed was estimated. Recognize. On the other hand, in the case of this frequency analysis method, it was confirmed that the error was small.

  As described above, the motion estimation apparatus to which the frequency analysis method is applied can perform motion estimation with high accuracy even in the case of a natural moving image.

[Effect of motion estimation device]
As described above, the motion estimation apparatus to which the present frequency analysis method is applied performs processing such as motion compensation, motion prediction, and interpolation in moving image coding, and greatly reduces the amount of computation and performs processing with high accuracy. Therefore, this motion estimation apparatus is suitable for use when estimating / predicting the motion of an object included in a moving image, performing frame interpolation, or the like.

  In addition, this motion estimation device can be significantly compressed as compared with the prior art by applying it to video encoding. Furthermore, this motion estimation apparatus can be used for development of a scalable encoding technique that can cope with qualitative and quantitative changes in processing without adding a large-scale change. Thereby, this motion estimation apparatus can contribute to the reduction of the network traffic which is increasing recently.

  The present invention is not limited to the embodiment described above. For example, in the above-described embodiment, the description has been made on the assumption that the motion estimation apparatus performs frequency analysis by software. However, the present invention is a DSP (Digital Signal Processor) or the like that implements a motion estimation processing algorithm including the frequency analysis method. If the product-sum operation can be performed, it can be realized by hardware.

  Thus, it goes without saying that the present invention can be modified as appropriate without departing from the spirit of the present invention.

11 CPU
12 ROM
13 RAM
14 storage unit 15 input operation control unit 16 display unit

Claims (3)

The moving image data to be analyzed is such that the sum of squares of the difference between the three-dimensional signal of the moving image data and the sinusoidal model signal represented by the phase and amplitude using the frequency and the initial phase becomes a minimum value. The frequency, the amplitude, and the initial phase are obtained as parameters of a Fourier transform formula of an aperiodic signal, a spatiotemporal spectrum is extracted, and the extracted spatiotemporal spectrum distribution is divided for each spectrum group forming a plane. A motion estimation method for estimating the motion of one or a plurality of moving objects in the moving image by estimating one or a plurality of plane groups and obtaining an inclination of the estimated plane group,
Clustering using the least squares method is performed on the spatiotemporal spectrum, and when estimating one or a plurality of plane groups, the existence probability variable when the spatiotemporal spectrum belongs to a predetermined plane is set to 1 or A motion estimation method, characterized in that it is set to a decimal number and clustering is performed using the least squares method. A moving image input means for inputting moving image data to be analyzed, a three-dimensional signal of the input moving image data, and a sine wave model signal represented by a phase and an amplitude using a frequency and an initial phase A spatio-temporal spectrum extracting means for extracting the spatiotemporal spectrum by obtaining the frequency, the amplitude and the initial phase such that the sum of squares of the difference is a minimum value as a parameter of a Fourier transform formula of the non-periodic signal; A plane estimation unit that estimates one or a plurality of plane groups by dividing the spatio-temporal spectrum distribution for each spectrum group that forms a plane, and an inclination calculation unit that calculates an inclination of the estimated plane group. A motion estimation device for estimating the motion of one or more moving objects in the moving image,
Clustering using the least squares method is performed on the spatiotemporal spectrum, and when estimating one or a plurality of plane groups, the existence probability variable when the spatiotemporal spectrum belongs to a predetermined plane is set to 1 or A motion estimation apparatus, characterized in that it is configured to perform clustering using a least-squares method set to a decimal number . A moving image input means for inputting moving image data to be analyzed, a three-dimensional signal of the input moving image data, and a sine wave model signal represented by a phase and an amplitude using a frequency and an initial phase A spatio-temporal spectrum extraction means for extracting the spatiotemporal spectrum by obtaining the frequency, the amplitude and the initial phase at which the sum of squares of the differences becomes a minimum value as a parameter of a Fourier transform formula of the aperiodic signal, and the spatiotemporal spectrum A computer is caused to function as plane estimation means for estimating one or a plurality of plane groups by dividing the distribution for each spectrum group forming a plane, and an inclination calculation means for obtaining an inclination of the estimated plane group, A computer-executable motion estimation program for estimating the motion of one or more moving objects in an image,
Clustering using the least squares method is performed on the spatiotemporal spectrum, and when estimating one or a plurality of plane groups, the existence probability variable when the spatiotemporal spectrum belongs to a predetermined plane is set to 1 or A motion estimation program that is set to a decimal number and performs clustering using the least squares method .

Images