JP2007228195A - Device, program, and method for automatically imparting video attribute - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automate edition when a specific image such as a highlight scene is clipped from the captured image of a moving object such as a runner in marathon game and selection of a section and edition are performed, by extracting a target scene automatically using the attribute information of the moving object from an acquired image. <P>SOLUTION: An RF tag recording the information of a moving object is attached to a moving object, attribute information of the moving object is generated from RF tag data received from the RF tag, the region of the moving object existing in the captured image is pursued, the moving object is specified by separating and identifying the moving object for every individual moving objects, and the attribute information of the specified moving object is imparted automatically to the captured image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a video editing device for generating a replay video by cutting out a highlight scene and the like at the same time as performing live broadcast of a sports video or the like. In particular, the present invention relates to an apparatus that automatically generates and edits attributes of each scene of a captured image.

  In recent years, it has become easy to distribute and broadcast digital contents using an Internet line or the like. Corresponding to this, there is a demand for an apparatus for processing and editing a large amount of video material in a short time.

Nonlinear editing apparatuses are known as video editing apparatuses. Non-linear editing devices convert video captured by a video camera or video recorded on a tape into digital data, which are captured and processed on a hard disk of a personal computer. For example, to cut a marathon competition finish scene, play the video stored on the hard disk and manually set the cut start and end positions and save the set section as a separate data file, or video tape To create a replay video.
However, in the non-linear editing apparatus, editing operations such as specifying a cut-out section and storing a replay video need to be performed manually, and thus there is a problem that much labor and cost are required.

  As a countermeasure to the problem of the non-linear editing device, a method has been proposed in which an attribute is assigned to each scene of a video and an extraction section is automatically specified by searching attribute information.

For example, in Non-Patent Document 1, there is a method of recognizing ID (Identification) information emitted from an infrared LED (Light Emitting Diode) attached to a subject and indexing the stored video data in real time. However, since the LED is used as a signal source, there is a problem that the ID information cannot be received when there is no light receiver in the light emitting direction or when there is an obstacle between the LED and the light receiver.
In Patent Document 1, when recording a camera image, there is a method for preventing falsification of an image by reading ID information such as an RF (Radio Frequency) tag attached to a subject and storing it together with image data. In the present invention, a signal is received only from an RF tag to which the camera is directed by a directional antenna. That is, there is no specific means relating to the method of associating the directional antenna and the camera field of view, and there is a problem that it is not guaranteed whether or not the corresponding photographing object is reflected.
In Patent Document 2, there is a method of making it easy to find a desired article by detecting the direction of an RF tag using a directional antenna and irradiating light beams such as LEDs in the same direction. However, although a directional antenna is used, there is no means for associating the antenna with the camera or associating the RF tag ID information with the moving object.

Non-Patent Document 2 discloses image processing using a local correlation method as a tracking method of which region in an image a moving object exists. However, it is impossible to detect what kind of attribute the detected moving object has and to automatically add the attribute to the captured image in real time.
The inventor of the present application discloses a method of detecting a moving object from a captured image, cutting out a region where the moving object exists into a desired size, and associating the RF tag information with the detected moving object. There is no description of object identification.

JP 2000-261551 A JP 2002-271229 A Japanese Patent Application No. 2005-96158 Kaku, Ito, Matsuguchi, Sidney Fels, Utsumi, Suzuki, Nakahara, Iwasawa, Kogure, Mase, and Tomita, “Collaborative Interaction Recording with Multiple Sensors”, Interaction 2003 Proceedings, Information Processing Society of Japan, pages 255-262 . Morita, “Motion Detection and Tracking by Local Correlation”, IEICE Transactions D-II, Vol. J84-D-II, NO2, pages 299-309, 2001. As described above, any known technology associates a directional antenna with a camera, and automatically links information of an RF tag attached to a moving subject to cut out a highlight scene of a captured image. Not disclosed.

  The problem to be solved is a problem that it takes man-hours, man-hours and time to select and edit a cutout section of a specific image such as a highlight scene from a photographed image of a moving object.

  1st invention attaches | subjects to the said image the attribute which matched the information of the said RF tag received with the moving object detected from the image which attached | attached the RF tag and image | photographed continuously and the radio | wireless antenna to the said image This is a video attribute automatic assigning device.

  The video attribute automatic assigning device generates means for receiving the captured image, means for receiving the RF tag information from the antenna, and attribute information for generating an attribute of the moving object attached to the RF tag from the received RF tag information. A moving object detecting means for detecting a moving section of a grid-like section in which a predetermined number of pixels constituting the received image are collected, and a section of a previous frame image of the section of the current frame image in which the motion is detected; Extracting a section where a moving object may be present from the speed and direction of movement of the area, a region extracting means for extracting a region combining the adjacent sections having the same motion, and the center of the extracted region Calculating the position, calculating the movement vector from the previous frame image of the center position, recognizing the moving object, and attaching the RF tag attached to the moving object in the extracted region And an attribute information addition means for automatically imparting attributes outgoing confirmed the RF tag moving object and the moving object identification device and the identified said image to identify the moving object.

  According to a second aspect of the present invention, in the video attribute automatic assigning apparatus according to the first aspect of the invention, the information from the RF tag is controlled by controlling the directional beam of the directional antenna based on the information from the attribute information generating means. An automatic video attribute assigning device further comprising an RF tag reader / writer equipped with a receiving RF tag receiving radio antenna.

  In a third aspect of the present invention, when there are a plurality of moving objects extracted by the moving object specifying means and the RF tag cannot be confirmed in the video attribute automatic assigning apparatus of the second invention, the position is located near the center of the area extracted by the area extracting means. The automatic image attribute assigning device further includes means for controlling the directional beam to receive information from the RF tag, separate the plurality of moving objects, and identify and identify the moving objects.

  According to a fourth aspect of the present invention, there is provided the moving object identified by separating the moving object into the image before the separation of the moving object identified by separating the plurality of moving objects in the video attribute automatic assigning apparatus of the third aspect of the invention. The video attribute automatic assigning device further includes means for automatically assigning an attribute of the RF tag.

  According to a fifth aspect of the present invention, there is provided a program for assigning to the image an attribute that associates the information of the RF tag with a moving object present in an image obtained by continuously capturing an object that is moving with an RF tag attached thereto by a computer. is there.

  The program generates an attribute of the moving object attached with the RF tag from the RF tag information, detects a moving section of a grid-like section in which a predetermined number of pixels constituting the image are collected, and detects the movement. Extract a section where a moving object may exist from the speed and direction of movement from the section of the previous frame image of the section of the current frame image, and combine the adjacent sections with the same motion Extracting, calculating a center position of the extracted region, calculating a movement vector from the previous frame image of the center position to recognize a moving object, and the RF tag attached to the moving object of the extracted region Is extracted, the moving object is specified, and attribute information for automatically assigning the attribute of the RF tag of the specified moving object is added to the image.

  According to the present invention, editing processing can be automated by automatically extracting a target scene using attribute information of a moving object from an acquired image. In addition, even when the angle detection resolution of the wireless antenna is rough, it is possible to accurately detect moving objects based on the editing processing result of the captured image, and the directional beam boundary of the antenna can be rotated to the intermediate position of multiple moving objects. By doing so, a plurality of moving objects can be separated and detected. Further, the angular resolution can be improved by using a plurality of directional antennas, and attribute data can be automatically given to an image area occupied by the moving object. Thereby, it is possible to efficiently edit the distribution / broadcast of the photographed image.

Example 1
FIG. 1 is a diagram showing a block configuration of a video attribute transmission system and a video attribute automatic assigning apparatus according to an embodiment of the present invention. A shooting object 2 that is a moving object equipped with an RF tag 21, a camera 3 that photographs an imaging object that is equipped with the RF tag 21, an RF tag reader / writer 4 that reads information of the RF tag 21 to be photographed and writes and holds it in a memory; The video attribute automatic assigning device 1 for identifying and processing the attribute information of the subject from the information of the RF tag 21 and the image from the camera, the network 5 for transmitting the information of the identified RF tag 21 to the transmission destination of the video device 7 or The recording medium 6 is a DVD or CDR. Note that it is also conceivable that the video attribute automatic assigning apparatus 1 in FIG. 1 is combined with the RF tag reader / writer 4 to form the video attribute automatic assigning apparatus.

The RF tag 21 is a device used for identifying an object that is also called a wireless IC tag or the like. The RF tag 21 includes a wireless IC chip antenna coil and can store information such as its own identification code. The RF tag 21 can wirelessly communicate with a device having functions such as reading and storing the RF tag information. The RF tag 21 may be configured as either a type that obtains driving power from electromagnetic waves supplied from the RF tag reader / writer 4 or a battery built-in type. When the RF tag 21 of the present application is worn on a person, it is desirable that the RF tag 21 be worn on the head or chest where movement is relatively stable.
In the memory in the RF tag 21, the name, characteristics (color, shape, ID number, etc.), specification, or network address in which these are described, depending on the object 2 to which the RF tag 21 is attached. Etc. are stored in advance. By using a writable RF tag, it is possible to rewrite the data contents stored in the memory while photographing with the camera 3 as necessary. An example of data of the RF tag 21 in one embodiment applied to the present application is shown in FIG. The description will be described later.

The RF tag reader / writer 4 includes a directional antenna 41, an A / D 42, and an I / F 43 that is an interface with the video attribute automatic assigning apparatus 1. An adaptive array antenna is used as the directional antenna 41, and the directivity of the adaptive array antenna is controlled to receive radio waves from the RF tag 21.
The modulated signal from the RF tag 21 is demodulated from the signal y (n) received by the directional antenna 41, and the analog signal is converted into a digital code by the A / D 42 to obtain a digital signal. The digital data is sent to the video attribute automatic assigning apparatus 1 via the interface unit of the I / F 43. Further, when data is written, information from the video attribute automatic assigning device 1 is written in the memory of the RF tag 21. For the sake of simplicity, FIG. 1 shows a unidirectional signal flow from the RF tag 21 to the video attribute automatic assigning apparatus 1, and the reverse direction is omitted.
The camera 3 forms an image at a constant sampling interval (for example, 33 ms) from the light intensity detected by the CCD or the like, and further generates a moving image. In the case of a color image, the pixel that is a unit of an image is given by gradation values of a red component (R), a green component (G), and a blue component (B). In the case of a black-and-white image, it is given as a luminance gradation value. For example, the gradation values of the red component (R), green component (G), blue component (B), and luminance (I) of the pixel at coordinates (x, y) indicated by integers x and y are digital values R, respectively. (X, y), G (x, y), B (x, y), and I (x, y). For the sake of simplicity, the following description will be made on the luminance (monochrome gray value) as a pixel for video processing. Image processing can be similarly performed on pixels of a color image.
Image information from the camera 3 and information on the RF tag 21 from the RF tag reader / writer 4 are input to the video attribute automatic assigning device 1 via the signal input I / F 20. The video attribute automatic assigning device 1 processes the information of the RF tag 21 inputted through the signal input I / F 20 and the image data from the camera, and the processed data is sent to the video device 7 through the signal output I / F 30 and the network 5. Send to. Alternatively, it is sent to a recording medium such as a DVD (Digital Video Disc) or a CDR (Compact Disc Recordable) via the signal output I / F 30.

  Before describing the configuration of the video attribute automatic assigning apparatus 1 and the contents of image processing, the directivity control of the directional antenna and the positional relationship between the camera and the directional antenna will be described. First, directivity control of the directional antenna will be described. The direction of arrival of radio waves from the RF tag 21 is defined as θ. When the antenna elements of the adaptive array antenna are arranged in a straight line at equal intervals D, the propagation delay time τi (θ) to the i-th antenna element when compared with the reference antenna element is

It is represented by c is the propagation velocity. When the modulation signal is m (n), the angular frequency of the carrier wave is ω, the i-th antenna element output is xi (n), and n is time,

It becomes.

  If the desired signal waveform is known and can be used as the reference signal d (n), the weighting factor can be determined by minimizing the mean square error u (n) from the output signal y (n) of the antenna array. u (n) is an error component such as interference and noise.

  The input signal xi (n) and weighting coefficient w i (θ) of each antenna element are expressed by vectors, and the combined output of each antenna element is substituted into the above equation as y (n).

  The subscript T represents transposition, and H represents complex conjugate transposition. At this time, the expected value J of the square of the error component is expressed by the following equation.

  The autocorrelation matrix of the input signal of each antenna element is Rxx = E {x (n) × d (n) H}, and the cross-correlation matrix of the reference signal and the input signal of each antenna element is rxd = E {x (n) Xd (n)}, the optimum weight is obtained by partial differentiation of the expected value J with the weight vector wi (θ),

Given in.

  As described above, the radio wave arrival direction θ can be associated with the weighting factor wi (θ). Therefore, when the directivity of the antenna element is changed in the specified range θ1 <θ <θ2, the corresponding weighting factor gradually changes within the range of w (θ1) to w (θ2), so that the direction of the antenna element Can be scanned. For example, when the direction of the antenna element is swung with a step size Δθ,

The directivity pattern of the antenna element can be controlled by changing the weighting coefficient w i (θ) according to the step value k · Δθ. The directional antenna may be composed of an antenna having a fixed directional pattern such as a Yagi antenna and a mechanism for mechanically rotating the antenna. In the case of this configuration, processing equivalent to the above-described adaptive array antenna can be realized by changing the rotation angle θ of the antenna in the specified range θ1 <θ <θ2.

  Next, the positional relationship between the directional antenna 41 and the camera 3 will be described. When detecting the imaging object 2 to which the target RF tag 21 is attached, a process of associating the radio wave arrival direction θ from the imaging object 2 obtained by the above method with a gaze area in a camera image, which will be described later, is performed. At this time, the subject 2, the directional antenna 41, and the camera 3 need to be in the same straight line. Since the position of the subject 2 is not specified, it is desirable that the center position of the antenna and the focal position of the camera lens coincide on the vertical axis in order to satisfy the condition of the same straight line.

The configuration and image processing of the video attribute automatic assigning apparatus 1 will be described. The video attribute automatic assigning apparatus 1 includes an attribute information automatic assigning unit 10, a signal input I / F 20, and a signal output I / F 30. The RF tag information and the image information received via the signal input I / F 20 are input to the attribute information automatic adding unit 10. The attribute information automatic adding unit 10 includes an attribute information generating unit 11, a moving object detecting unit 12, a region extracting unit 13, a moving object specifying unit 14, an attribute information adding unit 15, a received image, a video attribute adding image, and an image processing process. It comprises a memory 16 that holds data.
The attribute information automatic assigning device 1 may be an arithmetic processing device such as a personal computer with a built-in CPU that operates according to a program that processes each function of the attribute information automatic assigning unit 10 described later.

  Before describing a specific processing procedure, a method for separating and detecting a moving object from an image used in the present invention will be described. Background differential methods and local correlation methods are known as methods for separating and detecting moving objects from captured images.

  The background subtraction method is a method of performing image processing by passing a low-pass filter through an image from a video camera captured at a constant period (every sampling time). In separation detection of a moving object from a captured image, an image that does not include a moving object is prepared in advance as a background image, and then a difference from an image that includes a moving object is obtained. In this method, a portion having a large difference is used as a moving object.

Moving object separation detection by the background subtraction method uses a low-pass filter with a low cutoff frequency to obtain an image of an object that does not move, that is, a background image, and eliminates the flickering of the fluorescent lamp by increasing the cutoff frequency. This is done by acquiring an image containing a moving object and taking the difference between the two. Although this method can be processed in a shorter processing time, it has a drawback that it is vulnerable to noise such as brightness fluctuation.
On the other hand, the separation detection of the moving object by the local correlation method is a method that is performed based on the moving speed and moving distance of the pixel by the correlation calculation of the pixel of the current frame and the previous frame of the image. The procedure divides the image into grid-like blocks (for example, 8 × 8 pixels), and calculates the correlation value between neighboring blocks between a frame at a certain sampling time and a frame acquired at the next sampling time. Then, it is determined whether or not the block has moved according to the size. The separation and detection of a moving object from a captured image by the local correlation method has a problem in that it cannot follow a fast-moving object because the image size cannot be increased because the amount of correlation calculation is large and the distance between blocks on which the correlation calculation is performed is limited.

FIG. 2 is a diagram showing a processing procedure of the video attribute transmission system of the present invention. Procedures for collecting moving images and RF tag information, detecting and identifying moving objects from the collected images, linking RF tag data and images of the identified moving objects, and delivering images with video attributes Is shown. Detailed processing contents of steps S8 and S9 performed by the video attribute automatic assigning apparatus will be described later.
S1: Attribute information is detected from RF tag information, and initial setting of image editing processing is performed.
A. The step value of the control step of the adaptive array antenna is set, and the weighting factor of the directional antenna control and the angle of the antenna are associated using a reference table prepared in advance from past data or the like.
A. A conversion rule for converting the RF tag data into the attribute value is set.
C. Write data to the RF tag.
D. A threshold value used for detecting a moving object is set. As the thresholds to be set, a background separation threshold Tth used for moving object region candidate extraction and a movement comparison threshold Qth used for moving object region determination are set.
E. Conditions necessary for image processing such as a cutoff frequency of the image processing low-pass filter, a determination amount Δx used for associating the RF tag of the filter with the moving object, and a frame continuation condition for specifying the moving object are set.
S2: The camera starts photographing a moving object with an RF tag, and transmits the photographed image to the video attribute automatic assigning device. If it is not real-time distribution, it is stored and held in a recording medium.
S3: The direction of the antenna is controlled by changing the step value of the adaptive antenna set in S1 and changing the weighting factor of the adaptive antenna. The control method is as described above.
S4: The presence / absence of an RF tag in the direction of the antenna is determined. If an RF tag is present, the RF tag data is sent to the RF tag reader / writer. If there is no RF tag, the process returns to S3 to continue the direction control of the adaptive antenna.
S5: Send RF tag data to the RF tag reader / writer.
S6: The RF tag reader / writer converts the received RF tag data into digital data, and transmits the digital data to the attribute information generation unit of the video attribute automatic assigning device.
S7: If it is confirmed that the antenna angle (hereinafter referred to as RF tag detection angle) when the RF tag is detected in S4 is within the camera viewing angle, the detected RF tag is considered to be valid, and video attribute automatic assignment in S8 Perform image processing on the device. If it is determined that the RF tag is invalid, the process returns to S3 to control the adaptive antenna.
S8: The video attribute automatic assigning apparatus performs the following processing with the video attribute automatic assigning unit as the center.
A. The RF tag data is converted into attribute information of a moving object attached with the RF tag.
I. The presence / absence of a moving object is detected in each frame image.
C. Determine the area where the moving object exists.
D. A moving object in the determined area is specified.
E. The moving object specified in the image and the RF attribute are linked.
S9: A video attribute-assigned image in which attribute information (moving object identification number, tag number, attribute value) of the moving object is linked to the image is transmitted to the video device of the distribution destination. Alternatively, it is stored and held in a recording medium.
S10: An image with video attribute is delivered in real time as an image with video attribute. Alternatively, the image attributed image recorded on the recording medium obtained offline is distributed.
The detailed contents of the processing procedure described in FIG. 2 and the operation of the attribute information automatic assigning apparatus 1 will be described together with the function contents of the constituent blocks of the apparatus.
1) Attribute Information Generation Unit Applying a predetermined conversion rule to RF tag data output from the RF tag reader / writer 4 to convert digital data into attribute values and generate attribute information. FIG. 3 is a diagram showing an example of conversion of RF tag data into attribute values in the present invention. It is an example of the write-in data to RF which applied this method to marathon competition. In order to quickly detect the RF tag data, it is desirable to limit the write data to the minimum data required for the application system. In this example, the following data is set in the RF tag.
A. RF tag number a. Runner number number c. Competition day d. Start time RF tag detection angle These data are stored in the RF tag as numerical values and character string data. These numeric values and character string data are read by the RF tag reader / writer 4. Each data is converted into its attributes and values. For example, 1 on page 1 indicates RF tag number 1, 20050101 on page 2 is attribute date and value is January 1, 2005, page 3 data is attribute start time, value is 10:00 00 minutes 00 seconds.
FIG. 4 is a diagram showing detection of an RF tag detection angle in the present invention. The lion-shaped curve in the figure indicates that the directional antenna 41-1 has a directional beam with an angle-resolving field of 30 degrees. When it is detected that the RF tag is in the region Q using a directional antenna with an angular resolution of 30 degrees, the detection angle of the RF tag is given as -15 degrees to 15 degrees. The attribute information is generated only from digital data corresponding to a valid RF tag.
Whether the RF tag is valid or invalid is determined by the following method. FIG. 5 is a diagram showing the relationship between the camera viewing angle and the antenna direction in the present invention. Each upper and lower limits of the current viewing angle of the camera 3 in the shooting, θmax, θmin, the angle theta _R directional antenna when performing read operations for the RF tag .theta.max, if during .theta.min, namely

When the condition is satisfied, the RF tag is determined to be valid.
2) Moving object detection unit The moving object detection includes the background difference method and the local correlation method described above. As described above, the background subtraction method can be processed in a shorter processing time, but has a drawback that it is vulnerable to noise such as brightness fluctuation. On the other hand, the local correlation method has a problem that it cannot follow a fast-moving object because the amount of correlation calculation between blocks is large. In the present application, both methods are used in combination in order to realize without increasing the data processing scale. That is, a procedure is used in which a rough moving object region is obtained by the background subtraction method and then the moving object region is finely determined by the local correlation method.
Define sub-partitions, sub-partition templates, and movement tracking blocks for image processing. FIG. 6 is a diagram showing the relationship between the sub-partition of an image, the sub-partition template, and the tracking block in the present invention. The sub section is composed of a square of vertical n pixels and horizontal n pixels. The sub-partition template is a group of sub-partitions and is an area where an object may exist, and is a hatched part in FIG. The tracking block is an area that is set so as to surround the sub-partition template and is provided between the upper, lower, left, and right m pixels outside the sub-partition template.

  An area where a moving object may exist is obtained using the background subtraction method, and then an effective sub-partition for moving object detection is calculated using the local correlation method. FIG. 7 is a diagram showing a moving object detection procedure in the present invention. The procedure of effective sub-section detection for confirming whether a moving object is included in a captured image is shown.

S21: Capture an image obtained by the camera at a constant cycle (every sampling time). In the case of real-time distribution, it is a real-time image from a camera, but an image from a storage medium offline may be used. The acquired current timing image is divided into two series and passed through low-pass filters having different cutoff frequencies.

S22: A background image of the current frame with no motion is set with the cut-off frequency of the low-pass filter set low. The cut-off frequency set here can be obtained together with the moving speed of the moving object, and can be set variably according to the target moving object. The background image is divided into sub-partitions (n × n pixel squares).

S23: The cut-off frequency of the low-pass filter is set higher than that of the low-pass filter of S22, and an image from which the influence of the flickering of the fluorescent lamp is removed is obtained while leaving a moving object. This cut-off frequency can be set variably in accordance with the target moving object, similarly to the cut-off frequency in S22. This image is the current image at the current sampling time. Similarly to S22, the image is divided into sub-partitions (squares of n × n pixels).

S24: One of sub-sections (squares of n × n pixels) at the same position of both the current image (hereinafter referred to as current frame image) and the background image is taken out.

S25: The sum T of the differences between the pixels of the sub-partitions at the same position in both images is calculated by the following calculation formula.

Here, I ^C (x, y) is the black and white gray value of the pixel position (x, y) in the current image, I ^B (x, y) is the black and white gray value of the pixel position (x, y) in the background image, and R _SUB represents a sub-partition. When the calculated value T is equal to or greater than Tth, it indicates that it is different from the background image, and it is determined that there is a moving sub-section with movement, and the process proceeds to S26, otherwise the process returns to S24 and the next sub-section is extracted.

S26: Using the local correlation method, find out from which sub-section of the previous frame image the moving sub-part determined to have moved in S25. Correlation calculation is used to determine which sub-section S _q (x ′, y ′) of the previous frame image has moved from the sub-frame S _p (x, y) of the current frame image focused as a moving sub-section. Explore. The correlation value C with the sub-partitions S _p (x, y) and S _q (x ′, y ′) is calculated by the following equation, and it is determined whether or not the movement has moved from a predetermined threshold value.

In the above equation, I _p (x, y), I _q (x ′, y ′) is the black and white gray value of the pixels in each sub-partition S _p (x, y), S _q (x ′, y ′), R _p and R _q indicate that they are in the subcompartment region.

  The determination is made using the zero comparison method of Non-Patent Document 1 described above. The difference Q between the correlation value C0 between the sub-part of interest and the sub-part of the previous frame at the same position and the minimum correlation value C1 obtained from the correlation calculation with all sub-parts within the scanning range is expressed by the following equation: And is compared with the threshold value Qth for movement determination.

S27: When Q is larger than the threshold value Qth, the sub-part of interest S _p (x, y) is determined as a sub-part moved from S _q (x ′, y ′), and is set as an effective sub-part.
S28: The speed and direction of movement from the previous frame image of the effective sub-section are calculated.

  The upper left end point of the sub-partition of the previous frame image is (xLeftTop1, yLeftTop1). If yLeftTop2-yLeftTop1), the speed of movement and the direction of movement are as follows.

  S29: The process of S28 from S24 is repeated until the confirmation in all the sub-compartments is completed. Through a series of processing, all effective sub-partitions, movement speeds from the previous frame of each effective sub-partition, and direction of the movement vector are obtained.

3) Region Extraction Unit FIG. 8 is a diagram showing a moving object presence region extraction procedure in the present invention. An effective sub-partition template for confirming the moving object is generated by tracking from the set of effective sub-partitions obtained by the processing of FIG. 7, and moving tracking block candidates are set from the effective sub-partition template.

S31: Take out one of the effective sub-partitions and use it as an effective sub-partition template.

S32, S33: Take out the next one of the effective sub-sections and compare whether the moving speed and the moving direction of both match. If they match, the process proceeds to S34, and if they do not match, the process proceeds to S35.

S34: Combine both effective sub-partitions to generate a sub-partition template. This sub-partition template is set as an effective sub-partition template.

S35: The effective sub-partition is set as another effective sub-partition template.
S36: S32 to S35 are repeated until confirmation with all valid sub-partitions is completed.

S37: A region obtained by adding m pixels from the upper, lower, left, and right ends of the sub-partition templates of all the effective sub-partition templates defined in S34 and S35 is set as a movement tracking block candidate.

The area of the movement tracking block candidate is calculated. The top, bottom, left and right ends of the effective sub-partition template in S37 are (xsub _- min), (xsub _- max), (ysub _- min), and (ysub _- max). Assuming that an area given a predetermined pixel value m outside the area surrounded by the upper, lower, left, and right ends is (> 0), the area of the movement tracking block candidate is as follows.

Further, the center position (xtrk _- cen, ytrk _- cen) of the movement tracking block candidate is as follows.

4) Moving object specifying unit A moving object tracking block for specifying a moving object is extracted from the area of the moving tracking block candidate obtained by the processing of FIG. 8, and the moving object is specified. The moving object is specified by calculating the moving vector of the moving object. FIG. 9 is a diagram showing a moving object specifying procedure (part 1) in the present invention.

S41: The movement vector from the previous frame image of the moving object existing in the movement tracking block candidate is calculated. If the moving tracking block center position of the previous frame image is (xtrk _- cen1, ytrk _- cen1) and the moving tracking block candidate center position of the current frame image is (xtrk _- cen2, ytrk _- cen2), the moving object of the current frame image The moving speed v _trk ^(k) and moving direction vector (d _trk−x , d _trk−y ) ^(k) of the moving vector (xtrk ₋ cen2−xtrk ₋ cen1, ytrk ₋ cen2−ytrk ₋ cen1) It becomes.

  Here, the movement vector in the current frame image is calculated from the movement tracking block candidate of the current frame image and the movement tracking block of the previous frame image, but the movement tracking block candidate of the current frame image is the result of the processing of FIG. 9 described later. Since it becomes a movement tracking block of the next frame image, the size of the area is the same, and the movement vector can be calculated.

S42: One of the movement tracking block candidates of the current frame image is extracted.

S43: One of the movement tracking blocks of the previous frame image is taken out.

S44, S45: The movement vector of the current frame image is calculated, and the tracking condition is calculated from the movement vector of the movement vector in the previous frame image.

  The tracking conditions are as follows.

Here, v _trk ^(k−1) , (d _trk−x , d _trk−y ) ^(k−1) indicate the moving speed of the moving vector of the previous frame image and the direction of the moving vector.

When the speed difference between the moving tracking block candidate and the moving tracking block is less than a certain value and the inner product is more than a certain value, the moving objects of both moving tracking blocks are regarded as the same tracking target. If the tracking condition is satisfied, the process proceeds to S46, and if the tracking condition is not satisfied, the process proceeds to S47.
S46, S47: Set and confirm the number of continuations when the frame continuation condition is set as the tracking condition. Let FR be the number of consecutive frames, and add 1 to FR. If the frame continuity condition is satisfied, the process proceeds to S48, and otherwise, the process proceeds to S49.
S48: Recognizing that a moving object exists in the area of the moving tracking block candidate, replace the moving tracking block candidate with the moving tracking block of the current frame image and hold it. Proceed to S50.
S49: Not satisfying the tracking condition is regarded as another moving object newly appearing in the current frame image, and the moving tracking block candidate is held as a new moving tracking block. The tracking condition FR = 1 is set, and the process proceeds to S50.

S50: The processes from S43 to S49 are repeated until the confirmation of all the movement tracking blocks in the previous frame is completed. All the movement tracking blocks in which the moving object recognized by the current frame image acquired by this series of processes exists are obtained. The moving objects of each moving tracking block are different moving objects. Second Embodiment Moving objects before separation described below are regarded as the same moving tracking block.

  The moving object of the movement tracking block extracted by the moving object detection unit in 2) is associated with the RF tag information.

  FIG. 10 is a diagram showing a moving object specifying procedure (part 2) in the present invention. A procedure for identifying a moving object from an area where the moving object exists is shown.

S51: Take out one of the movement tracking blocks.
S52: The center position of the moving object is calculated, and the azimuth angle β of the moving object is calculated from the calculated center position.

FIG. 11 is an overhead view of a subject, a camera, and an image according to the present invention. The relationship between a moving object and the detection angle of a camera is shown. Subject is a moving object is located in the region Q, the detection angle of the RF tag is theta _R relative to the boundary of region P and the region Q, shows that the viewing angle of the camera is 2.alpha.

  The calculation of the center position is performed by cutting out the moving object image. FIG. 12 is a diagram for explaining the cutout range of the moving object image in the present invention. A rectangular area circumscribing a sub-partition template in which a moving object exists from an image that enters the field of view of the camera and a moving tracking block is used. In the case of FIG. 12, this is a range in which a white portion is added to the shaded portion where the moving object exists.

  If the upper left is BhleftTop (xLefttop, yLeftTop) and the lower right is BhrRightBottom (xRightBottom, yRightBottom), the region is enclosed by the following formula.

However, min ( _BhLeftTop ) and max ( _{BhRightBottom} ) are functions for extracting the minimum value and the maximum value, respectively. If the center position is (x _trk- cen, y _trk- cen), the center position can be calculated as the center of the rectangle from the coordinates of the sub-partitions. Since the calculation method is the same as the movement tracking block method of the movement vector in 2), the description is omitted.

Next, the azimuth angle of the moving object is calculated. FIG. 13 is a diagram showing calculation of the azimuth angle of a moving object in the present invention. In FIG. 13, it is assumed that the camera is facing the positive direction of the x-axis. The viewing angle of the camera is 2α. The azimuth angle β of the moving object detected from the relationship with the center position (x _trk-cen ) is calculated by the following equation, with the pixel value of the screen being 0 at the left end and X _{size at} the right end.

Here, the RF tag and the moving object are associated with each other in the horizontal direction, but the correspondence can be obtained in the same manner in the vertical direction. Description is omitted.
S53, S54: when changing the RF tag detection angle theta _R, checks whether the azimuth angle β is equal to theta _R. If they match, the process proceeds to S55, and if they do not match, the process proceeds to S57.
S55: The moving object is specified from the RF tag information of the moving object.

  FIG. 14 is a diagram showing association between a moving object and an RF tag in the present invention. FIG. 14 is a view of FIG. 10 as viewed from above.

A method for identifying a moving object is described. The boundary between the camera viewing area P and area Q is 0 degree, and the camera viewing angle is 2α. For simplicity of explanation, it is assumed that the angular resolution of the antenna is also α, and the boundary of the directivity angle (the boundary between the region P and the region Q in the figure) coincides with the left end direction of the camera field of view. When the moving object has progressed from the area P at time t to the region Q, the detection angle theta _R of the RF tag detected by the directional antenna, theta from θ _R = -α~0 (region P) _R = 0~α ( Switch to area Q). At the same time, the moving object appears in the camera field of view, and the azimuth angle β of the moving object is calculated using the center position (x _trk-cen ) obtained by the region extraction and moving object specifying unit. Thus, at time t, if the detection angle theta _R of the RF tag is changed from the direction of a region P in the direction of the region Q, were azimuth β substantially 0 degrees of the moving object captured in the camera field of view at the same time It is assumed that the RF tag is attached to the detected moving object. Once it is determined that the RF tag is attached to the moving object, the moving object is associated with the RF tag while being tracked by image processing.
Thereby, the moving object of the current frame image can be specified, and the RF tag information of the specified moving object is sent to the attribute information adding unit.
5) Attribute information adding unit The RF tag information of the moving object is associated with the image. The processing procedure is described first, and the association method will be described later.
S56: The attribute value of the extracted RF tag data of the moving object is associated with the image, and the process proceeds to S57.
S57: The process returns to S51 until confirmation is completed for all the movement tracking blocks, the next movement tracking block is taken out, and the processing from S52 to S56 is repeated.

The region where the moving object exists in the current frame image is detected by the series of processes shown in FIGS. 7, 8, 9, and 10, the moving object is identified, the RF tag information of the moving object is extracted, and the moving object is continuously detected. Can be tracked.
FIG. 15 is a diagram showing an association between an image and attribute information in the present invention. The normal camera outputs image data for one frame at an appropriate sampling period (for example, 33 ms). The number of the frame is associated with the identification number of the moving object output from the region extraction processing unit 13.
There are items of frame number, image data, and movement tracking block number. In the image data, a captured image with a frame number is described, and in the movement tracking block number, the number of the moving object to be tracked and the recognized tag number are described.
In frame numbers 1 and 2 in FIG. 15, no moving object is detected. The frame number 3 is reached, a moving object is detected at the movement tracking block numbers 1 and 3, and the movement tracking block number 1 indicates that the RF tag 1 can be recognized. In the movement tracking block number 1, the moving object number 1 and the tag 1 of the moving object are obtained from the region extraction and angle detection unit 13. The movement tracking number 3 indicates that the moving object can be detected, but the tag number cannot be recognized. The tag 5 can be recognized at the frame number 5 and the movement tracking block number 2, and the tag 8 can be recognized at the frame number 8 and the movement tracking block number 3.
The attribute information adding unit 15 associates the attribute information of the RF tag from the attribute attribute generating unit 11 with the image. An image with a video attribute is generated by linking the RF frame attribute information to the image frame number, and the video with the attribute is transmitted to the video apparatus via the network 5 via the signal output I / F 30. Note that, as an image output destination, a recording medium such as a streaming relay using a TV monitor Internet, a DVD, a video, or a CD-ROM can be considered. These processes are described in S9 and S10 of FIG.

Here, the RF tag and the moving object are associated with each other in the horizontal direction, but the correspondence can be obtained in the same manner in the vertical direction. Description is omitted.
(Example 2)
An identification method when a plurality of moving objects appear simultaneously in the camera field of view will be described. FIG. 16 is a diagram showing a separation process when a plurality of moving objects appear simultaneously in the camera field of view according to the present invention.

A separation process is shown when a plurality of moving objects A and B appear simultaneously in the camera field of view. Since the change timings of the detection angles of the RF tag A and RF tag B detected by the directional antenna match, both cannot be identified. Consider a case where one moving object A reaches the boundary between the region Q and the region R and the other moving object B is in the region Q at the time t2 after the time t1. At this time, the detection angle θ _RA detected by the directional antenna of the RF tag A of the moving object A is
θ _RA = switched to 0~α (region Q) from θ _RA = α~2α (area R). On the other hand, the detection angle θ _RB of the RF tag B of the moving object B is θ _RB = 0 to α (region Q). Therefore, the moving object region and the RF tag can be associated by comparing with the azimuth angles βA and βB of the moving objects A and B calculated by the moving object specifying unit 14.

As a result, the attribute information of the moving object A and the moving object B that were unknown from time t1 to time t2 can be given retroactively.
FIG. 17 is a diagram showing an update of association between a moving object and attribute information in the present invention. For example, at time t1 <t <t2 of 1) in FIG. 17, the moving object detected by the moving tracking block number i is a moving object that combines moving object A and moving object B between frame numbers 3 to 6, Separation is not possible. Between frame numbers 3 to 6, there is a moving object i in common with the moving tracking block number i. For this reason, RF tag information cannot be given between times t1 <t <t2.
At time t2, the moving object A and the moving object B can be separated, and can be associated with the attribute information of the RF tag. Therefore, as shown in 2), the tag 3 of the detected RF tag number of the moving object A is assigned to the retroactive moving tracking block number i to the frame number 3, and the moving object B assigns a new moving tracking block number m to the moving object. A tag 10 with an RF tag number of B is assigned.
By using the above method, even if multiple moving objects appear in the camera field of view at the same time and they cannot be identified, the moving object is separated and detected with the passage of time, and the RF tag is linked to the moving object. Information can be given automatically.
(Example 3)
An identification method in the case where a plurality of moving objects are present in the region having the same directivity angle from the imaging start time will be described. In this case, since the azimuth angles of the RF tag 1 and the RF tag 2 detected by the directional antenna have the same value, they cannot be identified. The identification method of both in such a case is demonstrated.
FIG. 18 is a diagram showing separation and identification of a moving object by rotation of a directional beam in the present invention.
By rotating the beam of the directional antenna and arranging the boundary of the directivity angle in the middle of a plurality of moving objects, the two can be separated. The rotation angle Δθ of the antenna beam is obtained as follows.

  When the antenna beam rotates and becomes the arrangement shown in FIG. 18 (2), the azimuth angle of the moving object detected by the directional antenna is based on the left end of the camera field of view.

It becomes. At this time, the azimuth angles βA and βB of the moving object A and the moving object B calculated by the angle detection unit are

Therefore, the moving object and the RF tag can be associated with each other.

Thereby, even when a directional antenna having a low angular resolution is used, attribute information in which an RF tag is associated with a moving object can be automatically given.
Example 4
A method for increasing the angular resolution using a plurality of directional antennas will be described. FIG. 19 is a diagram showing the improvement of the angular resolution by the multi-directional antenna in the present invention. FIG. 19A schematically shows that the beam bulge corresponds to the resolution of the angle at which the RF tag can be detected, and is 2α in this figure. FIG. 19 (2) schematically shows a case where the angular resolution is increased by using two directional antennas of FIG. 19 (1). The directional antennas are arranged so that the directional beams are located at the boundary between the beams of the other directional antennas. Thus, when the RF tag is at the position shown in the figure, the directional antenna 1 detects the RF tag in the region P, and the directional antenna 2 detects the RF tag in the region P ′. As a result, the detection angle of the RF tag that satisfies the conditions at the same time is in the direction of the shaded portion, so that the angular resolution is improved to α of ½.
(Appendix 1)
Video attribute automatic assigning device for automatically assigning to the image an attribute that associates the information of the RF tag received by a wireless object and a moving object detected from an image obtained by continuously capturing an object moving with an RF tag Because
The video attribute automatic assigning device includes means for receiving the captured image;
Means for receiving information of the RF tag from the antenna;
Attribute information generating means for generating an attribute of the moving object attached to the RF tag from the received RF tag information;
A moving object detecting means for detecting a moving section of a grid-like section in which a predetermined number of pixels constituting the received image are collected;
The section in which the moving object may be present is extracted from the speed and direction of the motion from the section of the previous frame image of the section of the current frame image in which the motion is detected, and the section having the same motion and adjacent thereto A region extracting means for extracting a region obtained by combining the regions, calculating a center position of the extracted region, calculating a movement vector from the previous frame image of the center position, recognizing a moving object, and determining a moving object of the extracted region Moving object specifying means for extracting and confirming the mounted RF tag and specifying a moving object;
Attribute information giving means for automatically giving the RF tag attribute of the identified moving object to the image;
An apparatus for automatically assigning video attributes, comprising:

(Appendix 2)
In the video attribute automatic assigning device according to attachment 1,
An RF tag reader / writer further comprising an RF tag receiving radio antenna that receives information from the RF tag by controlling the directional beam of the directional antenna based on the information from the attribute information generation unit based on the information from the RF tag. The apparatus for automatically assigning video attributes according to supplementary note 1, characterized by comprising:

(Appendix 3)
In the video attribute automatic assigning device described in appendix 2,
When there are a plurality of moving objects extracted by the moving object specifying means and the RF tag cannot be confirmed, information from the RF tag is controlled by controlling the directional beam so as to be positioned near the center of the area extracted by the area extracting means. The video attribute automatic assigning device according to claim 2, further comprising means for separating the plurality of moving objects and confirming and identifying the moving objects.

(Appendix 4)
In the video attribute automatic assigning device according to attachment 3,
The apparatus further comprises means for automatically assigning the separated moving object and the attribute of the RF tag to the image going back to a time before separating the moving object identified by separating the plurality of moving objects. The video attribute automatic assigning device according to appendix 3.

(Appendix 5)
A program for assigning to the image an attribute that associates information of the RF tag with a moving object that is present in an image obtained by continuously capturing an object that is attached and moving with an RF tag by a computer,
The program generates an attribute of the moving object attached to the RF tag from the RF tag information,
Detecting a moving section of a grid-like section in which a predetermined number of pixels constituting the image are collected;
Extracting a section where a moving object may exist from the speed and direction of movement from the section of the previous frame image of the section of the current frame image in which the motion is detected;
Let us extract a region that combines the adjacent sections with the same movement,
Calculating the center position of the extracted region;
Recognizing a moving object by calculating a movement vector from the previous frame image of the center position, extracting and confirming the RF tag attached to the moving object in the extracted region,
A video attribute automatic assigning program, wherein attribute information for automatically assigning an attribute of the RF tag of the identified moving object is added to the image.

(Appendix 6)
An apparatus for automatically adding subscripts for video attributes according to appendix 2,
A radio antenna of the RF tag receiving means of the RF tag receiving radio antenna means is composed of a plurality of radio antennas, and one directional beam of the plurality of radio antennas is at the boundary of the directional beam of the other radio antenna. The video attribute automatic assigning device according to appendix 2, further comprising means for receiving information on the RF tag by controlling to match.
(Appendix 7)
A video attribute imparting method for imparting to the image a video attribute in which a moving object detected from an image obtained by continuously capturing an object moving with an RF tag and information on the RF tag is associated with the image,
Generating an attribute of the moving object attached to the RF tag from the received RF tag information;
Generating an attribute of the moving object attached to the RF tag from the RF tag information;
Detecting a moving section of a grid-like section in which a predetermined number of pixels constituting the image are collected;
Extracting a section where a moving object may exist from the speed and direction of movement from the section of the previous frame image of the section of the current frame image in which the motion is detected;
Extract the region that combines the adjacent sections with the same movement,
Calculating the center position of the extracted region;
Recognizing a moving object by calculating a movement vector from the previous frame image of the center position, extracting and confirming the RF tag attached to the moving object in the extracted region,
A method of automatically assigning video attributes, wherein the attribute of the RF tag of the identified moving object is automatically assigned to the image.

It is a figure which shows the video attribute transmission system of one Embodiment of this invention, and a video attribute automatic provision apparatus block configuration. It is a figure which shows the process sequence of the video attribute transmission system of this invention. It is a figure which shows the example of conversion into the attribute value of RF tag data in this invention. It is a figure which shows the detection of the RF tag detection angle in this invention. It is a figure which shows the relationship between the camera viewing angle and antenna direction in this invention. It is a figure which shows the relationship between the subdivision of the image in this invention, a subdivision template, and a tracking block. It is a figure which shows the moving object detection procedure in this invention. It is a figure which shows the moving object presence area | region extraction procedure in this invention. It is a figure which shows the moving object specific procedure (the 1) in this invention. It is a figure which shows the moving object specific procedure (the 2) in this invention. It is an overhead view of a subject, a camera, and an image in the present invention. It is a figure explaining the cutout range of the moving object image in this invention. It is a figure which shows the azimuth calculation method of the moving object in this invention. It is a figure which shows the correlation of the moving object and RF tag in this invention. It is a figure which shows the correlation of the image and attribute information in this invention. It is a figure which shows the separation process when the several moving object in this invention appears in a camera visual field simultaneously. It is a figure which shows the update of matching with the image and attribute information in this invention. It is a figure which shows the separation identification of the moving object by rotation of the directional beam in this invention. It is a figure which shows the improvement of the angular resolution by the multi-directional antenna in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image | video attribute automatic provision apparatus 2 Image pick-up object 3 Camera 4 RF tag reader / writer 5 Network 6 Recording medium 7 Image | video apparatus 10 Attribute information automatic provision part 11 Attribute information generation part 12 Moving object detection part 13 Area extraction process part 14 Moving object specification part 15 Attribute information giving unit 16 Memory 20 Signal input I / F unit
21 RF tag 30 Signal output I / F section
41 antenna 41-1 directional antenna 42 A / D
43 I / F

Claims (5)

Video attribute automatic assigning device for automatically assigning to the image an attribute that associates the information of the RF tag received by a wireless object and a moving object detected from an image obtained by continuously capturing an object moving with an RF tag Because
The video attribute automatic assigning device includes means for receiving the captured image;
Means for receiving information of the RF tag from the antenna;
Attribute information generating means for generating an attribute of the moving object attached to the RF tag from the received RF tag information;
A moving object detecting means for detecting a moving section of a grid-like section in which a predetermined number of pixels constituting the received image are collected;
The section in which the moving object may be present is extracted from the speed and direction of the motion from the section of the previous frame image of the section of the current frame image in which the motion is detected, and the section having the same motion and adjacent thereto A region extracting means for extracting a region obtained by combining the regions, calculating a center position of the extracted region, calculating a movement vector from the previous frame image of the center position, recognizing a moving object, and determining a moving object of the extracted region Moving object specifying means for extracting and confirming the mounted RF tag and specifying a moving object;
Attribute information giving means for automatically giving the RF tag attribute of the identified moving object to the image;
An apparatus for automatically assigning video attributes, comprising: In the video attribute automatic grant device according to claim 1,
An RF tag reader / writer further comprising an RF tag receiving radio antenna that receives information from the RF tag by controlling the directional beam of the directional antenna based on the information from the attribute information generation unit based on the information from the RF tag. The video attribute automatic assigning device according to claim 1, further comprising: The device for automatically assigning video attributes according to claim 2,
When there are a plurality of moving objects extracted by the moving object specifying means and the RF tag cannot be confirmed, information from the RF tag is controlled by controlling the directional beam so as to be positioned near the center of the area extracted by the area extracting means. 3. The apparatus for automatically assigning video attributes according to claim 2, further comprising means for receiving and receiving and identifying the plurality of moving objects to identify and identify the moving objects. In the video attribute automatic grant device according to claim 3,
The apparatus further comprises means for automatically assigning the separated moving object and the attribute of the RF tag to the image going back to a time before separating the moving object identified by separating the plurality of moving objects. The video attribute automatic assigning device according to claim 3. A program for assigning to the image an attribute that associates information of the RF tag with a moving object that is present in an image obtained by continuously capturing an object that is attached and moving with an RF tag by a computer,
The program generates an attribute of the moving object attached to the RF tag from the RF tag information,
Detecting a moving section of a grid-like section in which a predetermined number of pixels constituting the image are collected;
Extracting a section where a moving object may exist from the speed and direction of movement from the section of the previous frame image of the section of the current frame image in which the motion is detected;
Let us extract a region that combines the adjacent sections with the same movement,
Calculating the center position of the extracted region;
Recognizing a moving object by calculating a movement vector from the previous frame image of the center position, extracting and confirming the RF tag attached to the moving object in the extracted region,
A video attribute automatic assigning program, wherein attribute information for automatically assigning an attribute of the RF tag of the identified moving object is added to the image.

Images