JP5650060B2 - Image tracking apparatus and image tracking method - Google Patents

Description

  The present invention relates to an image tracking apparatus and an image tracking method for automatically tracking a specific tracking image in a video.

  Video communication systems are already in widespread use and have been implemented not only in expensive business videoconferencing systems but also in videoconferencing systems using a general-purpose mobile phone as a video communication terminal.

  As a general problem in video communication systems, it is difficult to talk to objects shown in video by pointing to them with gestures and gestures such as “that” and “it” like normal face-to-face conversations. There are some. In order to solve this problem, the same video is shared between video communication terminals, a figure is written in the video that the user of the video communication terminal communicates in the shared video, and the video and the graphic are superimposed. A displayable system has also been put into practical use (for example, see Non-Patent Document 1). Many of these systems share a screen such as a whiteboard, an electronic file, and a Web page separately from each video communication terminal, and can write a figure on the video conference video. Image tracking systems are also known.

“Real Collaboration”, [online], NTT Software Corporation, [Search May 26, 2011], Internet <URL: http://www.ntts.co.jp/products/realcollabo/index.html>

  FIG. 7 is a diagram illustrating a user interface of a video communication terminal in a conventional image tracking system. With reference to FIG. 7, the problem of the prior art will be described. On the video display unit 31 of the user interface 3 shown in FIG. 7, a live video taken by the other party performing the video communication is displayed. A user using the user interface 3 specifies an image A (hereinafter referred to as a tracking image) A to be tracked in the live video. For example, a region of the tracking image A (hereinafter referred to as a tracking region) is designated by a mouse drag operation. At the same time, the user selects a mark image for indicating the tracking area from the mark image list 32, and the video display unit 31 displays the mark image superimposed on the tracking image A. On the image tracking device side, the process of calculating the feature of the tracking image A designated by the user and searching the live image for an image that closely matches the image feature is continuously performed after the user designates the tracking region. . Then, the video display unit 31 superimposes and displays the designated landmark image on the searched tracking image A.

  At this time, as shown in FIG. 7, if similar images B and C similar to the tracking image A designated by the user exist in the same frame, a problem occurs in the search process. Specifically, it may not be possible to determine which of the tracking image A and the similar images B and C designated by the user is the tracking image A.

  Therefore, the present invention has been made in view of the above problems, and prevents erroneous detection of another object or the like having a similar feature when a specific object in a live image is tracked by an image feature. To provide an image tracking apparatus and an image tracking method.

  In order to solve the above-mentioned problem, in the present invention, a physical marker for preventing confusion between a tracking area and a similar area is provided. Specifically, the area where the marker is to be placed is displayed in a superimposed manner in the video, thereby prompting the user to place the marker.

  In the present invention, the tracking area is corrected and enlarged so that the installed marker is included. The user who specifies the tracking area can manually enlarge the tracking area, but the marker area where the marker is placed is searched, and the marker area is automatically set to the original tracking area specified by the user interface. The convenience of operation is improved by including it in.

That is, the image tracking device according to the present invention is an image tracking device that automatically tracks a specific tracking image in a video, receives the video distributed from the first video communication terminal, and receives the video as the second. A video transmission unit that transmits to the video communication terminal; a tracking image information from the second video communication terminal is acquired; and a similar image search unit that searches for a similar image similar to the tracking image from the video; When a similar image exists, a marker installation area determination unit that determines a marker installation area indicating an area where a physical marker is installed within a predetermined range from the tracking image, and searches the tracking image from the video. In addition, when the similar image does not exist, a first synthesized video obtained by synthesizing a landmark image with the searched tracking image is transmitted to the first video communication terminal and the second video communication terminal, and Kind If the image is present, and a tracking and combination processing unit for transmitting a second composite image by superimposing the marker installation area tracking image the search to the first video communication terminal, the video transmission When the similar image exists, the unit receives the video in which the marker is installed in the marker installation area from the first video communication terminal, and transmits the video to the second video communication terminal. , the tracking and combination processing section, when said similar image exists, characterized that you search the modified tracking image to include some or all of the images of the markers from within the video .

The image tracking method according to the present invention is an image tracking method for automatically tracking a specific tracking image in a video, receiving the video distributed from the first video communication terminal, and receiving the video from the second video tracking method. Transmitting to the video communication terminal, obtaining information of the tracking image from the second video communication terminal, searching for a similar image similar to the tracking image in the video, and the tracking image from the video. And when the similar image does not exist, a first synthesized video obtained by synthesizing a landmark image with the searched tracking image is transmitted to the first video communication terminal and the second video communication terminal. And, if the similar image exists, transmitting a second composite image in which the marker installation area is superimposed on the searched tracking image to the first video communication terminal; and (f) the Kind If a similar image exists, receiving a video in which the marker is installed in the marker installation area from the first video communication terminal, and transmitting the video to the second video communication terminal; and g) searching for a tracking image modified to include a part or all of the image of the marker in the video when the similar image exists .

  According to the present invention, when a specific object in a video is tracked by an image feature, it does not automatically prevent a false detection but efficiently supports a user to install a physical marker for preventing a false detection. By doing so, erroneous detection can be prevented with a simple mechanism.

It is a block diagram which shows the structure of the image tracking system of 1st Embodiment by this invention. It is a figure which shows an example of the marker installation area | region in the image tracking system of 1st Embodiment by this invention. It is a flowchart which shows operation | movement of the image tracking system of 1st Embodiment by this invention. It is a block diagram which shows the structure of the image tracking system of 2nd Embodiment by this invention. It is a figure which shows an example of the expansion tracking area | region in the image tracking system of 2nd Embodiment by this invention. It is a flowchart which shows operation | movement of the image tracking system of 2nd Embodiment by this invention. It is a figure which shows the user interface of a video communication terminal.

  Embodiments of an image tracking apparatus and an image tracking method according to the present invention will be described below in detail with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram illustrating a configuration of an image tracking system according to the first embodiment. As shown in FIG. 1, the image tracking system 1 includes a first video communication terminal 11, an image tracking device 12, and a second video communication terminal 13. The first video communication terminal 11, the image tracking device 12, and the second video communication terminal 13 are connected via a network, respectively.

  The first video communication terminal 11 is a communication terminal that distributes video shared between video communication terminals. The first video communication terminal 11 includes a video distribution unit 111 and a video display unit 112. The video distribution unit 111 acquires the live video captured by the imaging device 10 and transmits the acquired live video to the image tracking device 12. The video display unit 112 displays the live video transmitted from the image tracking device 12 on the display and presents it to the user of the first video communication terminal 11 (hereinafter referred to as user U1).

  The image tracking device 12 is a device that automatically tracks where a specific object in a video captured by the imaging device 10 is reflected in the video by image processing. The image tracking device 12 includes a tracking / combination processing unit 121, a tracking image storage unit 122, a video transmission unit 123, a similar image search unit 124, and a marker installation region determination unit 125.

  The tracking image storage unit 122 stores the tracking image designated by the user of the second video communication terminal 13 (hereinafter referred to as the user U2) as a template image used by the tracking / combination processing unit 121 during the tracking process. . In addition to the template image, the tracking image storage unit 122 stores a set of identifiers i of mark images to be displayed superimposed on the tracking image. It is assumed that the set of the landmark image identifier i and the landmark image data is shared in advance between the second video communication terminal 13 and the image tracking device 12.

  The tracking / combination processing unit 121 outputs the live video received from the video distribution unit 111 to the video transmission unit 123 as it is in an initial state where the tracking area is not designated by the user U2. Further, when the tracking area T is designated by the user U2, the tracking / combination processing unit 121 acquires a tracking image from the tracking image storage unit 122 and tracks the live image received from the video distribution unit 111. Search for an image by image processing. For example, a matching process between the tracking image acquired from the tracking image storage unit 122 and an image having the same size as the tracking area in each frame is performed, and an image whose similarity with the tracking image exceeds a predetermined threshold is searched. Then, the tracking / combination processing unit 121 generates a composite image obtained by combining the mark image registered in advance with the searched tracking image, and outputs the generated composite image to the video transmission unit 123.

  The video transmission unit 123 transmits the live video (including the composite video) input from the tracking / combination processing unit 121 to the first video communication terminal 11 and the second video communication terminal 13.

  The similar image search unit 124 acquires the tracking region information from the tracking region specification unit 131 and searches for a similar image similar to the tracking image in the frame by the same processing as the tracking / combination processing unit 121. Specifically, the correlation coefficient rel (T, Sj) is obtained for a rectangular area Sj (j = 1, 2, 3,...) Having the same size as the tracking area T in the frame, and the predetermined threshold t is set. On the other hand, a set Ss = {Sj | rel (T, Sj)> t} of Sj is extracted as a similar region.

  The marker installation area determination unit 125 acquires the tracking area T, the set Ss, and the identifier i from the similar image search unit 124. If the set Ss is an empty set, that is, if there is no similar area, the marker installation area is determined. The tracking image A is registered in the tracking image storage unit 122 without setting. On the other hand, if the set Ss is not an empty set, that is, if there is a similar area, the marker installation area determination unit 125 does not register the template image, and installs a marker to prompt the user U1 to install the marker. The marker installation area indicating the area to be determined is determined.

  The marker placement area determination unit 125 registers the tracking image in the tracking image storage unit 122 when the original number of sets Ss (the number of similar images) is equal to or less than a predetermined threshold, and the original number of sets Ss is predetermined. If the threshold value is exceeded, the marker placement area may be determined. Alternatively, when the distance between the tracking image and the similar image exceeds a predetermined threshold, the marker installation region determination unit 125 registers the tracking image in the tracking image storage unit 122, and the distance between the tracking image and the similar image is a predetermined value. The marker placement area may be determined when it is equal to or less than the threshold value.

  The marker installation area determination unit 125 includes a tracking image and determines an image area within a predetermined distance from the tracking image as a marker installation area. FIG. 2 is a diagram illustrating an example of the marker installation area. The upper left coordinates of the tracking area T are (x1, y1), the lower right coordinates of the tracking area T are (x2, y2), and the center coordinates O of the tracking area T are ((x1 + x2) / 2, (y1 + y2) / Assuming that 2), in the example shown in FIG.

  Then, the marker installation area determination unit 125 instructs the tracking / combination processing unit 121 to display the marker installation area superimposed on the tracking image for a certain period so that the marker installation area is presented to the user. To do. By displaying the marker installation area, the user U1 and the user U2 can visually recognize it. As a result, the user U1 can install a physical marker in a portion corresponding to the marker installation area in the real space being photographed by the photographing apparatus 10. The physical marker only needs to be able to distinguish the image of the tracking area from the image of the similar area. For example, a seal of a specific color can be used as the marker.

  The second video communication terminal 13 is a communication terminal that designates a tracking area among terminals sharing a live video distributed from the first video communication terminal 11. The second video communication terminal 13 includes a tracking area designation unit 131, a tracking area transmission unit 132, and a video display unit 133.

  When the tracking area information is input from the user U 2 via the user interface 3, the tracking area specifying unit 131 outputs the tracking area information to the tracking area transmitting unit 132. For example, using the user interface 3 shown in FIG. 7, the user U2 designates the tracking image in the video by a drag operation of the mouse as a rectangular area, and inputs the tracking area information by selecting the mark image by clicking the operation. To do. The tracking area information includes a tracking area T {(x1, y1), (x2, y2)} and an identifier i.

  The tracking area transmitting unit 132 transmits the tracking area information input from the tracking area specifying unit 131 to the similar image searching unit 124.

  The video display unit 133 displays the video (including the synthesized video) distributed from the video transmission unit 123 on the display and presents it to the user U2 of the second video communication terminal 13.

  When the marker is installed, the user U2 modifies the tracking area so that the tracking area includes the marker. When the tracking area information after correction is input from the user U2 via the user interface 3, the tracking area designating unit 131 outputs the tracking area information after correction to the tracking area transmitting unit 132, and the similar image search unit 124 The tracking area information after correction is acquired from the tracking area transmission unit 132. The similar image search unit 124 confirms that an image similar to an image in the corrected tracking area (hereinafter referred to as an enlarged tracking image) does not exist in the same frame, and stores the enlarged tracking image in the tracking image storage unit 122. Remember. When an image similar to the enlarged tracking image exists in the same frame, the first video communication terminal 11 is notified to reset the marker.

  Next, the operation of the image tracking system configured as described above will be described. FIG. 3 is a flowchart showing the operation of the image tracking system 1. First, the image tracking device 12 transmits a video to the first video communication terminal 11 and the second video communication terminal 13 by the video transmission unit 123, and the video is transmitted to the first video communication terminal 11 and the second video communication terminal 13. Is displayed (step S101).

  The second video communication terminal 13 designates the tracking area by the tracking area designation unit 131 based on the instruction from the user U2 (step S102).

  Upon receiving the tracking area information from the second video communication terminal 13, the image tracking device 12 searches for a similar area using the similar image search unit 124 (step S103), and determines whether or not a similar area exists (step S104). ). If it is determined in step S104 that a similar area exists, the image tracking apparatus 12 determines the marker installation area by the marker installation area determination unit 125 (step S105).

  When the video on which the marker installation area is superimposed is displayed on the video display unit 112, the user U1 of the first video communication terminal 11 installs a physical marker in the marker installation area (step S106).

  The user U2 of the second video communication terminal 13 designates the enlarged tracking area Te through the user interface 3 (step S107).

  The image tracking device 12 tracks the image in the enlarged tracking area Te as a tracking image by the tracking / combination processing unit 121 (step S108). If it is determined in step S104 that there is no similar region, no marker is required, and the image tracking device 12 tracks the tracking image by the tracking / combining processing unit 121 (step S108). .

  As described above, according to the image tracking device 12 of the first embodiment, when tracking a tracking image, a false marker is not automatically prevented from being detected automatically, but a physical marker for preventing false detection by the user U1. By efficiently supporting the installation, it becomes possible to prevent false detection with a simple mechanism.

(Second Embodiment)
Next, an image tracking apparatus according to a second embodiment of the present invention will be described. FIG. 4 is a block diagram illustrating a configuration of an image tracking system according to the second embodiment. As shown in FIG. 4, the image tracking system 2 according to the second embodiment is further compared with the image tracking system 1 (see FIG. 1) according to the first embodiment, in that the image tracking device 12 further includes a marker detection unit 126 and The difference is that a tracking area correction unit 127 is provided. In addition, the same reference number is attached | subjected to the same component as 1st Embodiment, and description is abbreviate | omitted.

  The marker detection unit 126 detects a marker region including a marker image based on marker information (for example, information on the color and / or shape of the marker), and uses the marker region information indicating the detected marker region as a tracking region correction unit 127. Output to. The marker information may be notified to the image tracking device 12 by the user U1 who installed the marker, or may be determined in advance. For example, if the pixel value vector of the video is Icam (x, y), the marker area M is determined as M = {(x, y) | Icam (x, y) ⊂R}. Here, R represents a specific area in the pixel value vector space.

  Although it is assumed that the marker cannot be detected correctly due to fluctuations in illumination conditions, the detection rate is expressed by a method such as expressing Icam (x, y) in the HSV color system and defining R using parameters other than brightness. Can be increased. If the marker area includes a large amount of small mosquito noise-like partial areas, the above-described extraction processing is performed after the resolution of the video is reduced (x ′, y ′), and then (x, y ) To define a marker area, it is possible to extract a certain area. Through these processes, the marker detection unit 126 can improve the marker detection rate.

  The tracking area correction unit 127 acquires the tracking area information from the tracking area transmission unit 132, acquires the marker area in the recently acquired video from the marker detection unit 126, and corrects the tracking area based on the marker area. . FIG. 5 is a diagram illustrating a modification example of the tracking area. As shown in FIG. 5, a rectangular area including the tracking area T and the marker area M is set as a corrected tracking area Te. The corrected tracking area Te may include only a part of the marker area M. For example, for a predetermined threshold D, Mp = {(x, y) | (x, y) ⊂M, | (x, y) − ((x1 + x2) / 2, (y1 + y2) / 2) | <D } And a minimum rectangular area including all the subsets Mp and (x1, y1), (x2, y2) may be used as the corrected tracking area Te. After the marker is set, an image including the marker becomes a tracking target image, and is not confused with a similar image, and a mark image intended for the tracking image intended by the user U2 is displayed.

  Next, the operation of the image tracking system 2 configured as described above will be described. FIG. 6 is a flowchart showing the operation of the image tracking system 2. The operations from step S201 to step S206 are the same as the operations from step S101 to step S106 of the image tracking system 1 shown in FIG.

  When a physical marker is set by the user U1 in step S206, the image tracking device 12 detects an image of the marker from the video by the marker detection unit 126 (step S207). Subsequently, the image tracking device 12 corrects the tracking image into an image including a part or all of the marker image by the tracking region correcting unit 127 (step S208). That is, in the first embodiment, the user U2 manually corrects the tracking image. However, in the second embodiment, the image tracking device 12 automatically corrects the tracking image.

  As described above, according to the image tracking device 12 of the second embodiment, by further including the marker detection unit 126 and the tracking region correction unit 127, the tracking region can be automatically corrected, and the operation is easier. Become.

  Each of the above embodiments has been described as a representative example, but it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims.

DESCRIPTION OF SYMBOLS 1, 2 Image tracking system 10 Imaging device 11 1st video communication terminal 12 Image tracking device 13 2nd video communication terminal 111 Image | video delivery part 112,133 Image | video display part 121 Tracking / composition processing part 122 Tracking image memory | storage part 123 Image | video transmission part 124 Similar Image Search Unit 125 Marker Placement Area Determination Unit 126 Marker Detection Unit 127 Tracking Area Correction Unit 131 Tracking Area Designation Unit 132 Tracking Area Transmission Unit

Claims (8)

An image tracking device that automatically tracks a specific tracking image in a video,
A video transmission unit that receives the video distributed from the first video communication terminal and transmits the video to the second video communication terminal;
Obtaining a tracking image information from the second video communication terminal, and searching for a similar image similar to the tracking image from the video,
When the similar image exists, a marker installation area determination unit that determines a marker installation area indicating an area in which a physical marker is installed within a predetermined range from the tracking image;
When the tracking image is searched from the video and the similar image does not exist, a first synthesized video obtained by synthesizing a landmark image with the searched tracking image is used as the first video communication terminal and the second video communication terminal. Tracking / synthesizing process for transmitting to the first video communication terminal a second composite image in which the marker installation area is superimposed on the searched tracking image when the similar image exists, transmitted to the video communication terminal comprises a part, the,
When the similar image exists, the video transmission unit receives the video in which the marker is installed in the marker installation area from the first video communication terminal, and transmits the video to the second video communication terminal. To
The tracking and combination processing section, when said similar image exists, the image which is characterized that you search the modified tracking image to include some or all of the images of the markers from within the video Tracking device. A marker detection unit for detecting an image of the marker installed from the video;
A tracking area correction unit that corrects the tracking image to an image including a part or all of the image of the marker;
2. The image tracking according to claim 1, wherein, when the similar image exists, the tracking / combination processing unit searches the image for the tracking image corrected by the tracking region correcting unit. apparatus.   The image tracking device according to claim 1, wherein the similar image search unit sets, as the similar image, an image whose similarity with the tracking image exceeds a predetermined threshold.   The marker installation region determination unit determines the marker installation region when the number of the similar images exceeds a predetermined threshold value, or when the distance between the tracking image and the similar image is equal to or less than a predetermined threshold value. The image tracking device according to any one of claims 1 to 3, wherein An image tracking method for automatically tracking a specific tracking image in a video,
(A) receiving a video distributed from the first video communication terminal and transmitting the video to the second video communication terminal;
(B) obtaining information of a tracking image from the second video communication terminal, and searching for a similar image similar to the tracking image from the video;
(C) searching the tracking image from the video;
(D) when the similar image does not exist, a step of transmitting a first synthesized video obtained by synthesizing a landmark image to the searched tracking image to the first video communication terminal and the second video communication terminal; ,
(E) When the similar image exists, a marker installation region indicating a region where a physical marker is installed is determined within a predetermined range from the tracking image, and the marker installation region is included in the searched tracking image. Transmitting a second composite image superimposed with the first video communication terminal;
(F) When the similar image exists, the video having the marker installed in the marker installation area is received from the first video communication terminal, and the video is transmitted to the second video communication terminal. Steps,
(G) If the similar image exists, searching the video for a tracking image modified to include a part or all of the marker image;
An image tracking method comprising: ( H ) detecting an image of the marker placed from the video;
( I ) modifying the tracking image into an image including a part or all of the marker image ;
Wherein step (g), the if the similar image exists, wherein the benzalkonium be searched tracking image that has been modified by the step (i) from within the video image according to claim 5 Tracking method.   The image tracking method according to claim 5, wherein in the step (b), an image whose similarity with the tracking image exceeds a predetermined threshold is set as the similar image.   The step (e) determines the marker placement region when the number of the similar images exceeds a predetermined threshold value, or when the distance between the tracking image and the similar image is equal to or smaller than the predetermined threshold value. The image tracking method according to claim 5, wherein:

Images