JP2015191480A - Information processor, operation method of object and operation program of object - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor capable of smoothly moving an object synthesized and displayed on a dynamic image in combination with changes of the dynamic image.SOLUTION: An information processor 37 displays a dynamic image taken by an imaging apparatus on a display device and operates an object synthesized and displayed on the displayed dynamic image. The information processor 37 includes: optical flow generation means 374 that generates optical flows on the basis of at least two frame images constituting a dynamic image taken by an imaging apparatus; motion vector detection means 375 that detects motion vectors in a display position of the object from the generated optical flow; and object operation means 376 that operates the object on the basis of the detected motion vectors.

Description

  The present invention relates to an information processing apparatus that operates an object that is combined and displayed on a moving image captured by an imaging apparatus, an object operation method, and an object operation program.

2. Description of the Related Art Conventionally, a technique is known in which a finger part of a captured image is detected on a captured image captured by a camera, a GUI (Graphical User Interface) member is displayed on the detected finger part, and an image of the finger part is used as a pointer. (For example, refer to Patent Document 1).
According to this technique, the user can perform GUI operations such as instruction, selection, and movement on the GUI member displayed on the captured image.

JP 2012-53674 A

  However, the technique described in Patent Document 1 displays a GUI member on the image of the finger part, and the image of the finger part that operates the displayed GUI member is treated as a still image. Therefore, when the synthesized image is a moving image or the like, there is a problem that it is difficult to smoothly operate the GUI member according to the movement of the finger part on the moving image.

  An object of the present invention is to provide an information processing apparatus, an object operation method, and an object operation program that can smoothly operate an object combined and displayed on a moving image in accordance with a change in the moving image. is there.

An information processing apparatus according to the first aspect of the present invention includes:
An information processing device that displays a moving image captured by an imaging device on a display device, and operates an object that is combined and displayed on the displayed moving image,
An optical flow generating means for generating an optical flow based on at least two frame images constituting a moving image captured by the imaging device;
Motion vector detection means for detecting a motion vector at the display position of the object from the optical flow generated by the optical flow generation means;
And an object operating means for operating the object based on the motion vector detected by the motion vector detecting means.

Here, the optical flow generation means generates an optical flow from at least two frame images constituting the moving image, and the optical flow can be generated by a known block matching method, gradient method or the like.
According to the information processing apparatus according to the first aspect, since the optical flow of the moving image is generated by the optical flow generation means and the motion vector at the display position of the object is detected by the motion vector detection means, the object operation means detects the object When the movement operation is performed, the object movement operation can be smoothly performed according to the detected motion vector.

An information processing apparatus according to a second aspect of the present invention is the information processing apparatus according to the first aspect.
An operation area selecting means for selecting an area for operating the object is provided.

  According to the information processing apparatus according to the second aspect, since the operation area is selected by the operation area selection unit, it is possible to prevent the operation of the object from being obstructed by a motion vector other than the operation area.

An information processing device according to a third aspect of the present invention is the information processing device according to the second aspect,
Masking means for masking the operation area selected by the operation area selection means is provided.
According to the information processing apparatus according to the third aspect, since the operation area can be masked by providing the masking means, the object can be operated in a state in which the observer of the moving image can easily see.

An object operating method according to the fourth aspect of the present invention includes:
An object operation method for displaying a moving image captured by an imaging device on a display device and operating an object synthesized and displayed on the displayed moving image,
The control device for controlling the imaging device and the display device is:
Generating an optical flow based on at least two frame images constituting a moving image captured by the imaging device;
Detecting a motion vector at the display position of the object from the generated optical flow;
Performing the operation of the object based on the detected motion vector.

An object operation program according to the fifth aspect of the present invention is:
An object operation method for displaying a moving image captured by an imaging device on a display device and operating an object synthesized and displayed on the displayed moving image is provided in the control device for controlling the imaging device and the display device. An object operation program to be executed,
In the control device,
Generating an optical flow based on at least two frame images constituting a moving image captured by the imaging device;
Detecting a motion vector at the display position of the object from the generated optical flow;
And a step of operating the object based on the detected motion vector.

  According to the 4th aspect and the 5th aspect of this invention, it can enjoy the effect | action and effect same as the 1st aspect of this invention mentioned above.

The top view which shows an example of the portable terminal which concerns on embodiment of this invention. The block diagram which shows the structure of the portable terminal in the said embodiment. The block diagram which shows the structure of the control part in the said embodiment. The schematic diagram of the screen which displayed the moving image on the display part of the portable terminal in the said embodiment, and displayed the object image. The schematic diagram of the screen which displayed the moving image on the display part of the portable terminal in the said embodiment, and displayed the object image. The schematic diagram of the screen which displayed the optical flow in the said embodiment. The schematic diagram for demonstrating the detection of a motion vector from the optical flow in the said embodiment. The schematic diagram showing the state which deform | transformed the object image in the said embodiment. The schematic diagram showing the polyhedron set to the object image in the said embodiment. The schematic diagram showing the arbitrary vertex of the object image in the said embodiment in the space in a polyhedron. The schematic diagram for demonstrating the positional relationship of the arbitrary vertex of the object image at the time of changing the polyhedron in the said embodiment. The schematic diagram showing the state which divided | segmented the object image in the said embodiment for every part, and set the polyhedron for every part. The flowchart for demonstrating the effect | action in the said embodiment. The schematic diagram of the object image used as the deformation | transformation of the said embodiment.

Embodiments of the present invention will be described below.
FIG. 1 is a plan view showing an example of the mobile terminal 3 according to the embodiment of the present invention.
The portable terminal 3 synthesizes and displays an object on the moving image captured by the imaging unit 35. Such a portable terminal 3 can be comprised by a smart phone (multifunctional mobile phone), a tablet terminal, a portable game machine, etc.
In the present embodiment, as shown in FIG. 1, the mobile terminal 3 includes a housing 31, a display unit 32, an operation unit 33, an audio output unit 34, an imaging unit 35, a storage unit 36, and a control unit 37 (see FIG. 2). And a terminal device capable of executing a game program.

The casing 31 constitutes the exterior of the mobile terminal 3 and is formed in a substantially oval shape that is large enough to be held by the left and right hands. The housing 31 includes a front surface portion 31A, a back surface portion 31B (not shown in FIG. 1), an upper surface portion 31C, a bottom surface portion 31D, and left and right side surface portions 31E and 31F.
The display unit 32 corresponds to the display device of the present invention, and is disposed at the approximate center of the front unit 31A. The display unit 32 includes a display panel formed in a rectangular shape, and an image formed by the display control unit 371 under the control of a display control unit 371 (see FIG. 3) of the control unit 37 described later. Is displayed. Examples of such a display panel include organic EL (Electro-Luminescence) and liquid crystal display panels.

The operation unit 33 includes various buttons and sticks arranged around the display unit 32 in the front surface 31A and the upper surface 31C of the housing 31, and controls operation signals in accordance with user input operations. To the unit 37.
Specifically, the operation unit 33 is located on the left side with respect to the display unit 32, and can be input in each direction of up, down, left, and right when viewed facing the front unit 31A, and in each direction An operation stick 332 that can be tilted and an operation button 333 located on the bottom surface portion 31D side are provided. The operation unit 33 includes four operation buttons 334 positioned on the right side of the display unit 32, an operation stick 335 having the same configuration as the operation stick 332, and operation buttons 336 and 337 positioned on the bottom surface portion 31D side. And having. Further, the operation unit 33 includes operation buttons 338L and 338R which are respectively located on the left and right sides of the upper surface portion 31C.

The audio output unit 34 includes speakers 34L and 34R disposed near the left and right ends of the front portion 31A. The audio output unit 34 outputs audio corresponding to an audio signal input from an audio output control unit 373 (see FIG. 3) of the control unit 37, which will be described later, through speakers 34L and 34R.
The imaging unit 35 corresponds to the imaging device of the present invention, and includes a camera 351 as a first imaging unit disposed on the upper right side of the front unit 31A. The imaging unit 35 outputs an image captured by the camera 351 to the control unit 37.

FIG. 2 is a block diagram illustrating a configuration of the mobile terminal 3.
As shown in FIG. 2, the display unit 32, the operation unit 33, the audio output unit 34, the imaging unit 35, the storage unit 36, and the control unit 37 are connected to each other by a bus line BL.
Among these, the storage unit 36 stores programs and data necessary for the operation of the mobile terminal 3. For example, the storage unit 36 stores an OS (Operating System) that controls the operation of the mobile terminal 3, an application program such as a game, and an image display program. Such a storage unit 36 can be configured by a non-volatile memory such as a flash memory or an HDD (Hard Disk Drive).

3A and 3B are block diagrams illustrating the configuration of the control unit 37. FIG.
The control unit 37 includes a CPU (Central Processing Unit) and the like, and controls the operation of the mobile terminal 3. Specifically, the control unit 37 reads out and processes the program stored in the storage unit 36 according to the user's input operation (operation signal input from the operation unit 33), or This is a control device that autonomously controls the mobile terminal 3.
In addition, the control part 37 displays the moving image imaged by the imaging part 35 on the display part 32, when CPU runs the said image display program. Therefore, as shown in FIG. 3, the control unit 37 includes a display control unit 371, an imaging control unit 372, an audio output control unit 373, an optical flow generation unit 374, a motion vector detection unit 375, an object operation unit 376, an operation area. A selection unit 377 and a masking unit 378 are provided.

The display control unit 371 draws an image indicating the operation state of the mobile terminal 3 or an image captured by the imaging unit 35 in a drawing memory (not shown), and outputs an image signal corresponding to the image to the display unit 32. As a result, the image is displayed on the display unit 32.
Specifically, as shown in FIG. 4, a moving image captured by the imaging unit 35 is displayed on the display unit 32, and an object image OB is synthesized and displayed on the moving image.
The imaging control unit 372 controls the operation of the imaging unit 35. Specifically, the imaging control unit 372 causes the imaging unit 35 to capture images at a predetermined cycle (frame rate). Then, the imaging control unit 372 acquires a moving image that is a captured image input from the imaging unit 35 and stores the moving image in the storage unit 36.
The audio output control unit 373 controls the audio output unit 34 to output a predetermined audio.

The optical flow generation unit 374 generates an optical flow from at least two or more frame images G1 and G2 constituting a moving image.
Now, assuming that the frame image G1 shown in FIG. 4 has transitioned to the frame image G2 shown in FIG. 5, the face image CH1 in the moving image slides in the right direction, and the finger portion image CH2 rotates. Assuming that
The optical flow generator 374 generates an optical flow G3 as shown in FIG. 6 from the movement of the luminance points of the two frame images G1 and G2. The optical flow G3 can be obtained by using a known method such as a gradient method or a block matching method.
The gradient method is a method of deriving a relational expression between spatiotemporal differentiation and optical flow from the assumption that the brightness of a point on an object does not change after movement, and using this to estimate the motion of the object. .
On the other hand, in the block matching method, an image is divided into small areas of a certain size, and a motion vector of each block is obtained. This is a method of searching where each block corresponds to the previous frame, and using a difference in position from the corresponding block as a motion vector.
Using these methods, the optical flow G3 calculates the moving direction and moving speed of the images CH1 and CH2 from the two frame images G1 and G2 using the frame rate and the like, and generates a motion vector.

  The motion vector detection unit 375 detects a motion vector at a position where the object image OB is displayed in the optical flow G3 generated by the optical flow generation unit 374. In the change from FIG. 4 to FIG. 5, the image CH2 of the finger part is displayed around the object image OB, and the finger part is rotated leftward, and this is detected as the motion vector V1.

The object operation unit 376 is a part that moves and deforms the object image OB based on the motion vector V1 around the object image OB detected by the motion vector detection unit 375. The object operation operation unit 376A and the object deformation operation unit 376B.
The object moving operation unit 376A calculates the moving direction and moving amount of the object image OB according to the detected direction and size of the motion vector V1, and displays the object image OB at an appropriate position on the display unit 32. Let
Specifically, as shown in FIG. 7A, when the optical flow OF is detected around the object image OB, the motion vector detection unit 375 is near the vertex of the polyhedron POL set in the object image OB. The motion vector V1 of the polyhedron POL is detected from the optical flow OF. The optical flow OF is obtained discretely for each pixel of the display unit 32. The optical flow of the vertex to be obtained is obtained by interpolation from the optical flow of surrounding pixel coordinates.
Based on the motion vector V1 detected by the motion vector detection unit 375A, the object movement operation unit 376A deforms the polyhedron POL as shown in FIG. 7B, and then converts the polyhedron POL between the original vertices. As shown in FIG. 7C, the polyhedron POL is rotated so as to maintain the shape according to the constraint conditions and the object image OB is moved according to the rotational movement of the polyhedron POL.

The object deformation operation unit 376B calculates the deformation direction and deformation amount of the object image OB according to the direction and size of the motion vector V1 around the object image OB detected by the motion vector detection unit 375, and deforms the object image OB. Thus, it is synthesized and displayed on the display unit 32. As shown in FIG. 3B, the object deformation operation unit 376B includes a polyhedron setting unit 376C, a space division unit 376D, a space determination unit 376E, a decomposition coefficient storage unit 376F, and a deformation display unit 376G.
Specifically, as shown in FIG. 8, when the image CH2 of the finger portion moves up and down on the object image OB, the object image OB is deformed up and down by the motion vector V1 on the display unit 32. To display.

For deformation of the object image OB, as shown in FIG. 9, the polyhedron setting unit 376C sets the center point P0 at an arbitrary position in the object image OB, and sets the polyhedron POL centered on the center point P0.
In this embodiment, the polyhedron POL is set as a convex polyhedron having twelve vertices P1 to P12. However, the present invention is not limited to this as long as the space can be uniquely divided into regions. It can be arbitrarily set according to the shape and the like.
The polyhedron POL may be independent of the shape of the object image OB, and the polyhedron POL may be set using the vertex POB of the object image OB. Preferably, the polyhedron POL is set so that the portion of the object image OB that is desired to move, such as a limb, covers the entire object image OB and is as small as possible.
The space division unit 376D divides the space in the polyhedron POL into a plurality of vectors from the center point P0 to the vertices P1 to P12 of the polyhedron POL.

Next, the space discriminating unit 376E indicates that an arbitrary vertex POB of the object image OB is included in a space constituted by the center point P0, the vertex P1 of the polyhedron POL, the vertex P4, and the vertex P5. Determine.
As shown in FIG. 10, the decomposition coefficient storage unit 376F sets the position x of an arbitrary vertex POB to a vector a from the center point P0 to the vertex P1, a vector b from the center point P0 to the vertex P4, and a center point. It is stored and saved by the vector c from P0 to the vertex P5 and the respective decomposition coefficients p, q, r.
That is, the position x of an arbitrary vertex POB of the object image OB is expressed by the following equation (1).
x = pa + qb + rc (1)

For example, when the deformation display unit 376G detects the motion vector V1 that deforms the polyhedron POL, as shown in FIG. 11, the deformation display unit 376G detects the space in the polyhedron POL according to the detected motion vector V1, the vertex P1, FIG. The vertex P4 and the vertex P5 are transformed into a vertex P1 ′, a vertex P4 ′, and a vertex P5 ′.
Along with this, the deformation display unit 376G stores the stored expression (1) in accordance with the motion vector V1, and the position x of an arbitrary vertex POB of the object image OB is represented by the following expression (2). 'Store and save, and display the deformed object image OB on the display unit 32.
x ′ = pa ′ + qb ′ + qc ′ (2)

Further, as shown in FIG. 12, the object deformation operation unit 376B not only sets one polyhedron POL for the object image OB, but also converts the object image OB into a plurality of parts such as a head and limbs. Dividing into parts, setting part polyhedrons PA1 to PA5 for each part, and storing the positions of arbitrary vertices of each part in the form of the above formula (1) in each part polyhedron PA1 to PA5 You can also.
In this case, the deformation of the object image OB is realized by the following method.
First, the lower part polyhedrons PA1 to PA5 are regarded as objects, and the part polyhedrons PA1 to PA5 are deformed and moved by deformation of the upper polyhedron POL. Next, the object image OB is deformed and moved in accordance with the deformation of the lower component polyhedrons PA1 to PA5 to which the respective parts belong. The object image OB can be deformed by directly deforming the lower component polyhedrons PA1 to PA5 without setting the upper polyhedron POL.
For example, when the hand portion of the object image OB is desired to be deformed, only the component polyhedron PA3 set for the hand portion needs to be deformed.
On the other hand, when the deformation operation affects the entire object image OB, for example, when the object image OB is operated on a display image engulfed in the vortex of the sea, the upper polyhedron POL may be deformed or moved. .
Furthermore, a complicated operation combining the upper polyhedron POL and the component polyhedrons PA1 to PA6 is also possible. That is, if the object image OB is assumed to be a display image that hits a rock or the like in a river or the like, the upper polyhedron POL performs a deformation movement operation in the river flow, and the deformation operation when hitting a rock is a component polyhedron. It can be performed with PA1 to PA6. By adopting such a hierarchical structure, it is possible to match the deformation of the entire flow with the deformation of the part.

Which part polyhedrons PA1 to PA5 belong to each part of the object image OB can be determined by the decomposition coefficient of each vector. Each part of the object image OB is selected so that the part polyhedrons PA1 to PA5 do not overlap each other.
Further, when the vertex P13 is included in the component polyhedron PA3 like the vertex P13 of the object image OB, the decomposition coefficients p, q, r in the equation (1) are set in the range of 0-1. On the other hand, when the vertex P14 is also outside the component polyhedron PA4 as in the vertex P14 of the object image OB, the sum of squares of the decomposition coefficients p, q, r in the equation (1) is calculated as the component polyhedron PA3, PA4. And the vertex P14 of the object image OB may be deformed as belonging to the smaller component polyhedron.

The operation area selection unit 377 selects an area where the object image OB to be operated exists as the operation area AR.
The operation area AR is selected using a known image recognition technique such as face recognition, excluding the optical flow in the area of the face image CH1 in FIG. It can also be selected by generating.
Further, since the optical flow interval of the face image CH1 is widely detected and the finger image CH2 is generated narrowly, the operation area AR may be selected using this.

  The masking unit 378 is a means for facilitating the operation by masking the operation area AR. For example, the moving image other than the operation area AR is not displayed, or the image of the operation area AR is not left as a moving image but the contrast is increased. The visibility of the operation area is improved by using an emphasized binary image, displaying only a contour image, or replacing it with an object image such as an icon.

Next, the effect | action of this embodiment is demonstrated based on the flowchart shown in FIG.
First, the display control unit 371 composites and displays the moving image captured by the imaging unit 35 and the object image OB read from the storage unit 36 on the display unit 32 (step S1).
The optical flow generation unit 374 generates an optical flow G3 based on the frame images G1 and G2 constituting the moving image (step S2).

The motion vector detection unit 375 monitors the optical flow G3 in the operation area AR (step S3), and detects the direction and magnitude of the motion vector V1 when the motion vector in the operation area AR has changed (step S3). S4).
The object operation unit 376 calculates the moving direction and moving amount of the object image OB based on the detected motion vector V1, and displays the object image OB in an appropriate position on the display unit 32 (step S5).
A series of object operation methods from step S1 to step S5 are stored in the storage unit 36 as a computer-readable program, and when the moving image captured by the imaging unit 35 is displayed on the display unit 32, the control unit It is read by 37 and executed.

According to this embodiment, there are the following effects.
Since the optical flow G3 of the moving image is generated by the optical flow generation unit 374 and the motion vector V1 at the display position of the object image OB is detected by the motion vector detection unit 375, the moving operation of the object image OB is performed by the object operation unit 376. In performing the above, the moving operation of the object image OB can be smoothly performed according to the detected motion vector V1.

Since the operation area AR is selected by the operation area selection unit 377, the operation of the object image OB can be prevented from being obstructed by a motion vector other than the detection area. Since the area AR is masked, the object image OB can be manipulated in a state that is easy for the viewer of the moving image to see.

The object transformation operation unit 376B sets a polyhedron POL centered on an arbitrary center point P0 in the object image OB, and divides the space in the polyhedron POL by a vector from the center point P0 of the polyhedron POL toward the vertices P1 to P12. Then, it is determined in which of the divided spaces the vertex POB of the object image OB, and the decomposition coefficient of the position x of the object vertex is determined by three vectors constituting the space including the vertex of the object p, q, and r are stored, and the object image OB is deformed and displayed in accordance with the deformation operation applied to the polyhedron POL. Therefore, it is not necessary to perform weighted addition in the orthogonal coordinate system of a plurality of bones as compared with skinning by bones, and the processing load on the control unit 37 can be reduced.
Further, since the object deformation operation unit 376B can set the component polyhedrons PA1 to PA5 for each part of the object image OB and perform the deformation operation on each of the component polyhedrons PA1 to PA5, the object image OB having a complicated shape can be obtained. Even in such a case, the object image OB can be displayed in a deformed manner according to the deforming operation, and the processing load can be reduced.

The present invention is not limited to the embodiments described above, and includes modifications as described below.
In the above-described embodiment, the object operation unit 376 operates the character-like object image OB. However, the present invention is not limited to this. For example, the slide switch image OB2 as shown in FIG. An operation may be performed in which the image is fixed somewhere and slid on the image CH2 of the finger portion.
The embodiments of the present invention can employ other configurations as long as the object of the present invention can be achieved.

  DESCRIPTION OF SYMBOLS 3 ... Portable terminal, 31 ... Housing | casing, 31A ... Front part, 31C ... Upper surface part, 31E ... Side part, 32 ... Display part, 33 ... Operation part, 331 ... Direction button, 332 ... Operation stick, 333 ... Operation button, 334 ... Operation buttons, 335 ... Operation stick, 336 ... Operation buttons, 338L ... Operation buttons, 34 ... Audio output unit, 34L ... Speaker, 35 ... Imaging unit, 351 ... Camera, 36 ... Storage unit, 37 ... Control unit, 371 ... Display control unit, 372 ... Imaging control unit, 373 ... Audio output control unit, 374 ... Optical flow generation unit, 375 ... Vector detection unit, 376 ... Object operation unit, 376A ... Object movement operation unit, 376B ... Object deformation operation unit 376C: polyhedron setting unit, 376D ... space division unit, 376E ... space discrimination unit, 376F ... decomposition coefficient storage unit, 376 ... deformation display part, 377 ... operation area selection part, 378 ... masking part, AR ... operation area, BL ... bus line, CH1, CH2 ... image, G1, G2 ... frame image, G3 ... optical flow, OB ... object image, OB2 ... Slide switch image

Claims (5)

An information processing device that displays a moving image captured by an imaging device on a display device, and operates an object that is combined and displayed on the displayed moving image,
An optical flow generating means for generating an optical flow based on at least two frame images constituting a moving image captured by the imaging device;
Motion vector detection means for detecting a motion vector at the display position of the object from the optical flow generated by the optical flow generation means;
An information processing apparatus comprising: an object operation unit that operates the object based on a motion vector detected by the motion vector detection unit. The information processing apparatus according to claim 1,
An information processing apparatus comprising operation region selection means for selecting a region for operating the object. The information processing apparatus according to claim 2,
An information processing apparatus comprising masking means for masking an area selected by the operation area selecting means. An object operation method for displaying a moving image captured by an imaging device on a display device and operating an object synthesized and displayed on the displayed moving image,
The control device for controlling the imaging device and the display device is:
Generating an optical flow based on at least two frame images constituting a moving image captured by the imaging device;
Detecting a motion vector at the display position of the object from the generated optical flow;
And a step of operating the object based on the detected motion vector. An object operation method for displaying a moving image captured by an imaging device on a display device and operating an object synthesized and displayed on the displayed moving image is provided in the control device for controlling the imaging device and the display device. An object operation program to be executed,
In the control device,
Generating an optical flow based on at least two frame images constituting a moving image captured by the imaging device;
Detecting a motion vector at the display position of the object from the generated optical flow;
And a step of performing an operation of the object based on the detected motion vector.