JP2018129008A - Image compositing device, image compositing method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To create an image with an arbitrary field of view at an arbitrary time with higher quality by using a video obtained without imposing a strict constraint condition, such as limitation of the background to a uniform color.SOLUTION: An image composition device comprises: an input part that receives an input of a plurality of dynamic images, camera parameters related to a plurality of cameras that photograph the dynamic images, and parameters related to the three-dimensional shape of a subject photographed in the dynamic images; a three-dimensional joint information estimation part that estimates three-dimensional joint information on the subject for a frame at every time; a shape changing part that acquires information indicating the three-dimensional shape of the subject at every time on the basis of a result of estimation of the three-dimensional joint information in the frame at every time and the parameters related to the three-dimensional shape of the subject; and an image composition part that creates an image including an image of the subject with a designated field of view at a designated time on the basis of the information indicating the three-dimensional shape of the subject.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for generating a free viewpoint video.

  In a free viewpoint video, a video of an arbitrary viewpoint is synthesized using videos taken by cameras arranged at a plurality of positions. With such a composition process, it is possible to view images from any viewpoint. This kind of free-viewpoint video technology has long been studied as the next-generation video media. In the free viewpoint video, by restoring the three-dimensional shape of the subject in the scene, it is possible to generate a video with the viewpoint where the camera is not actually arranged.

  Collet et al. (See Non-Patent Document 1) is a representative study that can realize a high-quality free viewpoint video. This is a study that proposed a series of pipelines for shooting, compositing, and distributing free-viewpoint images. The technology of this research makes it possible to synthesize high-quality free viewpoint images. However, a large number of cameras and infrared cameras are required. Further, it is necessary to limit the background to a uniform color in order to extract the subject area. Furthermore, it is necessary to perform calibration specialized for these special environments. Thus, there are very strict constraints on the shooting environment. Therefore, it is difficult to use in actual scenes.

  As another research, a free viewpoint video composition method under a relatively realistic constraint by using a distance sensor has been proposed (see Non-Patent Document 2). However, with the technique according to this proposal, the synthesis quality is not sufficiently high. One of the major factors that degrade the synthesis quality is occlusion and time flicker. Regarding occlusion, since it is necessary to reproduce information that cannot be obtained, it is impossible in principle to solve it without using prior information. The flickering in the time direction is caused by variations in information acquired by the frames due to the interference of infrared light by the distance sensor. In this case, attempts have been made to solve the problem by filtering distance information. However, although it has been improved, it has not been fully resolved. Occlusion and flickering in the time direction are problems to be solved because even if they occur only slightly, the viewer feels uncomfortable.

A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, S. Sullivan, “High-quality streamable free-viewpoint video,” ACM Transactions on Graphics, 34 (4), 2015. D. Alexiadis, D. Zarpalas, P. Daras, “Fast and smooth 3D reconstruction using multiple RGB-Depth sensors,” in Visual Communications and Image Processing Conference 2014, pp.173-176.

As described above, there are still problems to be solved in the conventional free viewpoint video technology, and it has not been realized to generate an image with sufficient quality under the constraint that it can be used in an actual scene.
In view of the above circumstances, the present invention is a technique for generating an image at an arbitrary field of view and time with higher quality by using an image obtained without imposing severe restrictions such as limiting the background to a uniform color. The purpose is to provide.

  One embodiment of the present invention receives input of a plurality of moving images, camera parameters related to the fields of view of the plurality of cameras that captured the moving images, and parameters related to the three-dimensional shape of the subject captured in the moving images. An input unit, a three-dimensional joint information estimation unit for estimating the three-dimensional joint information of the subject for the frame at each time, an estimation result of the three-dimensional joint information in the frame at each time, and a three-dimensional shape of the subject A shape deforming unit that acquires information indicating the three-dimensional shape of the subject at each time based on the parameter, and the information on the specified field of view at the specified time based on the information indicating the three-dimensional shape of the subject. And an image composition unit that generates an image including an image of a subject.

  One aspect of the present invention is the above-described image composition device, wherein the parameters relating to the three-dimensional shape of the subject include reference shape information indicating the three-dimensional shape of the subject, and the three-dimensional shape indicated by the reference shape information. And reference joint information indicating information on each joint.

  One aspect of the present invention is the above-described image composition device, in which the frame at each time constituting the moving image is a frame that is a target for estimating joint information of the subject based on the image of the frame. A frame classification unit for determining a frame; and the three-dimensional joint information estimation unit estimates the three-dimensional joint information of the subject based on an image of the estimation frame for the estimation frame, For the frame, the 3D joint information is estimated by an interpolation process using the estimation result of the 3D joint information in the estimated frame.

  One aspect of the present invention is the above-described image composition device, further including a two-dimensional joint information estimation unit that estimates the two-dimensional joint information of the subject for the frame at each time and acquires the reliability of the estimation result. The three-dimensional joint information estimation unit estimates the three-dimensional joint information for the estimated frame based only on information about a part of frames for which high reliability is acquired by the two-dimensional joint information estimation unit.

  One embodiment of the present invention receives input of a plurality of moving images, camera parameters related to the fields of view of the plurality of cameras that captured the moving images, and parameters related to the three-dimensional shape of the subject captured in the moving images. An input step, a three-dimensional joint information estimation step for estimating the three-dimensional joint information of the subject for the frame at each time, an estimation result of the three-dimensional joint information in the frame at each time, and a three-dimensional shape of the subject A shape deformation step for acquiring information indicating the three-dimensional shape of the subject at each time based on the parameter, and the information on the specified field of view at the specified time based on the information indicating the three-dimensional shape of the subject. And an image composition step for generating an image including an image of the subject.

  One embodiment of the present invention is a computer program for causing a computer to function as the above-described image composition device.

  According to the present invention, it is possible to generate an image at an arbitrary field of view and time with higher quality by using an image obtained without imposing severe restrictions such as limiting the background to a uniform color.

It is a schematic block diagram which shows the structural example of the image synthesis apparatus 10 in embodiment. 3 is a diagram illustrating a configuration example of a frame classification unit 12. FIG. It is a figure which shows the structural example of the two-dimensional joint information estimation part. It is a figure which shows the structural example of the three-dimensional joint information estimation part. 3 is a diagram illustrating a configuration example of a shape deforming unit 15. FIG. 4 is a flowchart illustrating a specific example of processing of the image composition device 10. It is a flowchart which shows the detail of a process of step S106.

  FIG. 1 is a schematic block diagram illustrating a configuration example of an image composition device 10 according to the embodiment. The image composition device 10 includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, and the like connected by a bus, and executes an image composition program. By executing the image synthesis program, the image synthesis apparatus 10 includes an input unit 11, a frame classification unit 12, a two-dimensional joint information estimation unit 13, a three-dimensional joint information estimation unit 14, a shape deformation unit 15, and an image synthesis unit 16. Function as. Note that all or part of each function of the image composition device 10 may be realized by using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). . The image composition program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. The image composition program may be transmitted via a telecommunication line.

  First, the input unit 11 will be described. The input unit 11 includes a plurality of moving images, parameters of a camera that captures each moving image (hereinafter referred to as “camera parameters”), reference shape information indicating a three-dimensional shape of a predetermined subject, and three-dimensional joint information of the predetermined subject. Are received, and reference deformation information of a predetermined subject is received. The plurality of moving images are moving images obtained by shooting the same scene at the same time by the cameras arranged at a plurality of positions. For example, a moving image obtained by photographing the field at the same time (for example, 1 hour from 13:00 to 14:00 on the same day) is input by a plurality of cameras arranged so as to surround the field such as a soccer field. . The moving image may be a color moving image, a gray scale moving image, or a binary moving image. The moving image data may be input from each camera in real time, or a moving image recorded on a recording medium such as a hard disk drive (HDD) may be input. Each moving image need not be taken at the exact same time, and there may be some time lag before and after. Each frame of the moving image is preferably given a time when each frame was shot (hereinafter referred to as “frame time”). When the frame time is not given, the input unit 11 may give the frame time to each frame based on the shooting start time and the moving image reproduction time. In the following description, for the sake of simplicity, there is only one person in the moving image, and that person is a subject to be rendered using reference shape information or the like (hereinafter referred to as “target subject”). To do. When there are a plurality of subjects of interest, the input unit 11 may divide the moving image for each subject of interest by performing a process of extracting a person area in the moving image. By performing the same processing as that when there is only one subject on the moving image relating to each subject of interest, the same processing can be performed even when there are a plurality of subjects of interest.

  The camera parameters are parameters such as a viewpoint position, a line-of-sight direction, and a viewing angle of each camera that has captured a moving image. The camera parameter may include both a parameter inside the camera and a parameter outside the camera. The camera parameter may be acquired by performing camera calibration in each camera before acquiring the moving image.

The reference shape information indicates a three-dimensional shape of a subject (a subject of interest) that is predetermined based on a predetermined reference among subjects captured in the input moving image. For example, the reference shape information is input for a subject that is particularly likely to receive attention. For example, when a moving image of a soccer game is input, the reference shape information of all players (stamen players and players on a bench) participating in the soccer game may be input. The reference shape information may be obtained by performing a three-dimensional shape restoration process on the subject of interest in advance. For example, the reference shape information may be generated based on data obtained by performing measurement using a measuring device such as a distance sensor on the subject of interest. For example, the reference shape information may be generated by using still images taken by cameras at a plurality of positions. The reference shape information may be data (three-dimensional human model data) including, for example, the position of each joint of the person, the surface shape of the person, and an image (texture image) of the person's surface. By using the three-dimensional person model data, it is possible to generate an image of a person in a desired posture with a desired visual field. Note that the posture of the subject of interest in the reference shape information may be a posture such as a T pose or an A stance, or may be another posture. In addition, the input unit 11 uses a technique such as Poisson Surface Reconstruction (reference document 1) or general spatial filtering if there is a defect or noise in the input reference shape information. May also be performed. By performing such a treatment, the shape restored thereafter retains a continuous surface. As a result, defects due to changes in the viewpoint position are less likely to occur.
Reference 1: M. Kazhdan, M. Bolitho, H. Hoppe, “Poisson Surface Reconstruction,” Symposium on Geometry Processing 2006, 61-70.

  The reference joint information is three-dimensional joint information of each joint in the posture of the input reference shape information. The three-dimensional joint information represents a joint position and a joint angle. The position of the joint is expressed by, for example, xyz coordinates. The angle of the joint is represented by, for example, Euler angles with the xyz axis as the center. The reference joint information may be measured using a measurement technique such as motion capture when generating the reference shape information, or may be acquired by performing an estimation process on an image in which the subject of interest is captured. .

  The deformation parameter is a parameter that determines how the shape of the subject of interest in the moving image is deformed when the joint of the subject of interest corresponding to the reference shape information changes. The deformation parameter is a parameter that defines a change in shape according to the rotation of the joint, for example. The deformation parameter may be acquired in advance by measurement or the like. For example, the deformation parameter may be determined so as to be inversely proportional to the distance between the joint and the shape vertex by using a general skinning technique. The deformation parameter may be determined manually using software.

  Next, the frame classification unit 12 will be described. FIG. 2 is a diagram illustrating a configuration example of the frame classification unit 12. A plurality of moving images input from the input unit 11 are input to the frame classification unit 12. In the example of FIG. 2, L moving images captured by L cameras at different viewpoint positions are input to the frame classification unit 12. The frame classification unit 12 includes a frame separation unit 121 and a joint information estimation flag provision unit 122. The frame separation unit 121 separates each moving image into images for each frame. At this time, the frame separation unit 121 associates frames obtained from different moving images with frames satisfying a predetermined condition indicating that it is an image of a frame shot at the same time based on the frame time. The predetermined condition may be a condition indicating that the frame time is closest to a frame of a certain moving image from among other moving images. A set of frames associated with each frame estimated to have been shot at the same time is called a simultaneous frame set. In the following processing, each frame included in the same time frame set may be treated as being taken at the same time regardless of the actual frame time.

  The joint information estimation flag assigning unit 122 indicates, for each simultaneous frame set, a flag indicating whether or not to estimate the three-dimensional joint information based on the image of the moving image frame set (hereinafter referred to as “joint information estimation flag”). Value). The joint information estimation flag has two types of values: an estimation flag and a non-estimation flag. When the estimation flag is given, the 3D joint information is estimated using the frame image and camera parameters in the same time frame set. On the other hand, when the non-estimation flag is given, the 3D joint information is estimated by interpolation processing in the same time frame set. The joint information estimation flag assigning unit 122 may assign an estimation flag to the same time frame set at a predetermined cycle, for example, and may assign a non-estimation flag to another same time frame set. The joint information estimation flag giving unit 122 may give an estimation flag to the same time frame set that satisfies a predetermined condition in the image. The predetermined condition may be, for example, that the moving speed of the subject of interest in the image shows an extreme value. The moving speed may be the moving speed of the entire subject of interest, or may be the moving speed of some indirect or body parts (for example, arms or face). In this case, the two-dimensional joint information estimation unit 13 may determine the moving speed of the subject of interest in the image, and may add an estimation flag when the moving speed shows an extreme value. The two-dimensional joint information estimation unit 13 may assign an estimation flag to the same-time frame set when any one frame satisfies a predetermined condition in the same-time frame set, When a predetermined condition is satisfied in a predetermined number of frames or more in a frame set, an estimation flag may be assigned to the same-time frame set. A non-estimated flag is assigned to all the same-time frame sets to which no estimated flag is assigned. In the following description, an image of each frame of each simultaneous frame set to which the joint information estimation flag is assigned is referred to as a classified frame image.

Next, the two-dimensional joint information estimation unit 13 will be described. FIG. 3 is a diagram illustrating a configuration example of the two-dimensional joint information estimation unit 13. The two-dimensional joint information estimation unit 13 receives the same time frame set (images of the respective viewpoints) to which the estimation flag is assigned. The two-dimensional joint information estimation unit 13 estimates two-dimensional joint information (joint position on the image) in each viewpoint frame image included in the input simultaneous frame set. Further, the two-dimensional joint information estimation unit 13 acquires the reliability of the estimation result of the two-dimensional joint information in the frame image of each viewpoint. The reliability is a value indicating how close the estimated result of the two-dimensional joint information is to the actual value. For example, the higher the reliability, the closer the estimation result is to the actual value. For these processes, for example, DeepCut described in Reference Document 2 below may be used, or other methods may be used.
Reference 2: L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. Gehler, B. Schiele, “DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016).

  Next, the three-dimensional joint information estimation unit 14 will be described. FIG. 4 is a diagram illustrating a configuration example of the three-dimensional joint information estimation unit 14. The three-dimensional joint information estimation unit 14 estimates the three-dimensional joint information at the time of the subject of interest in each simultaneous frame set based on the estimation result in the two-dimensional joint information estimation unit 13. The three-dimensional joint information estimation unit 14 includes a three-dimensional joint information calculation unit 141 and a three-dimensional joint information interpolation unit 142.

  The three-dimensional joint information calculation unit 141 receives the two-dimensional joint information and reliability acquired by the two-dimensional joint information estimation unit 13 in the same time frame set to which the estimation flag is assigned, and the camera parameters of each camera. The The three-dimensional joint information calculation unit 141 repeatedly performs the following processing for all joints of the subject of interest. First, the three-dimensional joint information calculation unit 141 selects a predetermined number (for example, two) frames of the joints to be processed from the one with higher reliability. The three-dimensional joint information calculation unit 141 estimates the three-dimensional information of the joint to be processed using the two-dimensional joint information in the selected frame and the camera parameters in the selected frame. For example, two-dimensional joint information on a pair of frame images is projected as a straight line in a three-dimensional space using camera parameters, and the midpoint of a line segment connecting two points where the distance between the straight lines is the shortest is three-dimensional. You may estimate as a joint position of joint information. Note that the predetermined number of frames selected based on the reliability may be different for each joint to be processed.

  When the three-dimensional joint information calculation unit 141 estimates the three-dimensional positions for all the joints, the three-dimensional joint information calculation unit 141 estimates the angles of the joints based on the estimation results. The angle of the joint may be obtained by deriving a rotation matrix or may be obtained using a quaternion. The three-dimensional joint information calculation unit 141 executes the above-described processing for all the same-time frame sets to which the estimation flag is assigned. As described above, the three-dimensional joint information calculation unit 141 estimates the three-dimensional joint information using a predetermined number of two-dimensional joint information with high reliability. For this reason, it is possible to suppress errors included in the two-dimensional joint information.

  The three-dimensional joint information interpolation unit 142 repeatedly performs the following processing for all the same-time frame sets to which the non-estimation flag is assigned. First, the three-dimensional joint information interpolating unit 142 determines, for each frame of the same-time frame set to be processed, a frame that is closest in time and has an estimation flag attached to a frame at an earlier time. And later frames one by one. The three-dimensional joint information interpolation unit 142 acquires an estimation result (three-dimensional joint information) by the three-dimensional joint information calculation unit 141 for each selected frame. The three-dimensional joint information interpolation unit 142 performs the interpolation according to the difference of the frame time with respect to the joint position and the angle based on the obtained estimation result, so that the three-dimensional of each frame to which the non-estimation flag is given Estimate joint information. At this time, some weight may be given to the coefficient used for interpolation. In this way, the estimation process by the three-dimensional joint information calculation unit 141 is not performed on the frames at all times, but only on a part of the thinned frames (frames to which the estimation flag is given). An estimation process by the joint information calculation unit 141 is executed, and an interpolation process based on the estimation result is performed on the remaining frames. By such processing, flickering of the shape for each frame can be suppressed.

  Next, the shape deformation unit 15 will be described. The shape deforming unit 15 receives reference shape information, reference joint information, an estimation result (three-dimensional joint information) by the three-dimensional joint information estimating unit 14, and a deformation parameter. The shape deforming unit 15 compares the three-dimensional joint information estimated by the three-dimensional joint information estimating unit 14 with the reference joint information. The shape deforming unit 15 deforms the reference shape based on the comparison result and the deformation parameter. The shape deforming unit 15 performs the above processing on the three-dimensional joint information obtained in all the same time frame sets.

  FIG. 5 is a diagram illustrating a configuration example of the shape deforming unit 15. Hereinafter, the shape deforming portion 15 will be described in detail with reference to FIG. The shape deforming unit 15 includes a joint displacement calculating unit 151 and a deforming unit 152.

  The joint displacement calculation unit 151 calculates a difference between the three-dimensional joint information estimated by the three-dimensional joint information estimation unit 14 and the reference joint information. For example, the joint displacement calculation unit 151 calculates the difference between the three-dimensional coordinates in the estimated three-dimensional joint information and the three-dimensional coordinates in the reference joint information, and acquires the positional deviation in the x-axis, y-axis, and z-axis. To do. Further, the joint displacement calculation unit 151 calculates a difference between the three-dimensional angle in the estimated three-dimensional joint information and the three-dimensional angle in the reference joint information, with the x axis, the y axis, and the z axis as the centers. Get the deviation of the rotation angle.

  The deforming unit 152 deforms the reference shape based on the difference between the estimated three-dimensional joint information and the reference joint information and the deformation parameter. By such processing, the deforming unit 152 deforms the reference shape so as to have the same posture as that of the subject of interest at the frame time of the same-time frame set that is the processing target. The deformation unit 152 outputs information on the shape after being deformed in this way as deformed shape information. By executing such processing in all the simultaneous frame sets, the deformed shape information corresponding to each simultaneous frame set is acquired. The deformation unit 152 may record the deformation shape information in the storage device in association with each frame time.

  The image composition unit 16 generates a free viewpoint video at the designated field of view and time. The field of view and time may be specified, for example, by a device that reproduces a free viewpoint video, or may be specified by a person who views the free viewpoint video. The image composition unit 16 acquires the deformed shape information at the frame time corresponding to the designated time. The acquired deformed shape information indicates the position and orientation of the subject of interest at that time. The image composition unit 16 renders an image in the designated visual field using the acquired deformed shape information. At this time, the image of the subject of interest is obtained by rendering using the deformed shape information. The image composition unit 16 may render the background image of the subject of interest based on model data indicating the background obtained in advance, or one or a plurality of frames with a near field of view in the corresponding simultaneous frame set You may render by performing image processing, such as an affine transformation, using an image. The image synthesizing unit 16 synthesizes the background image and the image of the subject of interest to generate a free viewpoint video at the designated field of view and time. When a moving image is requested, the image composition unit 16 may generate a video at a free viewpoint by repeatedly executing the above processing along the time axis.

  FIG. 6 is a flowchart illustrating a specific example of processing of the image composition device 10. Hereinafter, a specific example of the processing flow of the image composition device 10 will be described. First, the input unit 11 includes a plurality of moving images, camera parameters of the camera that captured each moving image, reference shape information indicating the three-dimensional shape of the subject of interest, reference joint information indicating the three-dimensional joint information of the subject of interest, and the subject of interest. Are received (steps S101 and S102). Next, the frame classification unit 12 assigns an estimation flag or a non-estimation flag to each simultaneous frame set (step S103).

  A control unit (not shown) selects a simultaneous frame set to be processed from unprocessed simultaneous frame sets (step S104). When the processing target simultaneous frame set is provided with the estimation flag, the two-dimensional joint information estimation unit 13 estimates the two-dimensional joint information of the subject of interest in the processing target simultaneous frame set (step S105). ). Next, the three-dimensional joint information estimation unit 14 estimates the two-dimensional joint information of the subject of interest in the same-time frame set to be processed (step S106).

  FIG. 7 is a flowchart showing details of the process in step S106. When the same time frame set to be processed has an estimation flag (step S201—YES), the 3D joint information calculation unit 141 of the 3D joint information estimation unit 14 performs processing. Specifically, the three-dimensional joint information calculation unit 141 receives input of two-dimensional joint information and reliability, and camera parameters of each camera (step S202). The three-dimensional joint information calculation unit 141 selects a predetermined number (for example, two) of frames with higher reliability for the joint to be processed (step S203). Then, the three-dimensional joint information calculation unit 141 estimates the three-dimensional information of the joint to be processed using the image of the selected frame and the camera parameter corresponding to the selected frame ( Step S204).

  In the process of step S201, when the non-estimation flag is assigned to the processing target simultaneous frame set (step S201-NO), the 3D joint information interpolation unit 142 of the 3D joint information estimation unit 14 performs the process. Do. Specifically, the three-dimensional joint information interpolation unit 142 acquires the three-dimensional joint information by performing an interpolation process using the three-dimensional joint information in the same-time frame set to which the estimation flag is assigned (step S205). . This is the end of the description of FIG.

  Returning to the description of FIG. The processing of S104 to S106 is executed for all the same time frame sets (step S107). Thereafter, the shape deforming unit 15 generates deformed shape information by deforming the reference shape based on the estimation result of the three-dimensional joint information in each simultaneous frame set, the reference joint information, and the deformation parameter (step S108).

  Thereafter, at the timing of generating the free viewpoint video, the image composition unit 16 renders the image of the subject of interest at the specified time and field of view using the deformed shape acquired in advance by the shape deforming unit 15 (step S109). ). Then, the image synthesis unit 16 generates and outputs a synthesized image by synthesizing the background image with the obtained image (step S110).

  In the image composition device 10 configured as described above, the following processing is performed in order to generate a free viewpoint video. Based on a plurality of actually captured moving images, 3D joint information of each joint of the subject of interest at each frame time is estimated. Based on the estimation result of the three-dimensional joint information, the reference shape of the subject of interest obtained in advance is deformed, and the deformed shape information of the subject of interest at each frame time is obtained. When the free viewpoint video is actually generated, the video of the subject of interest is generated by performing rendering in the designated field of view using the deformed shape information of the subject of interest at the designated time. For this reason, for example, the occlusion problem also occurs in a portion of the video that was not obtained in the shadow of a plurality of actually captured moving images (for example, the side of the subject of interest or the portion under the chin). Can be suppressed.

  In addition, when acquiring the deformed shape information at each frame time, the 3D joint information of the subject of interest is not estimated independently from the moving image at every frame time, but a part of the frame time (estimation flag is assigned). 3D joint information is estimated from the moving image only in the same time frame set at (frame time). In the moving image frame set at the remaining frame time (frame time to which the non-estimation flag is assigned), the interpolation processing is based on the estimation result in the same time frame set to which the estimation flag is assigned, not from the moving image. Three-dimensional joint information can be obtained. Therefore, flickering in the time direction is unlikely to occur at least between the same-time frame set to which the estimation flag is assigned and the same-time frame set to which the next estimation flag is given. Such processing makes it possible to suppress flickering in the time direction.

(Modification)
Although the object of processing by the image composition device 10 described above is a person, the object of processing is not necessarily limited to a person. The object of processing may be any organism or object having a joint. For example, an animal may be a target for processing. In this case, reference shape information, reference joint information, and deformation parameters obtained in advance are all information related to animals. For example, a robot may be a processing target. In this case, reference shape information, reference joint information, and deformation parameters obtained in advance are all information related to the robot.

  The image composition device 10 described above may be configured as a system in which a plurality of information processing devices are combined. For example, an apparatus including an input unit 11, a frame classification unit 12, a two-dimensional joint information estimation unit 13, a three-dimensional joint information estimation unit 14, and a shape deformation unit 15, and an apparatus including the input unit 11 and the image composition unit 16 , A system may be constructed.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Image composition apparatus, 11 ... Input part, 12 ... Frame classification | category part, 121 ... Frame separation part, 122 ... Joint information estimation flag provision part, 13 ... Two-dimensional joint information estimation part, 14 ... Three-dimensional joint information estimation part, DESCRIPTION OF SYMBOLS 141 ... Three-dimensional joint information calculation part 142 ... Three-dimensional joint information interpolation part 15 ... Shape deformation part 151 ... Joint displacement calculation part 152 ... Deformation part 16 ... Image composition part 202 ... Image generation part 211 ... Network construction unit, 212 ... parameter learning unit, 21 ... image acquisition unit, 22 ... image processing unit

Claims (6)

An input unit that receives input of a plurality of moving images, camera parameters related to the fields of view of the plurality of cameras that captured the moving images, and parameters related to the three-dimensional shape of the subject captured in the moving images;
For a frame at each time, a three-dimensional joint information estimation unit for estimating the three-dimensional joint information of the subject,
A shape deforming unit that acquires information indicating the three-dimensional shape of the subject at each time based on the estimation result of the three-dimensional joint information in the frame at each time and a parameter related to the three-dimensional shape of the subject;
An image composition unit that generates an image including an image of the subject in a designated field of view at a designated time based on information indicating the three-dimensional shape of the subject;
An image synthesizing apparatus.   The parameter related to the three-dimensional shape of the subject includes reference shape information indicating the three-dimensional shape of the subject, and reference joint information indicating information of each joint in the three-dimensional shape indicated by the reference shape information. Item 2. The image composition device according to Item 1. A frame classifying unit for determining an estimated frame that is a target for estimating joint information of the subject based on an image of the frame for each time frame constituting the moving image;
The three-dimensional joint information estimation unit estimates the three-dimensional joint information of the subject based on the image of the estimated frame for the estimated frame, and the three-dimensional joint in the estimated frame for frames other than the estimated frame The image synthesis apparatus according to claim 1, wherein the three-dimensional joint information is estimated by an interpolation process using a joint information estimation result. For a frame at each time, further comprising a two-dimensional joint information estimation unit that estimates the two-dimensional joint information of the subject and acquires the reliability of the estimation result,
The three-dimensional joint information estimation unit estimates the three-dimensional joint information based on only information about a part of frames for which the high reliability is acquired in the two-dimensional joint information estimation unit for the estimation frame. Item 4. The image composition device according to Item 3. An input step for receiving input of a plurality of moving images, camera parameters related to the fields of view of the plurality of cameras that captured the moving images, and parameters related to the three-dimensional shape of the subject captured in the moving images;
For a frame at each time, a three-dimensional joint information estimation step for estimating the three-dimensional joint information of the subject;
A shape deformation step of obtaining information indicating the three-dimensional shape of the subject at each time based on the estimation result of the three-dimensional joint information in the frame at each time and a parameter relating to the three-dimensional shape of the subject;
An image synthesis step for generating an image including an image of the subject in a designated field of view at a designated time based on information indicating the three-dimensional shape of the subject;
An image composition method comprising:   A computer program for causing a computer to function as the image composition device according to any one of claims 1 to 4.

Images