JP5402166B2 - Image composition apparatus and program - Google Patents

Description

The present invention relates to an image composition apparatus and a program for synthesizing a plurality of images.

  Conventionally, a technique for generating a composite image by combining a composite image with a background image or a frame image is known (see, for example, Patent Document 1).

JP 2004-159158 A

  However, if the background image is a moving image, the moving image in a state where the composition still image is fixed at the initially set composition position even if the composition position of the composition still image with respect to the background image is set. The image is merely generated, and the image is not interesting.

Therefore, an object of the present invention is to provide an image composition device and a program that can give a motion to a synthesized still image and can generate a highly interesting image.

In order to solve the above-described problem, the image composition device according to the first aspect of the present invention includes an acquisition unit that acquires a background moving image and a still image to be combined with the moving image, and a moving image acquired by the acquisition unit. Specifying means for specifying a moving object in an image; detecting means for detecting motion information of the moving object specified by the specifying means; and based on the motion information detected by the detecting means, the still image is converted into the moving image for background Synthesis means for synthesizing the still image and the moving image so as to operate on the image to generate a synthesized moving image, wherein the synthesizing means has a synthesis position of the still image in the moving image. in the case body is a position including the, characterized in that synthesizing said still image and the moving image to displace in response to the still image in the moving object motion information in the moving image There.

According to a second aspect of the present invention, in the image composition device according to the first aspect,
The image processing apparatus further includes designation means for designating a synthesis position of the still image in any one of the plurality of image frames constituting the moving image, and the synthesis means includes the one of the plurality of image frames. In the image frame, the still image is synthesized at the synthesis position designated by the designation unit, and in each image frame other than the one image frame, the still image synthesized in the one image frame. On the other hand, the still image is synthesized so as to be displaced according to the movement of the moving body.

According to a third aspect of the present invention, in the image composition device according to the second aspect,
The synthesizing unit synthesizes the one image frame in each image frame other than the one image frame when the synthesis position designated by the designation unit is a position including a moving object in the moving image. The still image is synthesized so that the still image is displaced in accordance with movement information of the moving object.

According to a fourth aspect of the present invention, in the image composition device according to the second or third aspect,
The detection means detects movement information of a moving object between at least one adjacent image frame among the plurality of image frames, and a movement of the moving object between the at least one adjacent image frames detected by the detection means. The image processing apparatus further includes a combination position setting unit that sets a combination position of the still image in each image frame other than the one image frame based on the information.

The invention according to claim 5 is the image composition device according to any one of claims 1 to 4,
The still image is an image obtained by extracting a subject area including a subject from a subject existing image in which a subject is present in a background.

The invention according to claim 6 is the image synthesizing apparatus according to any one of claims 1 to 5,
The motion information of the moving object includes at least one of position change information, size change information, and rotation angle change information of the moving object.

According to a seventh aspect of the present invention, there is provided a program for an image synthesizing apparatus, wherein a computer of an image synthesizing apparatus obtains a moving image for background and a still image to be synthesized with the moving image, and a moving object in the moving image obtained by the obtaining unit. Identification means for identifying the moving object, detection means for detecting the motion information of the moving object identified by the identification means, and operating the still image on the background moving image based on the motion information detected by the detection means And combining the still image and the moving image so as to generate a combined moving image, and the combining unit includes a moving object in the moving image at a combining position of the still image in the moving image. If the position, the still image and synthesizing the said video image to displace in response to the still image in the moving object motion information in the moving image It is a symptom.

  ADVANTAGE OF THE INVENTION According to this invention, a motion can be given to the still image synthesize | combined using the moving image as a background, and a synthetic animation with high interest property can be produced | generated.

It is a block diagram which shows schematic structure of the imaging device of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to subject clipping processing by the imaging apparatus of FIG. 1. It is a figure which shows typically an example of the image which concerns on the to-be-photographed object cutting process of FIG. 3 is a flowchart illustrating an example of an operation related to a moving image generation process by the imaging apparatus of FIG. 1. It is a figure which shows typically an example of the image which concerns on the moving image generation process of FIG. 3 is a flowchart illustrating an example of an operation related to a synthetic moving image generation process performed by the imaging apparatus of FIG. 1. FIG. 7 is a continuation flowchart showing a continuation of the synthetic moving image generation process of FIG. 6. FIG. 8 is a continuation flowchart showing a continuation of the synthetic moving image generation process of FIGS. 6 and 7. FIG. It is a flowchart which shows an example of the operation | movement which concerns on the image composition process in the synthetic | combination moving image production | generation process of FIGS. It is a figure which shows typically an example of the image which concerns on the synthetic | combination moving image production | generation process of FIGS.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment to which the present invention is applied.
The imaging apparatus 100 according to the present embodiment detects the motion information of the moving object B (see FIG. 5) in the background moving image P4 (see FIG. 5), and matches the motion of the moving object B based on the detected motion information. The still image P3 and the moving image P4 are synthesized so that the subject area S (see FIG. 3C) of the still image P3 for synthesis is operated in a predetermined manner, and the synthesized moving image M (see FIG. 10B) is obtained. Generate.
Specifically, as illustrated in FIG. 1, the imaging apparatus 100 includes a lens unit 1, an electronic imaging unit 2, an imaging control unit 3, an image data generation unit 4, an image memory 5, and a motion detection unit 6. An image processing unit 7, a recording medium 8, a display control unit 9, a display unit 10, an operation input unit 11, and a CPU 12.
In addition, the imaging control unit 3, the motion detection unit 6, the image processing unit 7, and the CPU 12 are designed as, for example, a custom LSI 1A.

The lens unit 1 includes a plurality of lenses and includes a zoom lens, a focus lens, and the like.
Although not shown, the lens unit 1 includes a zoom drive unit that moves the zoom lens in the optical axis direction and a focus drive unit that moves the focus lens in the optical axis direction when imaging a subject. May be.

  The electronic imaging unit 2 is composed of an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-oxide Semiconductor), for example, and converts an optical image that has passed through various lenses of the lens unit 1 into a two-dimensional image signal. To do.

Although not shown, the imaging control unit 3 includes a timing generator, a driver, and the like. Then, the imaging control unit 3 scans and drives the electronic imaging unit 2 with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 2 every predetermined period, and the electronic imaging unit 2 Image frames are read out from the imaging area for each screen and output to the image data generation unit 4.
In addition, the imaging control unit 3 performs adjustment control of conditions when imaging a subject such as AF (automatic focusing process), AE (automatic exposure process), and AWB (automatic white balance).

The lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 configured in this way have a subject existing image P 1 (see FIG. 3A) in which a subject exists in the background and a background moving image P 4 (see FIG. 5). Image).
In addition, the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 capture the subject clipped image P3 (FIG. 3) in a state where the imaging conditions for capturing the subject presence image P1 are fixed after the subject presence image P1 is captured. A subject non-existence image P2 (see FIG. 3B) for generating (see (c)) is captured.

The image data generation unit 4 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the image frame transferred from the electronic imaging unit 2, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to an image memory 5 used as a buffer memory via a DMA controller (not shown).

  The image memory 5 is composed of, for example, a DRAM or the like, and temporarily stores data processed by the motion detection unit 6, the image processing unit 7, the CPU 12, and the like.

  The motion detection unit 6 analyzes the motion of the subject using a plurality of images taken at different times. Specifically, as shown in FIG. 1, the motion detection unit 6 includes a feature amount calculation unit 6a and a block matching unit 6b.

The feature amount calculation unit 6a performs feature extraction processing for extracting feature points from the subject non-existing image P2 with reference to the subject non-existing image P2. Specifically, the feature amount calculation unit 6a selects a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the subject non-existing image P2, and the block Is extracted as a template (for example, a square of 16 × 16 pixels).
Here, the feature extraction process is a process of selecting a feature having a high characteristic convenient for tracking from a large number of candidate blocks.

  The block matching unit 6b performs block matching processing for aligning the subject non-existing image P2 and the subject existing image P1. Specifically, the block matching unit 6b determines where the template extracted in the feature extraction process corresponds in the subject existing image P1, that is, the pixel value of the template optimally matches in the subject existing image P1. Search for a position (corresponding region). Then, an optimum offset between the subject non-existing image P2 and the subject existing image P1 having the best evaluation value (for example, sum of squared differences (SSD), sum of absolute differences (SAD), etc.) is calculated. Calculated as a template motion vector.

Further, the block matching unit 6b, for example, to an arbitrary reference image frame (for example, an image frame f3) desired by the user constituting the moving image P4 based on a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user. When the composite position of the subject region S in the subject cutout image P3 is designated and input, an image frame adjacent to the reference image frame (for example, the image frame f2 immediately before the reference image frame) using the composite position as a template. The position (corresponding region) where the pixel value of the template matches optimally is searched. Then, the reference image frame having the best evaluation value (for example, difference sum of squares (SSD), difference sum of absolute values (SAD), etc.) of the pixel value and the image frame immediately before the reference image frame are evaluated. An optimal offset between them is calculated as the motion vector of the template. Then, for each of the plurality of calculated motion vectors, one of the motion vectors is compared with another motion vector as an evaluation target, and it is determined whether or not to vote for the motion vector of the evaluation target. The motion vector of the moving object B in the image frame is calculated by majority vote.
Further, the same processing as described above is performed for an image frame (for example, the image frame f1 immediately before the reference image frame) adjacent to the image frame (for example, the image frame f2) for which the motion vector of the moving object B is calculated. To calculate the motion vector of the moving object B of the image frame. By performing these processes on all image frames (for example, image frames f1, f2, f4, and f5) other than the reference image frame constituting the moving image P4, the motion of the moving object B in each of the image frames f1 to f5 Calculate the vector.

The image processing unit 7 includes an alignment unit 7a that performs alignment between the subject presence image P1 and the subject non-existence image P2.
The alignment unit 7a calculates a coordinate conversion formula (projection conversion matrix) of each pixel of the subject presence image P1 with respect to the subject non-existence image P2 based on the feature points extracted from the subject non-existence image P2, and the coordinate conversion formula Accordingly, the subject presence image P1 is coordinate-converted to align with the subject non-existence image P2.

  Further, the image processing unit 7 generates difference information of corresponding pixels between the subject presence image P1 and the subject non-existence image P2 that are aligned by the alignment unit 7a, and the subject exists based on the difference information. A subject area extraction unit 7b that extracts a subject area S including a subject from the image P1 is provided.

In addition, the image processing unit 7 specifies a position of the subject region S extracted in the subject presence image P1, and generates a position information generation unit 7c that generates position information indicating the position of the subject region S in the subject presence image P1. It has.
Here, as the position information, for example, an alpha map can be cited. The alpha map is a weight for alpha blending the image of the subject area S with respect to a predetermined background for each pixel of the subject existing image P1. It is expressed as a value (0 ≦ α ≦ 1).

Further, the image processing unit 7 does not transmit a pixel having an alpha value of 1 out of each pixel of the subject existing image P1 to a predetermined single color image (not shown) based on the generated alpha map. In addition, a cut-out image generation unit that generates image data of the cut-out image P3 of the subject (see FIG. 3C) by combining the image of the subject with a predetermined single-color image so that the pixels with an alpha value of 0 are transmitted. 7d.
As a result, the cutout image generation unit 7d acquires a subject cutout image P3 obtained by cutting out a region including the subject from the subject existing image P1 in which the background and the subject exist.

In addition, the image processing unit 7 includes an acquisition unit 7e that acquires a moving image P4 including a plurality of image frames f1 to f5 and a subject cutout image P3 as a still image for synthesis.
Specifically, the acquisition unit 7e performs a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user among a plurality of pieces of image data recorded on the recording medium 8 in the synthetic moving image generation process (described later). The image data of the instructed moving image P4 and subject cutout image P3 is read and developed in the image memory 5.

In addition, the image processing unit 7 includes a composite position setting unit 7f that sets a composite position of the subject region S of the subject cutout image P3 in each of the image frames f1 to f5 of the moving image P4.
Specifically, the subject clipping in the reference image frame (for example, the image frame f3 (see FIG. 10A)) of the moving image P4 set based on a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user. When the reference composite position (for example, center coordinates) of the subject region S of the image P3 is designated, the composite position setting unit 7f is an image in which the subject cutout image P3 is superimposed around the reference composite position in the reference image frame. Set the region as a composite region.
The composite position setting unit 7f then generates a reference image frame (for example, an image) based on the motion vector of the moving object B between adjacent image frames (for example, the image frame f2 with respect to the image frame f3) detected by the motion detection unit 6. The composite position of the subject area S of the subject cutout image P3 in each image frame (for example, image frames f1, f2, f4, and f5) other than the frame f3 (see FIG. 10A) is calculated. Specifically, the composition position setting unit 7f is a reference image frame (e.g., image frame f3 (see FIG. 5) that is searched when the motion detection unit 6 detects the motion vector of the moving object B in each of the image frames f1 to f5. 10 (a))) is specified as the combined position of the subject region S of the subject cutout image P3 in the image frame (for example, the image frame f2) related to the detection of the motion vector. To do. For example, the composite position setting unit 7f calculates the composite position of the subject region S of the subject clipped image P3 in the image frame f2 based on the detection result of the motion vector of the moving object B in the image frame f2 by the motion detection unit 6. Then, an image area on which the subject cutout image P3 is superimposed with the synthesis position in the image frame f2 as the center is set as a synthesis area. Subsequently, the composite position setting unit 7f performs the same process on the image frame f1 with the image frame f2 as a reference. In this way, by repeating up to the first image frame f1 of the moving image P4, an image frame (for example, the image frame f1,...) Preceding the reference image frame (for example, the image frame f3 (see FIG. 10A)). For f2), the composition position of the subject region S in the subject cutout image P3 is set.
Similarly, the composition position setting unit 7f also applies to the image frame f4 that is the next image frame of the image frame f3 detected by the motion detection unit 6 and the image frame f5 that is the second image frame. An image area on which the subject cutout image P3 is superimposed is set as a synthesis area.
As described above, the composite position setting unit 7f is based on the motion information of the moving object B between at least one adjacent image frame (for example, the image frame f2 with respect to the image frame f3) detected by the motion detection unit 6. The composition position of the subject region S of the subject cutout image P3 in each image frame (for example, the image frames f1, f2, f4, and f5) other than the frame (for example, the image frame f3 (see FIG. 10A)) is set.

  The image processing unit 7 includes an image composition unit 7g that synthesizes the subject cut-out image P3 and the image frames f1 to f5 constituting the background moving image P4. Specifically, the image composition unit 7g designates each image frame f1 to f5 of the moving image P4 as a background image, and among the pixels of the designated image frames f1 to f5, a pixel having an alpha value of 0 Is transmitted, and the pixel with the alpha value of 1 is overwritten with the pixel value of the corresponding pixel of the subject cutout image P3, and among the pixels of the designated image frames f1 to f5, the alpha value is 0 <α <1. After generating an image (background image × (1-α)) obtained by cutting out the subject region S using 1's complement (1-α), 1's complement (1-α) in the alpha map is used. When the subject cutout image P3 is generated, a value blended with the single background color is calculated, and the value is subtracted from the subject cutout image P3, and the image is cut out from the subject region S (background image × (1− a)).

Further, the image composition unit 7g includes a composition control unit 7h that controls composition of the image frames f1 to f5 constituting the moving image P4 of the subject cutout image P3. That is, the image composition unit 7g adjusts the composition position, composition region, and the like of the subject area S in the subject cutout image P3 for each of the image frames f1 to f5 constituting the moving image P4 under the control of the composition control unit 7h. A plurality of image frames g1 to g5 are generated by image synthesis, and a synthesized moving image M including the plurality of image frames g1 to g5 is generated.
Specifically, the composition control unit 7h is designated based on a predetermined operation of the operation input unit by the user based on the composition position and composition region of the subject region S in the subject cutout image P3 set by the composition position setting unit 7f. In the reference image frame (for example, image frame f3), the subject region S is combined at the combination position designated by the user in the reference image frame. Then, the composition control unit 7h cuts out the subject in each of the image frames f1 to f5 other than the reference image frame so as to be displaced in accordance with the movement of the moving object B with respect to the subject region S synthesized with the image frames f1 to f5. The images P3 are sequentially synthesized. For example, as shown in FIGS. 10A and 10B, in the image frame f2 immediately preceding the reference image frame, the composition control unit 7h sets the subject area S set by the composition position setting unit 7f. Based on the combination position, the subject cutout image P3 is synthesized so that the subject region S is displaced to the left in accordance with the movement of the moving object B with respect to the subject region S in the image frame g3, and the image frame constituting the synthesized moving image M g2 is generated. Similarly, for another image frame (for example, image frame f1) before the reference image frame, an image frame (for example, image frame) immediately after the image frame to be processed (for example, image frame f1) is processed. An image frame g1 constituting the combined moving image M is generated so as to be displaced to the left in accordance with the movement of the moving body B with respect to the combined position of the subject area S in f2).
In addition, in the image frame f4 immediately after the reference image frame, the composition control unit 7h defines the subject area S as the subject in the image frame g3 based on the composition position of the subject area S set by the composition position setting unit 7f. The subject cutout image P3 is synthesized so as to be displaced to the right in accordance with the movement of the moving body B with respect to the region S to generate an image frame g4 constituting the synthesized moving image M. Similarly, for another image frame (for example, image frame f5) after the reference image frame, the image frame (for example, image frame) immediately before the image frame (for example, image frame f5) to be processed is processed. An image frame g5 constituting the synthesized moving image M is generated so as to be displaced to the right according to the movement of the moving body B with respect to the synthesized position of the subject area S in f4).
In this way, the image composition unit 7g performs the subject cutout image P3 to perform a predetermined operation in accordance with the motion of the moving body B of the moving image P4 based on the motion vector detected by the motion detection unit 6. The synthesized moving image M is generated by synthesizing P3 and the moving image P4.

The recording medium 8 is configured by, for example, a non-volatile memory (flash memory) or the like, and stores still image data of the subject cutout image P3 encoded by the JPEG compression unit (not shown) of the image processing unit 7.
The image data of the subject clipped image P3 is associated with the alpha map generated by the position information generating unit 7c of the image processing unit 7, and the extension of the image data of the subject clipped image P3 is set to “.jpe”. Saved.

Further, the recording medium 8 uses a coding unit (not shown) of the image processing unit 7 to generate a plurality of image frames f generated by the imaging of the moving body B by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. A moving picture P4 encoded in a predetermined compression format is stored.
The moving image P4 is obtained by recording the motion of the moving object B in a predetermined recording method (for example, Motion-JPEG format). For example, a plurality of continuous image frames (for example, n) captured at a predetermined imaging frame rate. Image frames) f,... In FIG. 5, the moving image P <b> 4 is typically represented by five image frames f <b> 1 to f <b> 5.

The display control unit 9 performs control for reading display image data temporarily stored in the image memory 5 and displaying the read image data on the display unit 10.
Specifically, the display control unit 9 includes a VRAM, a VRAM controller, a digital video encoder, and the like. The digital video encoder periodically reads the luminance signal Y and the color difference signals Cb and Cr read from the image memory 5 and stored in the VRAM (not shown) under the control of the CPU 12 from the VRAM via the VRAM controller. Are read out, a video signal is generated based on these data, and is output to the display unit 10.

  The display unit 10 is, for example, a liquid crystal display device, and displays an image captured by the electronic imaging unit 2 on the display screen based on a video signal from the display control unit 9. Specifically, the display unit 10 displays a live view image based on a plurality of image frames f generated by imaging the subject by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 in the imaging mode. Or a REC view image captured as the actual captured image.

  The operation input unit 11 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 11 selects and determines a shutter button 11a related to a subject photographing instruction, a selection instruction such as an imaging mode and a function, and a setting instruction for setting a reference composite position of the subject region S in the subject cutout image P3. A button 11b, a zoom button (not shown) relating to a zoom amount adjustment instruction, and the like are provided, and a predetermined operation signal is output to the CPU 12 in accordance with the operation of these buttons.

  The CPU 12 controls each unit of the imaging device 100. Specifically, the CPU 12 performs various control operations in accordance with various processing programs (not shown) for the imaging apparatus 100.

Next, the subject clipping process performed by the imaging apparatus 100 will be described with reference to FIGS.
FIG. 2 is a flowchart illustrating an example of an operation related to the subject clipping process. FIG. 3 is a diagram schematically illustrating an example of an image related to the subject clipping process.
The subject clipping process is executed when the subject clipping mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user. It is.

  As shown in FIG. 2, first, the CPU 12 performs live display on the display control unit 9 based on a plurality of image frames f generated by imaging the subject by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. The view image is displayed on the display screen of the display unit 10 and is superimposed on the live view image to display an imaging instruction message for the subject existing image P1 on the display screen of the display unit 10 (step S1).

Next, the CPU 12 determines whether an imaging instruction is input based on a predetermined operation of the shutter button 11a of the operation input unit 11 by the user (step S2). If it is determined here that an imaging instruction has been input (step S2; YES), the CPU 12 instructs the imaging control unit 3 to focus the focus lens, exposure conditions (shutter speed, aperture, amplification factor, etc.) and white. By adjusting conditions such as balance, an optical image of the subject existing image P1 (see FIG. 3A) is picked up by the electronic image pickup unit 2 under a predetermined condition, and the image data generation unit 4 causes the electronic image pickup unit 2 to YUV data of the image frame of the transferred subject presence image P1 is generated (step S3). The YUV data of the subject existing image P1 is temporarily stored in the image memory 5.
Further, the CPU 12 controls the imaging control unit 3 to maintain a state in which conditions such as a focus position, an exposure condition, and a white balance at the time of imaging the subject presence image P1 are fixed.

Then, the CPU 12 causes the display control unit 9 to display a live view image on the display screen of the display unit 10 based on a plurality of image frames generated by imaging the subject by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. In addition to the display, the image of the semi-transparent display mode of the subject presence image P1 and the imaging instruction message of the subject non-existence image P2 are superimposed on the live view image and displayed on the display screen of the display unit 10 (step S4).
Thereafter, the CPU 12 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 11a of the operation input unit 11 by the user (step S5). Then, after the user moves the subject out of the angle of view or waits for the subject to move, the user adjusts the camera position so that the subject non-existing image P2 overlaps the semi-transparent image of the subject existing image P1. When it is determined that the shutter button 11a of the operation input unit 11 is operated for a predetermined time and an imaging instruction is input (step S5; YES), the CPU 12 causes the imaging control unit 3 to display the subject non-existing image P2 (FIG. 3). The optical image (see (b)) is captured by the electronic imaging unit 2 under conditions fixed after the subject presence image P1 is captured, and the subject non-existence image P2 transferred from the electronic imaging unit 2 to the image data generation unit 4 is captured. YUV data of the subject non-existing image P2 is generated based on the image frame (step S6). The YUV data of the subject existing image P1 is temporarily stored in the image memory 5.

Next, the CPU 12 uses the YUV data of the subject non-existing image P2 temporarily stored in the image memory 5 in the feature amount calculating unit 6a, the block matching unit 6b, and the image processing unit 7 as a reference. A projection transformation matrix for projectively transforming data is calculated using a predetermined image transformation model (for example, a similarity transformation model or a joint transformation model) (step S7).
Specifically, the feature amount calculation unit 6a selects a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the subject non-existing image P2, and the block Is extracted as a template. Then, the block matching unit 6b searches the subject existing image P1 for a position where the pixel value of the template extracted by the feature extraction process is optimally matched, and the evaluation value of the difference between the pixel values is the best. An optimal offset between the subject non-existing image P2 and the subject existing image P1 is calculated as a motion vector of the template. Then, the alignment unit 7a of the image processing unit 7 statistically calculates the entire motion vector based on the motion vectors of the plurality of templates calculated by the block matching unit 6b, and performs feature point correspondence related to the motion vector. Using this, a projective transformation matrix of the subject existing image P1 is calculated.

  Next, the CPU 12 causes the alignment unit 7a to perform the projective transformation on the subject existing image P1 based on the calculated projective transformation row example, so that the YUV data of the subject existing image P1 and the YUV data of the subject nonexistent image P2 Is performed (step S8).

Then, the CPU 12 causes the subject region extraction unit 7b of the image processing unit 7 to perform a process of extracting the subject region S including the subject from the subject presence image P1 (step S9).
Specifically, the subject region extraction unit 7b applies a low-pass filter to each of the YUV data of the subject presence image P1 and the YUV data of the subject non-existence image P2 to remove the high-frequency component of each image. Thereafter, the subject area extraction unit 7b calculates a difference for each corresponding pixel between the subject existing image P1 and the subject non-existing image P2 subjected to the low-pass filter, and generates a difference map. Subsequently, the subject region extraction unit 7b binarizes the dissimilarity map relating to each pixel with a predetermined threshold value, and then performs a contraction process to remove a region in which the dissimilarity is caused by fine noise or camera shake from the dissimilarity map. Do. Thereafter, the subject region extraction unit 7b performs a labeling process to remove a region below a predetermined value or a region other than the maximum region, and then specifies the largest island pattern as the subject region S and corrects the contraction. Expansion processing is performed.

  Next, the CPU 12 causes the position information generation unit 7c of the image processing unit 7 to generate an alpha map indicating the position of the extracted subject region S in the subject existing image P1 (step S10).

Thereafter, the CPU 12 performs processing for generating image data of a subject cutout image P3 (see FIG. 3C) obtained by combining the subject image with a predetermined single color image in the cutout image generation unit 7d of the image processing unit 7. (Step S11).
Specifically, the cut-out image generation unit 7d reads out the subject existing image P1, the single color image, and the alpha map and develops them in the image memory 5, and thereafter, for all the pixels of the subject existing image P1, the pixels having an alpha value of 0. (Α = 0) for transmission, and for pixels with an alpha value of 0 <α <1, (0 <α <1), blending with a predetermined single color is performed, and for pixels with an alpha value of 1 (α = 1) Do nothing and do not transmit light to a predetermined single color.

Thereafter, the CPU 12 associates the alpha map generated by the position information generation unit 7c of the image processing unit 7 with the image data of the subject clipped image P3 in a predetermined storage area of the recording medium 8, and stores the clipped image P3 of the subject clipped image P3. The extension of the image data is stored as one file with “.jpe” (step S12).
Thereby, the subject clipping process is completed. As a result, for example, still image data of the subject cutout image P3 in which a person (see FIG. 3C) is extracted as the subject region S from the background is generated.

Next, background moving image generation processing by the imaging apparatus 100 will be described with reference to FIGS. 4 and 5.
FIG. 4 is a flowchart illustrating an example of an operation related to a background moving image generation process. FIG. 5 is a diagram schematically illustrating an example of an image related to the background moving image generation process.
The background moving image generation process is a normal moving image imaging process, and is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user. This is a process executed when a moving image capturing mode is instructed to be selected.

  As shown in FIG. 4, first, the CPU 12 causes the display control unit 9 to perform live view based on a plurality of image frames generated by imaging the moving image P4 by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. The image is displayed on the display screen of the display unit 10 (step S21).

Next, the CPU 12 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 11a of the operation input unit 11 by the user (step S22). If it is determined that an imaging instruction has been input (step S22; YES), the CPU 13 instructs the imaging control unit 3 to focus the focus lens, exposure conditions (shutter speed, aperture, amplification factor, etc.) and white. The moving image transferred from the electronic image capturing unit 2 to the image data generating unit 4 by adjusting the conditions such as the balance, causing the electronic image capturing unit 2 to capture an optical image of the moving image P4 (see FIG. 5) under a predetermined condition. YUV data of the image frames f1 to f5 of P4 is generated (step S23). Thereafter, the CPU 12 stores the YUV data of the image frames f1 to f5 of the moving image P4 in the Motion-JPEG format in a predetermined storage area of the recording medium 8 (step S24).
Thus, the moving image generation process is terminated. As a result, for example, image data of a moving image P4 including a plurality of image frames f1 to f5 in which the track travels from the left side to the right side is generated.

Next, the synthetic moving image generation process will be described in detail with reference to FIGS.
6 to 8 are flowcharts illustrating an example of operations related to the synthetic moving image generation process. FIG. 9 is a flowchart illustrating an example of an operation related to the image composition process in the composite moving image generation process. FIGS. 10A and 10B are diagrams schematically illustrating an example of an image that is combined in the combined moving image generation process.

The synthesized moving image generation process is executed when an image synthesis mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user. It is processing.
As shown in FIG. 6, based on a predetermined operation of the operation input unit 11 by the user, a subject cutout image P3 as a still image for synthesis is selected and specified from among a plurality of images recorded on the recording medium 8. Then, the acquisition unit 7e of the image processing unit 7 reads out the image data of the designated subject cutout image P3 and develops it in the image memory 5 (step S31).

  Next, when a desired moving image P4 is selected and specified from among a plurality of images recorded on the recording medium 8 based on a predetermined operation of the operation input unit 11 by the user, the image processing unit 7 obtains it. The unit 7e reads out the image data of the designated moving image P4 and develops it in the image memory 5. Then, the CPU 12 causes the display control unit 9 to switch the image frames f1 to f5 constituting the moving image P4 at a predetermined display frame rate based on the image data of the moving image P4 developed in the image memory 5. Reproduction is displayed on the display screen of the display unit 10 (step S32).

Subsequently, the CPU 12 determines whether or not an instruction to select the image frames f1 to f5 reproduced and displayed on the display screen of the display unit 10 is input based on a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user. Determination is made (step S33).
If it is determined that the selection instruction for the image frames f1 to f5 has been input (step S33; YES), the CPU 12 causes the display control unit 9 to input the image frame desired by the user who has input the selection instruction (for example, an image). The frame f3 (see FIG. 10A)) is temporarily displayed on the display screen of the display unit 10 as a reference image frame (step S34).

Then, based on a predetermined operation of the selection determination button 11b of the operation input unit 11 by the user, the CPU 12 determines whether or not an instruction to specify a composite position of the subject area S in the subject cutout image P3 with respect to the reference image frame is input. Determination is made (step S35).
Here, if it is determined that an instruction for specifying a composite position has been input (step S35; YES), the CPU 12 combines the image frames f1 to f5 of the moving image P4 and the subject cutout image P3 to generate a composite moving image M. The image composition unit 7g is caused to perform image composition processing for generating the image frames g1 to g5 to be configured (step S36).

Here, the image composition processing will be described in detail with reference to FIG.
As shown in FIG. 9, the image composition unit 7g reads the alpha map stored in association with the subject cutout image P3 and develops it in the image memory 5 (step S61).
Note that when the synthesis position of the subject area S in the subject cutout image P3 is designated in step S35 of FIG. 6, the reference image frame (for example, the image frame f3 (see FIG. 10A)) of the moving image P4 and For an area that deviates from the alpha map and is out of the alpha map range, α = 0 is set so that no alpha value exists.

  Next, the image composition unit 7g designates any one pixel (for example, the pixel at the upper left corner) of the reference image frame (step S62), and processes the pixel based on the alpha value of the alpha map. Branch (step S63). Specifically, the image composition unit 7g, for any one of the pixels of the reference image frame, for the pixel having an alpha value of 1 (step S63; α = 1), the corresponding pixel of the subject cutout image P3. The pixel value is overwritten (step S64), and the subject area S is cut out using the one's complement (1-α) for the pixel with an alpha value of 0 <α <1 (step S65; 0 <α <1). After the image (background image × (1-α)) is generated, the blended value with the single background color is calculated when the subject cutout image P3 is generated using the one's complement (1-α) in the alpha map. Then, the value is subtracted from the subject cutout image P3, and is combined with an image obtained by cutting out the subject area S (background image × (1-α)) (step S65). Step S63; α = ), The reference image frame so as to transmit without doing anything.

Subsequently, the image composition unit 7g determines whether or not all the pixels of the reference image frame have been processed (step S66).
If it is determined that all the pixels have not been processed (step S66; NO), the image composition unit 7g designates the next pixel as the processing target and moves the processing target to the pixel (step S66). (S67) The process proceeds to step S63.
By repeating the above processing until it is determined in step S66 that all pixels have been processed (step S66; YES), the image composition unit 7g composes the subject clipped image P3 and the moving image P4. One image frame g constituting the moving image M is generated.
Thereby, the image composition process is terminated.

As shown in FIG. 6, thereafter, the CPU 12 determines whether or not the reference image frame is the first image frame f1 among the image frames f1 to f5 constituting the moving image P4 (step S37).
If it is determined that the reference image frame is not the first image frame f1 (step S37; NO), the CPU 12 moves the processing target to the image frame immediately before the reference image frame (step S38). Specifically, as shown in FIG. 10 (a), when an image frame f3 is selected as a reference image frame, the image frame f2 is moved to the previous image frame f3.

  Then, the CPU 12 detects the motion vector of the moving object in the image frame (for example, the image frame f2) with respect to the reference image frame as the motion information of the moving object B between the adjacent image frames (for example, the image frame f3 and the image frame f2). It is made to detect by the part 6 (step S39).

Then, based on the motion vector (motion information) of the moving object B of the image frame (for example, image frame f2) detected by the motion detection unit 6, the CPU 12 cuts out the subject in the image frame (for example, image frame f2). The composite position setting unit 7f calculates the composite position of the subject area S of P3 (step S40).
Then, the composition control unit 7h, when compositing the subject cutout image P3 to the image frame (for example, the image frame f2) that has been moved forward one time based on the calculated composition position, It is determined whether or not the part is in the image frame f2 (step S41).
Here, when it is determined that a part of the synthesis area is within the image frame f2 (step S41; YES), the CPU 12 cuts out the subject clipped image in the immediately preceding calculated image frame (for example, the image frame f2). The composite position of the subject area S of P3 is set in the composite position setting unit 7f (step S42).

Then, the CPU 12 synthesizes an image frame (for example, image frame f2) of the moving image P4 and the subject cutout image P3 to generate an image frame (for example, image frame g2) that forms the combined moving image M. The image composition unit 7g is caused to perform the process (step S43; see FIG. 9).
The contents of the image composition process are substantially the same as those described above, and a detailed description thereof is omitted. In addition, when the composition position of the subject area S in the subject cutout image P3 is set in step S42 in FIG. 7, the image frames f1 to f5 of the moving image P4 are shifted from the alpha map, and are outside the range of the alpha map. For such a region, α = 0 is set so as not to generate a region having no alpha value.

As shown in FIG. 7, thereafter, the CPU 12 determines whether or not the synthesis process has been performed for all image frames (for example, image frames f1 and f2) before the reference image frame (step S44).
Here, if it is determined that the composition process has not been performed for all image frames (for example, image frames f1 and f2) before the reference image frame (step S44; NO), the process returns to step S38, and the reference image frame. The CPU 12 causes the processes in and after step S38 to be performed until all the previous image frames (for example, image frames f1 and f2) have been combined. On the other hand, if it is determined that all image frames prior to the reference image frame (for example, image frames f1 and f2) have been combined (step S44; YES), that is, a part of the combined region is determined in step S41. Is not within the image frame (step S41; NO), for example, when a part of the composite region of the subject region S of the subject cutout image P3 in the image frame f1 is not in the image frame f1. In step S44, the CPU 12 determines that all the image frames before the reference image frame (for example, the image frames f1 and f2) have been combined. Note that in step S41, for an image frame (for example, image frame f1) that is determined not to be part of the image frame (for example, image frame f1), the image frame (for example, image frame f1) of the moving image P4 is used. The image frame f1) is regarded as an image frame (for example, an image frame g1) of the synthesized moving image M.

  Thereafter, the CPU 12 moves the processing target to the next image frame (for example, the image frame f4) after the reference image frame (for example, the image frame f3) (step S45), and the CPU 12 performs the adjacent processing similarly to step S39. The motion detector 6 is caused to detect motion information of the moving object B between matching image frames (for example, the image frame f4 with respect to the image frame f3) (step S47).

Then, based on the motion vector (motion information) of the moving object B of the image frame (for example, image frame f4) detected by the motion detection unit 6, the CPU 12 cuts out the subject in the image frame (for example, image frame f4). The composite position setting unit 7f is caused to calculate the composite position of the subject area S of P3 (step S48).
Then, the composition control unit 7h, when compositing the subject clipped image P3 to the image frame (for example, the image frame f4) moved one after based on the calculated composition position, the subject region S in the subject clipped image P3. It is determined whether or not a part of the synthesis area is within the image frame (for example, image frame f4) (step S49).
Here, when it is determined that a part of the composite area is within the image frame (for example, image frame f4) (step S49; YES), the CPU 12 calculates the next image frame (for example, image frame). The composition position setting unit 7f sets the composition position of the subject region S of the subject cutout image P3 in step f4) (step S50).

Then, the CPU 12 combines image frames (for example, an image frame f4) of the moving image P4 and the subject cutout image P3 to generate an image combination process for generating an image frame (for example, an image frame g4) that forms the combined moving image M. The image composition unit 7g is caused to perform the process (step S51; see FIG. 9).
The contents of the image composition process are substantially the same as those described above, and a detailed description thereof is omitted. Also, when the composition position of the subject region S in the subject cutout image P3 is set in step S50 of FIG. 8, the image frames f1 to f5 of the moving image P4 are shifted from the alpha map, and are outside the range of the alpha map. For such a region, α = 0 is set so as not to generate a region having no alpha value.

As shown in FIG. 8, thereafter, the CPU 12 determines whether or not the synthesis process has been performed for all the image frames f1 to f5 (step S52).
If it is determined that the composition process has not been performed for all the image frames f1 to f5 (step S52; NO), the process returns to step S46, and the CPU 12 continues until the composition process is performed for all the image frames f1 to f5. Causes the processing after step S46 to be performed. On the other hand, if it is determined that the composition processing has been performed for all the image frames f1 to f5 (step S52; YES), the image composition unit 7g configures all the image frames g1 to g5 (FIG. ))). Note that in step S49, for an image frame (for example, image frame f5) that is determined to have a part of the synthesis area not in the image frame (for example, image frame f5), the image frame (for example, image frame f5) The image frame f5) is regarded as an image frame (for example, an image frame g5) of the synthesized moving image M.

If it is determined in step S37 of FIG. 6 that the reference image frame is the first image frame f1 (step S37; YES), the CPU 12 moves the processing target to the image frame f2 immediately after the reference image frame. (Step S46) As described above, the processing after Step S47 is performed.
As a result, the combined moving image generation process is terminated.

As described above, according to the imaging apparatus 100 of the present embodiment, among the plurality of image frames f1 to f5 of the background moving image P4, between adjacent image frames (for example, the image frame f2 with respect to the image frame f3). The motion detection unit 6 detects the motion information of the moving body B, and based on the motion information, a plurality of image frames f1 of the subject cutout image P3 and the moving image P4 so that the subject cutout image P3 is operated on the moving image P4. -F5 are combined with each other to generate a combined moving image M by the image combining unit 7g.
Therefore, when the image frames f1 to f5 of the background moving image P4 and the subject cutout image P3, which is a still image for combining, are combined, the combining still image is fixed at the initially set combining position. Therefore, the subject cutout image P3 can be operated in a predetermined manner in accordance with the movement of the moving object B in the moving image P4, and a highly interesting synthetic video M can be generated.

In addition, the image composition unit 7g uses a reference image frame (for example, image frame f3) designated based on a predetermined operation of the operation input unit 11 by the user among the plurality of image frames f1 to f5 constituting the moving image P4. Synthesizes the subject clipped image P3 at a desired reference composition position, and in each image frame other than the reference image frame, the composition position setting unit 7f detects the interval between adjacent image frames detected by the motion detection unit 6 ( For example, based on the motion information of the moving body B of the image frame f2) with respect to the image frame f3, the composition position of the subject cutout image P3 in each image frame other than the reference image frame is set, and the subject cutout synthesized with the reference image frame The subject cutout image P3 is synthesized so as to be displaced in accordance with the movement of the moving object B with respect to the image P3. Generating a plurality of image frames g1~g5 constituting the formed video M.
As a result, in the user-desired image frame (reference image frame), the composition position of the subject cutout image P3 is automatically set for each image frame other than the reference image frame by designating the composition position of the subject cutout image P3. Therefore, the operation related to the generation of the synthesized moving image M can be simplified. Therefore, it is possible to easily generate a highly interesting synthesized moving image M simply by appropriately specifying a user-desired synthesis position.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, as the above embodiment, the motion vector (motion information) of each moving object B between the image frames f1 to f5 constituting the moving image P4 is calculated. However, the present invention is not limited to this. If it is determined that the object is displaced in a certain direction at a constant interval, the combined position of the subject cutout image P3 for each of the image frames f1 to f5 is calculated based on the motion vector of at least one moving object B. You may do it.
Further, when the reference composite position designated in the reference image frame is an area including the moving object B, in each image frame other than the reference image frame, the moving object B between adjacent image frames detected by the motion detection unit 6 is displayed. Based on the motion information, the synthesis position of the subject cutout image P3 in each image frame other than the reference image frame may be set.

  In the above embodiment, the moving image data recorded in the Motion-JPEG format is exemplified as the moving image P4. However, the present invention is not limited to this. For example, a plurality of continuous images continuously shot at a predetermined shutter speed are used. The continuous shot image which consists of still images may be sufficient. Even in this case, as in the above-described embodiment, the motion information of the moving object B is detected between successive still images, and the subject region S of the subject clipped image P3 is caused to perform a predetermined operation based on the motion information. .

  In the above embodiment, the Motion-JPEG format has been exemplified as a moving image data recording method. However, an MPEG format may be used. In such a case, when detecting the motion of the moving object B, the moving image data in the MPEG format is generated in advance by decoding the MPEG moving image data to generate a plurality of image frames f1 to f5. Motion detection can be performed.

  In addition, the configuration of the imaging apparatus 100 is merely an example illustrated in the above embodiment, and is not limited thereto. In other words, the image pickup apparatus is exemplified as the image composition apparatus, but is not limited thereto. For example, the generation of data related to the combined position of the subject clipped image with respect to the moving image P4 set based on the motion information of the subject clipped image P3, the moving image P4, and the moving object B is generated by an imaging device different from the imaging device 100. It is also possible to use an image playback device that records only the data transferred from the imaging device and executes only the playback processing of the synthesized moving image.

In addition, in the above embodiment, the functions as the acquisition unit, the detection unit, the synthesis unit, and the synthesis position setting unit are controlled under the control of the CPU 13 by the motion detection unit 6 and the cutout image generation unit of the image processing unit 7. 7d, the acquisition unit 7e, the synthesis position setting unit 7f, and the image synthesis unit 7g are configured to be driven. However, the configuration is not limited to this, and is realized by the CPU 13 executing a predetermined program or the like. It is good also as a structure to be performed.
That is, a program memory (not shown) that stores a program stores a program including an acquisition process routine, a detection process routine, a synthesis position setting process routine, and a synthesis process routine. Then, the background processing image and the composition still image may be acquired by the CPU 13 by the acquisition processing routine. Further, the CPU 13 may detect the motion information of the moving object in the acquired background moving image by the detection processing routine. Further, the synthesis position setting processing routine causes the CPU 13 to set the still image synthesis position in each image frame other than the one image frame based on the detected motion information of the moving body between at least one adjacent image frame. Anyway. Further, the synthesis processing routine causes the CPU 13 to generate a synthesized moving image by synthesizing the still image and the moving image so that the synthesis still image is operated in a predetermined manner in accordance with the motion of the moving object based on the detected motion information. You may do it.

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Lens part 2 Electronic imaging part 3 Imaging control part 6 Motion detection part 7 Image processing part 7d Cutout image generation part 7e Acquisition part 7f Composition position setting part 7g Image composition part 7h Composition control part 8 Recording medium 10 Display part 12 CPU

Claims (7)

Acquisition means for acquiring a moving image for background and a still image to be combined with the moving image;
Specifying means for specifying a moving object in the moving image acquired by the acquiring means;
Detecting means for detecting motion information of the moving object specified by the specifying means;
Based on the motion information detected by the detection means, a synthesis means for synthesizing the still image and the moving image so as to operate the still image on the background moving image to generate a synthesized moving image;
Equipped with a,
The synthesis means includes
When the combined position of the still image in the moving image is a position including a moving object in the moving image, the still image and the moving image are displaced so as to be displaced according to movement information of the moving object in the moving image. An image synthesizing apparatus that synthesizes a moving image . Among a plurality of image frames constituting the moving image, further comprising designation means for designating the combined position of the still image in any one of image frames,
The synthesis means includes
Among the plurality of image frames, in the one image frame, the still image is synthesized at the synthesis position designated by the designation unit, and in each image frame other than the one image frame, the one image frame The image synthesizing apparatus according to claim 1, wherein the still image is combined with the still image combined with the image frame so as to be displaced in accordance with the movement of the moving object. The synthesis means includes
Synthetic positions specified by said designation means, said in each image frame other than one image frame, the relative one combined the still image to the image frame to displace in response to the movement information of the moving object The image synthesizing apparatus according to claim 2, wherein the still images are synthesized. The detection means includes
Detecting motion information of a moving object between at least one adjacent image frame among the plurality of image frames;
Composition position setting means for setting a composition position of the still image in each image frame other than the one image frame based on movement information of a moving body between the at least one adjacent image frames detected by the detection means; The image composition device according to claim 2 or 3, further comprising:   The image synthesizing apparatus according to claim 1, wherein the still image is an image obtained by extracting a subject area including a subject from a subject existing image in which a subject is present in a background.   The motion information of the moving object includes at least one of change information of a position of the moving object, change information of a size, and change information of a rotation angle. Image composition device. The computer of the image synthesizer
Acquisition means for acquiring a moving image for background and a still image to be combined with the moving image;
A specifying unit for specifying a moving object in the moving image acquired by the acquiring unit;
Detecting means for detecting movement information of the moving object specified by the specifying means;
Based on the motion information detected by the detection means, a synthesis control means for synthesizing the still image and the moving image so as to operate the still image on the background moving image to generate a synthesized moving image;
To function as,
The synthesis means includes
When the combined position of the still image in the moving image is a position including a moving object in the moving image, the still image and the moving image are displaced so as to be displaced according to movement information of the moving object in the moving image. A program characterized by combining a moving image .

Images