JP5493839B2 - Imaging apparatus, image composition method, and program - Google Patents

Description

  The present invention relates to an imaging apparatus, an image synthesis method, and a program for synthesizing a plurality of images.

  Conventionally, a technique for generating a composite image by combining a subject image with a background image or a frame image is known (see, for example, Patent Document 1).

JP 2004-159158 A

  However, in order to generate a composite image, it is necessary to select a background image and a subject image for composition, and to specify a composite position and size of the subject image with respect to the background image, which is troublesome. is there.

  Accordingly, an object of the present invention is to provide an imaging apparatus, an image composition method, and a program that can easily generate a composite image.

In order to solve the above problem, an imaging apparatus according to claim 1 is provided.
Designating means for designating a second image to be combined with the first image;
Imaging means for acquiring a subject presence image by imaging a subject;
Non-flatness determining means for determining whether or not the non-flatness of the background portion of the subject existing image is equal to or less than a predetermined value;
When the non-flatness determining unit determines that the non-flatness is equal to or less than a predetermined value , the second image specified by the specifying unit is transmitted through the subject existing image acquired by the imaging unit. Display control means for displaying an image of the state on the display means;
Subject extraction means for extracting a subject area including a subject from the subject presence image as the first image when a predetermined instruction is detected while the image is displayed by the display control means;
Combining means for generating a combined image by combining the first image extracted by the subject extracting means and the second image specified by the specifying means;
It is characterized by having.

The invention according to claim 2 is the imaging apparatus according to claim 1,
The display control unit displays the second image designated by the designation unit in a semi-transparent display mode when the non-flatness judgment unit judges that the non-flatness is a predetermined value or less. It is characterized by being displayed on the means.

The invention according to claim 3 is the imaging apparatus according to claim 1 or 2 ,
It further comprises designation determination means for determining whether or not the second image has been designated by the designation means, and the combining means further comprises a step of determining that the second image has been designated by the designation determination means. A composite image obtained by combining the first image and the second image is generated.

Invention of Claim 4 is an imaging device as described in any one of Claims 1-3 ,
The synthesizing unit further synthesizes the first image and the second image with reference to a position within the angle of view of the subject when the subject-existing image is captured by the imaging unit. .

Invention of Claim 5 is an imaging device as described in any one of Claims 1-4 ,
The synthesizing unit further synthesizes the first image and the second image on the basis of a size within the angle of view of the subject when the subject-existing image is captured by the imaging unit. Yes.

Invention of Claim 6 is an imaging device as described in any one of Claims 1-5 ,
The image processing apparatus further includes recording control means for causing the recording means to record the subject area extracted by the subject extracting means and the synthesized image generated by the synthesizing means in association with each other.

The invention according to claim 7 is the imaging apparatus according to any one of claims 1 to 6,
The imaging means acquires a plurality of the subject presence images by continuously imaging the subject a plurality of times,
The subject extracting means extracts a subject area including a subject for each of the plurality of subject existing images acquired by the imaging means;
The synthesizing unit synthesizes each of the subject areas extracted by the subject extracting unit with the second image to generate a plurality of synthesized images.

An image composition method according to an eighth aspect of the present invention includes:
An image composition method using an imaging device including an acquisition unit that acquires a subject existing image in which a subject is present in a background,
A process for designating a second image to be combined with the first image;
A process of determining whether or not the non-flatness of the background portion of the subject existing image is a predetermined value or less;
When the non-flatness is determined to be less than or equal to a predetermined value, a process for displaying on the display means an image in which the designated second image is transmitted with respect to the acquired subject existing image;
When a predetermined instruction is detected while an image is displayed on the display means, a process for extracting a subject area including a subject from the subject existing image as the first image;
A process of generating a composite image by combining the extracted first image and the designated second image;
It is characterized by executing.

The program of the invention according to claim 9 is:
A computer of an imaging apparatus including an acquisition unit that acquires a subject presence image in which a subject is present in the background,
Designating means for designating a second image to be synthesized with the first image;
Non-flatness determining means for determining whether or not the non-flatness of the background portion of the subject existing image is equal to or less than a predetermined value;
When the non-flatness determining means determines that the non-flatness is equal to or less than a predetermined value , the second image specified by the specifying means is transmitted through the subject existing image acquired by the acquiring means. Display control means for displaying an image of the state on the display means,
Subject extraction means for extracting a subject area including a subject from the subject presence image as the first image when a predetermined instruction is detected while the image is displayed by the display control means;
Combining means for generating a combined image by combining the first image extracted by the subject extracting means and the second image specified by the specifying means;
It is characterized by making it function as.

  According to the present invention, a composite image can be easily generated by a single photographing operation.

It is a block diagram which shows schematic structure of the imaging device of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to subject clipping processing by the imaging apparatus of FIG. 1. FIG. 3 is a flowchart showing a continuation of the subject clipping process in FIG. 2. FIG. 3 is a flowchart illustrating an example of an operation related to an extraction background generation process in the subject clipping process of FIG. 2. 3 is a flowchart illustrating an example of an operation related to an area detection process in the subject clipping process of FIG. 2. It is a figure which shows typically an example of the image for demonstrating the object clipping process of FIG. It is a figure which shows typically an example of the image for demonstrating the object clipping process of FIG. It is a figure which shows typically an example of the image for demonstrating the object clipping process of FIG. 10 is a flowchart illustrating an example of an operation related to a subject clipping process performed by the imaging apparatus according to the first modification. 10 is a flowchart showing a continuation of the subject clipping process in FIG. 9.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment to which the present invention is applied.

In the imaging apparatus 100 according to the present embodiment, the subject S exists in the background after specifying the composition background image P2 (see FIG. 6A) to be combined with the subject image A (see FIG. 7B). A subject existing image (for example, a subject existing image P1a; see FIG. 6C)) is captured, and a subject image A in a region including the subject S is extracted from the subject existing image, and the extracted subject image A and The subject composition image P6 (see FIG. 7C) is generated by compositing with the designated composition background image P2.
Specifically, as illustrated in FIG. 1, the imaging apparatus 100 includes a lens unit 1, an electronic imaging unit 2, an imaging control unit 3, an image data generation unit 4, an image memory 5, and a non-flatness calculation. A unit 6, a block matching unit 7, an image processing unit 8, a recording medium control unit 9, a display control unit 10, a display unit 11, an operation input unit 12, and a central control unit 13 are provided.
Further, the imaging control unit 3, the non-flatness calculation unit 6, the block matching unit 7, the image processing unit 8, and the central control unit 13 are designed as a custom LSI 1A, for example.

The lens unit 1 includes a plurality of lenses and includes a zoom lens, a focus lens, and the like.
Although not shown, the lens unit 1 includes a zoom drive unit that moves the zoom lens in the optical axis direction when the subject S is imaged, a focus drive unit that moves the focus lens in the optical axis direction, and the like. May be.

  The electronic imaging unit 2 is composed of an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-oxide Semiconductor), for example, and converts an optical image that has passed through various lenses of the lens unit 1 into a two-dimensional image signal. To do.

Although not shown, the imaging control unit 3 includes a timing generator, a driver, and the like. Then, the imaging control unit 3 scans and drives the electronic imaging unit 2 with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 2 every predetermined period, and the electronic imaging unit 2 Image frames are read out from the imaging area for each screen and output to the image data generation unit 4.
In addition, the imaging control unit 3 performs adjustment control of imaging conditions of the subject S such as AF (automatic focusing process), AE (automatic exposure process), AWB (automatic white balance), and the like.

Further, when the subject clipping mode (described later) is set as the imaging mode, the imaging control unit 3 sets the subject S in the background that has passed through the lens unit 1 based on the imaging instruction operation of the shutter button 12a by the user. An optical image of the existing subject presence image P1a (see FIG. 6C) is converted into a two-dimensional image signal by the electronic imaging unit 2 under a predetermined imaging condition, and the subject presence image P1a is converted from the imaging region of the electronic imaging unit 2. The image frame concerning is read.
Here, the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 constitute an imaging unit that captures the subject existing image P <b> 1 a in which the subject S exists in the background.

The image data generation unit 4 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the image frame transferred from the electronic imaging unit 2, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to an image memory 5 used as a buffer memory via a DMA controller (not shown).

  A demosaic unit (not shown) for developing the digital data after A / D conversion may be mounted on the custom LSI 1A.

  The image memory 5 is composed of, for example, a DRAM or the like, and temporarily stores data processed by the non-flatness calculation unit 6, the block matching unit 7, the image processing unit 8, the central control unit 13, and the like.

  The non-flatness calculation unit 6 calculates the non-flatness of the background image Pb generated as an image frame for live view image display and the subject presence image P1a. Specifically, the non-flatness calculation unit 6 includes a division unit 6a, a feature amount calculation unit 6b, and a variation amount calculation unit 6c.

  In the extraction background generation process, the dividing unit 6a divides the peripheral portion (see FIG. 7A) of the subject existing image P1a into a plurality of image blocks B,... Based on the YUV data of the subject existing image P1a. To do.

The feature amount calculation unit 6b performs feature extraction processing for extracting feature points from the background image Pb and the subject presence image P1a.
That is, the feature amount calculation unit 6b has a block region with a high feature with high frequency components (for example, a square of 16 × 16 pixels) based on the YUV data of the background image Pb generated as an image frame for live view image display. Are extracted as feature points.
Further, the feature amount calculation unit 6b extracts the feature points of each image block B of the subject existing image P1a divided by the dividing unit 6a in the extraction background generation processing. Specifically, the feature amount calculation unit 6b extracts, as feature points, a block region (for example, a square of 16 × 16 pixels) having a high feature with a high frequency component of each image block B.

  In addition, the feature amount calculation unit 6b performs the alignment process between the subject presence image P1b (see FIG. 8A) and the subject non-existence image P3, and uses the subject non-existence image P3 as a reference. The process which extracts the feature point from is performed. Specifically, the feature quantity calculation unit 6b is based on the YUV data of the subject non-existing image P3, and has a predetermined number (or more than a predetermined number) of high feature block regions (or more than a predetermined number) that are convenient for tracking from a large number of candidate blocks. The feature point is selected, and the contents of the block area are extracted as a template (for example, a square of 16 × 16 pixels).

なお、上記式（１）において、ｂ：各ブロック領域の画素値としては、例えば、輝度値が挙げられる。 The variation amount calculation unit 6c calculates the variation amount of the pixel value in the block area.
In other words, the variation amount calculation unit 6c calculates the standard deviation according to the following equation (1) as the variation amount in each block region of the background image Pb generated as the image frame for displaying the live view image.
Further, the variation amount calculation unit 6c calculates a standard deviation according to the following equation (1) as a variation amount for each of the plurality of image blocks B,... In the peripheral portion of the subject existing image P1a in the extraction background generation processing. To do.

In the above formula (1), b: a pixel value of each block area includes, for example, a luminance value.

  As described above, the feature amount calculation unit 6b extracts the feature points of the peripheral portion of the subject existing image P1a and the feature points of the background image Pb as the non-flatness. Further, the variation amount calculation unit 6c calculates the variation amount of the pixel values in the plurality of image blocks B,... In the peripheral portion of the subject existing image P1a and the variation amount in the plurality of block regions of the background image as non-flatness. To do.

  The block matching unit 7 performs block matching processing for aligning the subject non-existing image (reference image) P3 and the subject existing image (target image) P1b. Specifically, the block matching unit 7 optimally matches where the template extracted in the feature extraction process corresponds to in the subject existing image P1b, that is, the template pixel value in the subject existing image P1b. Search for a position (corresponding region). Then, the optimum offset between the subject non-existing image P3 and the subject existing image P1b having the best evaluation value (for example, sum of squared differences (SSD), sum of absolute differences (SAD), etc.) Calculated as a template motion vector.

  The image processing unit 8 includes a background designation determination unit 8a, a non-flatness determination unit 8b, an extraction background generation unit 8c, an alignment unit 8d, a mask generation unit 8e, a cutout image generation unit 8f, and an image composition 8g.

The background designation determination unit 8a determines whether or not the composition background image P2 is designated in the subject clipping process. That is, the background designation determination unit 8a selects a desired composition background image P2 from the plurality of images recorded on the recording medium 9a based on a predetermined operation of the operation input unit 12 by the user, for example. It is determined whether or not specified by.
Here, the background designation determination unit 8a constitutes a designation determination unit that determines whether or not the composition background image P2 has been designated by the central control unit (designation unit) 13.

The non-flatness determination unit 8b determines whether or not the non-flatness calculated in the non-flatness calculation process is equal to or less than a predetermined value.
Specifically, the non-flatness determination unit 8b is generated as an image frame for displaying a live view image, and a feature point (a block region having a high feature) of the background image Pb constituting the background portion of the subject existing image P1a is predetermined. It is determined whether or not it is less than or equal to the value. Further, the non-flatness determination unit 8b determines whether or not the variation amount of the pixel value in the block region of the background image Pb is equal to or less than a predetermined value.
That is, since the subject existing image P1a is configured by arranging the subject S within the same angle of view as the background image Pb, the background portion that does not overlap the subject S of the subject existing image P1a. Is substantially equal to the pixel value of the background image Pb. Then, the non-flatness determining unit 8b determines whether or not the non-flatness of the background image P1a is equal to or less than a predetermined value by determining whether or not the non-flatness of the background image Pb is equal to or less than a predetermined value. Non-flatness determining means for determining whether or not is configured.

In addition, the non-flatness determination unit 8b has the feature points extracted by the feature amount calculation unit 6b for the plurality of image blocks B,... It is determined whether or not there is (whether or not the feature amount of the peripheral portion of the subject existing image P1a is large). Further, the non-flatness determination unit 8b determines whether or not the variation amount of each image block B calculated by the variation amount calculation unit 6c is equal to or less than a predetermined value.
Further, the non-flatness determination unit 8b counts the image blocks B in which the feature points are determined to be equal to or smaller than a predetermined value and the variation amount of the pixel values in the block area is equal to or smaller than the predetermined value, It is determined whether or not there are a plurality of Bs.

  Note that, as described above, the non-flatness determination unit 8b determines whether or not the non-flatness of the background portion of the subject existing image P1a is equal to or less than a predetermined value, as described above. A non-flatness determination method for the image blocks B,... May be applied.

The extraction background generation unit 8c generates an extraction background image (not shown) for extracting the subject image A using the chroma key technique.
Specifically, in the extraction background generation unit 8c, the non-flatness determination unit 8b causes the feature point of the peripheral portion of the subject existing image P1a to be equal to or less than a predetermined value, and the variation amount of the pixel value in the block region is When it is determined that there are a plurality of image blocks B that are equal to or smaller than the predetermined value, that is, image blocks B with a small amount of feature, an extraction background image having the same color as the color of the image block B is generated. For example, as shown in FIG. 6B, when the subject S is photographed against a plain background with a small feature amount, the feature amount of the peripheral portion of the subject presence image P1a decreases, and the subject presence image P1a A background image for extraction having the same color as that of the peripheral edge portion is generated.
Note that the chroma key is a method of cutting out the subject S from one image data using a specific color background. In the chroma key, normally, a blue or green screen having a complementary color relationship with the subject S is used as the background, and extraction is performed by generating an extraction background image having the same color as the peripheral portion of the subject existing image P1a. The background portion and the subject portion can be separated based on the color information of the background image for the subject and the subject existing image P1a.

The alignment unit 8d performs alignment between the subject existence image P1b and the subject absence image P3 based on the feature points extracted from the subject absence image P3.
That is, the alignment unit 8d calculates a coordinate conversion formula (projection conversion matrix) of each pixel of the subject existing image P1b with respect to the subject nonexistent image P3 based on the feature points extracted from the subject nonexistent image P3, and the coordinates. The subject presence image P1b is coordinate-transformed according to the conversion formula and aligned with the subject non-existence image P3.
Specifically, the positioning unit 8d calculates the motion vectors of a plurality of templates calculated by the block matching unit 7 by majority, and the motion determined to be statistically greater than or equal to a predetermined percentage (for example, 50%). Using the vector as the overall motion vector, the projection transformation matrix of the subject existing image P1b is calculated using the feature point correspondence related to the motion vector. Then, the alignment unit 8d performs coordinate conversion of the subject existing image P1b according to the projective transformation matrix and performs alignment with the subject non-existing image P3.

なお、上記式にあっては、背景となる画像のＹＵＶデータを「Y」、「U」、「V」で表し、被写体存在画像Ｐ１ａのＹＵＶデータを「Yc」、「Uc」、「Vc」で表す。また、Ｇは、色差信号U、Vのゲインを表している。
そして、マスク生成部８ｅは、生成した相違度マップを所定の閾値で二値化（０、２５５）してマスク画像を生成する。 The mask generation unit 8e generates a mask image for extracting the subject image A from the subject presence image P1a (P1b).
That is, the mask generation unit 8e calculates the difference D between the corresponding pixels of the background image (extraction background image (not shown) or the subject non-existing image P3) and the subject existing image P1a (P1b) by the following equation (2). ) To generate a dissimilarity map.

In the above formula, the YUV data of the background image is represented by “Y”, “U”, “V”, and the YUV data of the subject existing image P1a is “Yc”, “Uc”, “Vc”. Represented by G represents the gain of the color difference signals U and V.
Then, the mask generation unit 8e generates a mask image by binarizing (0, 255) the generated difference map with a predetermined threshold.

  Further, the mask generation unit 8e performs a contraction process for removing fine noise to remove a pixel set smaller than a predetermined value, and then performs a expansion process for correcting the contraction. By performing a labeling process for assigning the same numbers to the pixel sets to be configured, a hole area is also filled by replacing an area having a predetermined ratio or less in the number of constituent pixels of the effective area with the effective area. Further, the mask generation unit 8e applies an averaging filter to the region information to add a composite gradation to the edge of the region.

The cut-out image generation unit 8f combines the image of the subject S with a predetermined single color background image P5 to generate image data of the subject cut-out image P4.
That is, the cut-out image generation unit 8f uses the chroma key technique to cut out the subject image A from the subject presence image P1a (P1b) using the mask image generated by the mask generation unit 8e, and the single color background image P5. The image data of the subject cutout image P4 is generated by combining.
In addition, since the synthetic gradation is given to the edge part of the mask image, it can be synthesized in a natural feeling in which the boundary part between the clipped subject image A and the single color background image P5 is not clear. .
Here, the cut-out image generation unit 8f constitutes a subject extraction unit that extracts a subject region (subject image A) including the subject S from the subject presence image P1a (P1b).

The image composition unit 8g composes the subject image A (first image) of the subject cutout image P4 and the composition background image P2 (second image) to generate a subject composition image P6.
Specifically, the image composition unit 8g allows the pixels covered by the portions other than the subject S of the image data of the mask image among the pixels of the composition background image P2 to pass through without doing anything while The partial pixel is overwritten with the pixel value of the corresponding pixel in the subject image A of the subject cutout image P4.

Further, when it is determined in the subject clipping process that the background designation determination unit 8a has designated the composition background image P2, the image composition unit 8g and the designated composition background image P2 and the subject clipped image P4. The subject image A is synthesized. At this time, the image synthesis unit 8g synthesizes the subject image A and the synthesis background image P2 with reference to the position and size within the angle of view of the subject image A when the subject presence image P1a is captured. Specifically, the image composition unit 8g, for example, the position of the subject image A (that is, the center position of the subject image A) and the size (that is, the subject image with respect to the subject composition image P6) in the generated subject composition image P6. The ratio of the number of constituent pixels of A) is adjusted to be substantially equal to the position and size of the subject image A within the angle of view when the subject existing image P1a is captured, and the subject image A and the composition background image P2 are adjusted. And synthesize.
On the other hand, if it is determined in the subject clipping process that the composition background image P2 is not designated, the image composition unit 8g performs composition with the subject image A of the subject clipped image P4 in the subject clipping process. After the background image P2 is not synthesized, the subject cutout image P4 and the composition background image P2 are synthesized by an image synthesis process executed at a user-desired timing.

  As described above, the image composition unit 8g composes the subject image A of the subject cutout image P4 extracted by the cutout image generation unit 8f and the synthesis background image P2 designated in advance to generate the subject composite image P6. It constitutes a synthesis means.

  Note that when the composition position of the subject image A is changed based on a predetermined operation of the operation input unit 12 by the user, the image composition unit 8g is arranged to shift the mask image according to the composition position. In such a state, for the region outside the range of the portion where the image data of the mask image is filled (portion other than the subject S) because the composition background image P2 and the mask image are shifted, the image composition unit 8g The filled portion is enlarged so as not to generate an area not covered by the mask image.

The recording medium control unit 9 is configured such that the recording medium 9a is detachable, and controls reading of data from the loaded recording medium 9a and writing of data to the recording medium 9a. That is, the recording medium control unit 9 causes the recording medium 9a to record the image data for recording the captured image encoded by the JPEG compression unit (not shown) of the image processing unit 8.
In addition, the recording medium control unit 9 compresses and associates the mask image generated by the mask generation unit 8e of the image processing unit 8 with the image data of the subject cutout image P4 generated by the cutout image generation unit 8f. Thus, the extension of the image data of the subject cutout image P4 is recorded on the recording medium 9a as “.jpe”.
Further, the recording medium control unit 9 associates the mask image generated by the mask generation unit 8e of the image processing unit 8 with the image data of the subject combined image P6 generated by the image combining unit 8g, and combines the subject composition. The extension of the image data of the image P6 is recorded on the recording medium 9a as “.jpg”.

  The recording medium 9a is composed of, for example, a non-volatile memory (flash memory) or the like. However, the recording medium 9a is an example and is not limited to this, and can be arbitrarily changed as appropriate.

The display control unit 10 performs control to read display image data temporarily stored in the image memory 5 and display the display image data on the display unit 11.
Specifically, the display control unit 10 includes a VRAM, a VRAM controller, a digital video encoder, and the like. The digital video encoder reads the luminance signal Y and the color difference signals Cb and Cr read from the image memory 5 and stored in the VRAM (not shown) under the control of the central control unit 13 via the VRAM controller. The data is periodically read from the VRAM, a video signal is generated based on these data, and is output to the display unit 11.
The display unit 11 is, for example, a liquid crystal display device, and displays a live view image, a REC view image, and the like on the display screen 11 a based on a video signal from the display control unit 10.
That is, the display control unit 10 in the imaging mode, a live view image or a main captured image based on a plurality of image frames generated by imaging the subject S by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. The REC view image captured as is displayed on the display screen 11 a of the display unit 11.

Further, the display control unit 10 causes the display unit 11 to display the synthesis background image P2 designated by the central control unit 13 in a semi-transparent display mode when the subject presence image P1a is captured. In other words, the display control unit 10 determines a predetermined value when the non-flatness determination unit 8b determines that the non-flatness of the live view image Pb (a highly characteristic block area or pixel value variation) is equal to or less than a predetermined value. Based on the image data of the composition background image P2 of the image size, the composition background image P2 is displayed on the display unit 11 in a semi-transparent display mode (semi-transparent composition background image P2 ′) on the live view image. Let
Here, as the image data of the composition background image P2, high-resolution image data recorded on the recording medium 9a for recording may be used, or image data is generated when the composition background image P2 is captured. Reduced image data having an intermediate resolution (for example, about VGA) generated by being reduced at a predetermined ratio in both horizontal (x-axis) and vertical (y-axis) by the unit 4 may be used, but the display control unit 10 In order to reduce the processing load and increase the processing speed, it is preferable to use reduced image data.
The translucent display mode of the compositing background image P2 (semi-transparent compositing background image P2 ′) is a state in which the compositing background image P2 is displayed between transparent and opaque, and the compositing background image P2 is displayed. The display mode is such that the outline, color, brightness, and the like of the live view image displayed on the rear side superimposed on the image P2 are transmitted. Further, by switching alternately between display and non-display of the synthesis background image P2 having a semi-transparent display mode at a predetermined timing, the normal live view image and the semi-transparent synthesis background image P2 are overlapped. The live view image may be continuously visible.

  Thus, the display control unit 10 displays the designated composition background image P2 in a semi-transparent display mode when the subject unit image P1a is captured by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. The display control means to display on the part (display means) 11 is comprised.

  The operation input unit 12 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 12 includes a shutter button 12a related to an instruction to shoot the subject S, a selection determination button 12b related to an instruction to select an imaging mode or a function on the menu screen, and a zoom related to an instruction to adjust the zoom amount. Buttons (not shown) and the like are provided, and predetermined operation signals are output to the central control unit 13 in accordance with operations of these buttons.

In addition, the selection determination button 12b of the operation input unit 12 gives a designation instruction for a desired image selected from a plurality of images recorded on the recording medium 9a based on a predetermined operation by the user. Output to.
The central control unit 13 designates an image corresponding to the designated designation instruction as the synthesis background image P2 to be synthesized with the subject image A.
Here, the operation input unit 12 and the central control unit 13 constitute a designating unit for designating the synthesis background image P2 to be synthesized with the subject image A.

  The composition background image P2 is designated based on a predetermined operation of the operation input unit 12 by the user, but the method for designating the composition background image P2 is not limited to this. That is, the central control unit 13 may be configured to automatically specify, for example, an image captured immediately before the subject clipping process or a user-desired image registered in advance as the synthesis background image P2. .

  The central control unit 13 controls each unit of the imaging apparatus 100. Specifically, the central control unit 13 includes a CPU, a RAM, and a ROM (all not shown), and performs various control operations according to various processing programs (not shown) for the imaging apparatus 100.

Next, the subject clipping process according to the image composition method performed by the imaging apparatus 100 will be described with reference to FIGS.
2 and 3 are flowcharts showing an example of an operation related to the subject clipping process.

The subject clipping process is executed when a subject clipping mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user. It is processing.
As shown in FIG. 2, first, the central control unit 13 writes a synthesis background image designation flag = 0 and an extraction background shooting flag = 0 in a flag table (not shown) of the image memory 5 (step S1).
Next, the background designation determination unit 8a of the image processing unit 8 selects a desired composition background image P2 (from the plurality of images recorded on the recording medium 9a based on a predetermined operation of the operation input unit 12 by the user. It is determined whether or not selection (see FIG. 6A) has been selected (step S2).
If it is determined that the composition background image P2 has been selected and designated (step S2; YES), the central control unit 13 sets the composition background image designation flag = 1 in the flag table (not shown) of the image memory 5. Write (step S3).

Thereafter, the display control unit 10 displays a live view image on the display screen 11 a of the display unit 11 based on image frames sequentially generated by imaging the subject S by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. At the same time, a background instruction message (for example, “Please turn to the background”) is displayed on the display screen 11a of the display unit 11 so as to overlap the live view image (step S4; see FIG. 6B).
If it is determined in step S2 that the composition background image P2 is not selected and designated (step S2; NO), the central control unit 13 skips the process of step S3 and performs the process. The display control unit 10 displays the live view image on the display screen 11a of the display unit 11 and displays a background instruction message on the display unit 11 so as to overlap the live view image in the same manner as described above. It is displayed on the screen 11a (step S4).

Then, the display control unit 10 performs control to update the live view image based on the image frames sequentially generated by the imaging of the subject S by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 (step S5). Based on the YUV data of the image frame (background image Pb) for live view image display, the central control unit 13 causes the feature amount calculation unit 6b to extract a block region with high features (feature point) and The standard deviation as the amount of variation is calculated by the variation amount calculation unit 6c according to equation (1) (step S6).

Next, the non-flatness determination unit 8b of the image processing unit 8 calculates whether or not the block region having a high feature extracted by the feature amount calculation unit 6b is equal to or less than a predetermined value and is calculated by the variation amount calculation unit 6c. By determining whether or not the variation amount of the pixel value in the block region of the background image is equal to or less than a predetermined value, it is determined whether or not there is a variation amount of the pixel value in the feature point and the block region (step) S7).
Here, if it is determined that there is no variation amount between the feature point and the pixel value in the block area (step S7; YES), for example, the subject S (for example, “ When photographing a “dog” or the like) (see FIG. 6B), the central control unit 13 determines whether or not the composition background image designation flag = 1 (step S8).

If it is determined in step S8 that the composition background image designation flag = 1 (step S8; YES), the display control unit 10 displays the composition background image P2 in a semi-transparent display on the live view image. The mode image (semi-transparent composition background image P2 ′) and the imaging instruction message (for example, “shoot the subject”) are displayed on the display screen 11a of the display unit 11 (step S9).
Thereafter, the display control unit 10 performs control to update the live view image based on the image frames sequentially generated by the imaging of the subject S by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 (step S10). The central control unit 13 determines whether or not the user has performed an imaging instruction operation on the shutter button 12a of the operation input unit 12 (step S11).

  If it is determined in step S8 that the composition background image designation flag is not 1 (step S8; NO), the display control unit 10 superimposes the live view image on the imaging instruction message (for example, “subject” , Please take a picture ”) on the display screen 11a of the display unit 11 (step S12), the central control unit 13 moves the process to step S10 and performs the subsequent processes.

  If it is determined in step S11 that the shutter button 12a has not been operated to perform an imaging instruction (step S11; NO), the central control unit 13 shifts the process to step S10 and performs the subsequent processes in step S11. Step S11 is repeatedly executed until it is determined that the shutter button 12a has been operated to instruct imaging (step S11; YES).

Thereafter, the user moves the subject S within the angle of view, or waits for the subject S to move (see FIG. 6C), and then determines in step S11 that the shutter button 12a has been operated for imaging. Then (step S11; YES), the imaging control unit 3 causes the electronic imaging unit 2 to capture an optical image of the subject existing image P1a under a predetermined imaging condition (step S13).
Then, the image data generation unit 4 generates YUV data of the subject existing image P1a based on the image frame of the subject existing image P1a transferred from the electronic imaging unit 2, and then temporarily stores the YUV data in the image memory 5. Let
As a result, a subject presence image P1a in which the subject S (for example, “dog” or the like) is photographed with a background having a small feature amount is generated.

On the other hand, if it is determined in step S7 that there is a variation amount of the pixel value in the feature point and the block area (step S7; YES), for example, the subject S is photographed against a background having a large amount of feature. In the case (see FIG. 8A), the central control unit 13 determines whether or not the user has performed an imaging instruction operation on the shutter button 12a of the operation input unit 12 (step S14).
Here, if it is determined that the shutter button 12a has not been operated to perform an imaging instruction (step S14; NO), the central control unit 13 shifts the process to step S5, and performs the subsequent processes in step S14. This is repeatedly executed until it is determined that the button 12a has been operated to instruct imaging (step S14; YES).
If it is determined in step S14 that the shutter button 12a has been instructed to perform an imaging operation (step S14; YES), the central control unit 13 stores an extraction background photographing flag = in the flag table (not shown) of the image memory 5. After 1 is written (step S15), the process proceeds to step S13, and the subject S is photographed.
As a result, a subject presence image P1b (see FIG. 8A) in which the subject S is photographed against a background with a large amount of feature is generated.

Next, as shown in FIG. 3, the central control unit 13 determines whether or not the extraction background photographing flag = 0 (step S16).
Here, if it is determined that the extraction background shooting flag = 0 (step S16; YES), the central control unit 13 sends the subject image P1a to the subject image to the non-flatness calculation unit 6 and the image processing unit 8. An extraction background generation process for generating an extraction background image for extracting A is performed (step S17).

Hereinafter, the extraction background generation processing will be described in detail with reference to FIG.
Here, FIG. 4 is a flowchart showing an example of an operation related to the extraction background generation processing.

As shown in FIG. 4, the dividing unit 6a of the non-flatness computing unit 6 divides the peripheral portion of the subject existing image P1a into a plurality of image blocks B,... Based on the YUV data of the subject existing image P1a. (See FIG. 7A), feature points of each image block B are extracted (step S41).
Subsequently, the variation amount calculation unit 6c of the image processing unit 8 calculates a standard deviation as a variation amount of the pixel value for each of the plurality of image blocks B in the peripheral portion of the subject existing image P1a according to the equation (1). (Step S42).

Next, the non-flatness determination unit 8b determines whether or not the feature points extracted for the plurality of image blocks B in the peripheral portion of the subject existing image P1a are equal to or less than a predetermined value, and each image block B Is determined to be less than or equal to a predetermined value, and as a result of the determination, it is determined that the feature point is equal to or less than the predetermined value and the variation value of the pixel value in the image block B is equal to or less than the predetermined value. The image blocks B are counted to determine whether there are a plurality of image blocks B. Further, the image blocks B of the same color among the image blocks B determined not to vary are counted. Then, it is determined whether there is no variation and the number of image blocks B of the same color is equal to or greater than a predetermined value (for example, half the number of blocks of the entire image) (step S43).
Here, when it is determined that there is no variation and the image block B having the same color has a predetermined value or more (step S43; YES), the extraction background generation unit 8c uses the same color as the color of the image block B as the background color. YUV data of the extraction background image is generated (step S44), and the extraction background generation is successful (step S45).

On the other hand, if it is determined in step S43 that there is no variation and the image block B of the same color does not satisfy the predetermined value (step S43; NO), for example, even if there is a stable image block B without variation, Since the background cannot be specified when there is no block of the same color or the like, the extraction background generation unit 8c fails to generate the extraction background without generating the extraction background image (step S46).
Thus, the extraction background generation process is terminated.

As shown in FIG. 3, next, the central control unit 13 causes the extraction background generation unit 8c to determine whether or not the extraction background generation has succeeded (step S18).
Here, if it is determined that the extraction background generation has been successful (step S18; YES), the central control unit 13 causes the image processing unit 8 to detect a subject region including the subject S from the subject existing image P1a. A detection process is performed (step S19).

Hereinafter, the region detection processing will be described in detail with reference to FIG.
Here, FIG. 5 is a flowchart showing an example of an operation related to the region detection processing.

As shown in FIG. 5, the mask generation unit 8e of the image processing unit 8 calculates the difference D of each pixel corresponding between the YUV data of the background image for extraction and the YUV data of the subject existing image P1a using the equation (2). To calculate the difference map and generate a difference map (step S51).

Next, the mask generation unit 8e binarizes the generated dissimilarity map with a predetermined threshold value to generate mask image data (step S52). Then, the mask generation unit 8e determines whether or not the binarization is successful by determining whether or not the background (binarization; 0) region is extremely small, for example (step S53).
Here, if it is determined that the binarization is successful (step S53; YES), the mask generation unit 8e corrects the remaining portion of the periphery of the subject region and removes fine noise, so that the mask image data is processed. After performing a contraction process to remove a pixel set smaller than a predetermined value (step S54), an expansion process for correcting the contraction is performed (step S55).

Subsequently, if the subject S has a color similar to the background color, the subject region of the mask image is lost, so the mask generation unit 8e performs a labeling process for assigning the same number to the pixel sets that constitute the same connected component. Thus, a hole is filled by replacing an area below a predetermined ratio in the number of constituent pixels of the effective area of the mask image data with the effective area (step S56).
Thereafter, the mask generation unit 8e applies an averaging filter to the mask image data, adds a synthetic gradation to the edge of the subject region (step S57), and determines that the region has been successfully detected (step S58).

On the other hand, if it is determined in step S53 that the binarization has not been successful (step S53; NO), for example, if the background (binarization; 0) region is extremely small, the mask generation unit 8e. Is assumed to have failed in binarization, and region detection fails (step S59).
Thereby, the region detection process is terminated.

  As shown in FIG. 3, when it is determined in step S18 that the extraction background generation has not succeeded (step S18; NO), the display control unit 10 determines a predetermined message related to the subject S clipping failure ( For example, “Subject clipping failed” or the like is displayed on the display screen 11a of the display unit 11 (step S20), and the subject clipping process is terminated.

  If it is determined in step S16 that the extraction background shooting flag = 0 is not satisfied (step S16; NO), that is, if the extraction background shooting flag = 1, the display control unit 10 The image of the semi-transparent display mode of the subject presence image P1b and the imaging instruction message of the subject non-existence image P3 (see FIG. 8B) are superimposed on the live view image (for example, “shoot the background”) ) Is displayed on the display screen 11a of the display unit 11 (step S21).

Thereafter, the display control unit 10 performs control to update the live view image based on the image frames sequentially generated by the imaging of the subject S by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 (step S22). The central control unit 13 determines whether or not the user has performed an imaging instruction operation on the shutter button 12a of the operation input unit 12 (step S23).
Here, if it is determined that the shutter button 12a has not been operated to instruct imaging (step S22; NO), the central control unit 13 shifts the process to step S22, and performs the subsequent processes in step S23. This is repeatedly executed until it is determined that the shutter button 12a has been operated to instruct imaging (step S23; YES).

Thereafter, after the user moves the subject S out of the angle of view or waits for the subject S to move, it is determined in step S23 that the shutter button 12a has been operated to instruct imaging (step S23; YES). The imaging control unit 3 causes the electronic imaging unit 2 to capture an optical image of the subject non-existing image P3 under a predetermined imaging condition (step S24).
Here, it is preferable that the imaging conditions such as the in-focus position, the exposure condition, and the white balance of the subject non-existing image P3 are the same as the imaging conditions for imaging the subject existing image P1b. That is, after the subject presence image P1b is captured, the imaging control unit 3 may maintain a state in which imaging conditions such as a focus position, an exposure condition, and a white balance at the time of capturing the subject presence image P1b are fixed. .
Then, the image data generation unit 4 generates YUV data of the subject non-existing image P3 based on the image frame of the subject non-existing image P3 transferred from the electronic imaging unit 2, and then stores the YUV data in the image memory 5. Temporary storage.
As a result, a subject non-existing image P3 in which a background with a large amount of feature without the subject S is captured is generated.

Next, the central control unit 13 causes the non-flatness calculation unit 6, the block matching unit 7, and the image processing unit 8 to use the YUV data of the subject non-existing image P3 as a reference and the YUV data of the subject existing image P1b and the non-existence of the subject. A process of aligning the YUV data of the image P3 is performed (step S25).
Specifically, the feature amount calculation unit 6b of the non-flatness calculation unit 6 has a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the subject non-existing image P3. And the contents of the block are extracted as a template. Then, the block matching unit 7 searches the subject existing image P1b for a position where the pixel value of the template extracted by the feature extraction process is optimally matched, and the evaluation value of the difference between the pixel values is the best. An optimum offset between the subject non-existing image P3 and the subject existing image P1b is calculated as a motion vector of the template.
Then, the alignment unit 8d of the image processing unit 8 statistically calculates the entire motion vector based on the motion vectors of the plurality of templates calculated by the block matching unit 7, and performs feature point correspondences related to the motion vector. After calculating the projection transformation matrix of the subject existing image P1b, the subject existing image P1b is subjected to projective transformation based on the projection conversion row example, so that the YUV data of the subject existing image P1b and the YUV data of the subject nonexisting image P3 are converted. To perform the alignment process.

  Next, the central control unit 13 causes the image processing unit 8 to determine whether or not the alignment is successful (step S26). That is, in step S24, the image processing unit 8 can statistically calculate the entire motion vector from the motion vectors of a plurality of templates, and uses the feature point correspondences related to the motion vector to projective transformation matrix of the subject existing image P1b. It is determined whether or not it has been calculated.

If it is determined in step S26 that the alignment has succeeded (step S25; YES), the central control unit 13 shifts the process to step S19 and causes the mask generation unit 8e of the image processing unit 8 to perform area detection processing. (See FIG. 5) is performed (step S19).
The area detection process here is substantially the same as the above-described content except that the subject non-existing image P3 is used as the extraction background image, and a detailed description thereof will be omitted. That is, the mask generation unit 8e calculates the difference D between the corresponding pixels between the YUV data of the subject non-existing image P3 as the background image and the YUV data of the subject existing image P1b according to the equation (2). A degree map is generated (step S51).

  Thereafter, as described above, the mask generation unit 8e binarizes the dissimilarity map to generate mask image data (step S52), and determines whether binarization is successful (step S53). Shrinkage processing for the mask image data (step S54), expansion processing for correcting the shrinkage (step S55), processing for replacing an area below a predetermined ratio of the mask image data with an effective area by labeling (step S56), A process of applying a composite gradation to the edge of the subject area by applying an averaging filter (step S57) and the like are performed.

  On the other hand, when it is determined in step S26 that the alignment has not been successful (step S26; NO), the central control unit 13 shifts the process to step S19, and performs a predetermined process related to the clipping failure of the subject S. After the message is displayed on the display screen 11a of the display unit 11 (step S20), the subject clipping process is terminated.

Then, after the region detection process (step S19) ends, the central control unit 13 causes the mask generation unit 8e to determine whether or not the subject region has been successfully detected (step S27).
Here, if it is determined that the detection of the subject region has succeeded (step S27; YES), the central control unit 13 uses the chroma key technique to perform the region detection process in the cut-out image generation unit 8f of the image processing unit 8. The subject image A is cut out from the subject presence image P1a (P1b) using the mask image generated in step S1 and is combined with a predetermined single-color background image P5 to create a subject cutout image P4 (see FIG. 7B). Data is generated (step S28).
Specifically, the cut-out image generation unit 8f reads out the image data of the subject presence image P1a (P1b), the single color background image P5, and the mask image and develops them in the image memory 5. The cut-out image generation unit 8f then applies a predetermined single color of the single-color background image P5 for the pixels covered by the portion of the subject existing image P1a (P1b) that is filled with the mask image data (the portion other than the subject S). On the other hand, the pixel of the subject portion is not subjected to anything and is not transmitted through a predetermined single color. In addition, since the synthetic gradation is given to the edge part of the mask image, it becomes a natural feeling that the boundary part between the cut subject image A and the single color background image P5 is not clear.

Thereafter, the central control unit 13 determines whether or not the composition background image designation flag = 0 (step S29).
If it is determined that the composition background image designation flag = 0 (step S29; YES), the central control unit 13 sends the subject image A of the subject cutout image P4 to the image composition unit 8g of the image processing unit 8. And the designated composition background image P2 are combined to generate image data of the subject composite image P6 (see FIG. 7C) (step S30).
Specifically, the image composition unit 8g reads the image data of the composition background image P2, the subject cutout image P4, and the mask image and develops them in the image memory 5. Then, the image composition unit 8g makes the position and size of the subject image A in the generated subject composition image P6 substantially equal to the position and size of the subject image A within the angle of view when the subject presence image P1a is captured. After the adjustment, the pixels covered by the filled portion (the portion other than the subject S) of the image data of the mask image among the respective pixels of the composition background image P2 are transmitted without doing anything while the subject The partial pixel is overwritten with the pixel value of the corresponding pixel in the subject image A of the subject cutout image P4. In addition, since the synthetic gradation is given to the edge part of the mask image, it becomes a natural feeling that the boundary part between the subject image A to be synthesized and the synthesis background image P2 is not clear.

Thereafter, the display control unit 10 displays the subject composite image P6 in which the subject image A is combined with the composition background image P2 on the display unit 11 based on the image data of the subject composite image P6 generated by the image composition unit 8g. It is displayed on the screen 11a (step S31; refer to FIG. 7C).
Next, the recording medium control unit 9 associates the mask image data with the image data of the subject composite image P6 in a predetermined storage area of the recording medium 9a, and sets the extension of the image data of the subject composite image P6 to “ .jpg "is saved as one file (step S32).
Thereby, the subject clipping process is completed.

On the other hand, when it is determined in step S29 that the composition background image designation flag is not 0 (step S29; NO), the display control unit 10 displays the image data of the subject cutout image P4 generated by the cutout image generation unit 8f. The subject cutout image P4 obtained by combining the subject image A with the predetermined single color background image P5 is displayed on the display screen 11a of the display unit 11 (step S33).
Thereafter, the recording medium control unit 9 associates the mask image data with the image data of the subject cutout image P4 in a predetermined storage area of the recording medium 9a, and sets the extension of the image data of the subject cutout image P4 to “. jpe "is saved as one file (step S34), and the subject clipping process is terminated.

  If it is determined in step S27 that the detection of the subject area has not been successful (step S27; NO), the central control unit 13 shifts the process to step S20 and relates to the failure in clipping the subject S. After a predetermined message is displayed on the display screen 11a of the display unit 11 (step S20), the subject clipping process is terminated.

  As described above, according to the imaging apparatus 100 of the present embodiment, it is determined that the synthesis background image P2 to be synthesized with the subject image A is designated, and the synthesis background image P2 is visible in the background. By capturing the subject existence image P1a in which the subject S exists, the subject image A in the region including the subject S is extracted from the subject existence image P1a, and the subject image A and the designated synthesis background image P2 Can be combined to generate a subject composite image P6. That is, by simply designating the composition background image P2 in advance before capturing the subject existence image P1a, the composition background image P2 and the subject image A of the subject existence image P1a can be synthesized, Based on the operation, the subject composite image P6 can be easily generated by a single photographing operation.

In addition, since the semi-transparent image P2 ′ of the designated composition background image P2 is displayed on the display unit 11 when the subject presence image P1a is captured, the composition is displayed in the semi-transparent display mode for the user. The subject presence image P1a can be captured while the background image P2 is visually recognized, whereby the composition of the subject composite image P6 is determined in consideration of the positional relationship between the composition background image P2 and the subject S. be able to. That is, when the subject existence image P1a is captured, not only the subject presence image P1a but also the composition of the subject composite image P6 generated after the imaging can be determined, and the subject composite image P6 is generated. It can be performed more simply.
In particular, when it is determined that the non-flatness of the background portion of the subject existing image P1a displayed as the live view image is equal to or less than a predetermined value, the designated composition background image P2 is displayed in a translucent display manner. 11, the composition background image P <b> 2 displayed in the semi-transparent display mode can be appropriately viewed by the user. That is, for example, when the subject S is photographed against a background with a large amount of feature, if the composition background image P2 is displayed overlaid on the live view image even in the translucent display mode, the subject existing image P1a is displayed. The background portion pixels of the subject S and the pixels of the composition background image P2 are mixed, making it difficult to visually recognize the composition background image P2 and determining the composition of the subject composition image P6. On the other hand, for example, when the subject S is imaged with a background having a small amount of feature, such as a plain background with a single color, the composite background image P2 is displayed in a semi-transparent display mode so as to be superimposed on the live view image. In addition, the composition background image P2 displayed on the background side of the subject S in the subject presence image P1a can be properly recognized, and the composition of the subject composition image P6 can be easily determined.

  Further, since the subject image A and the synthesis background image P2 are synthesized based on the position within the angle of view of the subject S when the subject presence image P1a is imaged, the subject image A in the generated subject synthesized image P6 is used. Can be adjusted so that the position of the subject image A is substantially equal to the position within the angle of view of the subject image A when the subject presence image P1a is captured, and the subject image A and the composition background image P2 can be combined. The composite image P6 can be generated more easily. In other words, by simply adjusting the position of the subject S within the angle of view when capturing the subject presence image P1a, the composition position of the subject image A in the composition background image P2 can be automatically determined. There is no need to determine the composition of the subject composite image P6 after taking the presence image P1a in consideration of the positional relationship between the composition background image P2 and the subject S.

  Further, since the subject image A and the synthesis background image P2 are synthesized based on the size of the subject S within the angle of view when the subject presence image P1a is imaged, the subject image in the generated subject synthesized image P6 is used. The subject image A can be combined with the composition background image P2 by adjusting the size of A so that it is substantially equal to the size of the subject image A within the angle of view when the subject existing image P1a is captured. The subject composite image P6 can be generated more easily. That is, the composite size of the subject image A in the composition background image P2 can be automatically determined simply by adjusting the size of the subject S within the angle of view when capturing the subject existing image P1a. It is not necessary to determine the composition of the subject composite image P6 after taking the subject presence image P1a in consideration of the magnitude relationship between the composition background image P2 and the subject S.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
Below, the modification of the imaging device 100 is demonstrated.
Here, the modified example of the imaging device 100 has substantially the same configuration as the imaging device 100 of the above-described embodiment except for the details described below, and the description thereof is omitted.

<Modification 1>
The imaging apparatus 100 of the first modification does not determine whether or not the non-flatness of the background portion of the subject existing image P1a is equal to or less than a predetermined value in the subject clipping process. That is, in the subject clipping process, it is not always necessary to perform a determination process as to whether or not the non-flatness of the background portion of the subject existing image P1a is equal to or less than a predetermined value. By setting the imaging mode to a mode in which the subject S is imaged in the background with a small feature amount in advance, the non-flatness determination process for the background portion of the subject existing image P1a is performed in the subject clipping process. Absent.

Next, the subject clipping process according to the image composition method performed by the imaging apparatus 100 according to the first modification will be described with reference to FIGS. 9 and 10.
9 and 10 are flowcharts illustrating an example of operations related to the subject clipping process.
In the following subject clipping process, a case where the subject S is imaged with a background having a small feature amount as a background will be described.

As shown in FIG. 9, first, the central control unit 13 writes a composition background image designation flag = 0 in a flag table (not shown) of the image memory 5 (step S101).
Next, the background designation determination unit 8a of the image processing unit 8 selects a desired composition background image P2 (from the plurality of images recorded on the recording medium 9a based on a predetermined operation of the operation input unit 12 by the user. It is determined whether or not selection (see FIG. 6A) has been selected (step S102).
If it is determined that the composition background image P2 is selected and designated (step S102; YES), the central control unit 13 sets the composition background image designation flag = 1 in the flag table (not shown) of the image memory 5. Write (step S103).

Next, the display control unit 10 superimposes the live view image on the composite background image P2 in a semi-transparent display mode image (semi-transparent composite background image P2 ′) and an imaging instruction message (for example, “ Please take a picture "is displayed on the display screen 11a of the display unit 11 (step S104).
Thereafter, the display control unit 10 performs control to update the live view image based on the image frames sequentially generated by the imaging of the subject S by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 (step S105). The central control unit 13 determines whether or not the user has performed an imaging instruction operation on the shutter button 12a of the operation input unit 12 (step S106).

  If it is determined in step S102 that the composition background image P2 is not selected and designated (step S102; NO), the display control unit 10 superimposes the live view image on an imaging instruction message (for example, After displaying “shoot a subject” or the like on the display screen 11a of the display unit 11 (step S107), the central control unit 13 shifts the processing to step S105 and performs the subsequent processing.

  If it is determined in step S106 that the shutter button 12a has not been operated to perform an imaging instruction (step S106; NO), the central control unit 13 shifts the process to step S105 and performs the subsequent processes in step S106. Step S106 is repeatedly executed until it is determined that the shutter button 12a has been operated to instruct imaging (step S106; YES).

Thereafter, when it is determined in step S106 that the shutter button 12a has been instructed to perform an imaging instruction (step S106; YES), the imaging control unit 3 uses the electronic imaging unit to capture the optical image of the subject existing image P1a under a predetermined imaging condition. 2 (step S106).
Then, the image data generation unit 4 generates YUV data of the subject existing image P1a based on the image frame of the subject existing image P1a transferred from the electronic imaging unit 2, and then temporarily stores the YUV data in the image memory 5. Let
As a result, a subject presence image P1a in which the subject S (for example, “dog” or the like) is photographed with a background having a small feature amount is generated.

Next, as shown in FIG. 10, the central control unit 13 causes the non-flatness calculation unit 6 and the image processing unit 8 to generate an extraction background image for extracting the subject image A from the subject existing image P1a. A background generation process is performed (step S109).
The extraction background generation process is substantially the same as the content described above in the above embodiment, and a detailed description thereof is omitted.

Subsequently, the central control unit 13 causes the extraction background generation unit 8c to determine whether or not the extraction background generation has succeeded (step S110).
Here, if it is determined that the extraction background generation is successful (step S110; YES), the central control unit 13 causes the image processing unit 8 to detect a subject region including the subject S from the subject existing image P1a. A detection process is performed (step S111).
Note that the area detection process is substantially the same as the contents described above in the above embodiment, and a detailed description thereof will be omitted.

  On the other hand, when it is determined in step S110 that the extraction background generation has not succeeded (step S110; NO), the display control unit 10 determines a predetermined message (for example, “subject of subject” “Cutout failed” etc.) is displayed on the display screen 11a of the display unit 11 (step S112), and the subject cutout process is terminated.

Then, after the region detection process (step S111) ends, the central control unit 13 causes the mask generation unit 8e to determine whether or not the detection of the subject region has succeeded (step S113).
Here, if it is determined that the detection of the subject region has succeeded (step S113; YES), the central control unit 13 uses the chroma key technique to perform the region detection process in the cut-out image generation unit 8f of the image processing unit 8. The subject image A is cut out from the subject existing image P1a using the mask image generated in step S1 and is combined with a predetermined single color background image P5 to generate image data of the subject cutout image P4 (see FIG. 7B). (Step S114).

Thereafter, the central control unit 13 determines whether or not the composition background image designation flag = 0 (step S115).
If it is determined that the composition background image designation flag = 0 (step S115; YES), the central control unit 13 sends the subject image A of the subject cutout image P4 to the image composition unit 8g of the image processing unit 8. And the designated synthesis background image P2 are combined to generate image data of the subject composite image P6 (see FIG. 7C) (step S116).

Thereafter, the display control unit 10 displays the subject composite image P6 in which the subject image A is combined with the composition background image P2 on the display unit 11 based on the image data of the subject composite image P6 generated by the image composition unit 8g. It is displayed on the screen 11a (step S117; refer to FIG. 7C).
Next, the recording medium control unit 9 associates the mask image data with the image data of the subject composite image P6 in a predetermined storage area of the recording medium 9a, and sets the extension of the image data of the subject composite image P6 to “ .jpg "is saved as one file (step S118).
Thereby, the subject clipping process is completed.

On the other hand, when it is determined in step S115 that the composition background image designation flag is not 0 (step S115; NO), the display control unit 10 displays the image data of the subject cutout image P4 generated by the cutout image generation unit 8f. The subject cutout image P4 obtained by combining the subject image A with the predetermined single color background image P5 is displayed on the display screen 11a of the display unit 11 (step S119).
Thereafter, the recording medium control unit 9 associates the mask image data with the image data of the subject cutout image P4 in a predetermined storage area of the recording medium 9a, and sets the extension of the image data of the subject cutout image P4 to “. jpe "is saved as one file (step S120), and the subject clipping process is terminated.

  If it is determined in step S113 that the detection of the subject area has not been successful (step S113; NO), the central control unit 13 shifts the process to step S112 and relates to the failure in clipping the subject S. After a predetermined message is displayed on the display screen 11a of the display unit 11 (step S112), the subject clipping process is terminated.

Therefore, according to the imaging apparatus 100 of the first modification, it is not necessary to determine whether or not the non-flatness of the background portion of the subject existing image P1a is equal to or less than a predetermined value in the subject clipping process. The clipping process can be further simplified, and the processing burden on each part of the imaging apparatus 100 can be reduced.
Furthermore, it is possible to increase the processing speed of the subject clipping process, to shorten the waiting time for shifting from the subject clipping process to another process, and to provide an imaging device 100 that is easier to use. can do.

In the above embodiment, the recording medium control unit (recording control unit) 9 includes the image data of the subject cutout image P4 generated by the cutout image generation unit 8f and the subject composite image generated by the image composition unit 8g. The image data of P6 may be associated and recorded on the recording medium 9a.
Accordingly, it is possible to facilitate the management of a plurality of image files having the same subject S, and it is possible to provide the imaging device 100 that is easier to use.

Further, in the above-described embodiment, when the background image for extraction is generated, the characteristics of each image block B are used as the determination processing of the non-flatness of the plurality of image blocks B in the peripheral portion of the subject existing image P1a. Although it is determined whether or not both the point and the variation amount of the pixel value in the block are equal to or less than the predetermined value, the present invention is not limited to this, and by determining at least one of the feature point and the variation amount The non-flatness of the image block B may be determined, and the extraction background image may be generated when it is determined that the non-flatness is equal to or less than a predetermined value.
In addition, when the live view image Pb is displayed, it is determined whether or not both the feature point of the image frame of the live view image Pb and the variation amount of the pixel value in the block are equal to or less than a predetermined value. However, the present invention is not limited to this. When the non-flatness of the background image is determined by determining at least one of the feature point and the variation amount, and the non-flatness is determined to be equal to or less than a predetermined value. Alternatively, the subject presence image P1a in which the subject S exists in the substantially same background as the background image may be captured.

  In the above embodiment, the mask image data and the image data of the subject cutout image P4 are associated with each other and saved as one file. However, the mask image data and the image data of the subject existing image P1a are associated with each other. One file may be stored in the recording medium (recording means) 9a. In this case, for the reproduction of the file, it is preferable to prepare a mode for reproducing the subject existing image P1a and two modes for combining and displaying the subject cutout image P4 by applying the mask image data during reproduction.

  In the above embodiment, the subject image is illustrated as the first image, and the background image for synthesis is illustrated as the second image. However, the present invention is not limited to this. For example, a frame image is used as the first image. A human face image or the like may be used as the second image. That is, a person's face image may be designated in advance, a frame image having a predetermined shape may be cut out and generated, and these images may be combined.

  In the above embodiment, when the synthesis background image P2 is designated, the semi-transparent synthesis background image P2 is displayed over the live view image. Not limited to this, it is only necessary to visually recognize the designated composition background image P2. For example, a frame for displaying the composition background image P2 may be displayed on the live view image, and the designated composition background image P2 may be displayed in the frame.

Further, the first image is a subject image obtained by extracting a subject region from each frame of a continuous shot image or a moving image obtained by continuous imaging, and a plurality of subject images are synthesized with a synthesis background image, The composite image may be generated.
A moving image may be used as the second image. That is, a moving image may be reproduced and displayed on the live view image, and the subject image may be combined with at least one frame image constituting the moving image.

  Furthermore, the configuration of the imaging apparatus 100 is merely an example illustrated in the above embodiment, and is not limited thereto.

In addition, in the above embodiment, the functions as the designation unit, the imaging control unit, the subject extraction unit, and the synthesis unit are realized by driving the central control unit 13, the imaging control unit 3, and the image processing unit 8. However, the present invention is not limited to this, and may be realized by executing a predetermined program or the like by the CPU of the central control unit 13.
That is, a program including a designation processing routine, an imaging control processing routine, a subject extraction processing routine, and a composition processing routine is stored in a program memory (not shown) that stores the program. Then, the CPU of the central control unit 13 may function as a designation unit that designates the second image to be combined with the first image by a designation processing routine. Further, the CPU of the central control unit 13 is caused to function as an imaging control unit that causes the imaging unit to capture the subject existing image P1a in a state where the second image is visible after the second image is designated by the imaging control processing routine. May be. Further, the CPU of the central control unit 13 may be caused to function as a subject extraction unit that extracts a subject region including the subject S from the subject presence image P1a captured by the imaging unit as a first image by the subject extraction processing routine. good. Further, the CPU of the central control unit 13 is caused to function as a combining unit that generates a combined image by combining the first image extracted by the subject extracting unit and the second image specified by the specifying unit by the combining specifying processing routine. You may do it.

  Furthermore, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory or a portable recording medium such as a CD-ROM is applied in addition to a ROM or a hard disk. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Lens part 2 Electronic imaging part 3 Imaging control part 8 Image processing part 8a Background designation determination part 8b Non-flatness determination part 8f Cutout image generation part 8g Image composition part 9 Recording medium control part 9a Recording medium 10 Display control part 11 Display unit 13 Central control unit

Claims (9)

Designating means for designating a second image to be combined with the first image;
Imaging means for acquiring a subject presence image by imaging a subject;
Non-flatness determining means for determining whether or not the non-flatness of the background portion of the subject existing image is equal to or less than a predetermined value;
When the non-flatness determining unit determines that the non-flatness is equal to or less than a predetermined value , the second image specified by the specifying unit is transmitted through the subject existing image acquired by the imaging unit. Display control means for displaying an image of the state on the display means;
Subject extraction means for extracting a subject area including a subject from the subject presence image as the first image when a predetermined instruction is detected while the image is displayed by the display control means;
Combining means for generating a combined image by combining the first image extracted by the subject extracting means and the second image specified by the specifying means;
An imaging apparatus comprising: The display control means includes
When the non-flatness determining means determines that the non-flatness is less than or equal to a predetermined value, the display means displays the second image specified by the specifying means in a semi-transparent display mode. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized. Further comprising designation determination means for determining whether or not the second image is designated by the designation means;
The synthesis means further includes:
The composite image obtained by combining the first image and the second image is generated when it is determined that the second image is specified by the specification determination unit. The imaging device described. The synthesis means further includes:
The first image and the second image are combined based on a position within the angle of view of the subject when the subject-existing image is captured by the imaging unit. The imaging device according to claim 1. The synthesis means further includes:
5. The composition of claim 1, wherein the first image and the second image are synthesized based on a size of the subject within an angle of view when the subject is imaged by the imaging unit. The imaging device according to one item.   6. The recording control unit according to claim 1, further comprising: a recording control unit that causes the recording unit to record the subject area extracted by the subject extracting unit and the synthesized image generated by the synthesizing unit in association with each other. The imaging device according to claim 1. The imaging means acquires a plurality of the subject presence images by continuously imaging the subject a plurality of times,
The subject extracting means extracts a subject area including a subject for each of the plurality of subject existing images acquired by the imaging means;
The said synthetic | combination means synthesize | combines each object area | region extracted by the said subject extraction means with the said 2nd image , and produces | generates a some synthetic | combination image, It is characterized by the above-mentioned. Imaging device. An image composition method using an imaging device including an acquisition unit that acquires a subject existing image in which a subject is present in a background,
A process for designating a second image to be combined with the first image;
A process of determining whether or not the non-flatness of the background portion of the subject existing image is a predetermined value or less;
When the non-flatness is determined to be less than or equal to a predetermined value, a process for displaying on the display means an image in which the designated second image is transmitted with respect to the acquired subject existing image;
When a predetermined instruction is detected while an image is displayed on the display means, a process for extracting a subject area including a subject from the subject existing image as the first image;
A process of generating a composite image by combining the extracted first image and the designated second image;
An image composition method characterized by causing A computer of an imaging apparatus including an acquisition unit that acquires a subject presence image in which a subject is present in the background,
Designating means for designating a second image to be synthesized with the first image;
Non-flatness determining means for determining whether or not the non-flatness of the background portion of the subject existing image is equal to or less than a predetermined value;
When the non-flatness determining means determines that the non-flatness is equal to or less than a predetermined value , the second image specified by the specifying means is transmitted through the subject existing image acquired by the acquiring means. Display control means for displaying an image of the state on the display means,
Subject extraction means for extracting a subject area including a subject from the subject presence image as the first image when a predetermined instruction is detected while the image is displayed by the display control means;
Combining means for generating a combined image by combining the first image extracted by the subject extracting means and the second image specified by the specifying means;
A program characterized by functioning as

Images