JP4271648B2 - Image composition apparatus, imaging means, and program - Google Patents

Description

  The present invention relates to an image synthesizing apparatus, an imaging unit, and a program for synthesizing a plurality of acquired image data to generate one image data.

  Conventionally, in order to obtain an image focused on a wide range in an imaging device such as a camera, in addition to adjusting the depth of field by adjusting the aperture of a lens, image processing has been performed on image data acquired by an imaging device. It has been broken. High-quality images that cannot be obtained only by adjustment using an optical method such as lens aperture adjustment, such as when the distance between distant and close-up scenes in the same image is large due to image processing, or when high sharpness is required for the image. Obtainable.

  Conventionally, as an image processing method for acquiring an image that is in focus over a wide range, conventionally, with respect to image data of a plurality of original images obtained by imaging the same subject, a pixel at a specific position in the image data or a region where a specific subject is captured (Hereinafter referred to as “subject region”) is specified as the target pixel or the target region, and the amount of change in image data with respect to the target pixel or the peripheral pixel of the target region or the peripheral region is determined from each of the RGB color regions using an edge detection filter or the like. A method is known in which a pixel or region having the largest amount of change is detected and selected as a focused original image, and image data of the selected pixel or region is synthesized over the entire screen to generate a composite image (for example, a patent) Reference 1).

JP-A-6-70212

  However, since the invention described in Patent Document 1 forms a composite image after performing in-focus determination for each subject, it is necessary particularly when there are a plurality of subjects in the image data. In order to perform accurate focus determination, it is necessary to capture a considerable number of image data. Therefore, a large memory capacity for temporarily storing image data is required. In addition, since an area is formed for each subject in the image data and processing such as focusing is performed for each area, the processing amount of the CPU increases, which may increase the processing load on the CPU and increase the processing time. There is.

  On the other hand, a method of reducing the processing load on the CPU required for image processing by combining images based on a plurality of image data without performing focus determination for each subject area is conceivable. In order to prevent the generation of a composite image with a sense of incongruity, some image processing is required.

  The present invention has been made on the basis of the above problems, and an object of the present invention is to provide high-quality image data focused on the entire image, a small processing load and a small memory capacity based on a small number of image data. Providing an image synthesizing device, an imaging means, and a program.

The above object of the present invention is an image synthesizing apparatus for forming a one composite image data based on the plurality of image data acquired continuously shooting imaging device by changing the focus of the same object, each image Feature amount information extracting means for extracting feature amount information included in the data for each processing region formed in a predetermined pixel group unit, and the feature amount information extracted by the feature amount information extracting means for each of the plurality of image data compared for each of the processing regions corresponding, comparison means for extracting as a comparison result a set of processing area data of the most common image data and the feature amount information of each of the processing region, among the comparison result, different image data Peripheral area forming means for extracting a part where processing areas are adjacent to each other and forming it as a peripheral area, and presence / absence of continuity of feature amount information existing in the processing area corresponding to the peripheral area Determination, and for been processed region it determined that the continuity of the feature amount information is present, the continuity determination to output as a result determine a corresponding set of processing area data of the most common image data and the feature amount information in the processing region Means for comparing the comparison result with the determination result and outputting a correction result obtained by correcting the comparison result, and a correspondence among the plurality of image data for each processing region using the correction result The image synthesizing apparatus includes image data synthesizing means for synthesizing the processing area data of the image data to be synthesized to form the synthesized image data.

  According to the image composition device, a plurality of acquired image data is divided into a plurality of processing areas, and a plurality of image data is synthesized in units of processing areas to form composite image data, thereby placing a heavy processing burden on the CPU. Image composition processing can be performed.

  According to the above image composition device, the image data is synthesized using a clearer image for each processing region by using the one with more feature amount information in the processing region corresponding to the different image data for forming the composite image data. be able to.

  According to the image composition device, feature amount information such as a boundary region between a foreground subject and a distant subject is obtained by using the result of determining the continuity of feature amount information included in adjacent processing regions for image data composition. Therefore, it is possible to form composite image data that is almost visually uncomfortable because it is difficult for position shifts, size shifts, and the like to occur in a continuous portion.

In the image composition device according to the present invention , the plurality of pieces of image data include: a shutter detection unit that detects an operation of a shutter by an operation of a shutter button of the imaging unit by a user of an imaging device including an imaging unit and a shutter driving unit; A focus adjusting unit that adjusts a focus of the lens unit of the imaging unit in response to detection of the operation of the shutter by the shutter detecting unit, and a focus adjustment unit that adjusts a focus of the lens unit by the focus adjusting unit. The same subject in which the same subject is continuously photographed by the imaging unit in response to the operation of the shutter button of the imaging unit by the user by the shutter driving unit having the shutter operating unit for operating the shutter, and the focus is changed. A plurality of image data is acquired.

  According to the imaging means, it is possible to continuously acquire two image data by operating the shutter button once, and it is possible to acquire image data to be used for an image composition apparatus that performs image composition using the two image data. realizable.

  According to the present invention, even when a foreground and a distant view coexist in one image data, the CPU can form a high-quality composite image that is focused on the entire screen without causing a sense of incongruity at the boundary between the foreground and the distant view. Thus, it can be easily realized even in a small terminal device incorporating a CPU with a small processing capacity and a memory with a small capacity, such as a mobile phone with a camera.

  Hereinafter, one embodiment of the present invention will be described with reference to the block diagram of FIG. 1 will be described with reference to the image diagrams shown in FIGS. 9 to 21. FIG.

  FIG. 1 is a configuration block diagram showing configurations of an imaging apparatus and an image composition apparatus according to the present invention. The imaging device 1A according to the present invention is built in various mobile terminals including various imaging devices such as a camera-equipped mobile phone, a camera-equipped PHS (Personal Handy-Phone System), and a camera-equipped personal digital assistant. The imaging device 1A includes an image composition device 1, an imaging unit 2, a shutter drive unit 3, and an AD conversion unit 4.

  The imaging unit 2 is a CCD camera for taking an image, and includes a lens unit 2a and an imaging element unit 2b. The imaging unit 2 has an autofocus mechanism and can adjust the focus. The focus adjustment may be performed by increasing the depth of field by the aperture stop of the lens or by adjusting the focal length using the zoom lens.

  The lens unit 2a of the imaging unit 2 includes a plurality of lenses, and includes, for example, a unit that adjusts the focal length like a zoom lens, or a unit that adjusts the depth of field like an aperture stop. The imaging element unit 2b is a CCD (change coupled device) imaging device, and a pixel region integrated on the light receiving surface 2c, which is a surface facing the lens on the lens unit 2a side, is provided, and the pixel region is collected by the lens unit 2a. Receives light and forms an image.

  A large number of pixels are arranged in a matrix in the pixel region of the light receiving surface 2c. The pixels forming the pixel region are formed by light receiving elements such as photodiodes, and color filters of red (R), green (G), and blue (B), which are the three primary colors of light, are provided on the surface of the pixel. It is pasted. The intensity of light received by each pixel is converted into the magnitude of an electric signal for each color of the filter attached to the surface (photoelectric conversion), and the converted electric signal is a pixel for a certain period of time as a charge corresponding to the signal amount. After being stored therein, it is output as signal charges for each pixel line in the vertical or horizontal direction by CCD transfer.

  The shutter drive unit 3 shown in FIG. 1 is a control unit for automatically operating the lens unit 2a and the shutter (not shown) of the imaging unit 2, and the shutter switch (not shown) of the imaging unit 2. Is operated once, the image pickup unit 2 is caused to photograph the same subject twice consecutively, and the image pickup unit 2 is caused to acquire image data twice consecutively. The shutter driving unit 3 includes a shutter detection unit 3a, a focus adjustment unit 3b, and a shutter operating unit 3c.

  The shutter detection unit 3a is a sensor that detects the operation of the shutter based on the manual operation of the user. When the user presses a shutter switch (not shown) of the image pickup unit 2 by the manual operation, the shutter operation signal or An opening / closing operation is detected, and a detection signal notifying that a shutter switch (not shown) has been pressed is output.

  The focus adjustment unit 3b receives the detection signal output from the shutter detection unit 3a, and sets the size of the aperture stop of the lens unit 2a or the position of the zoom lens in a predetermined state (hereinafter referred to as “setting state”). To adjust the focus of the lens unit 2a. In the setting state in the present embodiment, in view of the fact that a normal camera can take a deep depth of field with respect to a distant subject, when the lens unit 2a has an aperture stop, the aperture stop becomes small and the subject field is far away. When the depth is set to be deep, and the lens unit 2a is a zoom lens, the focus is set to a distance. However, depending on the state of the imaging apparatus 1A and the imaging target, the setting state may be a state where the aperture stop of the lens unit 2a is large or a position where the focal length of the zoom lens is close to the imaging unit 2. In addition, it is desirable that the time required for adjusting the lens unit 2a is extremely short in view of the need to photograph the same subject. When the adjustment of the lens unit 2a is completed, the focus adjustment unit 3b outputs a signal notifying that the adjustment has been completed.

  When the shutter operating unit 3c receives a signal indicating that the adjustment has been completed from the focus adjusting unit 3b, the shutter operating unit 3c transmits a shutter operating signal to the imaging unit 2. When the imaging unit 2 receives a shutter operation signal from the shutter operating unit 3c, the image capturing unit 2 automatically opens and closes the shutter again to take a second photo, and acquires the second image data. In this specification, the image data generated by the imaging unit 2 from the first photograph taken after the user presses the shutter switch of the imaging unit 2 is referred to as “first image data”, and the first photographing is performed. Thereafter, the image data generated by the imaging unit 2 from photographs taken continuously is referred to as “second image data”.

  FIG. 9 shows an image diagram of the first image data acquired by the imaging unit 2, and FIG. 10 shows an image diagram of the second image data acquired by the imaging unit 2. The first image data 21 shown in FIG. 9 is image data captured in a state where the focus is close to the imaging unit 2, and the solid line indicates a sharp contour line in focus, and the dotted line is out of focus. A blurred outline is shown. In FIG. 9, a cylindrical near-field subject 22 is captured with a sharp outline, and a free-form subject (showing a ridgeline of a mountain) is captured with a blurred outline. In FIG. 10, the distant subject 23 is imaged with a sharp outline, and the foreground subject 22 is imaged with a blurred outline.

  Returning to FIG. 1, the AD conversion unit 4 is a quantizer, which receives the image data signal output from the imaging device unit 2 b and quantizes it for each of red (R), green (G), and blue (B). Convert from analog signal to digital signal.

  The image synthesizer 1 is formed by a CPU or various image processors incorporated in a camera-equipped mobile phone or a camera-equipped PHS in which the image pickup apparatus 1A is incorporated, and a computer program executed by the arithmetic processing of the CPU. A quantity information extraction unit 7, a comparison unit 8, a labeling unit 9, a peripheral region forming unit 10, a continuity determination unit 11, a correction unit 12, an image data synthesis unit 13, and a storage unit 14 are provided. Yes.

  The feature amount information extraction unit 7 receives the first image data and the second image data output from the AD conversion unit 4, and extracts feature amount information included in each of the first image data and the second image data. The feature amount extracted by the feature amount information extraction unit 7 may be edge information included in the first image data and the second image data, color information such as RGB, and the like.

  The feature amount information extraction unit 7 extracts feature amounts in units of processing regions formed for each predetermined pixel group.

  FIG. 2 shows a schematic diagram of the processing area. The processing region 2d is formed by a matrix pixel group of r rows × s columns (r and s are integers of 1 or more), and the processing region 2d group has p rows × q columns (p and q are 1 or more). It is an integer, and in FIG. 2, p = 10 and q = 14). Each processing area 2d is associated with address information (i, j) formed by a row number and a column number. Here, i is a row number, j is a column number, and 1 ≦ i ≦ p and 1 ≦ j ≦ q. In order to correctly perform the processing in the image composition device 1, it is desirable that the number of pixels forming each processing region 2d is the same, and the areas of the processing regions 2d are all equal.

  Returning to FIG. 1, when the edge information included in the first image data and the second image data is used as the feature amount information in the feature amount information extraction unit 7, the first image data and the second image data are convolved with each other. Calculation processing using an operator is performed to extract feature amount information included in each of the first image data and the second image data. The extracted edge amount is calculated by a specific numerical value for each processing region 2d.

  FIG. 3 shows a specific example of a convolution operator used when the feature quantity information extraction unit 7 extracts edge information as a feature quantity. In FIG. 3, a-1 and a-2 are Prewitt edge detection operators (a-1 is an x direction difference, a-2 is an y direction difference operator), and b-1 and b-2 are Sobel. Edge detection operators (b-1 is an x direction difference and b-2 is an y direction difference operator). The operators a-1, a-2, b-1, and b-2 are all convolution filters for 9 pixels of 3 × 3 pixels, the center value is the value of the weighting coefficient for the pixel of interest, and the surrounding 8 One value is a value of a weighting coefficient for neighboring pixels in the vicinity of 8 of the target pixel. The feature amount information extraction unit 7 applies any of FIGS. 3A-1 and 3A-2, or B-1 and B-2 to the image data, and the image data of the pixel of interest and the surrounding area. By calculating the density value difference from the image data of the pixel and repeatedly performing the process while shifting the target pixel, it is possible to extract the feature amount portion of the entire processing region 2d and the entire image data.

  3C is a moving average operator of 3 rows × 3 columns, and FIGS. 3D and 3E are weighted average operators of 3 rows × 3 columns, which are selectively used in this embodiment. Used for. The operators in FIGS. 3C, 3D, and 3E each have a center value as a weighting coefficient value for the target pixel, and eight neighboring values as weighting coefficient values for the neighboring pixels in the vicinity of the target pixel. is there. By applying one of the operators shown in FIGS. 3C, 3D, and 3E to the image data, and obtaining the difference in density value between the image data of the pixel of interest and the image data of the surrounding pixels while shifting the pixel of interest Further, it is possible to smooth the image data, and to remove noise and blur an unnatural connection state of the boundary portion in a portion where noise exists or a boundary portion between the main subject and the background subject.

  Note that all of the operators shown in FIG. 3 are sizes applicable to 3 × 3 pixels, but the size of the operator is not limited to 3 × 3 pixels, and other sizes such as 5 × 5 pixels and 7 × 7 pixels, for example. It may be.

  FIG. 11 shows an image diagram of data obtained by extracting the edge component from the first image data by the feature amount information extracting unit 7. FIG. 12 shows an image diagram of data obtained by extracting the edge component from the second image data by the feature amount information extracting unit 7. Show. In FIGS. 11 and 12, the solid line indicates edge information of a strong edge component, and the dotted line indicates edge information of a weak edge component. In the edge component extraction data 25 of the first image data shown in FIG. 11, a strong edge component is extracted from the foreground subject 22 and a weak edge component is extracted from the distant subject 23. In the edge component extraction data 26 of the second image data shown in FIG. 12, weak edge components are extracted from the foreground subject 22 and strong edge components are extracted from the distant subject 23.

  When color information such as RGB is extracted by the feature amount information extraction unit 7 shown in FIG. 1, desired color information is extracted using a filter of the color to be extracted.

  In FIG. 1, the comparison unit 8 compares the amount of feature amount information included in the first image data and the second image data for each processing region 2 d, and the comparison result indicates that the feature amount component is larger. The processing area 2d is selected, and the data selected by the set of the selected processing areas 2d is output as a comparison result.

  FIG. 13 shows an image diagram of comparison result data output by the comparison unit 8. Each square of the comparison result data 27 is the processing area 2d shown in FIG. 2, and the processing area 2d indicated by diagonal lines is the first image data estimation area in which the first image data 21 is determined to have a larger edge amount. 28, the processing area 2d other than the hatched area is the second image data estimation area 29 in which it is determined that the second image data 24 has a larger edge amount. In FIG. 13, all edge information of the distant subject 23 is included in the second image data estimation area 29, and most of the edge information of the foreground subject 22 is included in the first image data estimation area 28. The edge information of the upper end is included in the second image data estimation area 29.

  Returning to FIG. 1, the labeling unit 9 extracts the address information of each processing area 2d forming the comparison result data output from the comparison unit 8, and associates the image identifier with the address information for each processing area 2d. The image identifier is a character or symbol for identifying the first image data and the second image data. In the present embodiment, the character “A” is applied to the image identifier of the first image data, and the second The character “B” is applied to the image identifier of the image data. In this specification, associating the image identifier of each image data with the address information of the processing area 2d forming the image data is referred to as “labeling”, and the data formed by labeling is referred to as “labeling data”. The labeling unit 9 outputs the first labeling data labeled “A” or “B” for each processing region 2 d and the data of the comparison result supplied from the comparison unit 8.

  FIG. 14 shows an image diagram of the first labeling data formed by the labeling unit 9. In the first labeling data 30, the image identifier “A” is labeled in the first image data estimation area 28 (dot display area) for each processing area 2d, and the second image data estimation area 29 (solid color is displayed). The image identifier “B” is labeled for each processing area 2d.

  Returning to FIG. 1, the peripheral region forming unit 10 labels the processing region 2 d formed based on the first image data in the comparison result data output from the labeling unit 9 (that is, the labeling unit 9 labels the image identifier “A”). A boundary region where the processing region 2d) and the processing region 2d formed based on the second image data (that is, the processing region 2d in which the labeling unit 9 labels the image identifier “B”) is adjacent is extracted, A peripheral region, which is a region formed by the processing region 2d adjacent to the portion, is formed. Here, the state in which the processing area 2d is “adjacent” means a state having one side shared by the two processing areas 2d, and is located above, below, left and right of the attention area when the one processing area 2d is the attention area. The processing area 2d (that is, the address information in the case where the address information of the attention area is (x, y)) is (x-1, y), (x + 1, y), (x, y-1), The processing area 2d) that is any of (x, y + 1) corresponds to the adjacent processing area 2d.

  FIG. 15 shows an image diagram of the peripheral area data formed in the peripheral area forming unit 10. In FIG. 15, the processing area 2 d displayed in dots is the processing area 2 d in which the image identifier “A” and the image identifier “B” are adjacent to each other in FIG. 14, and this area forms the peripheral area data 31. In this specification, as shown in FIG. 15, data included in the peripheral area formed by the peripheral area forming unit 10 is referred to as “peripheral area data”.

  Returning to FIG. 1, the continuity determination unit 11 supplies the feature amount information of the first image data and the second image data output from the feature amount information extraction unit 7 and the peripheral region data formed by the peripheral region formation unit 10. In response to the supply, it is determined whether or not there is continuity of the feature amount information existing in the processing area 2d corresponding to the peripheral area in the first image data and the second image data. Note that the continuity in which presence / absence is determined by the continuity determination unit 11 refers to a possibility that feature quantity information continuous over adjacent processing regions 2d is included. The continuity determination unit 11 forms the second labeling data by labeling the appropriate one of the image identifiers “A” or “B” in the portion where the determination result data is determined to have continuity. Which one of the image identifiers “A” and “B” is appropriate is determined by selecting the first image data and the second image data having the larger amount of feature information in the processing region 2d determined to be continuous. decide.

  FIG. 16 shows an image diagram of the continuity determination result by the continuity determination unit 11. As a result of the continuity determination unit 11 determining the continuity for the surrounding area data 31, it is determined that the portion where the edge information of the foreground subject 22 in the surrounding area data 31 is continuous is continuous. . In FIG. 16, the region 32 determined to have continuity is displayed as a dot-displayed region.

FIG. 17 shows an image diagram of the second labeling data. Second labeling data 33 in the region corresponding to the region 32 where it is determined that there is Oite continuity in Figure 16 are formed. In the region 32 determined to have continuity shown in FIG. 16, the first image data has a larger amount of edges than the second image data. Therefore, each processing region 2 d forming the second labeling data 33 includes An image identifier “A” meaning the first image data is associated.

  Returning to FIG. 1, the correction unit 12 compares the image identifier of the second labeling data output from the continuity determination unit 11 with the image identifier of the first labeling data formed by the labeling unit 9, and the image identifiers of both are different. Extract the location. When there is a processing area 2d in which both the image identifiers are different, the image identifier of the first labeling data is replaced with the image identifier of the second labeling data. The correction result by the correction unit 12 is output as third labeling data.

  FIG. 18 shows an image diagram of the third labeling data. In the third labeling data 34, the processing area 2d (the area indicated by diagonal lines in FIG. 18) at the upper left and right portions of the first image data estimation area 28 (area indicated by dots in FIG. 18) is the second image data estimation area 29. To the first image data estimation area 28 (hereinafter, the replaced area is referred to as “replacement area”). The replacement areas 35 a and 35 b form a part of the first image data estimation area 28.

  Returning to FIG. 1, the image data synthesis unit 13 receives the third labeling data output from the correction unit 12 and the first image data and the second image data output from the AD conversion unit 4, and receives the third labeling data. Based on the address information and the image identifier, the first image data and the second image data are synthesized to form synthesized image data. The composite image data is output to various recording media such as a main memory, transmission means, and flash memory.

FIG. 19 is an image diagram illustrating a process of combining the first image data and the second image data based on the third labeling data. In FIG. 19, a dot-displayed area 36 corresponds to the first image data estimation area 28 (dot-displayed area) 34 of the third labeling data shown in FIG. 18, and hatched areas 37a and 37b are shown in FIG. 18 corresponds to replacement areas 35a, 35b (hatched areas) of the third labeling data 34 shown in FIG. 18, and the first image data 21 is synthesized by both areas 36, 37a , 37b . In FIG. 19, the area displayed as plain corresponds to the second image data estimation area 29 (area displayed as plain) of the third labeling data 34 shown in FIG. 18, and the second image data 24 is synthesized.

  FIG. 20 is an image diagram of the composite image data synthesized by the image data synthesis unit 13.

  All the foreground subjects 22 in the composite image data 38 shown in FIG. 20 are composed of the first image data, and all the distant subjects 23 are composed of the second image data.

  For comparison with the synthesized image data 38 in FIG. 20, FIG. 21 shows an image diagram of virtual synthesized image data formed when synthesized only by the comparison result data 27. In FIG. In the virtual composite image data 39 shown in FIG. 21, image data including clear outline information is synthesized for most subjects, but a part of the foreground subject 22 (in the vicinity of the left and right upper ends of the foreground subject 22 in FIG. 21). The second image data is synthesized, and the image data is somewhat uncomfortable compared to the synthesized image data 38 shown in FIG.

  Returning to FIG. 1, the storage means 14 is various built-in memories such as DRAMs, and temporarily stores data used for image processing and data being processed. In the present embodiment, the first image data and second image data output from the AD conversion unit 4 and the first labeling data output from the labeling unit 9 are stored.

  When the adjustment in the focus adjustment unit 3b is the position of the zoom lens of the lens unit 2a, the feature amount information extraction unit 7, the continuity determination unit 11, the image data synthesis unit 13, and the like perform the first image data. It is desirable to have a function for correcting the difference in the size of the subject in the second image data.

  Here, a specific example of a continuity determination method in the continuity determination unit 11 shown in FIG. 1 will be described.

(First method)
The first method is used when the feature amount information extraction unit 7 extracts edge information as a feature amount. FIG. 4 is a flowchart showing a processing procedure of the first method, and FIG. 5 is a diagram showing a model of processing by the first method. The continuity determination method in the first method will be described below with reference to FIGS. 4 and 5.

  As shown in FIG. 4, for the first image data and the second image data that have been subjected to the edge extraction process by the feature amount information extraction unit 7, the edge amount of each pixel is greater than or equal to a predetermined threshold value. Binarization is performed by determining whether it is less than the value. For example, the amount of edge less than the threshold is formed as pixel data of 0 (no edge), and the amount of edge greater than the threshold is formed as pixel data of 1 (with edge) (step P1).

  From the first image data and the second image data obtained by binarizing the edge amount, the peripheral area data is extracted from the processing area 2d that is set as the peripheral area in the peripheral area forming unit 10 (step P2).

  Of the processing regions 2d extracted as the peripheral region, one processing region is formed as the attention region (step P3). When the attention area is formed, a predetermined range (hereinafter referred to as “determination area”) for determining the continuity of the edge formed across the attention area and the processing area 2d adjacent to the attention area is formed. (Step P4).

  FIG. 5 schematically shows the attention area formed in step P3 and the determination area formed in step P4. A predetermined range formed over one side 2h shared by the attention area 2g and one processing area 2d adjacent to the attention area 2g is the determination area 2i.

As shown in FIG. 4, the continuity determination unit 11 measures the number of data of numerical value 1 indicating “with edge” included in the determination region 2i (step P5), and the number of data of numerical value 1 in the determination region 2i. Is determined to be present in a predetermined amount or more (step P6). As a result of the determination, as the data numbers 1 is in the continuity of the edge information to the data contained between the processing region 2d adjacent to the target region 2g region of interest 2 g if it is judged that there more than a predetermined amount Judgment is made (step P7).

  For example, in FIG. 5, the determination area 2 i is a range of 36 pixel data in total of 6 vertical pixel data × 6 horizontal pixel data (including 3 pixel data in the attention area 2 g and 3 pixel data in the processing area 2 d). It is determined whether or not pixel data having edge amount 1 of 60% (that is, 21) or more is edge amount 1 data. As a result of the determination, if 21 or more pieces of pixel data are data of edge amount 1, it is determined that the data includes a continuous edge between the attention area 2g and the adjacent processing area 2d. To do.

  As shown in FIG. 4, after performing the process of step P7, if there is a processing area 2d for which the presence / absence of continuity of edge information has not yet been determined with respect to the adjacent processing area 2d (step P8). At least one of x and y in the address information of the attention area 2g is incremented by 1 to shift the attention area 2g by 1 (step P9), and the processes after step P4 are repeated. When it is determined in step P6 that the data of the numerical value 1 does not exist in a predetermined amount or more, the data in the determination area 2i includes edge information between the attention area 2g and the processing area 2d adjacent to the attention area 2g. It is determined that there is no continuity (step P10). If there is an area that has not been determined yet in the attention area 2g and the processing area 2d adjacent to the attention area 2g (step P11), the determination area 2i is moved along one side sharing one pixel row (step P12). The processes after P5 are repeated. That is, if even one determination result shows that there is a location where pixel data with a numerical value 1 exists in a predetermined amount or more (for example, there is a determination region 2i in which there are 21 or more pixel data with a numerical value 1 in FIG. 5). A continuous edge exists between the target attention area 2g and the determination target processing area 2d adjacent to the attention area 2g, and an edge exists between the attention area 2g and the processing area 2d adjacent to the attention area 2g. Judge that there is continuity of information.

  If it is determined in step P11 that the determination has been made for all the areas in the attention area 2g and the processing area 2d adjacent to the attention area 2g, at least one of x and y in the address information of the attention area 2g is determined. The attention area 2g is moved up by 1 and shifted by 1 to form a new attention area 2g (step P9), and the processes after step P4 are repeated.

  If it is determined in step P8 that all the processing areas 2d have been subjected to the determination of the continuity of the edge information with the adjacent processing area 2d, the processing area 2d in which the included data is determined to be continuous is determined. Then, the labeling data formed by the labeling unit 9 is extracted, and the number of the image identifiers “A” and “B” is compared with each other to measure which number “A” or “B” is larger (step P13). As a result of the measurement, if it is determined that there are more image identifiers “A”, all the processing regions 2d determined to have edge information continuity are associated with the image identifier “A” (step P15). If it is determined that there are more image identifiers “B”, the image identifier “B” is associated with all the processing areas 2d determined to have continuity of edge information (step P16). The processing area 2d associated with the image identifier in step P15 or step P16 forms second labeling data (step P17).

  According to the first method, by determining the continuity based on the amount of edge information in the determination area 2i, it is possible to determine the continuity of the edge information included in the adjacent processing areas 2d with high certainty. it can.

(Second method)
The second method is used when the feature amount information extraction unit 7 extracts edge information as feature amount information. FIG. 6 is a flowchart showing the processing procedure of the second method. The continuity determination method in the second method will be described below with reference to FIG.

  The feature amount information extraction unit 7 performs quantization that is subdivided into binary values for the edge amounts of the pixel data of the first image data and the second image data subjected to the edge extraction processing. For example, the maximum value 255 of the edge amount in the first image data and the second image data is set, the minimum value is set to 0, and 256 steps are converted (8-bit quantization), and the edge amount of each pixel data is set from 0 to 255. (Step P21).

  Each pixel data obtained by quantizing the edge amount is divided into a plurality of processing blocks using the value size as a parameter. For example, an 8-bit quantized edge amount is divided into four equal parts, a set of pixel data with an edge amount of 0 to 63 (first block), a set of pixel data with an edge amount of 64 to 127 (second block), and an edge amount of 128 to Four processing blocks are formed, a set of 191 pixel data (third block) and a set of pixel data with edge amounts 192 to 255 (fourth block) (step P22).

  Next, a predetermined threshold value is provided for each processing block, and the value of the edge amount for each block is binarized using the threshold value (step P23). For example, the median value of the first to fourth blocks (31 for the first block, 95 for the second block, 159 for the third block, 223 for the fourth block) is used as the threshold value, and for each of the first to fourth blocks The edge amount values of all the pixel data are compared with a threshold value, an edge amount less than the threshold value is 0 (no edge), and an edge amount equal to or greater than the threshold value is 1 (with edge).

  Then, the same processing as Steps P2 to P16 of the “first method” is performed for each processing block, and it is determined whether or not there is continuity of the edge information between the attention area 2g and the processing area 2d adjacent to the attention area 2g (Step). P24).

  When all the determinations of the presence or absence of continuity for each processing block are completed, temporary labeling data is formed for each processing block based on the determination result (step P25). In FIG. 6, a total of four types of temporary labeling data are formed for each of the first to fourth processing blocks.

  When provisional labeling data for each processing block is formed, all provisional labeling data are compared (step P27). For the comparison, for example, for the processing area 2d determined to have continuity of edge information in all the provisional labeling data, a method such as calculation of logical product, calculation of logical sum, or calculation of average can be considered.

  Based on the comparison result of the temporary labeling data, second labeling data is formed (step P28).

  According to the second method, the image data is binarized after being divided into a plurality of stages, and the result of the determination made for each stage is comprehensively determined to determine the continuity. Therefore, it is possible to determine the continuity for each processing region 2d with higher accuracy than the determination of continuity according to the above.

(Third method)
The third method is used when the feature amount information extraction unit 7 extracts edge information as feature amount information. Similar to the first method, the edge amount 0 and the edge amount 1 are binarized for the first image data and the second image data that have been subjected to the edge extraction processing by the feature amount information extraction unit 7, and the peripheral region The forming area 10 extracts the processing area 2d that is the peripheral area data.

  Several arbitrary points of edge amount 1 in the processing area 2d are extracted, and individual straight lines are extracted using the Hough transform (Hough transform) to determine the continuity of the edge information.

  FIGS. 7A and 7B are explanatory diagrams of the principle of Hough transform used in the third method. As shown in FIG. 7 (a), when an arbitrary straight line 1 exists on the XY plane, the perpendicular foot drawn from the origin O to the straight line 1 is H, and OH = ρ, the angle formed by OH and the X axis Where θ and H are coordinates (x, y), the straight line 1 can be expressed by the following equation (1).

ρ = x cos θ + ysin θ (1)

  Equation (1) can be applied to all straight lines on the XY plane. If the values of ρ and θ are determined, only one straight line on the XY plane can be specified.

An arbitrary point (xi, yi) on the XY plane is converted into a sine wave shown in Expression (2) and FIG. 7B on the ρ-θ plane.

ρ = Asin (θ + α) (2)
However, A = (x _i ² + y _i ² ) ^1/2 and α = cos (y _i / A).

  Then, as shown in FIG. 7B, a point on one straight line on the XY plane always passes through a unique passing point on the ρ-θ plane, and the value (θ, ρ) of the passing point Q is , The straight line on the XY plane coincides with the straight line in the case of the expression (1).

Accordingly, there are a plurality of known points (x ₁ , y ₁ ), (x ₂ , y ₂ ),... (X _n , y _n ) on an arbitrary arbitrary straight line l ₀ on the XY plane. In this case, the values of the points (x ₁ , y ₁ ), (x ₂ , y ₂ ),... (X _n , y _n ) are substituted into the equation (2), and the passing point Q ₀ (ρ ₀ , θ by obtaining a value of _0), it is possible to determine the value of the linear l _0.

Here, the XY plane of FIG. 7A is an image space, and a plurality of points 101, 102, 103, 104, 104, which are set to an edge amount 1 as a result of binarization on the XY plane, are obtained. 105 are distributed, and the points 101, 102, 103, 104, and 105 exist on the unknown straight line l, the points 101, 102, 103, 104, and 104 are in accordance with the above theory. By substituting the x and y coordinates of 105 (that is, the address information of the pixels) into Equation (2), the value of the passing point Q _j (ρ _j , θ _j ) is obtained, and the straight line l is obtained. A region where the detected straight line exists is determined to have continuity of edge information. In view of the fact that errors of various factors actually occur, it is desirable to select a point where a plurality of curves on the ρ-θ plane intersect most frequently as the passing point Q _j (ρ _j , θ _j ).

  When a plurality of edge information is extracted from the image data by the Hough transform and a region surrounded by the edge information can be formed, it can be determined that the edge information surrounding the region is continuous.

  By determining the continuity using the Hough transform shown in the third method, it is possible to reliably determine the continuity of the edge information existing over the adjacent processing region 2d. In particular, it is possible to determine the continuity of edge information with high certainty in image data including a large number of subjects whose contours are formed with straight lines. In addition, even if the edge information is actually cut off in some places because the edge component is weak even though it is edge information obtained from continuous contour information, the broken part is complemented and the original It is possible to determine continuity that correctly corresponds to the contour information.

(Fourth method)
The fourth method is used when the feature amount information extraction unit 7 extracts edge information as feature amount information.

  In the fourth method, the attention area 2g and the processing area 2d adjacent to the attention area 2g as shown in FIG. 5 are determined by using the determination area 2i.

  In the fourth method, as in the first and third methods, the edge amount 0 and the edge amount are applied to the first image data and the second image data subjected to the edge extraction processing by the feature amount information extraction unit 7. 1 is binarized. However, the threshold value is set to a sufficiently high value so that only a strong edge has an edge amount of 1.

  With respect to the first image data and the second image data in which the edge amount is binarized, the processing region 2d that is the peripheral region data is extracted by the peripheral region forming unit 10.

  One of the extracted processing areas 2d is set as the attention area 2g, and a predetermined area (for example, vertical 10 pixel data ×× 2) extending over the adjacent processing area 2d having a specific edge amount 1 in the processing area 2d adjacent to the attention area 2g. The determination area 2i is formed in the range of the horizontal 10 pixel data. If it is determined that pixel data of edge amount 1 exists in the attention area 2g and the processing area 2d within the determination area 2i, the processing area 2d adjacent to the attention area 2g and the attention area 2g is continuous. Judge that there is.

  According to the fourth method, the continuity of the edge information included in the attention area 2g and the processing area 2d adjacent to the attention area 2g is determined only by determining whether or not the edge information exists in the attention area 2i. Therefore, determination of continuity can be performed with a very small processing amount.

(Fifth method)
The fifth method is used when the feature amount information extraction unit 7 extracts color information such as RGB as a feature amount.

  The fifth method is performed by comparing the attention area 2g as shown in FIG. 5 with the processing area 2d adjacent to the attention area 2g. However, the determination area 2i shown in FIG. 5 is not provided. The first image data and the second image data from which color information such as RGB is extracted by the feature amount information extraction unit 7 is noted between the attention area 2g and the processing area 2d adjacent to the attention area 2g. A determination is made using how much color information exists. An upper limit is set for the ratio of the noticed color information in the pixel data contained in the attention area 2g and the noticed color information in the pixel data contained in the processing area 2d adjacent to the attention area 2g. A threshold value and a lower limit threshold value are provided, and if the ratio is within the threshold value, it is determined that there is continuity between the attention area 2g and the processing area 2d adjacent to the attention area 2g.

  According to the fifth method, determination of continuity based on CPU processing using color information as feature amount information can be realized. In addition, by using color information such as RGB for the feature amount information, the image data input from the imaging unit 2 to the image processing unit 4 can be used as it is for extracting the feature amount information. The processing performed can be reduced.

  Note that any method other than the first to fifth methods described above may be used as long as it is a method that can determine the continuity of the feature amount at the location specified as the peripheral region.

  FIG. 8 is a flowchart of the image composition process when the edge information included in the image data is used as the feature amount information in the image composition apparatus 1. FIGS. 9 to 19 are steps in the image composition process shown in FIG. It is an image figure of the data produced | generated. Hereinafter, an image data acquisition process performed by the imaging apparatus 1A and an image composition process performed by the image composition apparatus 1 when edge information included in the image data is used as feature amount information will be described with reference to FIG. In the description, the same numbers and symbols used in the configuration and data are the same as the numbers and symbols used in FIGS. 1, 2, 5, and 9 to 21.

  In FIG. 8, first, the user of the imaging apparatus 1A presses the shutter switch of the imaging unit 2 to photograph the subject and obtains first image data (step S1). When the shutter detection unit 3a of the second image data acquisition unit 3 detects the operation of the shutter based on the user's operation, the focus adjustment unit 2b immediately adjusts the lens unit 2a to the set state (step S2). When the lens unit 2a is adjusted to the set state, the focus adjusting unit 3b transmits a signal notifying that the adjustment is completed to the shutter operating unit 3c when the adjustment of the lens unit 2a is completed. A shutter operation signal is sent to the imaging unit 2 to acquire second image data (step S3).

  The first image data and the second image data acquired by the imaging unit 2 are converted into digital signals by the AD conversion unit 4 and then transmitted to the image composition device 1.

  The feature amount information extraction unit 7 of the image composition apparatus 1 performs edge component extraction processing using a convolution operator on each of the first image data and the second image data. The edge information is extracted with the processing area 2d as a unit. (Step S4).

  The feature amount information extraction unit 7 calculates the edge amount as a specific numerical value for each processing region 2d for the edge component extracted in step S4 (step S5).

  The edge amounts calculated by the feature amount information extraction unit 7 for the first image data and the second image data are respectively transmitted to the comparison unit 8. The comparison unit 8 compares the amount of edge components included in the first image data and the second image data for each processing region 2d, and selects the image data containing more edge components for each processing region 2d. Comparison result data is formed (step S5).

  The comparison result data 27 formed by the comparison unit 8 in step S5 is transmitted to the labeling unit 9. If the image data identification signal is the first image data for all the processing areas 2d, the labeling unit 9 uses the character “A” as the image identifier, and if the image data identification signal is the second image data, the labeling unit 9 selects the character “B”. Labeling is performed as an image identifier to form first labeling data.

  The first labeling data 30 formed by the labeling unit 9 in step S5 is transmitted to the storage means 14 and recorded (step S6). In addition, the labeling unit 9 transmits the first labeling data 30 and the comparison result data 27 to the peripheral region forming unit 10.

  In the peripheral area forming unit 10, the processing area 2 d formed based on the first image data 21 and the processing area 2 d formed based on the second image data 24 in the comparison result data 27 received from the labeling unit 9 are adjacent to each other. Then, the boundary area is extracted, the processing area 2d adjacent to the boundary area is set as the peripheral area, and the peripheral area data is formed based on the image data existing in the peripheral area (step S8).

  The peripheral area data 31 formed by the peripheral area forming unit 10 in step S8 is supplied to the continuity determining unit 11. The continuity determination unit 11 receives the first image data 21 and the second image data 24 output from the edge information extraction unit 7, and obtains the address information of the peripheral region data of the first image data 21 and the second image data 24. The data of the processing area 2d included is extracted, and it is determined whether or not there is continuity between the edge information existing in the adjacent processing areas 2d in the peripheral area data 31 (step S9). For the determination of the presence or absence of continuity, the determination methods as shown in the first method to the fourth method are used.

  The continuity determination unit 11 labels the region 32 determined to have continuity with an appropriate image identifier based on the determination result for each processing region 2d to form second labeling data (step S10).

  The second labeling data formed by the continuity extraction unit 11 in step S10 is supplied to the correction unit 12. The correction unit is further supplied with the first labeling data formed by the labeling unit 9 stored in the storage unit 14, and the correction unit 12 compares the first and second labeling data and is associated with a different screen identifier. It is confirmed whether there is 2d (step S11). If there is a processing area 2d associated with a different screen identifier, a screen identifier different from the second labeling data in the first labeling data is replaced with the screen identifier of the second labeling data (step S12). When the image identifier of the labeling data formed by the continuity determining unit 11 and the image identifier of the labeling data formed by the labeling unit 9 completely match, the correcting unit 12 does not perform the process of step S12. The labeling data formed as a result of the comparison between the first labeling data and the second labeling data by the second determination unit 12 is output as the third labeling data as the correction result regardless of the replacement of the screen identifier.

  The correction unit 12 outputs the third labeling data 34, and the third labeling data 34 is supplied to the image data synthesis unit 13. The image data supply unit 13 further receives the supply of the first image data and the second image data from the storage unit 14, and the first image data based on the address information and the image identifier information for each processing region 2d of the third labeling data 34. And the second image data are synthesized for each processing region 2d to generate synthesized image data (step S13).

  In the embodiment shown in FIGS. 8 to 21, the composite image data is formed by extracting the edge amount included in the image data in the feature amount information extraction unit 7. However, when color information such as RGB is used. The same processing is performed in step. However, the feature amount information extraction unit 7 extracts color information, and the continuity determination unit 11 uses the method as described above (fifth method) for the possibility that the color information for each adjacent processing region 2d is continuous. to decide.

  As described above, in the image composition device 1 according to the present embodiment, the image data can be divided into a plurality of processing regions 2d, and high-quality image data can be selected for each processing region 2d to form the composite image data 38. High-quality composite image data 38 can be acquired using only the two image data of the first image data 21 and the second image data 24.

  In the image composition device 1 according to the present embodiment, the comparison result based on the amount of feature amount information included in the corresponding processing area 2d of different image data and the characteristics included in the adjacent processing area 2d of the same image data. Since the processing region 2d for forming the composite image data 38 is selected using two different criteria, ie, the determination result of the determination result of the continuity of the quantity information, the composite image data 38 with higher quality can be formed.

  In the image composition device 1 according to the present embodiment, it is determined that the processing region 2d formed based on different image data in the comparison result is adjacent to the peripheral region, and the peripheral region data 31 has continuity. By using the determined region 32 as a determination result, composite image data is formed in such a manner that the comparison result of the feature amount information is corrected by the determination result of continuity, and a boundary region between the foreground subject 22 and the distant subject 23 is formed. Since the image data can be aligned with either the first image data or the second image data, it is difficult for the subject to shift in position or size, so that the composite image data 38 that is almost visually uncomfortable. Can be formed.

  In the image composition device 1 according to the present embodiment, the comparison result in the comparison unit 8, the determination result in the continuity determination unit 11, and the correction result in the correction unit 12 are formed as labeling data based on image identifiers. The comparison between the result and the determination result of continuity and the transmission of the comparison result to the image data synthesizing unit 13 can be reliably performed by simple processing.

1 is a configuration block diagram illustrating configurations of an imaging device and an image composition device according to an embodiment of the present invention. It is an image figure of the processing area formed in an image composition device same as the above. It is a specific example of the convolution operator which the feature-value information extraction part of an image composition apparatus same as the above uses. It is a flowchart which shows the process sequence by the 1st method of the continuity judgment part of an image composition apparatus same as the above. It is a figure which shows the model of the process by the 1st method of a continuity judgment part same as the above. It is a flowchart which shows the process sequence by the 2nd method of a continuity judgment part same as the above. It is explanatory drawing of the principle of the Hough transform used in the 3rd method of a continuity judgment part same as the above. It is a flowchart of the image composition process in an image composition apparatus same as the above. It is an image figure of the 1st image data which the image composition apparatus same as the above acquired. It is an image figure of the 2nd image data which the image composition apparatus same as the above acquired. It is an image figure of the data which extracted the edge component from 1st image data in the image composition apparatus same as the above. It is an image figure of the data which extracted the edge component from 2nd image data in the image composition apparatus same as the above. It is an image figure of the comparison result data formed in an image composition apparatus same as the above. It is an image figure of the 1st labeling data formed in an image composition apparatus same as the above. It is an image figure of the peripheral region data formed in an image composition apparatus same as the above. It is an image figure of the judgment result of the continuity of the continuity judgment part in an image composition apparatus same as the above. It is an image figure of the 2nd labeling data formed in an image composition apparatus same as the above. It is an image figure of the 3rd labeling data formed in an image composition apparatus same as the above. It is an image figure which shows the process which synthesize | combines 1st image data and 2nd image data in an image composition apparatus same as the above. It is an image figure of the synthesized image data synthesize | combined in the image synthesizer same as the above. It is an image figure of the virtual synthetic | combination image data formed when it synthesize | combines only by the comparison result data in an image synthesis apparatus same as the above.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1A ... Imaging device, 1 ... Image composition apparatus, 2 ... Imaging part, 2a ... Lens part, 2d ... Processing area, 2i ... Determination area, 3 ... Shutter drive means 3a ... shutter detection unit, 3b ... focus adjustment unit, 3c ... shutter operating unit, 7 ... feature quantity information extraction unit, 8 ... comparison unit, 9 ... labeling unit, 10 ... peripheral area forming unit, 11 ... continuity determination unit, 12 ... correction unit, 13 ... image data synthesis unit, 21 ... first image data, 24 ... second image data 30 ... first labeling data, 33 ... second labeling data, 34 ... third labeling data, 38 ... composite image data

Claims (9)

An image synthesizing apparatus for forming a one composite image data based on the plurality of images data that the imaging device is acquired continuously shooting by changing the focus of the same subject,
Feature amount information extracting means for extracting feature amount information included in each image data for each processing region formed in a predetermined pixel group unit;
It compared the feature amount information in which the feature amount information extraction means has extracted for each of the processing regions corresponding of the plurality image data of each of each of the processing region and the feature amount information of processing area data of the most common image data A comparison means for extracting a set as a comparison result;
Out of the comparison results, a peripheral region forming unit that extracts a portion where processing regions of different image data are adjacent to each other, and forms a peripheral region;
It is determined whether or not the feature amount information existing in the processing region corresponding to the peripheral region is continuous. For a processing region that is determined to have continuity of feature amount information, the feature amount information is the most in the processing region. Continuity determination means for outputting a set of processing area data corresponding to a large amount of image data as a determination result;
Correction means for outputting a correction result obtained by correcting the comparison result by comparing the comparison result and the determination result;
Image data synthesizing means for synthesizing processing area data of corresponding image data in the plurality of image data in the processing area unit using the correction result to form the synthesized image data. Image composition device. The feature amount information is an edge component information,
The image synthesizer according to claim 1 , wherein the continuity is a possibility that edge component information included in each of the adjacent processing regions is continuous. The feature amount information is color information,
The image synthesizer according to claim 1 , wherein the continuity is a possibility that color information included in each of the adjacent processing regions is continuous. The continuity determination unit forms a determination region for measuring feature amount information included in two adjacent processing regions included in a peripheral region in units of pixel data,
The image synthesizing apparatus according to claim 1 , wherein continuity of feature amount information between adjacent processing regions is determined based on whether or not the feature amount information is present in a predetermined amount or more in the determination region. . The image synthesizer according to claim 4 , wherein the continuity determination unit determines continuity of feature amount information between adjacent processing regions with respect to binarized feature amount information. The continuity determination means may be divided into a plurality of stages that depends feature amount information of each of the pixel data to the tone of the feature amount information, binarizing the feature amount information divided per the steps The image synthesizing device according to claim 5 . The image synthesizer according to claim 1 , wherein the continuity determination unit determines continuity of feature amount information between two adjacent processing regions included in a peripheral region by using a Hough transform. . The imaging device
An imaging unit;
Shutter driving means and
With
The shutter driving means includes
Shutter detection means for detecting the operation of the shutter by the operation of the shutter button of the imaging unit by the user;
A focus adjusting unit b that adjusts the focus of the lens unit of the imaging unit in response to detection of the operation of the shutter by the shutter detecting unit;
In response to adjustment of focus of the lens unit by said focusing means have a shutter operating means for operating the shutter of the imaging unit,
Receiving the operation of the shutter button of the imaging unit by a user, causing the imaging unit to continuously capture the same subject, and acquiring a plurality of image data of the same subject with the focus changed. The image composition device according to claim 1.   A program for causing a computer to execute the image composition apparatus according to any one of claims 1 to 9.

Images