JP2008092515A - Electronic camera, image processing apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a means of highly accurately and quickly extracting a subject from a photographic image. <P>SOLUTION: An electronic camera comprises an imaging unit, a region dividing unit, a first arithmetic unit, and a subject extracting unit. The imaging unit photographs a subject image and generates data of a photographic image. The region dividing unit determines, in the photographic image, the range of a first partial region including a main subject and the range of a second partial region excluding the first partial region. The first arithmetic unit analyzes a spatial frequency of data in the first partial region. The subject extracting unit extracts a main subject from the range of the first partial region in the photographic image based on an arithmetic result of the first arithmetic unit. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an electronic camera, an image processing apparatus, and a program for an image processing apparatus having a function of extracting a main subject from a captured image.

Conventionally, a number of methods have been proposed for techniques for extracting a main subject from an image captured by an electronic camera. For example, as an example of the prior art, Patent Document 1 discloses a technique for detecting a main subject in a captured image by edge extraction processing based on a change in luminance signal level.
JP-A-9-233384

  However, since the subject extraction processing is performed on the entire captured image in the conventional technology, the calculation amount of the processor has become very large. Moreover, when the size of the captured image that is the subject of the subject extraction process increases, there is a high possibility that a plurality of similar subjects coexist in one image, and the accuracy of the subject extraction process tends to decrease. As a result, in the prior art, it is very difficult to quickly extract a subject from a captured image with high accuracy, and there has been a demand for improvement.

  The present invention is to solve the above-mentioned problems of the prior art. An object of the present invention is to provide means capable of quickly extracting a subject from a captured image with high accuracy.

  An electronic camera according to a first aspect includes an imaging unit, a region dividing unit, a first calculation unit, and a subject extraction unit. The imaging unit captures a subject image and generates captured image data. The area dividing unit determines a range of the first partial area including the main subject and a range of the second partial area excluding the first partial area in the captured image. The first calculation unit analyzes the spatial frequency of the data in the first partial region. The subject extraction unit extracts a main subject from the range of the first partial region of the captured image based on the calculation result of the first calculation unit.

  An electronic camera of a second invention includes an imaging unit, a light emitting unit, a region dividing unit, a second calculating unit, and a subject extracting unit. The imaging unit captures a subject image and generates captured image data. The light emitting unit irradiates the subject with light. The area dividing unit determines a range of the first partial area including the main subject and a range of the second partial area excluding the first partial area in the captured image. The second calculation unit is configured to generate the first partial region of the first captured image and the second captured image based on the first captured image acquired by causing the light emitting unit to emit light and the second captured image acquired without causing the light emitting unit to emit light. At least one of the brightness difference from the first partial region and the brightness ratio. The subject extraction unit extracts a main subject from the range of the first partial region of the captured image based on the calculation result of the second calculation unit.

  An electronic camera according to a third invention includes an imaging unit, a light emitting unit, a region dividing unit, a first calculation unit, a second calculation unit, a subject extraction unit, and a control unit. The imaging unit captures a subject image and generates captured image data. The light emitting unit irradiates the subject with light. The area dividing unit determines a range of the first partial area including the main subject and a range of the second partial area excluding the first partial area in the captured image. The first calculation unit analyzes the spatial frequency of the data in the first partial region. The second calculation unit is configured to generate the first partial region of the first captured image and the second captured image based on the first captured image acquired by causing the light emitting unit to emit light and the second captured image acquired without causing the light emitting unit to emit light. At least one of the brightness difference from the first partial region and the brightness ratio. The subject extraction unit extracts a main subject from the range of the first partial region of the captured image based on the calculation result of at least one of the first calculation unit and the second calculation unit. A control part changes ON / OFF of operation | movement of a 1st calculating part and a 2nd calculating part, respectively.

In a fourth aspect based on the third aspect, the control section turns off the operation of the second calculation section when the shooting mode setting is in a state where the light emission section prohibits light emission.
In a fifth aspect based on the third aspect, the electronic camera further includes an operation unit. The operation unit receives input of setting of operations of the first calculation unit and the second calculation unit from the user. And a control part changes ON / OFF of operation | movement of a 1st calculating part and a 2nd calculating part based on a setting input.

In a sixth aspect based on any one of the first to third aspects, the electronic camera further includes a subject recognition unit. The subject recognizing unit detects a characteristic portion of the main subject from the captured image, and estimates the approximate position of the main subject in the captured image. Then, the region dividing unit determines the range of the first partial region and the range of the second partial region based on the output of the subject recognition unit.
In a seventh aspect based on the sixth aspect, the subject recognition unit detects a human face from the captured image.

In an eighth aspect based on the seventh aspect, the region dividing unit determines the size of the range of the first partial region in proportion to the size of the face detected by the subject recognition unit.
In a ninth aspect based on any one of the first to third aspects, the electronic camera further includes an operation unit. The operation unit receives a designation input for designating the approximate position of the main subject from the user. Then, the region dividing unit determines the range of the first partial region and the range of the second partial region based on the designated input.

In a tenth aspect based on any one of the first to third aspects, the imaging unit captures a main image for recording and a through image having a resolution lower than that of the main image. The area dividing unit, the first calculation unit, and the second calculation unit perform respective processes on the data of the main image or the through image. Then, the subject extraction unit extracts the main subject from the main image corresponding to the through image.
In an eleventh aspect based on the tenth aspect, the electronic camera further includes a monitor. The monitor displays a through image together with a display that allows the user to recognize the range of the first partial region.

In a twelfth aspect based on any one of the first to third aspects, the electronic camera further includes an image processing unit. The image processing unit performs image processing on the captured image data based on the extraction range of the main subject by the subject extraction unit.
In a thirteenth aspect based on the twelfth aspect, the image processing section performs a blur enhancement process for increasing the degree of blur in the background area excluding the main subject extraction range from the photographed image.

In a fourteenth aspect based on any one of the first to third aspects, the electronic camera further includes a photographing condition determining unit. The imaging condition determination unit determines the imaging condition of the imaging unit based on the extraction range of the main subject by the subject extraction unit.
In a fifteenth aspect based on any one of the first to third aspects, the electronic camera further includes a recording unit. The recording unit records the data of the extraction range of the main subject by the subject extracting unit together with the captured image data.

  An image processing apparatus according to a sixteenth aspect includes a data reading unit, a region dividing unit, a first calculating unit, and a subject extracting unit. The data reading unit reads data of a photographed image obtained by photographing the subject image. The area dividing unit determines a range of the first partial area including the main subject and a range of the second partial area excluding the first partial area in the captured image. The first calculation unit analyzes the spatial frequency of the data in the first partial region. The subject extraction unit extracts a main subject from the range of the first partial region of the captured image based on the calculation result of the first calculation unit.

  An image processing apparatus according to a seventeenth aspect includes a data reading unit, a region dividing unit, a second calculating unit, and a subject extracting unit. The data reading unit reads the data of the first photographed image taken by irradiating the subject with light and the data of the second photographed image taken without irradiating the subject with light. The region dividing unit determines a range of the first partial region including the main subject and a range of the second partial region excluding the first partial region in the first captured image and the second captured image. The second calculation unit obtains at least one of a brightness difference and a brightness ratio between the first partial region of the first captured image and the first partial region of the second captured image. The subject extraction unit extracts a main subject from the range of the first partial region of at least one of the first captured image and the second captured image based on the calculation result of the second calculation unit.

  An image processing apparatus according to an eighteenth aspect includes a data reading unit, a first calculation unit, and a subject extraction unit. The data reading unit reads data of a photographed image obtained by photographing the subject image and data indicating the range of the first partial area including the main subject in the photographed image. The first calculation unit analyzes the spatial frequency of the data of the partial region in the captured image. The subject extraction unit extracts a main subject from the range of the first partial region of the captured image based on the calculation result of the first calculation unit.

  An image processing apparatus according to a nineteenth invention includes a data reading unit, a second calculation unit, and a subject extraction unit. The data reading unit includes data of a first captured image captured by irradiating the subject with light, data of a second captured image captured without irradiating the subject with light, the first captured image, and the first captured image. Data indicating the range of the first partial area including the main subject in the two captured images is read. The second calculation unit obtains at least one of a brightness difference and a brightness ratio between the first partial region of the first captured image and the first partial region of the second captured image. The subject extraction unit extracts a main subject from the range of the first partial region of at least one of the first captured image and the second captured image based on the calculation result of the second calculation unit.

  In addition, what expressed the structure regarding said image processing apparatus converted into a program, an image processing method, a recording medium, etc. is also effective as a specific aspect of this invention.

  According to the electronic camera or the image processing apparatus of the present invention, it is possible to quickly extract a subject from a captured image with high accuracy.

(Description of the first embodiment)
FIG. 1 is a block diagram showing the configuration of the electronic camera of the first embodiment.
The electronic camera includes a photographic lens 11, a lens driving unit 12, an image sensor 13, an analog processing unit 14, a memory 15, an image processing unit 16, a compression / decompression processing unit 17, a recording I / F 18, an external An I / F 19, a communication unit 20, a light emitting unit 21, an operation unit 22, a monitor 23, a CPU 24 and a system bus 25 are included. The memory 15, the image processing unit 16, the compression / decompression processing unit 17, the recording I / F 18, the external I / F 19, the communication unit 20, the monitor 23, and the CPU 24 are connected via a system bus 25.

The photographing lens 11 is composed of a plurality of lens groups including a zoom lens and a focusing lens for adjusting a focus position. The position of the photographic lens 11 is driven and adjusted in the optical axis direction by the lens driving unit 12. The input / output of the lens driving unit 12 is connected to the CPU 24.
The image sensor 13 photoelectrically converts the light beam that has passed through the photographing lens 11 to generate an analog image signal of the subject image. In the shooting mode for shooting a subject, the image sensor 13 captures a recording image (main image) at the time of shooting, and outputs a through image by thinning out at predetermined intervals even during standby for shooting. The through image data is used for various arithmetic processes by the CPU 24 and display on the monitor 23.

  The analog processing unit 14 is an analog front-end circuit that performs analog signal processing on the output of the image sensor 13. The analog processing unit 14 includes a CDS circuit, a gain circuit, and an A / D conversion circuit. The CDS circuit of the analog processing unit 14 reduces the noise component of the output of the image sensor 13 by correlated double sampling. The gain circuit of the analog processing unit 14 amplifies the gain of the input signal based on an instruction from the CPU 24. In the gain circuit described above, the imaging sensitivity corresponding to the ISO sensitivity can be adjusted. The A / D conversion circuit of the analog processing unit 14 performs A / D conversion on the analog image signal output from the gain circuit. The output of this A / D conversion circuit is connected to the memory 15.

The memory 15 is a buffer memory for temporarily recording image data in a pre-process or post-process of image processing described later.
The image processing unit 16 performs various kinds of image processing on the digital image signal at the time of shooting in the above shooting mode to generate a main image. Further, the image processing unit 16 generates a display image (view image) from the image signal of the through image in accordance with an instruction from the CPU 24 in the above shooting mode. On the other hand, the image processing unit 16 can also perform image processing by post-processing on the data of the main image captured in advance. The specific contents of the image processing by the image processing unit 16 will be described later.

The compression / decompression processing unit 17 is an ASIC having an analysis unit and a compression unit (a separate illustration of the analysis unit and the compression unit is omitted). The analysis unit of the compression / decompression processing unit 17 performs spatial frequency analysis on the captured image data. On the other hand, the compression unit of the compression / decompression processing unit 17 executes compression / decompression processing in JPEG (Joint Photographic Experts Group) format.
Here, in subject extraction processing (first extraction processing) of the electronic camera, spatial frequency analysis processing by an analysis unit can be used. When the analysis unit performs spatial frequency analysis for subject extraction, image data compression processing by the compression unit is not necessarily required. That is, what is necessary for the subject extraction process is a spatial frequency analysis result, and image data compression processing is unnecessary. Accordingly, in the case of subject extraction processing, data output from the compression / decompression processing unit 17 is not recorded on the recording medium 26 described later.

  A connector for connecting the recording medium 26 is formed in the recording I / F 18. The recording I / F 18 executes writing / reading of main image data and the like with respect to the recording medium 26 connected to the connector. The recording medium 26 is composed of a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is illustrated as an example of the recording medium 26.

The external I / F 19 has a connection terminal of a serial communication standard such as USB (Universal Serial Bus). The external I / F 19 controls data transmission / reception with an external device (such as a computer) connected via a connection terminal in accordance with the communication standard.
The communication unit 20 controls communication with an external device in accordance with specifications defined in a known wireless communication standard such as the wireless LAN standard IEEE 802.11 or Bluetooth (registered trademark). For example, when transmitting captured image data to an external device, the communication unit 20 converts the captured image data into data that can be wirelessly communicated, and transmits the converted data to the external device.

The light emitting unit 21 irradiates a subject with a flash according to an instruction from the CPU 24 at the time of photographing. The light emitting unit 21 includes, for example, a xenon arc tube, a main capacitor for storing light emission energy, a reflector or lens member for efficiently irradiating a subject with flash light, a light emission control circuit, and the like.
The operation unit 22 includes a release button, a mode dial, an operation button, and the like. The release button of the operation unit 22 receives an instruction input for starting an exposure operation from the user. The mode dial of the operation unit 22 receives a switching input of the photographing mode of the electronic camera from the user. The operation buttons of the operation unit 22 are configured with, for example, a cross key. This operation button receives a setting input on a menu screen described later and a range designation input in a subject extraction process described later from the user.

  The monitor 23 displays various images according to instructions from the CPU 24. The monitor 23 in this embodiment is a liquid crystal monitor. The monitor 23 displays a moving image of the view image during shooting standby in the shooting mode. Further, the monitor 23 displays various information necessary for photographing by an on-screen function (for example, display of a frame of a face area detected by a face detection process described later, various information in a subject extraction process described later, etc.). Can be superimposed on the view image.

  Further, the monitor 23 can display a menu screen or the like that allows input in GUI (Graphical User Interface) format. On the menu screen, various settings for subject extraction processing, which will be described later, can be input to the CPU 24. For example, the user turns on / off the subject extraction process, turns on / off each of the first extraction process (spatial frequency analysis) and the second extraction process (analysis of brightness between two images) in the subject extraction process, image processing You can set on / off of the on the menu screen. Note that the display processing on the menu screen is executed by the CPU 24 based on a program (the menu screen is not shown).

The CPU 24 controls the operation of each part of the electronic camera according to the mode dial of the operation unit 22 or the mode selected on the menu screen. Further, the CPU 24 executes various calculations necessary for subject extraction processing described later. Furthermore, the CPU 24 of the present embodiment also functions as a shooting condition determination unit 27 and a face detection unit 28.
The imaging condition determination unit 27 performs known autofocus (AF) by a contrast detection method based on the data of the through image. The photographing condition determination unit 27 performs an automatic exposure (AE) calculation and an auto white balance (AWB) calculation based on the through image data.

  The face detection unit 28 detects the face area of the subject, the size of the face, and the like from the data of the through image or the main image. For example, the face detection unit 28 extracts a face region by a feature point extraction process described in Japanese Patent Laid-Open No. 2001-16573. Examples of the feature points include eyebrow, eye, nose, and lip end points, face contour points, head apexes, and chin lower end points. Alternatively, the face detection unit 28 extracts the outline of the skin color area based on the color information of the subject as in Japanese Patent Laid-Open No. 8-63597, and performs matching with a template of a face component prepared in advance. An area may be detected.

The operation of the CPU 24 at the time of shooting in the electronic camera of the first embodiment will be described below with reference to the flowchart of FIG.
Step 101: The CPU 24 drives the imaging device 13 to start acquiring a through image. The imaging element 13 acquires an image signal of a through image by thinning out reading at predetermined intervals. The image processing unit 16 sequentially generates view images based on the through image data. The view image is displayed as a moving image on the monitor 23 at the time of shooting standby. Therefore, the user can perform framing for determining the shooting composition based on the view image on the monitor 23.

Step 102: The CPU 24 determines the range of the first partial area and the range of the second partial area in the through image corresponding to the shooting screen. As a result, the photographing screen is divided into two parts by the range of the first partial region and the range of the second partial region.
The first partial area is an image that is a calculation target of a subject extraction process, which will be described later, and its outer edge is determined so as to include the entire main subject to be extracted. That is, the range of the first partial area includes the entire main subject to be extracted. The second partial area is an image of the remaining range obtained by removing the range of the first partial area from the above-described shooting screen. Since the second partial region does not include the main subject, the second partial region is not a calculation target for subject extraction processing described later.

Here, in S102, the CPU 24 determines the range of the first partial area and the second partial area by any one of the following methods (1) to (3).
(1) The face detection unit 28 of the CPU 24 executes face detection processing on the through image. Then, the CPU 24 estimates the approximate position of the main subject based on the detected position of the face area, and determines the range of the first partial area. In this case, there is a high possibility that a person will be the main subject. In this case, since the range of the first partial area is automatically set according to the result of face detection, the operability of the electronic camera is greatly improved.

When a plurality of faces are detected, the CPU 24 determines the range of the first partial area based on either the face area that is estimated to be the largest and closest to the face, or the face area that is located closest to the center. decide.
Hereinafter, a method for determining the range of the first partial region by face detection will be described more specifically.
First, the CPU 24 executes face detection processing. Then, the CPU 24 obtains the face size and the vertical direction of the face from the positional relationship of each feature point and face part. At this time, the CPU 24 displays a rectangular frame superimposed on the face position of the main subject in the view image of the monitor 23 (see FIG. 3).

  Further, in a scene that is not suitable for subject extraction processing for extracting the entire person, such as when the face on the shooting screen is too large or too small, the CPU 24 warns the user. Based on the presence or absence of this warning, the user can easily determine whether or not the photographing composition is suitable for subject extraction processing. Therefore, the usability of the electronic camera becomes better. Examples of the warning method include a method in which a warning display is superimposed on the monitor 23 and a method in which a warning is given to the user by voice or vibration (illustration of the warning display on the monitor 23). Is omitted).

  Next, the CPU 24 temporarily sets a candidate area for the first partial area based on the face position of the main subject and the vertical direction of the face. In the example of FIG. 4, the candidate area of the first partial area has a humanoid shape. The CPU 24 enlarges or reduces the size of the candidate area in proportion to the face size of the main subject. Thereafter, the CPU 24 sets the candidate area as the range of the first partial area. Here, FIG. 4A shows the range of the first partial region in the scene corresponding to FIG. FIG. 4B shows the range of the first partial region in a scene in which a human face is larger than that in FIG.

(2) The CPU 24 receives a designation input for manually designating the approximate position of the main subject from the operation unit 22 from the user. Then, the CPU 24 determines the range of the first partial area based on the above specified input. In this case, subject extraction processing is performed by the first partial region selected by the user himself / herself.
As an example, the CPU 24 superimposes and displays a candidate area having a shape such as a rectangle, a circle, and an ellipse on the view image of the monitor 23. The user uses the operation buttons on the operation unit 22 to change the size and move the candidate area. Then, when the candidate area has a desired position and size as the range of the first partial area, the user performs designation input for setting the candidate area as the range of the first partial area to the CPU 24.

  (3) When shooting is performed in a state where a composition assist frame for determining the composition is superimposed on the view image of the monitor 23, the range of the first partial region is determined based on the position of the composition assist frame. In this case, it is considered that the main subject is located in the composition assistance frame. The CPU 24 may match the range of the first partial area with the range of the composition auxiliary frame, or may set the range of the first partial area wider than the range of the composition auxiliary frame.

  Step 103: The CPU 24 displays a display indicating the range of the first partial area determined in S102 in a superimposed manner on the view image of the monitor 23. Thereby, the user can confirm the range of the first partial area determined in S102. When the range of the first partial area is inappropriate, the user changes the shooting composition, resets the range of the first partial area, or the like.

Here, in the display in S103, the CPU 24 may enclose the outer edge portion of the first partial region with a frame. Or CPU24 may display the image which changed the display state of the color and the brightness in the range of the 1st partial field and the 2nd partial field. FIG. 5 shows an example in which the range of the first partial area corresponding to FIG. 4A is displayed as a frame in the view image.
Step 104: The CPU 24 determines whether or not the release button is half-pressed. If the release button is pressed halfway (YES side), the process proceeds to S105. On the other hand, when the release button is not half-pressed (NO side), the CPU 24 returns to S102 and repeats the above operation.

Step 105: The CPU 24 determines whether or not to omit subject extraction processing for the first partial region. For example, when the subject extraction processing item is set to OFF on the menu screen, the CPU 24 performs the determination of omitting the subject extraction processing of the first partial area in the above determination.
When the subject extraction process for the first partial region is omitted (YES side), the process proceeds to S106. On the other hand, when subject extraction processing is executed for the first partial region (NO side), the process proceeds to S108.

  Step 106: The photographing condition determination unit 27 of the CPU 24 executes AF with reference to the AF area within the first partial area. In addition, the shooting condition determination unit 27 of the CPU 24 performs AE calculation and AWB calculation based on the through image data corresponding to the first partial region, and determines shooting conditions (exposure time, aperture value, and imaging sensitivity). When face recognition AF is performed with the face of the main subject as a reference together with the face detection process, the CPU 24 performs AF lock by pressing the release button halfway.

Step 107: The CPU 24 shoots the subject in response to the release button being fully pressed. Then, the CPU 24 causes the image processing unit 16 to generate main image data. The data of the main image is finally recorded on the recording medium 26. In the case of S107, the photographing operation is terminated without performing subject extraction processing for the first partial region.
In S107, the CPU 24 may record attached data indicating the range of the first partial area and the range of the second partial area in the data of the main image. The attached data is recorded using, for example, a MakerNote tag of the Exif (Exchangeable image file format for digital still cameras) standard. According to the data of the main image, it becomes easy to execute a subject extraction process described later later by an image processing apparatus or the like.

Step 108: In this case, the CPU 24 determines whether or not a main subject is to be extracted from the range of the first partial region only by a first extraction process (spatial frequency analysis) described later. That is, in S108, the CPU 24 determines whether or not a second extraction process described later is omitted in the subject extraction process.
For example, (1) when the item of the second extraction process is set to OFF on the menu screen, (2) when a shooting mode for prohibiting light emission of the light emitting unit 21 is selected, (3) battery of the electronic camera When the level corresponds to any of the cases where the level is equal to or lower than the threshold value, the CPU 24 performs a determination of omitting the second extraction process in the determination.

When the subject extraction process is performed only by the first extraction process (YES side), the process proceeds to S109. On the other hand, when subject extraction processing involving the second extraction processing is performed (NO side), the process proceeds to S110.
Step 109: The CPU 24 extracts a main subject from the first partial region by frequency analysis (first extraction processing) using the compression / decompression processing unit 17. Note that after the processing of S109, the CPU 24 proceeds to S113.

  In the first extraction process of S109, first, the CPU 24 divides the first partial area of the through image into pixel blocks of about 8 × 8 pixels. Next, the compression / decompression processing unit 17 performs a discrete cosine transform (DCT) process on each pixel block in accordance with an instruction from the CPU 24, and acquires a DCT coefficient for each pixel block. The CPU 24 extracts a pixel block constituting the outline of the main subject from the pixel blocks in the first partial area.

  Here, the relationship between the DCT coefficient and the main subject extraction process will be briefly described. FIG. 6 is a diagram showing a change in gradation at a contour portion in a monochrome image. When the change in brightness is steep (an image with a clear outline) as shown in FIG. 6A, the image data includes many high-frequency spatial frequency components. Therefore, the DCT coefficient obtained from the image corresponding to FIG. 6A has a non-zero value up to a high spatial frequency.

  On the other hand, when the change in brightness is gentle as shown in FIG. 6B (an image that does not include an outline), the high frequency spatial frequency component is reduced from the image data as compared with the case of FIG. The value of the DCT coefficient obtained from the image corresponding to FIG. 6B is 0 from the lower spatial frequency component than in the case of FIG. Therefore, the CPU 24 pays attention to the distribution of the DCT coefficient for each frequency component, and extracts a pixel block whose DCT coefficient corresponding to the high frequency component is not zero as the outline of the main subject. Note that the CPU 24 may group the areas partitioned by the outline based on information such as the color difference and brightness of the image.

  Step 110: In this case, the CPU 24 determines whether or not to omit the first extraction process in the subject extraction process. Specifically, when the item of the first extraction process is set to OFF on the menu screen, the CPU 24 performs a determination of omitting the first extraction process in the above determination. When the first extraction process is omitted (YES side), the process proceeds to S111. On the other hand, when both the first extraction process and the second extraction process are performed in the subject extraction process (NO side), the process proceeds to S112.

Step 111: The CPU 24 extracts the main subject from the first partial region by analyzing the brightness between the two images captured by changing the light emission state of the light emitting unit 21 (second extraction process). Note that after the processing of S111, the CPU 24 proceeds to S113.
In the second extraction process of S111, the CPU 24 acquires images obtained by photographing substantially the same subject in a state where the light emitting unit 21 emits light and a state where the light emitting unit 21 does not emit light. For convenience of explanation, an image captured by causing the light emitting unit 21 to emit light is defined as a first captured image. A through image captured without causing the light emitting unit 21 to emit light is defined as a second captured image. In the first embodiment, the first captured image and the second captured image are both through images.

Next, the CPU 24 extracts a main subject from the first partial area by the following method (1) or (2) (see FIG. 7).
(1) The CPU 24 calculates a brightness difference value for each pixel in the first partial region of the first captured image and the second captured image. Then, the CPU 24 extracts a main subject from the first partial area of the first photographed image or the second photographed image based on the brightness difference value.

  In general, the illuminance during flash photography decreases in proportion to the square of the distance from the light source. That is, during flash photography, the main subject is illuminated brightly, while the brightness of the background portion located farther from the main subject is low. Therefore, since the difference value of brightness between the flash photography and the non-flash photography changes according to the distance from the camera, the main subject can be extracted from the photographed image by using such a difference value.

(2) The CPU 24 calculates the ratio between the brightness of the first captured image and the brightness of the second captured image in the first partial region of the first captured image and the second captured image. Then, the CPU 24 extracts a main subject from the first partial region of the first captured image or the second captured image based on the brightness ratio.
If subject extraction is performed with the difference value of (1) above, the extraction accuracy may be reduced for subjects in which a portion with a high reflectance and a portion with a low reflectance are adjacent. For example, when shooting the face of a black-haired person, the dark-hair part has a low reflectance, so the difference in brightness is small, while the face part has a relatively high reflectance, so the brightness is high. The difference value of becomes larger. Therefore, in the above case, there is a possibility that the person does not recognize the hair part and the face part as the same subject.

  Here, the brightness of the first captured image is a value ((illumination light component + environment light component) × reflectance) obtained by multiplying the sum of the illumination light and the ambient light by the light emitting unit 21 by the reflectance of the subject. The brightness of the second captured image is a value obtained by multiplying the ambient light by the reflectance of the subject (environmental light component × reflectance). That is, when the subject is extracted based on the brightness ratio, the reflectance of the subject is canceled out. Therefore, it is possible to extract with high accuracy a subject in which a portion with a high reflectance and a portion with a low reflectance are adjacent as described above.

  Step 112: In this case, the CPU 24 performs both the first extraction process (S109) and the second extraction process (S111) to extract the main subject from the first partial area. For example, the CPU 24 narrows down the main subject from the first partial region in either one of the first extraction processing and the second extraction processing, and then performs the other processing to correct the extraction result and thereby reduce the range of the main subject. Finalize. In this case, the main subject can be extracted with higher accuracy.

  Step 113: The photographing condition determination unit 27 of the CPU 24 executes AF with reference to an AF area in the range of the main subject in the first partial area. Further, the shooting condition determination unit 27 of the CPU 24 executes AE calculation and AWB calculation based on the through image data corresponding to the range of the main subject to determine the shooting conditions (exposure time, aperture value, imaging sensitivity). To do. When face recognition AF is performed with the face of the main subject as a reference together with the face detection process, the CPU 24 performs AF lock by pressing the release button halfway.

Step 114: The CPU 24 captures the subject in response to the release button being fully pressed. Then, the CPU 24 causes the image processing unit 16 to generate main image data.
Step 115: The CPU 24 determines whether or not an item of image processing that requires subject extraction processing is set on in the menu screen. When the setting of the above item is on (YES side), the process proceeds to S116. On the other hand, if all of the above items are set off (NO side), the process proceeds to S117.

  Step 116: The CPU 24 causes the image processing unit 16 to perform image processing of the main image data (S114) based on the result of the subject extraction processing. The image processing unit 16 performs predetermined image processing set in the menu screen item on either the main subject range or a range other than the main subject (background region). Thereafter, the CPU 24 records the data of the main image on the recording medium 26 or the like. Hereinafter, an example of image processing in S116 will be described.

  (1) The image processing unit 16 performs a blur enhancement process for increasing the degree of blur of the image on the background area of the main image (see FIG. 8). Specifically, the image processing unit 16 performs a convolution operation with a point spread function (PSF) filter on the background region of the main image. As a result, it is possible to easily obtain an image expressed so that the main subject appears due to the background blur.

Further, the image processing unit 16 may further perform a known correction process (tone correction, edge enhancement filter process, etc.) for enhancing the contrast on the range of the main subject of the main image. In this case, since the main subject of the main image is emphasized, the stereoscopic effect of the image after image processing can be further enhanced.
(2) The image processing unit 16 performs correction to increase the saturation and brightness of the main subject range of the main image over the background area. In this case, it is possible to obtain a preferable captured image in which the skin color dullness of the person who is the main subject is reduced. Alternatively, the image processing unit 16 may perform correction for lowering the saturation of the main subject range of the main image so as to express the human skin white.

(3) The image processing unit 16 applies a gradation characteristic table of a soft gradation curve in which the contrast is relatively lower than the gradation curves of other areas to the range of the main subject of the main image. In this case, it is possible to obtain a preferable photographed image with softness on the face of the main subject.
(4) The image processing unit 16 executes a process of synthesizing a character such as a cartoon, a character, a figure, or the like in the range of the background area of the main image. Alternatively, the image processing unit 16 may generate the data of the main image by extracting only the range of the main subject by trimming. In this case, a photographed image with high entertainment can be provided to the user.

  Step 117: The CPU 24 records the attached data indicating the range of the main subject extracted by the subject extraction process on the recording medium 26 together with the main image data. The attached data is recorded using, for example, an Exif standard MakerNote tag. In this case, by using the attached data after shooting, the image processing apparatus can easily edit the image as described in S116. This is the end of the description of the series of operations in FIG.

  Note that in the electronic camera of the first embodiment, two main images corresponding to the first captured image and the second captured image are considered in consideration of the case where the second extraction process (S111) is performed afterwards in the image processing apparatus. It is also possible to generate an image file including the data. The image file may be recorded in association with attached data indicating the range of the first partial area and the range of the second partial area.

Hereinafter, effects of the first embodiment will be described.
In the electronic camera of the first embodiment, subject extraction processing is performed on the first partial region of the captured image. Therefore, it is possible to quickly extract the main subject with high accuracy compared to the case where subject extraction processing is performed on the entire captured image. In the electronic camera according to the first embodiment, the subject extraction process described above can be used to easily obtain, for example, a portrait image and the like by image processing.

(Description of Second Embodiment)
FIG. 9 is a flowchart for explaining image processing by the electronic camera in the second embodiment. In the second embodiment, an example will be described in which an electronic camera having the same configuration as that of the first embodiment functions as an image processing apparatus, and the subject extraction process is performed by reading data of a main image that has been captured in advance. Note that the configuration of the electronic camera in the second embodiment is the same as that of the first embodiment, and thus a duplicate description is omitted.

Step 201: The CPU 24 reads the main image data to be edited from the outside. For example, the CPU 24 may read main image data from the recording medium 26, or may read main image data from an external device (such as a computer) connected via the external I / F 19 or the communication unit 20.
Step 202: The CPU 24 determines whether or not the data to be edited read in S201 includes attached data indicating the range of the main subject extracted by the subject extraction process. That is, the CPU determines whether the data of the main image generated in S117 of the first embodiment is an editing target. If the above-mentioned attached data is included (YES side), the process proceeds to S210. On the other hand, if the above-mentioned attached data is not included (NO side), the process proceeds to S203.

Step 203: The CPU 24 determines whether or not the data to be edited read in S201 is an image file including two main images corresponding to the first captured image and the second captured image. If the image file corresponds to the above (YES side), the process proceeds to S204. On the other hand, if the image file does not correspond (NO side), the process proceeds to S207.
Step 204: The CPU 24 determines whether or not the data to be edited (image file) read in S201 includes attached data indicating the range of the first partial area and the range of the second partial area. When the above-mentioned attached data is included (YES side), the process proceeds to S206. On the other hand, if the above-mentioned attached data is not included (NO side), the process proceeds to S205.

  Step 205: The CPU 24 determines the range of the first partial area and the range of the second partial area by using either the first captured image or the second captured image. In this S205, a face detection process for the main image (a process corresponding to S102 (1) of the first embodiment) or a designation input of the range of the first partial area by the user (S102 (2) of the first embodiment) CPU 24 determines the range of the first partial area based on the processing to be performed.

Step 206: The CPU 24 extracts the main subject from the first partial region of any one of the main images by analyzing the brightness of the first and second captured images (second extraction process). Note that the second extraction process in S206 corresponds to the second extraction process in S111 of the first embodiment, and thus a duplicate description is omitted.
Step 207: The CPU 24 determines whether or not the data to be edited read in S201 includes attached data indicating the range of the first partial area and the range of the second partial area. When the above-mentioned attached data is included (YES side), the process proceeds to S209. On the other hand, if the above-mentioned attached data is not included (NO side), the process proceeds to S208.

  Step 208: The CPU 24 determines the range of the first partial area and the range of the second partial area in the main image. In S208, as in S205 described above, face detection processing for the main image (processing corresponding to S102 (1) in the first embodiment) or designation input of the range of the first partial area by the user (in the first embodiment) CPU 24 determines the range of the first partial area based on the process corresponding to S102 (2).

Step 209: The CPU 24 extracts a main subject from the first partial region of the main image by frequency analysis (first extraction processing) using the compression / decompression processing unit 17. Note that the first extraction process of S209 corresponds to the first extraction process of S109 of the first embodiment, and thus a duplicate description is omitted.
Step 210: The CPU 24 causes the image processing unit 16 to perform image processing of the main image data based on the result of the subject extraction processing. The image processing unit 16 performs predetermined image processing set in the menu screen item on either the main subject range or a range other than the main subject (background region). Thereafter, the CPU 24 records the data of the main image on the recording medium 26 or the like. Note that the description of each image processing is the same as that of S116 of the first embodiment, and thus a duplicate description is omitted.

The electronic camera of the second embodiment can perform subject extraction processing that is substantially the same as that of the first embodiment in image editing after shooting. Therefore, the convenience of the user in image editing after shooting is greatly improved.
(Description of the third embodiment)
FIG. 10 is a block diagram of an image processing apparatus according to the third embodiment. The third embodiment is a modification of the second embodiment, in which an image processing program is executed by a personal computer or the like to configure an image processing apparatus similar to the second embodiment. The image processing apparatus according to the third embodiment can obtain substantially the same effect as that of the second embodiment.

The image processing apparatus includes a control unit 31, a memory 32, an external I / F 33, a display driver unit 34, a disk module 35, and a system bus 36 that connects the units. In addition, an input device 37 such as a keyboard or a pointing device and a monitor 38 are connected to the image processing apparatus.
The control unit 31 controls and controls each unit of the image processing apparatus. In addition, the control unit 31 reproduces the functions of the image processing unit 16, the compression / decompression processing unit 17, and the CPU 24 of the electronic camera of the above embodiment on software by executing the image processing program. The memory 32 stores programs and data, and stores data transmitted to the control unit 31 and processing results returned from the control unit 31. The external I / F 33 controls data transmission / reception with peripheral devices (for example, the electronic camera, the recording medium 26, and the like) connected to the image processing apparatus in accordance with a serial communication standard such as USB. The display driver unit 34 outputs an image to the monitor 38. The disk module 35 reads / writes data to / from a recording medium such as an optical disk or a magneto-optical disk.

(Supplementary items of the embodiment)
(1) In the first embodiment, the example in which the subject extraction process is executed with the through image before the main image is captured and the extraction result of the main subject is reflected in the image processing of the main image has been described. However, in the present invention, the main subject may be extracted by performing the first extraction process and the second extraction process on the main image without performing the subject extraction process on the through image. Further, when performing the second extraction process in the first embodiment, the brightness between the images is analyzed by using one of the first photographed image and the second photographed image as a through image and the other as the main image. May be.

(2) The configuration of the electronic camera of the present invention is not limited to the above embodiment. For example, the CPU 24 may perform division photometry with a sensor dedicated to photometry without performing AE calculation based on through image data. Further, the communication unit 20 and the like may be omitted from the electronic camera.
(3) In the first extraction process of the above embodiment, orthogonal transform other than DCT such as wavelet transform may be used. In the first extraction process, for example, other known frequency analysis such as fast Fourier transform may be performed.

(4) In the above-described embodiment, the electronic camera or the image processing apparatus may perform subject recognition on a subject other than a human face by pattern matching processing or the like to determine the range of the first partial region.
(5) In the first embodiment, the example of the electronic camera that can selectively execute both the first extraction process and the second extraction process in the subject extraction process has been described. However, the electronic camera of the present invention may be configured to execute only one of the first extraction process and the second extraction process.

The block diagram which shows the structure of the electronic camera of 1st Embodiment. The flowchart which shows operation | movement of CPU at the time of imaging | photography in 1st Embodiment. The figure which shows the example which displayed the frame of the face area in the view image The figure which shows an example of the range of a 1st partial region, and the range of a 2nd partial region The figure which shows the example which displayed the range of the 1st partial area in the view image in the frame The figure which shows the gradation change in the outline part in the monochrome image Conceptual diagram for explaining the second extraction process The conceptual diagram explaining the blur enhancement process in 1st Embodiment Flowchart for explaining image processing by electronic camera in second embodiment The block diagram which shows the structure of the image processing apparatus of 3rd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 13 ... Image sensor, 16 ... Image processing part, 17 ... Compression / decompression processing part, 18 ... Recording I / F, 19 ... External I / F, 20 ... Communication part, 21 ... Light emission part, 22 ... Operation part, 23 ... Monitor , 24 ... CPU, 26 ... recording medium, 27 ... imaging condition determination unit, 28 ... face detection unit, 31 ... control unit, 33 ... external I / F, 35 ... disk module

Claims (23)

An imaging unit that captures a subject image and generates data of the captured image;
An area dividing unit that determines a range of a first partial area including a main subject and a range of a second partial area excluding the first partial area in the captured image;
A first arithmetic unit that analyzes a spatial frequency of the data of the first partial region;
A subject extraction unit that extracts the main subject from a range of the first partial region of the captured image based on a calculation result of the first calculation unit;
An electronic camera comprising: An imaging unit that captures a subject image and generates data of the captured image;
A light emitting unit that emits light to the subject;
An area dividing unit that determines a range of a first partial area including a main subject and a range of a second partial area excluding the first partial area in the captured image;
Based on the first captured image acquired by causing the light emitting unit to emit light and the second captured image acquired without causing the light emitting unit to emit light, the first partial region of the first captured image and the second captured image A second computing unit for obtaining at least one of a brightness difference and a brightness ratio with the first partial region;
A subject extraction unit that extracts the main subject from a range of the first partial region of the captured image based on a calculation result of the second calculation unit;
An electronic camera comprising: An imaging unit that captures a subject image and generates data of the captured image;
A light emitting unit that emits light to the subject;
An area dividing unit that determines a range of a first partial area including a main subject and a range of a second partial area excluding the first partial area in the captured image;
A first arithmetic unit that analyzes a spatial frequency of the data of the first partial region;
Based on the first captured image acquired by causing the light emitting unit to emit light and the second captured image acquired without causing the light emitting unit to emit light, the first partial region of the first captured image and the second captured image A second computing unit for obtaining at least one of a brightness difference and a brightness ratio with the first partial region;
A subject extraction unit that extracts the main subject from a range of the first partial region of the captured image based on a calculation result of at least one of the first calculation unit and the second calculation unit;
A control unit for changing on / off of the operations of the first calculation unit and the second calculation unit, and
An electronic camera comprising: The electronic camera according to claim 3.
The electronic control device, wherein the control unit turns off the operation of the second calculation unit when the shooting mode is set to prohibit light emission of the light emitting unit. The electronic camera according to claim 3.
An operation unit that receives a setting input of an operation of the first calculation unit and the second calculation unit from a user;
The control unit is configured to change the on / off of the operation of the first calculation unit and the second calculation unit based on the setting input. The electronic camera according to any one of claims 1 to 3,
A subject recognition unit that detects a characteristic portion of the main subject from the captured image and estimates a rough position of the main subject in the captured image;
The electronic camera according to claim 1, wherein the region dividing unit determines a range of the first partial region and a range of the second partial region based on an output of the subject recognition unit. The electronic camera according to claim 6.
The electronic camera, wherein the subject recognition unit detects a human face from the captured image. The electronic camera according to claim 7,
The electronic camera according to claim 1, wherein the region dividing unit determines the size of the range of the first partial region in proportion to the size of the face detected by the subject recognition unit. The electronic camera according to any one of claims 1 to 3,
An operation unit for accepting a designation input for designating an approximate position of the main subject from the user;
The area dividing unit determines the range of the first partial area and the range of the second partial area based on the designated input. The electronic camera according to any one of claims 1 to 3,
The imaging unit captures a main image for recording and a through image having a lower resolution than the main image,
The region dividing unit, the first calculation unit, and the second calculation unit perform respective processes on the data of the main image or the through image,
The electronic camera, wherein the subject extraction unit extracts the main subject from the main image corresponding to the through image. The electronic camera according to claim 10.
The electronic camera further comprising a monitor that displays the through image together with a display that allows a user to recognize the range of the first partial region. The electronic camera according to any one of claims 1 to 3,
An electronic camera, further comprising: an image processing unit that performs image processing on the captured image data based on an extraction range of the main subject by the subject extraction unit. The electronic camera according to claim 12,
The electronic camera according to claim 1, wherein the image processing unit performs a blur enhancement process for increasing a degree of blur of a background area excluding the extraction range of the main subject from the photographed image. The electronic camera according to any one of claims 1 to 3,
The electronic camera further comprising: a shooting condition determining unit that determines a shooting condition of the imaging unit based on an extraction range of the main subject by the subject extracting unit. The electronic camera according to any one of claims 1 to 3,
An electronic camera further comprising: a recording unit that records data of an extraction range of the main subject by the subject extraction unit together with data of the captured image. A data reading unit for reading data of a photographed image obtained by photographing a subject image;
An area dividing unit that determines a range of a first partial area including a main subject and a range of a second partial area excluding the first partial area in the captured image;
A first arithmetic unit that analyzes a spatial frequency of the data of the first partial region;
A subject extraction unit that extracts the main subject from a range of the first partial region of the captured image based on a calculation result of the first calculation unit;
An image processing apparatus comprising: A data reading unit that reads data of a first captured image captured by irradiating the subject with light and data of a second captured image captured without irradiating the subject with light;
An area dividing unit for determining a range of a first partial area including a main subject and a range of a second partial area excluding the first partial area in the first captured image and the second captured image;
A second computing unit for obtaining at least one of a brightness difference and a brightness ratio between the first partial area of the first captured image and the first partial area of the second captured image;
A subject extraction unit that extracts the main subject from a range of the first partial region of at least one of the first photographed image and the second photographed image based on a computation result of the second computation unit;
An image processing apparatus comprising: A data reading unit that reads data of a photographed image obtained by photographing a subject image and data indicating a range of a first partial area including a main subject in the photographed image;
A first arithmetic unit that analyzes a spatial frequency of data of the first partial region in the captured image;
A subject extraction unit that extracts the main subject from a range of the first partial region of the captured image based on a calculation result of the first calculation unit;
An image processing apparatus comprising: Data of a first photographed image taken by irradiating a subject with light, data of a second photographed image taken without irradiating the subject with light, the first photographed image and the second photographed image A data reading unit for reading data indicating the range of the first partial area including the main subject in the image;
A second computing unit for obtaining at least one of a brightness difference and a brightness ratio between the first partial area of the first captured image and the first partial area of the second captured image;
A subject extraction unit that extracts the main subject from a range of the first partial region of at least one of the first photographed image and the second photographed image based on a computation result of the second computation unit;
An image processing apparatus comprising: A program for a control unit of an image processing apparatus,
Reading data of a photographed image obtained by photographing a subject image into the control unit;
Determining a range of a first partial area including a main subject and a range of a second partial area excluding the first partial area in the captured image;
Analyzing the spatial frequency of the data of the first partial region;
Extracting the main subject from a range of the first partial region of the captured image based on a calculation result of the first calculation unit;
Is executed by the control unit. A program for a control unit of an image processing apparatus,
Reading data of a first photographed image taken by irradiating the subject with light and data of a second photographed image taken without irradiating the subject with light to the control unit;
Determining a range of a first partial area including a main subject and a range of a second partial area excluding the first partial area in the first captured image and the second captured image;
Obtaining at least one of a brightness difference and a brightness ratio between the first partial area of the first captured image and the first partial area of the second captured image;
Extracting the main subject from a range of the first partial region of at least one of the first photographed image and the second photographed image based on a computation result of the second computation unit;
Is executed by the control unit. A program for a control unit of an image processing apparatus,
Reading into the control unit data of a photographed image obtained by photographing a subject image and data indicating a range of a first partial area including a main subject in the photographed image;
Analyzing the spatial frequency of the data of the first partial region in the captured image;
Extracting the main subject from a range of the first partial region of the captured image based on a calculation result of the first calculation unit;
Is executed by the control unit. A program for a control unit of an image processing apparatus,
Data of a first photographed image taken by irradiating a subject with light, data of a second photographed image taken without irradiating the subject with light, the first photographed image and the second photographed image Reading data indicating the range of the first partial area including the main subject in the image into the control unit;
Obtaining at least one of a brightness difference and a brightness ratio between the first partial area of the first captured image and the first partial area of the second captured image;
Extracting the main subject from a range of the first partial region of at least one of the first photographed image and the second photographed image based on a computation result of the second computation unit;
Is executed by the control unit.

Images