JP5935581B2 - Imaging apparatus, image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an imaging device, an image processing device, an image processing method, and an image processing program.

  2. Description of the Related Art Conventionally, a camera that extracts a subject from a photographed image and automatically sets photographing conditions (such as a photographing distance and exposure) suitable for the subject is known (see, for example, Patent Document 1). In recent years, a technique has been proposed in which this setting is repeated according to a moving image acquired by the camera in real time.

JP 2002-512255 A

  However, when the user is at a loss in the composition of the camera, the subject extraction result becomes unstable, which may adversely affect shooting.

  An image pickup apparatus according to the present invention includes an extraction unit that extracts a plurality of subject candidate regions based on a plurality of types of extraction criteria from each of a plurality of frames that are sequentially generated by an image sensor, and a plurality of subject candidate regions that are extracted by the extraction unit. Creating an evaluation history by storing at least a predetermined frame period, an evaluation means for evaluating the likelihood of the subject for each of the above, and a type of extraction criterion for the subject candidate area highly evaluated by the evaluation means And means for selecting a subject area of the current frame based on the evaluation result regarding the current frame and the evaluation history regarding the past frame.

  An image processing apparatus according to the present invention includes: an extraction unit that extracts a plurality of subject candidate regions from each of a plurality of frames that are sequentially generated by an image sensor, and a plurality of subjects that are extracted by the extraction unit; An evaluation history is created by storing at least a predetermined frame period the evaluation means for evaluating the likelihood of subject for each of the candidate areas, and the types of extraction criteria for the subject candidate areas highly evaluated by the evaluation means. And creating means for selecting the subject area of the current frame based on the result of the evaluation relating to the current frame and the evaluation history relating to the past frame.

  An image processing method according to the present invention includes an extraction procedure for extracting a plurality of subject candidate areas from each of a plurality of frames that are sequentially generated by an image sensor according to a plurality of types of extraction criteria, and a plurality of subjects extracted by the extraction procedure. An evaluation history is created by storing at least a predetermined frame period an evaluation procedure for evaluating the likelihood of a subject for each candidate region, and the types of extraction criteria for the subject candidate region highly evaluated by the evaluation procedure. And a selection procedure for selecting a subject area of the current frame based on the evaluation result regarding the current frame and the evaluation history regarding the past frame.

  An image processing program according to the present invention includes an extraction procedure for extracting a plurality of subject candidate regions based on a plurality of types of extraction criteria from each of a plurality of frames sequentially generated by an image sensor, and a plurality of subjects extracted by the extraction procedure. An evaluation history is created by storing at least a predetermined frame period an evaluation procedure for evaluating the likelihood of a subject for each candidate region, and the types of extraction criteria for the subject candidate region highly evaluated by the evaluation procedure. And a selection procedure for selecting a subject area of the current frame based on the creation procedure to be performed, the result of the evaluation for the current frame, and the evaluation history for the past frame.

  According to the present invention, a subject selection result can be stabilized.

It is a figure which shows the structural example of an electronic camera. 4 is an operation flowchart of the image processing engine 14 in an imaging mode. It is an example of a through image. It is a figure which shows the example of the extracted subject candidate area | region for every extraction division. It is a figure explaining an evaluation value. It is an image of the evaluation history regarding frame numbers # 1 to # 24. It is a figure explaining step S16. It is a figure which shows the evaluation log | history regarding the frame numbers # 1- # 24, and the selection log | history. It is a figure which shows the mode of the frame display regarding frame number # 1- # 24.

[Embodiment]
Hereinafter, an imaging apparatus (electronic camera) will be described as an embodiment of the present invention.

  FIG. 1 is a diagram illustrating a configuration example of an electronic camera. As shown in FIG. 1, the electronic camera 11 includes a photographing optical system 12, an imaging unit 13, an image processing engine 14, a first memory 15, a second memory 16, a recording I / F 17, and a display unit 18. And an operation unit 19. Among these, the imaging unit 13, the first memory 15, the second memory 16, the recording I / F 17, the display unit 18, and the operation unit 19 are each connected to the image processing engine 14. The operation unit 19 is a switch that receives a user operation (for example, a shooting instruction).

  The photographing optical system 12 includes a plurality of lenses including a focusing lens for AF. In FIG. 1, the photographing optical system 12 is represented by a single lens for the sake of simplicity.

  The imaging unit 13 is a module that captures an image of the object scene formed by the imaging optical system 12. For example, the imaging unit 13 includes an imaging element that performs photoelectric conversion, and a signal processing circuit that performs analog signal processing, A / D conversion processing, and the like on the output of the imaging element. Further, for example, RGB color filters are arranged on the imaging surface of the imaging device in accordance with a known Bayer array. Thereby, the image sensor can acquire a color image.

  In addition, when the electronic camera 11 is in the imaging mode, the imaging unit 13 repeatedly acquires images for observation (through images). In addition, the imaging unit 13 acquires a recording image (detailed image) in accordance with a user's shooting instruction. In addition, when the electronic camera 11 is in the imaging mode, the image data acquired by the imaging unit 13 is input to the image processing engine 14.

  The image processing engine 14 is a processor that comprehensively controls the operation of the electronic camera 11. For example, the image processing engine 14 performs AF control and AE control using a through image in the imaging mode. The image processing engine 14 performs image processing such as color interpolation, white balance correction, gradation conversion, and color space conversion on the image data acquired by the imaging unit 13. Furthermore, the image processing engine 14 functions as a candidate extraction unit 21, an evaluation unit 22, a history creation unit 23, and a subject selection unit 24 by executing a program. The operations of the candidate extraction unit 21, the evaluation unit 22, the history creation unit 23, and the subject selection unit 24 will be described later.

  The first memory 15 is a memory that temporarily stores image data and the like, and is configured by an SDRAM that is a volatile storage medium, for example. The second memory 16 is a memory that stores a program executed by the image processing engine 14 and various data used in the program, and is configured by a non-volatile memory such as a flash memory, for example.

  The recording I / F 17 has a connector for connecting the nonvolatile storage medium 20. The recording I / F 17 executes writing / reading of image data with respect to the storage medium 20 connected to the connector. The storage medium 20 is, for example, a hard disk or a memory card incorporating a semiconductor memory. In FIG. 1, a memory card is illustrated as an example of the storage medium 20.

  The display unit 18 is, for example, a liquid crystal monitor provided on the back surface of the electronic camera 11. The display unit 18 performs real-time display of a through image, reproduction display of a detailed image, confirmation display after photographing of a detailed image, and the like.

  FIG. 2 is an operation flowchart of the image processing engine 14 in the imaging mode. Hereafter, each step of FIG. 2 is demonstrated in order. This imaging mode has a “subject tracking function” that is effective for photographing a static subject.

  Step S11: The image processing engine 14 drives the imaging unit 13 to acquire a through image, and sends the through image to the display unit 18. Note that this through image acquisition and display is repeated, for example, at a frame period of 1/30 seconds, and as a result, the state of the camera's object scene is displayed in real time on the display unit 18 (viewfinder display).

  Here, it is assumed that through images as shown in FIG. 3 are acquired sequentially. In FIG. 3, a number (frame number) indicating the acquisition order is assigned to the through image of each frame. Incidentally, each through image shown in FIG. 3 is acquired under the following conditions.

  Frame numbers # 1 to # 10: A period during which the dolphin play equipment is captured as the main part of the camera composition.

  Frame numbers # 11 to # 13: A period during which the main part of the camera composition transitions from a dolphin playground equipment to a duck playground equipment.

  Frame numbers # 14 to # 23: A period in which a duck playground equipment is captured as the main part of the camera composition.

  Frame number # 24: A period during which the main part of the camera composition transitions from a duck playground equipment to a dolphin playground equipment.

  In the present embodiment, it is assumed that the following steps S12 to S19 are executed every time a through image of each frame is acquired (that is, within one frame period).

  Step S12: The candidate extraction unit 21 of the image processing engine 14 extracts a subject candidate area from the through image by a plurality of types of extraction processes with different extraction criteria. Although details of the extraction process will be described later, it is assumed here that the following nine extraction classification extraction processes are employed.

  First extraction category: extraction processing based on a threshold value Y1 in the luminance direction (Y direction).

  Second extraction category: extraction processing based on a threshold B1 in the first color difference direction (Cb direction).

  Third extraction category: extraction processing based on the threshold R1 in the second color difference direction (Cr direction).

  Fourth extraction category: extraction processing based on a Y-direction threshold Y2 (Y2 <Y1).

  Fifth extraction category: extraction processing based on the threshold value B2 (B2 <B1) in the Cb direction.

  Sixth extraction category: extraction processing based on Cr-direction threshold R2 (R2 <R1).

  Seventh extraction category: extraction processing based on a Y-direction threshold Y3 (Y3 <Y2).

  Eighth extraction category: extraction processing based on the threshold value B3 (B2 <B2) in the Cb direction.

  Ninth extraction category: extraction processing based on a Cr direction threshold R3 (R3 <R2).

  FIG. 4 is a diagram showing the extracted subject candidate areas for each extraction section. In FIG. 4, a group of white pixels (pixel value = 1) indicates individual subject candidate regions, and an existence region of black pixels (pixel value = 0) indicates a region other than the subject candidate regions. However, the candidate extraction unit 21 in this step excludes those extracted subject candidate areas that do not satisfy certain criteria from the subject candidate areas. What does not satisfy a certain standard is, for example, a case where the area is less than a certain amount compared to the total area of the subject candidate regions in the frame, or a portion covering the edge of the frame.

  Step S13: The evaluation unit 22 of the image processing engine 14 calculates an evaluation value of the subjectness for each of the subject candidate areas extracted in step S12. The evaluation value of each subject candidate area can be calculated by, for example, the following equation.

Evaluation value = ((number of pixels in region) ^ γ) / (moment of inertia),
Inertia moment = (the square of the distance from the moment center to the pixel) in the region “γ” in the equation is a predetermined value of 1 or more, and “moment center” is the subject region selected from the previous frame ( The coordinates are set in accordance with the position (to be described later). Specifically, it is set to either the center of gravity of the subject area selected from the previous frame or the center of the envelope frame of the subject area selected from the previous frame. Instead of the center of gravity of the subject area (considering the distribution of pixel values in the area), the center of the subject area (not considering the distribution of pixel values in the area) may be used. Further, when the current frame is an initial frame (a frame at the start of the imaging mode) or when no subject area is selected from the previous frame, the center of the frame is set as the moment center.

  Therefore, the subject candidate region having a larger area, higher unity, and closer to the subject region (or the center of the frame) of the previous frame has a higher subject-like evaluation value. For example, when the two subject candidate regions 5A and 5B shown in FIG. 5 are compared, the difference in area and unity degree is small between the two subject candidate regions 5A and 5B, but from the moment center (x mark in FIG. 5). Since the subject candidate area 5B is surely shorter, the subject candidate area 5B has a smaller moment of inertia, and the subject candidate area 5B has a higher subject evaluation value.

  Subsequently, the evaluation unit 22 ranks the subject candidate areas according to their evaluation values. Specifically, a ranking is assigned to each subject candidate region in descending order of evaluation value.

  Here, the evaluator 22 excludes those whose rank is finally lower than the fourth place from the subject candidate area, and leaves only those whose rank is higher than the third place as the subject candidate area.

  The evaluation unit 22 in this step determines priorities for the nine extraction categories described above, and if the evaluation values are equal among a plurality of subject candidate regions, the subject candidates It is desirable to rank the subject candidate areas according to the area extraction category and the predetermined priority order.

  Step S14: The history creation unit 23 of the image processing engine 14 writes the numbers (section numbers) indicating the respective extraction categories of the subject candidate areas that have been ranked first and second in step S13 in the evaluation history.

  However, when the current frame is an initial frame, the history creation unit 23 clears all the division numbers (history data) in the evaluation history before writing the division number. In addition, when the second-ranked subject candidate area does not exist, the history creation unit 23 writes a code indicating that the classification number corresponding to the second rank does not exist in the evaluation history, and the first-ranked subject candidate area Is not written, a code indicating that there is no division number corresponding to the first place is written in the evaluation history.

  FIG. 6 is an image of the evaluation history regarding the frame numbers # 1 to # 24. Although FIG. 6 shows the history data for 24 frames, the history data used for selecting the subject area (step S16) is only the history data for the latest 5 frames including the current frame (details). Here, the data size of the evaluation history is set to 5 frames, and when the history data (section number) of the current frame is written to the evaluation history, the frame including the current frame is older than the latest 5 frames. It is assumed that the history data is erased (overwritten) from the evaluation history. Thus, if unnecessary history data is deleted when updating the evaluation history, the data size of the evaluation history can be minimized.

  Step S15: The image processing engine 14 refers to the updated evaluation history, and determines whether or not the latest five frames of history data including the current frame have been accumulated in the evaluation history. If the history data for the latest five frames has been accumulated, the process proceeds to step S16 to select a subject area. If not, the selection of the subject area is omitted and the process proceeds to step S11. Return.

  Step S16: The subject selection unit 24 of the image processing engine 14 is based on the evaluation history (= the evaluation history for the latest five frames including the current frame), for example, the subject of the current frame by the following procedures (a) to (c). A subject area is selected from the candidate areas. Here, for the sake of simplicity, it is assumed that the number of selected subject areas is 1.

  (A) The subject selection unit 24 counts the number of times that the classification number of the first candidate candidate region in the current frame has appeared in the evaluation history (= the evaluation history for the latest five frames including the current frame), and the number of times Is 3 or more, the subject candidate area is selected as the subject area of the current claim. However, if the number of times is 2 or less, the procedure proceeds to step (b) in order to examine the second candidate subject region.

  (B) The subject selection unit 24 counts the number of times that the classification number of the second highest candidate candidate area in the current frame has appeared in the evaluation history (= the evaluation history for the latest five frames including the current frame), and the number of times Is 3 or more, the subject candidate area is selected as the subject area of the current claim. However, if the number is 2 or less, the procedure proceeds to step (c) in order to examine the third candidate subject region.

  (C) The subject selection unit 24 counts the number of times that the classification number of the third candidate subject area in the current frame has appeared in the evaluation history (= the evaluation history for the latest five frames including the current frame), and the number of times Is 3 or more, the subject candidate area is selected as the subject area of the current claim. However, if the number of appearances is 2 or less, the process proceeds to the next step S17 without selecting the subject area.

  For example, as shown in FIG. 7, the first division number in the current frame (#i) is “4”, the second division number in the current frame (#i) is “9”, and the current frame (#i) The third section number in #i) is “3”, the number of appearances of the section number “4” in the evaluation history for the latest 5 frames is 1, and the section number “9” in the evaluation history for the latest 5 frames is # 1. ”Is 5 and the number of appearances of the classification number“ 3 ”in the evaluation history for the latest 5 frames is zero.

  In this case, since the first division number “4” in the current frame does not satisfy the criterion (3 or more) of the number of appearances, the subject candidate region with the division number 4 is not selected as the subject region. The second candidate subject area of category number 9 is selected as the subject area.

  That is, the subject selection unit 24, even if it is a higher-order subject candidate region in the current frame, if the subject candidate region category number has not appeared in the evaluation history more than a certain number of times, the subject candidate portion 24 Do not select the area as the subject area.

  Therefore, the timing at which a certain subject candidate area is selected as the subject area is not the timing at which the subject candidate area is higher, but the timing at which the category number of the subject candidate area appears in the candidate history more than a certain number of times.

  In this way, if a certain delay is provided before a certain subject candidate area is selected as the subject area, a period when the camera composition is unstable or even one subject candidate area that satisfies a certain standard appears. It is possible to prevent the selection result of the subject area from becoming unstable (frequent switching) during a period when there is not.

  FIG. 8 shows the selection history for the frame numbers # 1 to # 24 together with the evaluation history for the frame numbers # 1 to # 24 (the bottom row in FIG. 8). As apparent from FIGS. 3 and 8, during a period when the composition of the camera is unstable, or a period during which no subject candidate area that satisfies a certain criterion is expressed (# 11 to # 13, # 24), Of course, no subject area is selected. Further, even during a period (# 1 to # 10, # 14 to # 23) in which the composition of the camera is stable and a subject candidate area that satisfies a certain standard is expressed (until that period exceeds a certain period ( (Up to # 4 and # 15), no subject area is selected.

  Step S17: The image processing engine 14 determines whether or not a subject area has been selected in step S16. If selected, the process proceeds to step S18 to perform frame display control, AF control, and AE control. If not, the control is omitted and the process returns to step S11.

  Step S18: The image processing engine 14 superimposes (overlays) the image of the envelope frame (frame image) of the selected subject area on the display unit 18.

  Further, the image processing engine 14 performs AF control on the selected subject area or its envelope frame. Specifically, the position of the focusing lens is adjusted so that the object in the selected subject area or the envelope frame is in focus.

  In addition, the image processing engine 14 performs AE control on the selected subject area or its envelope frame. Specifically, the exposure of the image sensor (the shutter speed of the image sensor and the aperture value of the imaging optical system) is set to an exposure (appropriate exposure) suitable for the brightness of the object in the selected subject area or envelope frame. ).

  Therefore, frame display control, AF control, and AE control are suspended during a period when the composition of the camera is unstable, or during a period when no subject candidate area that satisfies a certain criterion is expressed. Also, even if the camera composition is stable and a subject candidate area that satisfies a certain standard is expressed, each of the frame display control, AF control, and AE control is performed until the period exceeds the certain period. You will be suspended.

  FIG. 9 shows the state of frame display for frame numbers # 1 to # 24. As apparent from FIGS. 8 and 9, during a period when the composition of the camera is unstable, or a period during which no subject candidate area satisfying a certain standard is expressed (# 11 to # 13, # 24), Since no subject area is selected, no frame is displayed. Further, even during a period (# 1 to # 10, # 14 to # 23) in which the composition of the camera is stable and a subject candidate area that satisfies a certain standard is expressed (until that period exceeds a certain period ( (Up to # 4, up to # 15), since no subject area is selected, no frame is displayed.

  Step S19: The image processing engine 14 determines whether or not a shooting instruction is input from the user via the operation unit 19. If it is input, the process proceeds to step S20. If not input, the process proceeds to step S11. To do.

  Step S20: When the image processing engine 14 drives the imaging unit 13 to acquire a detailed image for recording, the image processing engine 14 records the detailed image on the storage medium 20 in an appropriate format, and ends the flow.

[Effects of First Embodiment]
As described above, the imaging device (11) of the present embodiment has extracted the extraction means (21) that extracts a plurality of subject candidate regions from each of a plurality of frames that are sequentially generated by the imaging device, using a plurality of types of extraction criteria. The evaluation means (22) for evaluating the subjectivity for each of the plurality of subject candidate areas, and the type (category number) of extraction criteria for the subject candidate areas that have obtained a high evaluation are stored over at least a predetermined frame period. Thus, a creation means (23) for creating an evaluation history, and a selection means (24) for selecting a subject area of the current frame based on an evaluation result relating to the current frame and an evaluation history relating to the past frame are provided.

  Therefore, the imaging device (11) of the present embodiment can stabilize the selection result of the subject area as compared with the case where the evaluation history is not used.

  Note that the selection means (24) determines the subject region only when the extraction criterion (segment number) of the subject candidate region that has been highly evaluated in the current frame has appeared in the evaluation history for a certain number of times. Select.

  Therefore, in the imaging device (11) of the present embodiment, a subject candidate region is selected as a subject region, not at the timing when the subject candidate region is higher, but as a candidate for extraction criteria for the subject candidate region. This is the timing at which the number of appearances in the history exceeds a certain level.

  The imaging device (11) of the present embodiment further includes display control means for displaying a mark indicating the range of the subject area on the monitor. The imaging apparatus (11) of the present embodiment further includes an exposure control unit that performs exposure control on the subject area. In addition, the imaging apparatus (11) of the present embodiment further includes a focus adjustment unit that performs focus adjustment on the subject area.

  Therefore, the imaging device (11) of this embodiment can stabilize each of the mark display destination, the exposure of the imaging device, and the focal point of the imaging device.

  Further, the imaging device (11) of the present embodiment pauses the selection means (24) during a period from when the creation of the evaluation history is started until a predetermined frame period elapses.

  Therefore, the imaging apparatus (11) of the present embodiment can omit useless processing during a period in which the subject region selection result is likely to be unstable.

  The extraction means (21) generates a plurality of binarized images by binarizing the frame with a plurality of threshold values, and extracts subject candidate regions from each of the plurality of binarized images. Therefore, the extraction of the subject candidate area based on a plurality of types of extraction criteria is performed reliably.

  The evaluation means (22) evaluates the subject likeness of the subject candidate region based on the moment of inertia of the subject candidate region and the area of the subject candidate region. Therefore, the evaluation of the subjectness is accurately performed.

  The evaluation means (22) sets the center of the moment of inertia in accordance with the position of the subject area selected in the previous frame in the evaluation for the current frame. Accordingly, the evaluation of the subjectness is performed more accurately.

[Description of subject extraction processing]
Hereinafter, the subject extraction process (step S12) will be described in detail. This extraction process includes, for example, the following procedures (a) to (f). Among these, the order of procedures (b), (c), and (d) can be changed. It is also possible to execute at least two of the procedures (b), (c), and (d) in parallel.

  (A) The candidate extraction unit 21 performs color space conversion on the through image, and converts the color space of the through image from the RGB space to the YCbCr space.

  (B) The candidate extraction unit 21 applies a plurality of binarization processes (here, three patterns) using different thresholds to the Y component (Y image) of the through image, and binarized image. M (Y1), M (Y2), and M (Y3) are acquired (see the left column in FIG. 4). Among them, the binarized image M (Y1) has white pixels (pixel value = 1) in a region larger than the first threshold Y1 in the Y image, and black pixels (pixel value = 0) in other regions. In the binarized image M (Y2), white pixels are arranged in an area larger than the second threshold Y2 (Y2 <Y1) and black pixels are arranged in other areas in the Y image. The binarized image M (Y3) is a Y image in which white pixels are arranged in an area smaller than the third threshold Y3 (Y3 <Y2) and black pixels are arranged in other areas. . The threshold values Y1, Y2, and Y3 are set as follows, for example.

Y1 = Ave + ασ,
Y2 = Ave + σ,
Y3 = Ave−βσ,
Where Ave is the average pixel value of the Y image, σ is the standard deviation of the pixel value of the Y image, and α and β are one or more predetermined values.

  (C) The candidate extraction unit 21 performs a plurality of binarization processes (here, three patterns) using different thresholds on the Cb component (Cb image) of the through image, and binarized image M (B1), M (B2), and M (B3) are acquired (see the center column in FIG. 4). Among these, in the binarized image M (B1), white pixels (pixel value 1) are arranged in an area larger than the first threshold B1 in the Cb image, and black pixels (pixel value 0) are arranged in other areas. The binarized image M (B2) is a Cb image in which white pixels are arranged in an area larger than the second threshold B2 (B2 <B1) and black pixels are arranged in other areas. Yes, the binarized image M (B3) is a Cb image in which white pixels are arranged in an area smaller than the third threshold B3 (B3 <B2) and black pixels are arranged in other areas. The threshold values B1, B2, and B3 are set as follows, for example.

B1 = Ave + ασ,
B2 = Ave + σ,
B3 = Ave−βσ,
Where Ave is the average pixel value of the Cb image, σ is the standard deviation of the pixel value of the Cb image, and α and β are one or more predetermined values.

  (D) The candidate extraction unit 21 performs a plurality of binarization processes (three in this case) using different thresholds on the Cr component (Cr image) of the through image, and binarized image. M (R1), M (R2), and M (R3) are acquired (see the right column of FIG. 4). Among these, in the binarized image M (R1), white pixels (pixel value 1) are arranged in a region larger than the first threshold R1 in the Cr image, and black pixels (pixel value 0) are arranged in other regions. The binarized image M (R2) is obtained by arranging white pixels in a region larger than the second threshold value R2 (R2 <R1) and black pixels in other regions in the Cr image. Yes, the binarized image M (R3) is an image in which white pixels are arranged in an area smaller than the third threshold value R3 (R3 <R2) and black pixels are arranged in other areas in the Cr image. The threshold values R1, R2, and R3 are set as follows, for example.

R1 = Ave + ασ,
R2 = Ave + σ,
R3 = Ave−βσ,
Where Ave is the average pixel value of the Cr image, σ is the standard deviation of the pixel value of the Cr image, and α and β are one or more predetermined values.

  (E) The candidate extraction unit 21 performs binarized images M (Y1), M (Y2), M (Y3), M (B1), M (B2), M (B3), M (R1), M ( A labeling process is performed on each of R2) and M (R3), and a group of white pixels in each of the binarized images is extracted as a subject candidate area.

  The extraction segment using the binarized image M (Y1) is the first extraction segment described above, and the extraction segment using the binarized image M (B1) is the second extraction segment described above. The extraction segment using the binarized image M (R1) is the third extraction segment described above, and the extraction segment using the binarized image M (Y2) is the fourth extraction segment described above. The extraction segment using the binarized image M (B2) is the above-described fifth extraction segment, the extraction segment using the binarized image M (R2) is the above-described sixth extraction segment, and the binarized image The extraction segment using M (Y3) is the seventh extraction segment described above, and the extraction segment using the binarized image M (B3) is the eighth extraction segment described above, and the binarized image M ( The extraction section using R3) is the ninth extraction section described above.

  (F) The candidate extraction unit 21 has an area that is smaller than the total area of the subject candidate regions in the binarized image, for example, those that do not satisfy a certain standard from the subject candidate regions extracted in step (e). Those that are less than a certain value and those that are on the edge of the binarized image are removed from the subject candidate area. It should be noted that the reference in this procedure (f) is preferably set for each category.

[Supplement of embodiment]
In step S14 described above, the category number to be written in the evaluation history is only the category number of the second or higher rank, but may be only the category number of the Nth or higher rank (where N is a natural number).

  Further, in the present embodiment, an evaluation history in which the evaluation result (category number) of the subjectness is accumulated is created, and the subject area of the current frame is selected based on the evaluation history. However, in the present embodiment, a selection history that accumulates the selection results (category numbers) of the subject areas may be created together with the evaluation history, and the subject area of the current frame may be selected based on both the evaluation history and the selection history. .

  In the present embodiment, the number of subject areas selected from one frame is singular, but may be plural. In this case, the frame display is performed for each of the selected plurality of subject areas, and the AF control and AE control may be performed on the highest subject area. When a plurality of frame images are displayed simultaneously in one frame, the frame images may be color-coded according to the order.

  Further, in this embodiment, the subject candidate region extraction categories are set to the following nine types.

  First extraction category: extraction processing based on a threshold value Y1 in the luminance direction (Y direction).

  Second extraction category: extraction processing based on a threshold B1 in the first color difference direction (Cb direction).

  Third extraction category: extraction processing based on the threshold R1 in the second color difference direction (Cr direction).

  Fourth extraction category: extraction processing based on a Y-direction threshold Y2 (Y2 <Y1).

  Fifth extraction category: extraction processing based on the threshold value B2 (B2 <B1) in the Cb direction.

  Sixth extraction category: extraction processing based on Cr-direction threshold R2 (R2 <R1).

  Seventh extraction category: extraction processing based on a Y-direction threshold Y3 (Y3 <Y2).

  Eighth extraction category: extraction processing based on the threshold value B3 (B2 <B2) in the Cb direction.

  Ninth extraction category: extraction processing based on a Cr direction threshold R3 (R3 <R2).

  However, the combinations of extraction categories in the present embodiment are not limited to the above nine combinations. For example, an extraction process that uses a predetermined surface in the YCbCr space as a threshold value, an extraction process that uses a threshold value other than color, and the like may be adopted as one of the extraction categories of this embodiment.

  In the present embodiment, the number of subject candidate region extraction categories is nine, but it may be eight or less, or ten or more.

  Further, in step S16 of the present embodiment, the condition for selecting a certain subject candidate region as the subject region is set as “the category number of the subject candidate region appears three times or more in the evaluation history for the latest five frames. However, it is not limited to this. This condition only needs to be “the section number of the subject candidate area appears K or more times in the evaluation history for the latest M frames” (provided that M> K, and both K and M are natural numbers). .)

  In step S16 of the present embodiment, the appearance frequency threshold value K or the appearance frequency threshold value K / M for selecting a certain subject candidate region as the subject region may be variable. Incidentally, as the threshold value is larger, the control (frame display control, AF control, AE control) in step S18 can be stabilized. As the threshold value is smaller, the control responsiveness in step S18 can be improved. it can.

  In this embodiment, the subject area selection result is used for frame display control, AF control, and AE control. However, it may be used for other processing such as white balance control.

  In this embodiment, the frame image is used as a mark indicating the position of the selected subject area. However, other marks (for example, an outline image indicating the outline of the subject area) may be used.

  Further, the camera of the present embodiment can be applied to both a lens interchangeable camera and a compact camera. However, the camera of the present embodiment is considered suitable for a compact camera that often shoots a static subject rather than a dynamic subject.

  In the present embodiment, all of the subject candidate region extraction processing, subject region selection processing, and each processing (frame display control, AF control, AE control) based on the selection result are realized by software. Some or all of these processes may be realized in hardware by, for example, an ASIC.

DESCRIPTION OF SYMBOLS 11 ... Electronic camera, 12 ... Imaging | photography optical system, 13 ... Imaging part, 14 ... Image processing engine, 15 ... 1st memory, 16 ... 2nd memory, 17 ... Recording I / F, 18 ... Display part, 19 ... Operation part , 20 ... Storage medium, 21 ... Candidate extraction, 22 ... Evaluation part, 23 ... History creation part, 24 ... Subject selection part, 31 ... Computer, 32 ... Data reading part, 33 ... Storage device, 34 ... CPU, 35 ... Memory 36 ... Input / output I / F, 37 ... Bus, 38 ... Input device, 39 ... Monitor

Claims (12)

An extraction means for extracting a plurality of subject candidate regions from each of a plurality of frames that are sequentially generated by the imaging device according to a plurality of types of extraction criteria;
Evaluation means for evaluating the subjectivity for each of the plurality of subject candidate areas extracted by the extraction means;
Creating means for creating an evaluation history by storing at least a predetermined frame period the type of extraction criterion of the subject candidate area that has been highly evaluated by the evaluation means;
Selection means for selecting a subject area of the current frame based on the result of the evaluation relating to the current frame and the evaluation history relating to a past frame;
An imaging apparatus comprising: The imaging device according to claim 1,
The selection means is:
The imaging device, wherein the selection of the subject area is performed when the extraction criterion of the subject candidate area that has been highly evaluated in the current frame has appeared in the evaluation history for a certain number of times or more. . In the imaging device according to claim 1 or 2,
An imaging apparatus, further comprising display control means for displaying a mark indicating the position of the subject area on a monitor. In the imaging device according to any one of claims 1 to 3,
An imaging apparatus, further comprising an exposure control unit that performs exposure control on the subject area. In the imaging device according to any one of claims 1 to 4,
An imaging apparatus, further comprising: a focus adjustment unit that performs focus adjustment on the subject area. In the image processing apparatus according to any one of claims 1 to 5,
The imaging apparatus characterized in that the operation of the selection means is suspended during a period from when the creation of the evaluation history is started until a predetermined frame period elapses. In the imaging device according to any one of claims 1 to 6,
The extraction means includes
An imaging apparatus comprising: binarizing a frame with a plurality of threshold values to generate a plurality of binarized images and extracting a subject candidate region from each of the plurality of binarized images. The imaging apparatus according to claim 7,
The evaluation means includes
An imaging apparatus, wherein subject-likeness evaluation of the subject candidate region is performed based on a moment of inertia of the subject candidate region and an area of the subject candidate region. The imaging device according to claim 8,
The evaluation means includes
In the evaluation relating to the current frame, the center of the moment of inertia is set according to the position of the subject area selected in the previous frame. An extraction means for extracting a plurality of subject candidate regions from each of a plurality of frames that are sequentially generated by the imaging device according to a plurality of types of extraction criteria;
Evaluation means for evaluating the subjectivity for each of the plurality of subject candidate areas extracted by the extraction means;
Creating means for creating an evaluation history by storing at least a predetermined frame period the type of extraction criterion of the subject candidate area that has been highly evaluated by the evaluation means;
Selection means for selecting a subject area of the current frame based on the result of the evaluation relating to the current frame and the evaluation history relating to a past frame;
An image processing apparatus comprising: An extraction procedure for extracting a plurality of subject candidate areas from each of a plurality of frames that are sequentially generated by the image sensor, based on a plurality of types of extraction criteria;
An evaluation procedure for performing subjectivity evaluation for each of a plurality of subject candidate areas extracted in the extraction procedure;
A creation procedure for creating an evaluation history by storing at least a predetermined frame period, the type of extraction criterion of the subject candidate area that has been highly evaluated by the evaluation procedure;
A selection procedure for selecting a subject area of the current frame based on the result of the evaluation for the current frame and the evaluation history for the past frame;
An image processing method comprising: An extraction procedure for extracting a plurality of subject candidate areas from each of a plurality of frames that are sequentially generated by the image sensor, based on a plurality of types of extraction criteria;
An evaluation procedure for performing subjectivity evaluation for each of a plurality of subject candidate areas extracted in the extraction procedure;
A creation procedure for creating an evaluation history by storing at least a predetermined frame period, the type of extraction criterion of the subject candidate area that has been highly evaluated by the evaluation procedure;
A selection procedure for selecting a subject area of the current frame based on the result of the evaluation for the current frame and the evaluation history for the past frame;
An image processing program for causing a computer to execute.

Images