JP5697429B2 - Imaging apparatus, control method therefor, and program - Google Patents

Description

  The present invention relates to an imaging apparatus, a control method thereof, and a program, and more particularly, to an imaging apparatus capable of capturing moving images and multiple exposure imaging, a control method thereof, and a program.

  In an imaging device or a printing device, a system is disclosed in which pixels having similar hue and saturation information are grouped into the same region, and whether the region is a subject or background according to the position within the angle of view of the region is disclosed. (For example, refer to Patent Document 1).

  By using this technique, it is possible to specify an area occupied by a subject other than the background in the image. When a plurality of subjects are detected, an evaluation value is obtained for each subject, and the subject having the highest evaluation value can be considered as the main subject. For example, a system is disclosed in which an evaluation value is given according to the distance from the image center of the position of the subject or the size of the subject (see, for example, Patent Document 2).

  Furthermore, a system for performing tracking processing on a subject detected as a main subject in a video signal constituting a moving image is disclosed (for example, see Patent Document 3).

JP 2010-233236 A JP 2008-005438 A JP 2010-154286 A

  However, in the conventional technique for determining whether or not the area is a subject according to the position within the angle of view of the classified area, if the zoom is performed during imaging, the angle of view of the captured image changes. For this reason, the determination result of the subject or the background is affected. In other words, the main subject may be erroneously detected when the angle of view is not stable during zooming, and if this function is applied to an imaging device having a function of tracking the detected main subject, the erroneously detected main subject is detected. A case of tracking the subject will occur.

  As described above, in the conventional technique, when the angle of view changes, the main subject may be erroneously detected.

  An object of the present invention is to provide an imaging apparatus capable of accurately determining a main subject even when the angle of view changes, a control method therefor, and a program.

In order to achieve the above object, an imaging apparatus according to a first aspect captures an object and zooms to change a field angle, and an angle of view from an image showing the object captured by the imaging unit. A first determination unit that determines a first main subject candidate by calculating a first evaluation value indicating a main subject possibility that the main subject candidate is a main subject using information that depends on the image, and the imaging unit picked up the image Second determination means for determining a second main subject candidate by calculating a second evaluation value indicating the possibility of the main subject using information independent of the angle of view from an image showing the subject; and the first evaluation value the second using the evaluation value, and a third determining means for determining a main object candidate corresponding to one of the evaluation value as the main subject, the third determination means, when it is in the zoom, the as compared with the case not being zooming, the main Correcting the first evaluation value so as to lower the Utsushitai potential, and performs at least one of correcting the second evaluation value as the main subject possibility is high.

  According to the present invention, the main subject can be accurately determined even when the angle of view is changing.

1 is a diagram illustrating a schematic configuration of an imaging apparatus according to an embodiment of the present invention. 2 is a flowchart illustrating a procedure of a still image recording process executed by a system control unit in FIG. It is a flowchart which shows the procedure of the subject frame display process performed by step S201 of FIG. It is a flowchart which shows the procedure of the main subject frame determination process performed by step S301 of FIG. It is a flowchart which shows the procedure of the tracking frame determination process performed by step S307 of FIG. It is a flowchart which shows the procedure of the modification of the to-be-photographed object frame display process performed by step S201 of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment of the present invention.

  In FIG. 1, a lens 10 collects external light. The lens driving circuit 42 can adjust the focus and change the angle of view by moving the lens position back and forth along the optical axis. Further, based on the camera shake amount measured by the camera shake amount detection circuit 44, the lens 10 is driven by the camera shake correction circuit 40, and the optical axis is changed in a direction to cancel the camera shake, thereby optical camera shake. It is also possible to adopt a configuration for performing correction. The shake amount detection circuit 44 includes a gyro sensor. In FIG. 1, camera shake correction is realized by driving the lens 10, but camera shake can be similarly corrected by driving the CCD 16.

  The amount of light that has passed through the lens 10 can be adjusted by the diaphragm 14. The system control unit 60 can control the diaphragm 14 by transmitting the diaphragm control information to the diaphragm driving circuit 26. The diaphragm 14 includes an iris diaphragm composed of a plurality of wings and a round diaphragm in which holes are punched out in advance with various diameters. The system control unit 60 uses the diaphragm 14 and the diaphragm drive circuit 26 to control the diaphragm so as to reduce the amount of light when the subject brightness is high. When the subject brightness is low, the system controller 60 opens the diaphragm and captures a large amount of light. It is possible to control as follows.

  The system control unit 60 can control the mechanical shutter 12 by transmitting mechanical shutter control information to the mechanical shutter drive circuit 28. The exposure time during still image shooting is determined by the opening / closing time of the mechanical shutter 12, and this time is determined by the system control unit 60 and an instruction is given to the mechanical shutter drive circuit 28.

  Light that has passed through the lens 10, the mechanical shutter 12, and the aperture 14 is received by the CCD 16. The system control unit 60 can control the CCD 16 by transmitting an image sensor control signal to a TG (Timing Generator) 24. The TG 24 drives the CCD 16 based on the control information received from the system control unit 60. The CCD 16 periodically performs exposure of the element and reading of the exposed data, and this work is performed based on a drive signal from the TG 24. The TG 24 can control the exposure time of the CCD 16. This is made possible by outputting a drive signal from the TG 24 to the CCD 16 so as to release the charge charged by the element at an arbitrary timing.

  The image data read from the CCD 16 passes through a CDS (Correlated Double Sampler) circuit 18. The main role of CDS is to remove noise components of image data by a correlated double sampling method. Thereafter, the image data level of the image data can be attenuated / amplified by a PGA (Programmable Gain Amplifier) circuit 20. The system control unit 60 can control the amplification amount by transmitting the amplification level to the PGA circuit 20.

  Normally, the exposure of the CCD 16 is made appropriate by using the aperture 14 and the shutter. However, the PGA circuit 20 attenuates / amplifies the image data signal to change the exposure of the image data in a pseudo manner. Can bear. This is one of the exposure conditions at the time of shooting along with the aperture and shutter speed, and can provide a user with a function based on the concept of sensitivity.

  Further, the strobe 46 can be caused to emit light via the strobe controller 48. Shooting with flash photography when the subject is dark, such as in a backlit state or under low illuminance, makes it possible to photograph the subject brightly.

  The image data is converted from an analog signal to an image signal which is a digital signal by an A / D (Analog / Digital Converter) circuit 22. The bit width of the image signal is 10 bits, 12 bits, 14 bits, and the like, and the image processing circuit 50 in the subsequent stage can cope with a plurality of types of bit widths.

  In FIG. 1, the CDS circuit 18, the PGA circuit 20, and the A / D circuit 22 are expressed as separate blocks, but it is also possible to adopt a circuit in which these functions are mounted in one IC package. is there.

  The digitized image data from the A / D circuit 22 is input to the image processing circuit 50. The image processing circuit 50 includes a plurality of blocks and realizes various functions.

  The CCD 16 generally extracts a specific color component for each pixel through a color filter. The image signal from the A / D circuit 22 is in a data format corresponding to the pixel and color filter arrangement of the CCD 16, and is used for automatic exposure control (AE) that performs exposure control by evaluating only the luminance component. It is a format that is not suitable for use. The image processing circuit 50 has a function of excluding color information from an image signal and extracting only luminance information. Conversely, it also has a function of extracting color information, and can be used for white balance processing that identifies the light source of the subject and adjusts the color appropriately.

  Further, the image processing circuit 50 has a function of extracting only the frequency component of the signal read from the CCD 16, and this function is used in automatic focus control (AF). Further, the image processing circuit 50 has a function of extracting which frequency component of the data read from the CCD 16 is to be extracted, and dividing the area. The image processing circuit 50 also has a function of generating a histogram, and can acquire an image histogram by inputting image data.

  Further, the image processing circuit 50 has a function of adjusting the level of the image signal digitized by the A / D circuit 22 and operating the color effect of the image, and also plays a role of adjusting the image quality of the photographed image. . Regarding the level of the image signal, the image processing circuit 50 has a function of increasing / decreasing the level with a uniform amplification factor for the entire image, and a tone curve (gamma) function for converting the signal level according to the magnitude of the original signal level. Alternatively, the image processing circuit 50 has a function of increasing / decreasing the level with an amplification factor corresponding to the frequency component for each region in the screen. With these functions, various image signal level adjustments are possible.

  The digitized image data from the A / D circuit 22 is input to the image processing circuit 50, and at the same time, the image data can be stored in the temporary storage memory 30. The image data once stored in the temporary storage memory 30 can be read again, the image data can be referred to from the system control unit 60, and the read image data can be input to the image processing circuit 50. Further, the image data processed by the image processing circuit 50 can be written back to the temporary storage memory, or arbitrary data can be written from the system control unit 60.

  Image data appropriately processed by the image processing circuit 50 can be input to the image recognition circuit 38. The image recognition circuit 38 is capable of recognizing a person's face, its facial expression, and character information when there is a character, in addition to recognizing the brightness state, focus state, and color state of the input image. It has become. It is possible to input a plurality of images to the image recognition circuit 38. For example, it is possible to determine whether the images are the same by inputting two images and comparing the characteristics of the two images. It is possible.

  In addition to the method of recognizing an image by the image recognition circuit 38, the system control unit 60 can also perform image recognition processing. The system control unit 60 can execute a pre-coded program on the CPU, but reads the image data stored in the temporary storage memory 30 from the system control unit 60 and analyzes the image data to determine the scene status. Can be recognized.

  When outputting an image to the image display device 108 such as an LCD, the image data processed by the image processing circuit 50 is developed on the VRAM 34 and converted into analog data by the D / A 36 to display the image. Display on the device 108. In order to realize the electronic viewfinder, it is possible to sequentially update the continuous display of images read from the CCD 16 on the image display device 108. When image data is expanded on the VRAM 34, the image data is displayed on the VRAM 34 so that one image data is maximized on the image display device 108 or a plurality of images are displayed on a multi-screen. Can be deployed.

  The image display device 108 can display not only images but also arbitrary information alone or together with images. Camera status display, character information such as shutter speed, aperture value, sensitivity information, etc. selected by the user or determined by the camera, graphs such as luminance distribution measured by the image processing circuit 50, face recognition results, scene recognition Results and the like can also be displayed. The display position and display color of information can be arbitrarily selected. By displaying these various types of information, a user interface can be taken.

  The image display device 108 can also display image data stored in the image storage medium 82. If the image data is compressed, the data is decompressed by the compression / decompression unit 32 and the data is expanded in the VRAM 34. This data is converted into analog data by the D / A 36 and output.

  The image storage medium 82 is non-volatile and can mainly store captured image data. Regarding the storage of image data, it is possible to have a folder hierarchy or to name folders and file names in ascending order in the order of shooting. Shooting information such as aperture value, shutter speed, ISO sensitivity, shooting time, and the like can be added to each image data, and these can be stored together with the image data. Furthermore, the stored image data can be read out, and the image can be copied, moved, and erased.

  The operation unit 70 includes a power switch 102 that can switch the system power on / off, a shutter switch SW1 (104), SW2 (106), a mode change switch 110, and a parameter selection switch 112 for giving a shooting instruction. It is.

  The shutter switch SW1 (104) is turned ON during the operation of a shutter button (not shown), and instructs to start a shooting preparation operation such as automatic exposure control or focus control. The shutter switch SW2 (106) is turned on when the operation of a shutter button (not shown) is completed, and an operation for instructing still image shooting or image recognition can be realized.

  The mode switch 110 can switch camera operation modes such as a still image shooting mode, a moving image shooting mode, and a playback mode.

  The parameter selection switch 112 allows the user to select shooting conditions such as a distance measurement area and a photometry mode, page feed when shooting a captured image, and overall camera operation settings. Furthermore, the above-mentioned electronic viewfinder On / Off can be selected. Further, the image display device 108 can display an image and can be configured as an input device as a touch panel.

  In FIG. 1, a portion surrounded by the imaging system corresponds to an imaging unit that captures an image of a subject and can change the angle of view.

  FIG. 2 is a flowchart showing the procedure of the still image recording process executed by the system control unit 60 in FIG.

  In FIG. 2, the system control unit 60 executes a subject frame display process of FIG. 3 to be described later, and determines the subject region by analyzing the image data read from the CCD 16. Then, a subject frame indicating the main subject is displayed superimposed on the image indicating the captured subject (step S201).

  Next, it is determined whether or not the shutter switch SW1 (104) is ON (step S202). If the result of determination is OFF (NO in step S202), the process returns to S201. If ON (YES in step S202), the process proceeds to step S203, and photometric processing for obtaining subject luminance is performed (step S203). In this photometry process, subject brightness is obtained from the difference between the signal level of the image data read from the imaging unit and the appropriate level, exposure at the time of photometry, etc., and is stored in the internal memory (not shown) of the system control unit 60 as a photometric value. Remember. It is also possible to perform the focus metering on the position of the subject frame displayed in step S201.

  Next, the system control unit 60 controls the exposure to be suitable for the distance measurement process based on the photometric value stored in step S203 (step S204), and performs the distance measurement process (step S205). In step S205, distance measurement is performed on the position of the subject frame displayed in step S201, and the subject can be focused.

  Next, based on the photometric value stored in step S203, the exposure used while the shutter switch SW1 (104) is kept ON is controlled (step S206). Here, in order to shorten the shooting time lag after the shutter switch SW2 (106) is turned on, the aperture is controlled in advance to the aperture value for still image exposure.

  Next, it is determined whether or not the shutter switch SW2 (106) is ON (step S207). If it is OFF (NO in step S207), it is determined whether or not the shutter switch SW1 (104) is ON (step S208). If it is OFF (YES in step S208), the process returns to step S201. If it is ON (NO in step S208), the process returns to step S207.

  If the result of determination in step S207 is ON (YES in step S207), the still image exposure is set based on the photometric value stored in step S203 (step S209). In addition, in the case of flash photography, a light control operation is performed, and light emission settings for the main exposure are also made. Next, main exposure is performed (step S210). The data subjected to the main exposure is stored in the temporary storage memory 30. Then, the imaged data is developed (step S211), the developed data is recorded on the image storage medium 82 through the image recording medium I / F 80 (step S212), and the process returns to step S201 again.

  FIG. 3 is a flowchart showing the procedure of the subject frame display process executed in step S201 of FIG.

  In the following description, changing the angle of view is expressed as zoom, the telephoto side is expressed as the tele side, and the wide angle side is expressed as the wide side. In the flowchart, α is represented using an arrow as αT → W, but is omitted here.

  In FIG. 3, the system control unit 60 executes main subject detection frame determination processing to determine a main subject detection frame (first main subject candidate) as a subject frame candidate (step S301) (first determination means). ).

  FIG. 4 is a flowchart showing a procedure of main subject frame determination processing executed in step S301 of FIG.

  In FIG. 4, the system control unit 60 divides the obtained image data into M × N blocks, and acquires luminance / hue / saturation information for each M × N block (step S501). Here, the image processing circuit 50 and / or the system control unit 60 is used to convert the image data into M × N luminance / hue / saturation information (for example, M = 32, N = 24). Thereafter, this M × N luminance / hue / saturation information is used for the analysis.

  Next, based on the acquired luminance / hue / saturation information, the M × N block is divided into a plurality of regions (step S502). Specifically, first, the saturation information is used to recognize blocks with high and low saturation. Next, hue information is used for blocks with high saturation, and luminance information is used for blocks with low saturation, and blocks that fall within a certain range are grouped as the same region. By doing so, the M × N block is divided into regions having similar colors and / or regions having similar brightness.

  Next, it is determined whether or not each of the divided areas is the background, and the area determined to be the background is excluded from the main subject area candidates (step S503). Specifically, an area with many parts touching the screen edge or an area having a hue close to that area is determined as a background area, and other areas are determined as areas other than the background. Since there is a high possibility that the main subject area touches the screen lower direction even at the same screen edge, the screen lower direction may be excluded from the background removal target.

  The above-described evaluation value as the first evaluation value is calculated for each of the regions other than the background remaining in the candidate without being excluded from the main subject region candidates (step S504). This evaluation value is calculated from at least one information among the distance from the center of the screen, the area, and the flatness ratio, and the larger the value, the higher the possibility of being the main subject region. For example, the evaluation value is increased as the area is closer to the center of the screen, the area is larger, and the flatness is lower. In addition, the evaluation value of the area determined to be the same as the area having the largest evaluation value in the previous main subject frame determination process can be increased. For determination of the same area, luminance / hue / saturation information and position may be used.

  Next, the main subject confidence level indicating the main subject possibility as the main subject is calculated (step S505). The main subject confidence level is increased as the difference between the evaluation value of the region having the largest evaluation value and the evaluation value of the upper two regions having the largest evaluation value increases. Further, the main subject confidence level may be increased when the same region continuously becomes the region having the largest evaluation value.

  Then, the main subject detection frame as the first main subject candidate described above is determined (step S506), and this process ends. Here, if the main subject confidence level calculated in step S505 is equal to or greater than the threshold value, the region having the largest evaluation value is determined as the main subject detection frame. If the main subject confidence level has not reached the threshold, “no main subject detection frame” is set. Also, when there are a large number of main subject region candidates remaining as a result of background removal in step S503, the region with the largest evaluation value is not stable due to variations in the evaluation value, so that “no main subject detection frame” may be used. .

  Note that this main subject frame determination process is a procedure for excluding, from the main subject region candidate, a region with many parts touching the edge of the screen or a region having a hue close to that region. It is easy to be affected. For example, even a subject in an area determined as a main subject detection frame may be treated as a background when the number of portions in contact with the end of the angle of view increases due to subsequent zooming.

  Returning to FIG. 3, it is determined in step S302 whether tracking is in progress (step S302). This tracking process is a process of tracking the position of the subject region in the image data that is continuously captured based on the color of the designated subject region. Processing related to the start and end of the tracking process is performed in S306, S318, S610, and the like, but the system control unit 60 can determine the current tracking state by managing and referring to the flag stored in the internal memory.

  If the result of determination in step S302 is that tracking is not in progress (NO in step S302), it is determined whether or not there is a main subject detection frame that is a subject frame candidate obtained by the main subject detection frame determination processing in step S301 ( Step S303). When there is no main subject detection frame (NO in step S303), the subject frame on the image display device 108 is not displayed (step S304), and this process is terminated. In step S304, if the subject frame has been displayed so far, it is not displayed. If there is no subject frame displayed, the subject frame is not displayed as it is.

  If the result of determination in step S303 is that there is a main subject detection frame (YES in step S303), the main subject detection frame is displayed on the image display device 108 as the subject frame (step S305), and tracking processing is started (step S305). (S306) This process is terminated.

  If the result of determination in step S302 is that tracking is in progress (YES in step S302), tracking frame determination processing is executed to determine a tracking frame (second main subject candidate) that is a candidate for the subject frame (step S307) ( Second determining means).

  FIG. 5 is a flowchart showing the tracking frame determination process performed in step S307 of FIG.

  In FIG. 5, the system control unit 60 determines whether or not the initial characteristic color to be tracked has been stored (step S601). If the result of determination is that it has been stored (YES in step S106), the process proceeds to step S604, and if it has not been stored (NO in step S601), the color information of the target area is set as the initial characteristic color (predetermined color). Store (step S602). The initial feature color is for storing the color information of the subject to be tracked immediately after the start of tracking. If the color difference from this initial feature color becomes large during tracking, the judgment is made to stop tracking. Used for etc.

  Next, the initial characteristic color is set as the tracking target color (step S603). The tracking process operates so as to track the area of the color closest to the tracking target color. The tracking target color is updated every time the tracking frame determination process is performed, but step S603 is a process of initializing the tracking target color with the initial characteristic color immediately after the tracking is started.

  Next, a region in the image obtained by imaging the color region closest to the tracking target color is searched and determined as the tracking target region (step S604). It is sufficient to use the R / G / B signal of the image data to determine whether or not it is the closest color. For example, the difference between the R / G / B data and the R / G / B component of the tracking target color is within the threshold value. Find out if it fits. In addition, since the image data itself indicating the image to be searched may have a large data size, the corresponding area may be searched from data obtained by processing the image data.

  Next, the tracking target color is updated to the color of the tracking target area determined in step S604 and stored (step S605). By updating the tracking target color every time the tracking frame determination process is performed, it is possible to continue tracking even when the color of the subject gradually changes due to the influence of external light or the like.

  Next, a tracking frame as a final second main subject candidate is determined (step S606). Basically, an area having a color closest to the predetermined color is set as a tracking frame. However, if there is no region having a color similar to the tracking target color, or if the color of the region found in step S604 has changed to some extent from the initial feature color, the region determined in step S604 is determined. The final tracking frame is not adopted and “no tracking frame” is set.

  Next, the tracking confidence level as the second evaluation value is calculated (step S607). The tracking confidence level is calculated based on the similarity between a predetermined color and the color of the area in the tracking frame. Specifically, a large value is set when the difference between the tracking target color and the initial characteristic color is small. This is because when the difference between the tracking target color and the initial characteristic color is small, it can be determined that there is a high possibility of tracking the same subject as when tracking started. Further, the degree of tracking confidence may be increased when the size variation of the tracking frame is small, the movement distance of the tracking frame is small, or when the time elapsed from the start of tracking is short.

  Next, it is determined whether or not there is a frame determined as a tracking frame in step S606 (step S608), and when there is a tracking frame (YES in step S608), this process ends. If the result of determination in step S606 is that there is no tracking frame (NO in step S608), the tracking information is cleared (step S609), tracking is stopped (step S610), and this process ends.

  The tracking information is color information such as an initial characteristic color and a tracking target color. In addition, when the tracking frame size at the time of tracking start is stored, it is also cleared. When a flag for managing whether tracking is in progress is prepared regarding the cancellation of tracking, the flag is cleared.

  Since the tracking frame determination process searches for an area of the color closest to the tracking target color, even if a part of the subject to be tracked protrudes outside the screen, Tracking can be performed from the remaining portion. Therefore, compared to the main subject frame determination process, it is less susceptible to the influence of zoom that causes a change in the angle of view.

  Returning to FIG. 3, it is determined whether or not zooming is currently being performed (step S308). As a result of the determination, when zooming is in progress (YES in step S308), it is determined whether or not the zoom is in a direction (W → T) from the wide side (W) to the tele side (T) (step S309). When the zoom is W → T (YES in step S309), αWT is set to the coefficient α (step S310). When the zoom is not W → T, that is, in the direction T → W from the tele side (T) to the wide side (W) (NO in step S309), αTW is set to the coefficient α (step S311).

  Here, the coefficient α satisfies 0 ≦ αWT ≦ αTW <1. That is, the value when the zoom is in the direction from the wide side (W) to the tele side (T) is equal to or less than the value when the zoom is in the direction from the tele side (T) to the wide side (W). Therefore, α is evaluated to be larger when the field angle to be imaged is changed from the wide side to the tele side, and to reduce the possibility of the main subject more than when the angle of view is changed from the tele side to the wide side. The value is corrected. The coefficient α is a coefficient for reducing the main subject confidence level (first evaluation value) determined in the main subject detection frame determination process (step S301). That is, α is a coefficient for correcting the main subject confidence level so as to reduce the possibility of the main subject.

Next, the main subject confidence level multiplied by α is changed to the main subject confidence level (step S312) ( third determination unit). The main subject confidence level is an evaluation value that increases when it is determined during the main subject detection frame determination process that the main subject detection frame is influential as a subject frame.

  In this step S312, αWT is equal to or less than αTW, so that the main subject confidence level is further reduced when the zoom is in the direction from the wide side (W) to the telephoto side (T). This is because when the zoom is in the direction from the wide side (W) to the tele side (T), a subject that was within the angle of view before zooming may go out of the angle of view as a result of zooming. Further, the distance from the center also varies greatly, and the accuracy of the main subject detection frame determination process is likely to decrease.

  On the other hand, in the case of the direction from the tele side (T) to the wide side (W), it is highly possible that what was within the field angle before zooming is still within the field angle even after zooming. Further, since the variation in the distance from the center is also reduced, it is considered that the accuracy of the main subject detection frame determination process does not decrease so much. Considering these, the main subject confidence level is further reduced when the zoom is in the direction from the wide side (W) to the tele side (T).

  Next, it is determined whether or not the main subject confidence level is greater than or equal to a threshold value (step S313). If it is smaller than the threshold value (NO in step S313), the process proceeds to step S315, and if the main subject confidence level is equal to or greater than the threshold value (YES in step S313), whether the tracking confidence level (second evaluation value) is equal to or smaller than the threshold value. Is determined (step S314). This tracking confidence level is an evaluation value that increases when it is determined that the tracking frame determined in the tracking frame determination process (step S307) is influential as a subject, and is calculated in the tracking frame determination process.

  As a result of the determination in step S314, when the tracking confidence level is larger than the threshold (NO in step S314), the process proceeds to step S315. Since the main subject confidence level is low and the tracking confidence level is high, the tracking frame instead of the main subject detection frame is determined as the subject frame, and the tracking frame is displayed on the image display device 108 (step S315) (display means). End the process. If the tracking frame is not determined in the tracking frame determination process, the tracking frame is naturally not displayed as the subject frame.

  As a result of the determination in step S314, when the tracking confidence level is equal to or less than the threshold (YES in step S314), the process proceeds to step S316. Since the main subject confidence level is high and the tracking confidence level is low, the main subject detection frame instead of the tracking frame is determined as the subject frame, and the subject frame is displayed on the image display device 108 (step S316) (display means).

  Next, the tracking information thus far is cleared (step S317), the tracking process is resumed (step S318), and this process is terminated. In step S318, the stored color information of the tracking target area is cleared, the color information of the new subject frame area is stored again, and tracking is resumed.

  Steps S313 to S316 correspond to the third determining unit. A description will be given of determining a main subject candidate corresponding to an evaluation value having a higher main subject possibility as a main subject in these processes. If zooming is in progress, the process branches from step S308 to correct α, and the process of step S313 or further step S314 is performed there. The case where the main subject frame becomes the tracking frame during zooming may be when the main subject confidence level is less than the threshold value regardless of the tracking confidence level, or the main subject confidence level is greater than or equal to the threshold value and the tracking confidence level may be greater than the threshold value. sell. Since the tracking frame is less susceptible to fluctuations in the angle of view due to zooming, the tracking confidence level is more easily selected as the main subject frame. Conversely, the case where the main subject frame becomes the tracking frame during zooming is only when the main subject confidence level is greater than or equal to the threshold and the tracking confidence level is less than the threshold.

  According to the process of FIG. 3, firstly, a first main subject candidate is determined by calculating a first evaluation value indicating the main subject possibility using information depending on the angle of view (step S301). Next, a second evaluation value indicating the main subject possibility is calculated using information independent of the angle of view, and a second main subject candidate is determined (step S307). Next, when the angle of view is changed (YES in step S308), the first evaluation value is corrected so as to reduce the possibility of the main subject (steps S310 and 311), and the corrected first evaluation value and the first evaluation value are corrected. 2 The evaluation value is compared. Then, the main subject candidate corresponding to the evaluation value with the higher main subject possibility is determined as the main subject (steps S313 to S316). With this configuration, the main subject can be accurately determined even when the angle of view changes. The same effect can be obtained by increasing the tracking confidence level by multiplying the tracking confidence level by 1 / α instead of multiplying the main subject confidence level by α.

  Also, during zooming, the accuracy of the main subject detection frame determination process for determining the subject region based on the evaluation value that depends on the angle of view, such as the distance from the center of the screen, due to the change in the angle of view is reduced. Subject confidence level is lower than when zooming is not performed. On the other hand, with regard to the tracking confidence level for the tracking frame determination process that determines the subject region mainly based on color information that does not depend on the angle of view, no particular reduction process or the like is performed. In particular, with regard to the tracking frame determination process, an area of a color similar to the initial characteristic color stored before the start of zooming is tracked, so the object area is determined based on information when the object is stable. There is a high possibility that the subject frame can be correctly determined even during zooming.

  Furthermore, during zooming, the accuracy of the main subject detection frame determination process that determines the subject area based on the evaluation value that depends mainly on the angle of view, such as the distance from the center of the screen, due to fluctuations in the angle of view decreases. The degree is reduced. In this way, a tracking frame that is determined based on color information that is mainly independent of the angle of view is easily selected as the subject frame. As a result, it is possible to reduce the probability that the erroneously detected subject is tracked by erroneously detecting the main subject detection frame determination process during zooming.

  FIG. 6 is a flowchart showing the procedure of a modified example of the subject frame display process executed in step S201 of FIG.

  In FIG. 6, it is determined whether or not zooming is currently being performed (step S701). If the result of determination is that zooming is not in progress (NO in step S701), the main subject detection frame determination process of FIG. 4 is performed (step S702), and the process proceeds to step S703. If zooming is in progress (YES in step S701), step In step S703, it is determined whether tracking is in progress.

  If the result of determination in step S703 is that tracking is not in progress (NO in step S703), it is determined whether or not there is a main subject detection frame that is a subject frame candidate by the main subject detection frame determination processing in step S702 (step S703). S704). If there is no main subject detection frame (NO in step S704), the subject frame on the image display device 108 is not displayed (step S705), and the process is terminated. At this time, if the subject frame has been displayed so far, it is not displayed, and if there is no displayed subject frame, the subject frame is not displayed as it is.

  If the result of determination in step S704 is that there is a main subject detection frame (YES in step S704), the main subject detection frame is displayed as a subject frame on the image display device 108 (step S706), and tracking processing is started (step S706). S707), the process is terminated.

  As a result of the determination in step S703, when tracking is in progress (YES in step S703), a tracking frame that is a subject frame candidate is determined by the tracking frame determination processing in FIG. 5 (step S708), and whether or not zooming is currently in progress. A determination is made (step S709). If the result of determination in step S709 is that zooming is in progress (YES in step S709), the tracking frame is selected as the subject frame, the tracking frame is displayed on the image display device 108 (step S712), and this processing ends. To do. If the tracking frame is not determined in the tracking frame determination process, the tracking frame is naturally not displayed as the subject frame.

  If the result of determination in step S709 is that zooming is not in progress (NO in step S709), it is determined whether or not the main subject confidence level is greater than or equal to a threshold (step S710), and if it is less than the threshold (NO in step S710), The process proceeds to step S712 described above. If S702 is not executed during zooming, the main subject confidence level is not calculated, and it is assumed that the main subject confidence level is less than the threshold in the determination in step S710.

  If the result of the determination in step S710 is that the main subject confidence level is greater than or equal to the threshold value (YES in step S710), it is determined whether or not the tracking confidence level is less than or equal to the threshold value (step S711). If so, the process proceeds to step S712 described above. When step S712 is reached via step S711, both the main subject confidence level and the tracking confidence level are high, so that the tracking frame is preferentially determined as the subject frame instead of the main subject detection frame, and the tracking frame is determined. Is displayed on the image display device 108.

  As a result of the determination in step S711, if the tracking confidence level is equal to or less than the threshold (YES in step S711), the main subject detection frame is not the tracking frame but the main subject detection frame because the main subject confidence level is high and the tracking confidence level is low. Determine as. At the same time, the main subject detection frame is displayed on the image display device 108 (step S713). Next, the tracking information thus far is cleared (step S714), the tracking process is resumed (step S715), and this process ends. At this time, the stored color information of the tracking target area is cleared, the color information of the new subject frame area is stored again, and tracking is resumed.

  In the above processing, the accuracy of the main subject detection frame determination process for determining the subject area based on the evaluation value that depends mainly on the angle of view, such as the distance from the screen center, decreases due to the change in the angle of view during zooming. The detection frame determination process is not performed during zooming. On the other hand, a tracking frame determining process for determining a subject area is performed mainly based on color information that is not influenced by the angle of view, and the tracking frame is adopted as the subject frame. A decrease in subject frame display accuracy due to a decrease in accuracy of the main subject detection frame determination process can be avoided, and an increase in processing load on the system control unit 60 due to the main subject detection frame determination process during zooming can be suppressed.

  In the main subject detection frame determination process described above, an evaluation value can be calculated using history information in steps S504 and S505. However, the first judgment that cannot use history information is to determine the main subject detection frame based on the index that is affected by the angle of view such as position, area, and shape. By starting, an incorrect subject is tracked. By stopping the main subject detection determination process during zooming, which is likely to reduce accuracy, it is possible to track only the subject detected during non-zoom with a stable angle of view, making it difficult to track incorrectly I can do it.

  Furthermore, in the present embodiment, the main subject confidence level is reduced and the main subject detection frame determination process is stopped depending on whether zooming is being performed as a condition for changing the angle of view. Similar processing may be performed based on the output. For example, when the gyro signal is large or the output of the acceleration sensor is large, there is a high possibility that the angle of view has been changed. Therefore, the main subject confidence level may be reduced or the main subject detection frame determination process may be stopped.

  As described above, in the flowchart of the subject frame display process shown in FIG. 3, the value of the main subject confidence level with respect to the tracking confidence level is reduced when not zooming compared to when zooming. It is corrected to. In the flowchart of the subject frame display process shown in FIG. 6, the main subject confidence level is referred to when the zoom is not performed, but the subject frame is determined without referring to the subject confidence level when the zoom is being performed. Yes. Thus, in the present embodiment, when zooming, the tracking confidence level that does not depend on the angle of view is higher than the main non-subject confidence level that depends on the angle of view, compared to when the zoom is not performed. By determining the subject frame using the priority, it is possible to determine the main subject with high accuracy.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program code. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

10 Lens 16 CCD
38 Image recognition circuit 42 Lens drive circuit 50 Image processing circuit 60 System control unit 100 Imaging device 104 Shutter switch SW1
106 Shutter switch SW2
108 Image display device

Claims (18)

An imaging means capable of imaging a subject and zooming to change the angle of view;
A first evaluation value indicating the main subject possibility that the main subject candidate is the main subject is calculated from the image showing the subject imaged by the imaging means using information depending on the angle of view, and the first main subject candidate is selected. First determining means for determining;
Second determination means for calculating a second evaluation value indicating the main subject possibility by using information independent of an angle of view from an image showing the subject imaged by the imaging means and determining a second main subject candidate; ,
Third determination means for determining a main subject candidate corresponding to any of the evaluation values as a main subject using the first evaluation value and the second evaluation value;
Said third determination means, when it is in said zooming, as compared with the case not being the zoom, correcting the first evaluation value so as to lower the main subject possibilities, and, the main An imaging apparatus that performs at least one of correcting the second evaluation value so as to increase a subject possibility .   An imaging means capable of imaging a subject and zooming to change the angle of view;
  A first evaluation value indicating the main subject possibility that the main subject candidate is the main subject is calculated from the image showing the subject imaged by the imaging means using information depending on the angle of view, and the first main subject candidate is selected. First determining means for determining;
  Second determination means for calculating a second evaluation value indicating the main subject possibility by using information independent of an angle of view from an image showing the subject imaged by the imaging means and determining a second main subject candidate; ,
  When the zoom is not in progress, a main subject candidate corresponding to any one of the evaluation values is determined as a main subject using the first evaluation value and the second evaluation value, and when the zoom is in progress, An imaging apparatus comprising: a third determining unit that determines a main subject candidate corresponding to the second evaluation value as a main subject.   An imaging means capable of imaging a subject and zooming to change the angle of view;
  Any one of the divided regions is obtained by dividing the region based on at least one of luminance and color information from the image showing the subject imaged by the imaging unit and using the distance from the center of the screen of the divided region. A first determination unit that calculates a first evaluation value indicating a main subject possibility as a main subject and determines a first main subject candidate;
  A region having information similar to information included in a predetermined main subject is determined from an image showing the subject captured by the imaging unit, and the similarity between the information in the region and the predetermined information is determined. A second determining means for calculating a second evaluation value indicating the main subject possibility and determining a second main subject candidate;
  Third determination means for determining a main subject candidate corresponding to any of the evaluation values as a main subject using the first evaluation value and the second evaluation value;
  The third determining means corrects the first evaluation value so as to reduce the possibility of the main subject when the zoom is being performed, compared to when the zoom is not being performed, and the main subject An image pickup apparatus that performs at least one of correcting the second evaluation value so as to increase the possibility.   An imaging means capable of imaging a subject and zooming to change the angle of view;
  Any one of the divided regions is obtained by dividing the region based on at least one of luminance and color information from the image showing the subject imaged by the imaging unit and using the distance from the center of the screen of the divided region. A first determination unit that calculates a first evaluation value indicating a main subject possibility as a main subject and determines a first main subject candidate;
  A region having information similar to information included in a predetermined main subject is determined from an image showing the subject captured by the imaging unit, and the similarity between the information in the region and the predetermined information is determined. A second determining means for calculating a second evaluation value indicating the main subject possibility and determining a second main subject candidate;
  When the zoom is not in progress, a main subject candidate corresponding to any one of the evaluation values is determined as a main subject using the first evaluation value and the second evaluation value, and when the zoom is in progress, An imaging apparatus comprising: a third determining unit that determines a main subject candidate corresponding to the second evaluation value as a main subject.   The third determining means determines a main subject candidate corresponding to the first evaluation value if the first evaluation value is equal to or greater than a first threshold and the second evaluation value is equal to or less than a second threshold. The main subject candidate corresponding to the first evaluation value is determined as the main subject when the main subject is determined and the second evaluation value exceeds a second threshold value. 3. The imaging device according to 3. The third determining means, when the angle of view is changed from the wide angle side to the telephoto side by the zoom , is larger than the case where the angle of view is changed from the telephoto side to the wide angle side, and reduces the possibility of the main subject. The first evaluation value is corrected so as to be corrected, or the second evaluation value is corrected so as to increase the possibility of the main subject to be larger . The imaging apparatus according to item 1 . If the first evaluation value is equal to or greater than a first threshold and the second evaluation value is equal to or less than a second threshold when the zoom is not in progress , the third determination means A main subject candidate corresponding to a value is determined as the main subject, and if the second evaluation value exceeds a second threshold value, a main subject candidate corresponding to the first evaluation value is determined as the main subject. The imaging device according to claim 2, wherein: The first determination unit determines a region other than the background using luminance, hue, and saturation information, and uses the first evaluation value as information depending on the angle of view, as a distance and an area from the center of the image. , and the imaging apparatus according to claim 1 or 2, characterized in that calculated using at least one information of the aspect ratio. The information independent of the angle of view is information indicating a predetermined color,
The second determining means determines a region in the image having a color closest to the predetermined color as the second main subject candidate, and the predetermined color and the region in the second main subject candidate the imaging apparatus according to claim 1 or 2, characterized in that for calculating said second evaluation value by the similarity of the color of. 2. The display device according to claim 1, further comprising: a display unit configured to display an image indicating that the main subject is determined by the third determination unit so as to be superimposed on an image indicating the subject captured by the imaging unit. The imaging device according to any one of 1 to 9 . A method for controlling an imaging apparatus including an imaging unit capable of imaging a subject and zooming to change a field angle,
A first evaluation value indicating the main subject possibility that the main subject candidate is the main subject is calculated from the image showing the subject imaged by the imaging means using information depending on the angle of view, and the first main subject candidate is selected. A first determining step for determining;
A second determination step of determining a second main subject candidate by calculating a second evaluation value indicating the main subject possibility using information independent of an angle of view from an image showing the subject imaged by the imaging means; ,
A third determination step of determining a main subject candidate corresponding to any of the evaluation values as a main subject using the first evaluation value and the second evaluation value;
The third determining step, if it is in the zoom, compared to the case not being the zoom, correcting the first evaluation value so as to lower the main subject possibilities, and, the main A control method comprising performing at least one of correcting the second evaluation value so as to increase a subject possibility .   A method for controlling an imaging apparatus including an imaging unit capable of imaging a subject and zooming to change a field angle,
  A first evaluation value indicating the main subject possibility that the main subject candidate is the main subject is calculated from the image showing the subject imaged by the imaging means using information depending on the angle of view, and the first main subject candidate is selected. A first determining step for determining;
  A second determination step of determining a second main subject candidate by calculating a second evaluation value indicating the main subject possibility using information independent of an angle of view from an image showing the subject imaged by the imaging means; ,
  When the zoom is not in progress, a main subject candidate corresponding to any one of the evaluation values is determined as a main subject using the first evaluation value and the second evaluation value, and when the zoom is in progress, And a third determination step of determining a main subject candidate corresponding to the second evaluation value as a main subject.   A method for controlling an imaging apparatus including an imaging unit capable of imaging a subject and zooming to change a field angle,
  Any one of the divided regions is obtained by dividing the region based on at least one of luminance and color information from the image showing the subject imaged by the imaging unit and using the distance from the center of the screen of the divided region. A first determination step of determining a first main subject candidate by calculating a first evaluation value indicating the possibility of a main subject as a main subject,
  A region having information similar to information included in a predetermined main subject is determined from an image showing the subject captured by the imaging unit, and the similarity between the information in the region and the predetermined information is determined. A second determination step of calculating a second evaluation value indicating the main subject possibility and determining a second main subject candidate;
  A third determination step of determining a main subject candidate corresponding to any of the evaluation values as a main subject using the first evaluation value and the second evaluation value;
  The third determining step corrects the first evaluation value so that the possibility of the main subject is lower when the zoom is being performed than when the zoom is not being performed; and A control method comprising performing at least one of correcting the second evaluation value so as to increase the possibility.   A method for controlling an imaging apparatus including an imaging unit capable of imaging a subject and zooming to change a field angle,
  Any one of the divided regions is obtained by dividing the region based on at least one of luminance and color information from the image showing the subject imaged by the imaging unit and using the distance from the center of the screen of the divided region. A first determination step of determining a first main subject candidate by calculating a first evaluation value indicating the possibility of a main subject as a main subject,
  A region having information similar to information included in a predetermined main subject is determined from an image showing the subject captured by the imaging unit, and the similarity between the information in the region and the predetermined information is determined. A second determination step of calculating a second evaluation value indicating the main subject possibility and determining a second main subject candidate;
  When the zoom is not in progress, a main subject candidate corresponding to any one of the evaluation values is determined as a main subject using the first evaluation value and the second evaluation value, and when the zoom is in progress, And a third determination step of determining a main subject candidate corresponding to the second evaluation value as a main subject. A program for causing a computer to execute a control method of an image pickup apparatus including an image pickup unit capable of taking an image of a subject and zooming to change an angle of view,
The control method is:
A first evaluation value indicating the main subject possibility that the main subject candidate is the main subject is calculated from the image showing the subject imaged by the imaging means using information depending on the angle of view, and the first main subject candidate is selected. A first determining step for determining;
A second determination step of determining a second main subject candidate by calculating a second evaluation value indicating the main subject possibility using information independent of an angle of view from an image showing the subject imaged by the imaging means; ,
A third determination step of determining a main subject candidate corresponding to any of the evaluation values as a main subject using the first evaluation value and the second evaluation value;
The third determining step, if it is in the zoom, compared to the case not being the zoom, correcting the first evaluation value so as to lower the main subject possibilities, and, the main A program that performs at least one of correcting the second evaluation value so that a subject possibility is high .   A program for causing a computer to execute a control method of an image pickup apparatus including an image pickup unit capable of taking an image of a subject and zooming to change an angle of view,
  The control method is:
  A first evaluation value indicating the main subject possibility that the main subject candidate is the main subject is calculated from the image showing the subject imaged by the imaging means using information depending on the angle of view, and the first main subject candidate is selected. A first determining step for determining;
  A second determination step of determining a second main subject candidate by calculating a second evaluation value indicating the main subject possibility using information independent of an angle of view from an image showing the subject imaged by the imaging means; ,
  When the zoom is not in progress, a main subject candidate corresponding to any one of the evaluation values is determined as a main subject using the first evaluation value and the second evaluation value, and when the zoom is in progress, A third determination step of determining a main subject candidate corresponding to the second evaluation value as a main subject.   A program for causing a computer to execute a control method of an image pickup apparatus including an image pickup unit capable of taking an image of a subject and zooming to change an angle of view,
  The control method is:
  Any one of the divided regions is obtained by dividing the region based on at least one of luminance and color information from the image showing the subject imaged by the imaging unit and using the distance from the center of the screen of the divided region. A first determination step of determining a first main subject candidate by calculating a first evaluation value indicating the possibility of a main subject as a main subject,
  A region having information similar to information included in a predetermined main subject is determined from an image showing the subject captured by the imaging unit, and the similarity between the information in the region and the predetermined information is determined. A second determination step of calculating a second evaluation value indicating the main subject possibility and determining a second main subject candidate;
  A third determination step of determining a main subject candidate corresponding to any of the evaluation values as a main subject using the first evaluation value and the second evaluation value;
  The third determining step corrects the first evaluation value so that the possibility of the main subject is lower when the zoom is being performed than when the zoom is not being performed; and A program that performs at least one of correcting the second evaluation value so as to increase the possibility.   A program for causing a computer to execute a control method of an image pickup apparatus including an image pickup unit capable of taking an image of a subject and zooming to change an angle of view,
  The control method is:
  Any one of the divided regions is obtained by dividing the region based on at least one of luminance and color information from the image showing the subject imaged by the imaging unit and using the distance from the center of the screen of the divided region. A first determination step of determining a first main subject candidate by calculating a first evaluation value indicating the possibility of a main subject as a main subject,
  A region having information similar to information included in a predetermined main subject is determined from an image showing the subject captured by the imaging unit, and the similarity between the information in the region and the predetermined information is determined. A second determination step of calculating a second evaluation value indicating the main subject possibility and determining a second main subject candidate;
  When the zoom is not in progress, a main subject candidate corresponding to any one of the evaluation values is determined as a main subject using the first evaluation value and the second evaluation value, and when the zoom is in progress, A third determination step of determining a main subject candidate corresponding to the second evaluation value as a main subject.

Images