JP2019101997A - Image processing apparatus and image processing method reducing noise by composing plural captured images - Google Patents

Abstract

To provide an image processing apparatus that reduces influence of brightness variation and that generates a composite image having a noise improvement effect even when brightness varies between a plurality of image frames.SOLUTION: An image processing apparatus comprises: imaging means that captures a reference frame image and a plurality of other frame images; change amount calculation means that calculates a change amount of a signal of the plurality of other frame images with respect to the reference frame image; extraction means that extracts a background region by comparing the other frame image with the reference frame image; ratio calculation means that calculates a composition rate of the plurality of other frame images on the basis of the change amount of the signal with respect to the frame image being a reference of the plurality of other frame images; and composite means that composes two images of a target using the composition rate calculated by the ratio calculation means, in which the ratio calculation means calculates the composition rate by giving larger weight to other frame image in which the change amount is smaller.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a multiple sheet combining technique for combining a plurality of captured images and reducing noise.

  2. Description of the Related Art Conventionally, as a still image processing technique, a technique for obtaining a noise reduction effect by performing image synthesis on continuously captured images is known (hereinafter referred to as multi-shot NR). Multi-shot NR technology is now beginning to be processed not only for still image shooting but also for moving image shooting.

  As a method of realizing the multi-shot NR as a moving image, the moving image is treated as a continuous still image frame, and several frames are cut out as a plurality of still image frames from a predetermined reproduction position of the moving image. Next, the synthesized still image frame subjected to the noise reduction can be generated by performing image synthesis of the cut-out still image frame. At this time, when moving objects are combined, the objects become multiple images, and the images become unattractive as a noise reduction effect.

  On the other hand, Patent Document 1 detects the moving body likelihood based on the difference information between the first image and the second image, and controls the synthesis ratio so that the second image is not synthesized in the region where the moving body likelihood is high. Since the image is synthesized, it is possible to prevent the moving image from becoming a multiple image.

JP, 2013-102554, A

  However, in a still image frame cut out from a moving image, when there is a change in the brightness of the scene, the control until AE (automatic brightness adjustment) is reflected in the exposure control is slower than in still image shooting and several frames later It may be controlled to be a proper AE. As a result, the brightness is different even for a non-moving subject region, and the difference between the first image and the second image becomes large between a plurality of frames, and the moving object likelihood is detected high and can not be synthesized. . Therefore, in a region where a change in brightness is detected, a composite image with a small noise improvement effect is generated.

  In view of the above problems, it is an object of the present invention to provide an image processing apparatus that reduces the influence of brightness variation and generates a composite image having a noise improvement effect even when brightness variation occurs between a plurality of image frames as in a moving image An object of the present invention is to provide an image processing method.

  In order to solve the above-mentioned subject, the image processing device of the present invention finds an acquisition means which acquires a standard frame picture and a plurality of other frame pictures, and finds the amount of change of the signal of the plurality of other frame pictures to the standard frame picture. Based on the amount of change in the signal of the plurality of other frame images relative to the reference frame image, the amount of change calculation means, the extraction means for extracting the background area by comparing the reference frame image and the other frame image A ratio calculating unit configured to calculate a combination ratio of the plurality of other frame images; and a combining unit configured to combine two target images at the combination ratio calculated by the ratio calculating unit, the ratio calculating unit including: It is characterized in that the combining ratio is calculated by giving larger weight to the other frame image having smaller amount of change.

  In the image processing method according to the present invention, an acquisition step of acquiring a reference frame image and a plurality of other frame images, and a change amount calculation step of calculating an amount of change of signals of the plurality of other frame images with respect to the reference frame image. Extracting the background area by comparing the reference frame image and the other frame image, and the plurality of other frames based on a change amount of a signal of the plurality of other frame images with respect to the reference frame image The image processing apparatus further includes a ratio calculation step of calculating a synthesis ratio of the images, and a synthesis step of synthesizing the two target images with the synthesis ratio calculated in the ratio calculation step, and in the ratio calculation step, the change amount is smaller. It is characterized in that the weighting is given to the other frame image to calculate the composition ratio.

  According to the present invention, it is possible to reduce the influence of brightness variation and generate a composite image having a noise improvement effect.

Block diagram showing a configuration example of an imaging apparatus for realizing the image processing technology of the present invention Process flowchart in the first embodiment A diagram to explain the difference in exposure set in AE processing and actual imaging Diagram explaining the change in brightness of the subject Diagram explaining how to detect background subject area Diagram explaining the vector detection method Diagram for explaining the feature points of alignment Process flowchart in the present embodiment 2 An illustration of composition ratio conversion based on the brightness change amount A diagram for explaining the target image for alignment

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration example of an imaging apparatus including an image processing apparatus to which the image processing technique of the present invention can be applied. Hereinafter, an example in which the present invention is applied to a digital camera will be described as an example of an image processing apparatus and an imaging apparatus. However, the image processing apparatus according to the present invention is also applicable to any electronic device capable of acquiring a captured image. These electronic devices may include, for example, a cellular phone, a game machine, a tablet terminal, a personal computer, and a watch-type or glasses-type information terminal.

  The system control unit 101 is, for example, a CPU, reads out an operation program of each block included in the camera 100 which is an imaging device from the ROM 102 described later, expands it in the RAM 103 described later, and executes it. Control. The ROM 102 is a rewritable non-volatile memory, and stores parameters and the like necessary for the operation of each block in addition to the operation program of each block included in the camera 100. The RAM 103 is a rewritable volatile memory, and is used as a temporary storage area of data output in the operation of each block included in the camera 100.

  The optical system 104 is composed of a lens group including a zoom lens and a focus lens. A subject image obtained via the optical system 104 forms an image on the imaging unit 105. The imaging unit 105 is an imaging element such as a CCD or a CMOS sensor, for example, and photoelectrically converts an optical image formed on the imaging unit by the optical system, and sequentially A / D converts a pair of obtained analog image signals Output to unit 106. The A / D conversion unit 106 converts the input analog image signal into a digital image signal, and outputs the obtained digital image data to the RAM 103.

  The image processing unit 107 applies various image processing such as white balance adjustment, color interpolation, gamma processing, electronic zoom processing, and the like to the image data recorded in the RAM 103.

  The display unit 108 is a display device such as an LCD, and displays an image after image processing by the image processing unit recorded in the RAM 103 and an operation user interface for receiving an image recorded in the recording unit 109 described later and an instruction from the user. Do. A recording unit 109 is a recording medium such as a memory card or a hard disk, and stores final output image data subjected to the image processing of the present embodiment.

  The configuration of the imaging apparatus (hereinafter referred to as a camera) 100 and the basic operation have been described above.

  The operation of the image processing unit 107, which is a feature of this embodiment, will be described below with reference to the flowchart shown in FIG. The following processing is realized by the control unit 101 controlling each unit of the apparatus according to a program stored in the ROM 102 and controlling input and output to the image processing unit 107.

  In step S201 in FIG. 2, in order to perform a shooting operation, camera settings such as ISO sensitivity, shutter speed, exposure such as aperture are performed, and an image is acquired.

  In the present embodiment, by fixing the camera 100 to a tripod or the like and photographing, positional deviation does not occur between the two images due to the influence of camera shake or the like, and can be ignored. At this time, the image processing unit 107 performs an AE (automatic exposure) process using a frame of a moving image taken by the imaging unit 105 before the main photographing. In the AE processing, an appropriate exposure for setting an exposure parameter of aperture, sensitivity, and shutter speed is calculated so that an image signal can be acquired with an appropriate brightness in a scene to be photographed. However, unlike the AE processing of a still image in the AE processing of a moving image, the exposure setting at the time of imaging is controlled so as to meet the appropriate exposure after a predetermined frame so that the change in brightness does not change significantly between the previous and subsequent frames of the moving image. .

  Step S202 in FIG. 2 determines the reference position from the frame of the moving image. The moving image frame of the reference position is a composite image of the final output, which is used as a reference of the angle of view, and it is determined by accepting the designation of the frame from the user.

  The processes of steps S201 to S202 will be described in detail with reference to FIGS. 3 and 4.

  FIG. 4 shows three frames extracted from a moving image frame at a predetermined interval (for example, every 100 frames) when the weather changes from sunny to cloudy and the brightness of the scene changes. From the moving image frame 401 in the clear state, the brightness of the object changes between the moving image frames 402 where the clouds are spreading and 403 moving images which are completely cloudy. Therefore, AE processing is performed to properly adjust the brightness of the moving image frames 401 to 403, and the exposure parameters of the aperture, sensitivity, and shutter speed are calculated. An index that accurately represents the brightness of a subject is called a BV value, and can be expressed by BV = EV (aperture value + shutter speed value) + SV (sensitivity value). In this embodiment, it is assumed that the photographing sensitivity is constant. Therefore, in the embodiment, the brightness of the subject is expressed by the EV value. Here, FIG. 3 shows changes in the EV value controlled by the camera 100 and the target EV value with respect to the moving image frame.

  In the graph of FIG. 3, the horizontal axis represents an image frame captured in time series, and the vertical axis represents an EV value. The EV value represents the amount of light obtained by the combination of the aperture value of the lens and the shutter speed. The EV target values (dotted lines 301, 302, and 303) correspond to appropriate brightness when shooting an actual scene, and the control EV values (solid lines of 311, 312, and 313) are moving image frames captured by the camera 100 Represents the brightness of the signal value of. Section 341 is a moving image frame corresponding to 401 in FIG. 4, section 342 is 402 in FIG. 4, and section 343 is a scene in 403 in FIG. At this time, there is no difference between the EV target value and the control EV value in sections 341 and 343 in which the brightness of the scene does not change (the solid line and dotted line are separated on the display in the figure but they actually overlap). The moving image data is captured at However, in the section 342, a difference 331 occurs between the EV target value and the control EV value, and in this section 342, moving image data is captured darker than appropriate. At this time, assuming that the frame of the moving image serving as the reference of synthesis selected by the user is the reference position 351, the reference frame image 351 is included, and a predetermined number of frames behind it are frames of the moving image to be synthesized (hereinafter It is called another frame image).

  It returns to description of the flowchart of FIG.

  In step S203 of FIG. 2, a background area is detected for another frame image. The background area in the present embodiment indicates an area in which the movement of the subject is small between the reference frame image and the images of the other frame images. Specifically, as background area extraction processing, a vector is detected as the amount of movement of another frame image with respect to the reference frame image, and a moving body (an area not a background) and a non-moving body based on the vector movement direction or the magnitude of vector movement. Determine the area (background area). The background area is described with reference to FIG.

  FIG. 5 is a diagram showing an area detected as a background area for a certain subject scene. FIG. 5A shows the case where the background area is wide. The background area 531 in FIG. 5A is a background area detected from the reference frame image 501 and a frame 502 representing one frame of other frame images. On the other hand, FIG. 5 (b) shows the case where the background area is narrow. The background area 533 in FIG. 5B is a background area detected from the frame of the reference frame image 503 and the frame of one frame 504 among the other frame images. In the determination of the background area, the motion vector of the subject between frames is detected, and an area with a small vector size is used as the background area. As a method of determining a region using vectors, it is possible to apply an existing vector clustering algorithm in addition to the method described above.

  The motion area 532 has, for example, a large vector such as the motion vector 522, and an area close in magnitude and direction to the surrounding vector as one moving body area (judged by a predetermined value in design and threshold determination). On the other hand, with respect to a stationary and non-moving area such as 521, it is determined that the vector has a value near 0 and is different from the moving area. Calculated using the template matching method as the vector detection method. In the template matching method, since the vector is obtained based on the relative difference in the predetermined area between the reference frame image and the other frame image, the brightness change of the subject can be made robust. The vector detection using the template matching method will be described in detail with reference to FIG. 6 below.

  FIG. 6A shows a template position for performing vector search. For example, 620 in FIG. 6A indicates a plurality of template image groups arranged at equal intervals. One of the template images is described as 621 for explanation.

  FIG. 6B is a view for explaining setting of the position and range of vector search, which is an image of a reference image 600 (corresponding to a reference frame image) and an image of a vector detection image 601 (corresponding to another frame image). Starting from the area and position of the template 621 determined from the vector detection image, the position of the template 622 is set as the vector search start position in the reference image. Then, a hatched area around the area is set as a vector search range 641 and compared with the template 621 to detect a position shift between a position where the correlation is high and the template 621 as a motion vector. This operation is performed on all areas of the template image group 620 to detect vectors for the number of templates. At this time, for example, the contrast of each template image of the template image group is determined, and template matching may not be performed for a template image with low contrast.

  It returns to description of the flowchart of FIG.

  In step S204 in FIG. 2, the reference frame image and the region on the image of the other frame image to be combined are divided, and change amount calculation processing is performed to detect the brightness change for each of the divided predetermined regions. As a method of detecting a change in brightness, brightness integration is performed for each of the regions divided by the reference frame image and the other frame images, and the difference between the brightness integral values obtained by integrating the brightness value between two frames is calculated as an evaluation value. Do. The method of comparing the brightness between two frames is not limited to the above, and other methods such as comparison of average values of luminance values may be used as an operation of comparing image data of two frames and acquiring an evaluation value.

  In step S205 of FIG. 2, a combination ratio calculation process is performed to calculate the combination ratio based on the brightness change amount obtained as the evaluation value. For example, using the graph of FIG. 9A, the combining ratio is calculated from the brightness change amount. In the graph of FIG. 9A, the horizontal axis represents the brightness change amount, and the vertical axis represents the combining ratio. The composition ratio of another frame image to the reference frame image is controlled by a curve 902 corresponding to the amount of change in brightness. As the amount of change in brightness is larger, the combining ratio with the reference frame image of another frame image is controlled to be smaller. When the amount of change in brightness is small, control is performed to increase the composition ratio with the reference frame image of another frame image.

  Subsequently, in step S206 in FIG. 2, it is checked whether the area to be synthesized is the background. If it is determined that the area to be synthesized is the background, the process proceeds to step S207 in FIG.

  Conversely, if it is determined that the area to be synthesized is not the background, the synthesis ratio according to the amount of change in brightness determined based on the graph of FIG. The ratio is determined, and the process proceeds to step S208.

  In step S207 in FIG. 2, processing is performed to control the composition ratio with the reference frame image of another frame image to be higher than the composition ratio based on the brightness change amount obtained in step 205. Specifically, description will be made with reference to FIG.

  In FIG. 9B, the combination ratio is further controlled according to the background determination result with respect to the combination ratio according to the brightness change amount. Even if the amount of change in brightness is large, if the area is the background, the combined image is not multiplexed by combining and the noise reduction effect can be expected. Therefore, when it is determined to be the background area, control is performed so that the composition ratio of 913 is obtained in order to increase the composition ratio with the reference frame image of another frame image. Reference numeral 912 denotes the composition ratio of the area other than the background area to the reference frame image.

  In step 208 of FIG. 2, addition average combination of the reference frame image and the other frame image is performed using the combination ratio determined in step S205 and step S207 to realize noise reduction.

  Specifically, a general method will be described in which the combination of two moving image frames is cyclically repeated and finally the addition average combination of a plurality of moving image frames is finally performed.

  In addition to the reference frame image, when combining three other frame images, the ratio of the reference frame image and the first other frame image at the upper limit 1/2 in the first combination with respect to one reference frame image Composite to generate a composite image. The second cyclic synthesis can be realized by synthesizing the first composite image and the second other frame image at a ratio of the upper limit 1/3. Similarly, in the third cyclic combining, an average combining combining four moving image frames including the reference frame image at an equal ratio by combining the third other frame image at the ratio of the upper limit 1/4. Can be performed in a cyclic process. The above-described addition average combining process can be written as an equation below.

Here, img ₀ represents the signal value of the reference frame image, img _{n ≠ 0} represents the signal value of the moving image frame in the synthesis range, and n corresponds to the number of times of synthesis.

  Further, in this case, in order to perform addition average combining in consideration of the combining ratio determined in steps S205 and S207, combining processing is performed according to the following equation.

  The ratio represents the composition ratio with the reference frame image of the other frame image determined in step S205 and step S207. The composition in step S208 is realized by performing the composition processing in which the composition ratio (ratio) with the reference frame image of another frame image is added to the equation (1).

  This is the end of the description of the process of the flowchart of FIG.

  By performing the process of the flowchart of FIG. 2, even when the brightness varies among a plurality of image frames, the moving subject component between the reference frame image and the other frame image is excluded, and the composite image having the noise improvement effect is output. An image processing apparatus can be provided.

  As mentioned above, although the preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  In the present embodiment, the method of controlling the combining ratio in accordance with whether or not it is the background area has been described, but it is possible to control the combining ratio more finely based on the reliability when performing the background determination. Specifically, in the process of step S207 of FIG. 2, control of the synthesis ratio shown by the graph of FIG. 9C is performed.

  924 in FIG. 9C is a composition ratio with another frame image of the reference frame image when the reliability of the background area is high. Reference numeral 923 denotes a composition ratio with respect to the reference frame image of another frame image when the reliability of the background area is low. Reference numeral 922 denotes a composition ratio with respect to a reference frame image of another frame image other than the background area. The reliability of the background area is determined according to the size of the area of the detected background area (the reliability is changed in FIGS. 5A and 5B), or the template area at the time of detecting a vector. It can be calculated based on the texture amount (standard deviation) of (area 621 of FIG. 6). As described above, by calculating the reliability of the background area and finely controlling the synthesis ratio of the other frame images, it is possible to calculate the synthesis ratio in consideration of the influence of the erroneous detection of the background area.

  In the following, in the second embodiment, an example in which the present invention is applied not to a moving image frame shot with a fixed tripod but to a moving image frame shot by hand is described.

  The functional configuration of a camera 100 which is a digital camera according to an embodiment of the present invention is the same as that of the first embodiment, and therefore the description thereof is omitted (FIG. 1).

  The operation of the image processing unit 107, which is a feature of this embodiment, will be described using the flowchart shown in FIG. The following processing is realized by the control unit 101 controlling each unit of the apparatus according to a program stored in the ROM 102 and controlling input and output to the image processing unit 107. The entire process of the second embodiment will be described with reference to the flowchart of FIG. In the flowchart of FIG. 8 of the overall processing of the second embodiment, the steps performing the same processing as the flowchart of FIG. 2 of the first embodiment are described using the same symbols as the flowchart of FIG. Since the processing steps different from the first embodiment are the processing of step S801, the processing after step S801 will be described.

  In step S801 in FIG. 8, the moving image frame whose reference position is determined in step S202 is matched with the position of the background subject of the moving image frame to be synthesized. Processing that calculates the motion vector between the corresponding position of the moving image frame at the reference position and the moving image frame to be synthesized, performs geometric deformation based on the vector, and aligns the position of the moving image frame to be synthesized with the moving image frame at the reference position I do. The alignment will be described in detail with reference to FIGS.

  In order to perform alignment processing, an alignment coefficient for correcting the amount of deformation between images is calculated. First, alignment processing will be described with reference to FIG. Images 1001 and 1002 in FIG. 10 are an alignment reference image (1001) and an alignment correction target image (1002), respectively. In the alignment correction processing, in addition to the translation component, a shake or shake occurs to generate a component of rotation or rotation, and as a result, an image in which the image is affected by the rotation or rotation is acquired like an image 1002. In such a case, geometric transformation coefficient calculation is performed as a coefficient for correcting the translation component, the rotation component, and the tilt component by the geometric transformation process. The geometric transformation coefficient which performs this geometric transformation process is called an alignment coefficient. For example, a diagram schematically representing the image 1002 is 1003, and the position of the image 1001 to be corrected is indicated by 1004. At this time, the alignment coefficient used when performing geometric deformation processing from 1003 to 1004 is 1011. The registration coefficient A of 1005 is generally expressed by the equation (2). Assuming that the coordinates of the image are I (x coordinate, y coordinate), the geometric transformation processing of the equation (3) is performed to obtain the positions 1003 to 1004. It is a coefficient that can be performed.

  In order to calculate the alignment coefficient, an image set is set between two images of the alignment reference image (1001) and the alignment correction target image (1002). In the image set, for example, an image whose imaging time is one frame before the correction target image is used as a reference for alignment, and another image is used as the correction target image.

  Then, a vector is calculated using the movement direction / movement amount of the feature point (or feature area) as the feature amount for each area of the image between the image sets. Then, based on the calculated vector, a geometric transformation coefficient representing a positional relationship (translation, rotation, reduction, enlargement) between the image sets is obtained as an alignment coefficient.

  In FIG. 7A, reference numeral 701 denotes the entire image, and reference numeral 711 denotes feature points (or feature regions) uniformly set on the entire image. As feature points, blocks at equally-spaced positions in the image area may be simply taken out, but it is more accurate to select the vector between the image sets if the texture including many edge components or high frequency to low frequency frequency components is selected be able to.

  Reference numeral 702 in FIG. 7B denotes a registration target image, and reference numeral 703 denotes an image as a reference of registration between image sets. The position of the corresponding feature point is calculated for the image 702 and the image 703, and the direction and the movement amount of the corresponding feature point are calculated. For example, 721 to 723 and 731 to 733 are vectors in which arrows are shown as vectors as feature amounts representing the direction and the movement amount. Similarly, vectors are calculated for the remaining feature points.

  Generally, a block matching method is used as a method of obtaining the correspondence between feature points between images. According to this block matching method, hierarchical vector detection can also be performed by performing block matching not only on a photographed equal-magnified image but also on a stepwise reduced image. By hierarchically performing vector detection, it is possible to suppress an increase in processing time required for vector detection even if the search range is broadened.

  Subsequently, geometric transformation coefficients are determined using the vector group determined. For example, the vector X1 is a three-dimensional coordinate of the coordinate X '= (x'1, y'1, 1) of the feature point of the reference image and the coordinate X = (x1, y1, 1) of the feature point of the alignment target image I have information including the system. Therefore, as in equation (4), the coordinates X ′ obtained by multiplying the coordinates X of the feature points of the alignment target image by the predetermined conversion coefficient A, and the coordinates and difference of the feature points of the reference image become the smallest. Find a conversion factor A.

  The conversion coefficient A is determined using a known optimization method such as the Newton method or Gauss-Newton method. The obtained conversion factor A is adopted as a registration factor.

  Referring back to the flowchart of FIG.

  In step S203 of FIG. 8, the background frame is detected by performing the process of step S203 of the first embodiment using the reference frame image and the image of the moving image frame to be combined aligned in step S801. Since the detection of the background area uses another frame image after geometric transformation for correcting camera shake with respect to the reference frame image, the motion vector of the background area becomes almost zero, as shown in FIG. The background area can be detected in the same manner as (b).

  The processes in steps S204 to S208 in FIG. 8 are similar to those in the first embodiment, and thus the description thereof is omitted.

  This is the end of the description of the process of the flowchart of FIG.

  By performing the process of the flowchart of FIG. 8, it is possible to provide an image processing apparatus that reduces the influence of noise on a moving image frame captured by hand.

  As mentioned above, although the preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  In addition, detection of the background area in step S204 in FIG. 8 may be performed by using the vector size and direction of another frame image before geometric conversion for correcting camera shake on the reference frame image. It is possible. Specifically, for each vector start position (the vector start point 741 in FIG. 7) calculated in another frame image before alignment, the equation (4) transform coefficient A is determined. Thereafter, a background area is detected by detecting a vector in which an error between coordinates calculated by the conversion coefficient A for each vector start position and each vector end position (the vector end point 742 in FIG. 7) is smaller than a predetermined value. It is possible to determine

  Note that the process of the above-described embodiment may provide a system or apparatus with a storage medium storing a program code of software in which a procedure for realizing each function is described. Then, the computer (or CPU or MPU) of the system or the apparatus can read out and execute the program code stored in the storage medium to realize the functions of the above-described embodiment. In this case, the program code itself read out from the storage medium implements the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. As a storage medium for supplying such a program code, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk or the like can be used. Alternatively, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card, a ROM or the like can be used.

  In addition, the functions of the above-described embodiments are not only realized by executing the program code read by the computer. Based on the instructions of the program code, the operating system (OS) operating on the computer performs part or all of the actual processing, and the processing of the above-described embodiments is realized by the processing. It is done.

  Furthermore, the program code read out from the storage medium may be written to a memory provided to a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on an instruction of the program code, a CPU or the like provided in the function expansion board or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing. Is also included.

101 System control unit 102 ROM
103 RAM
104 optical system 105 imaging unit 106 A / D conversion unit 107 image processing unit 108 display unit 109 recording unit

Claims (11)

Acquisition means for acquiring a reference frame image and a plurality of other frame images;
Variation amount calculation means for obtaining variation amounts of the plurality of other frame image signals with respect to the reference frame image;
Extracting means for extracting a background area by comparing the reference frame image and the other frame image;
Ratio calculation means for calculating a synthesis ratio of the plurality of other frame images based on the amount of change of the signal of the plurality of other frame images with respect to the reference frame image;
And combining means for combining two images of the object at the combining ratio calculated by the ratio calculating means,
3. The image processing apparatus according to claim 1, wherein the ratio calculation unit calculates the synthesis ratio by giving larger weight to the other frame image having a smaller amount of change.   The image processing apparatus according to claim 1, wherein the change amount calculation unit calculates using the difference information of the luminance value between the reference frame image and the other frame.   The image processing apparatus according to claim 1, wherein the ratio calculation unit increases the synthesis ratio of the other frame image as the reliability of the background area is higher.   The image processing apparatus according to claim 3, wherein the reliability of the background area is higher as the area of the area detected as the background is larger.   5. The image processing apparatus according to claim 3, wherein the reliability of the background area increases as the amount of texture of the subject determined by the standard deviation of the image signal indicating the image increases.   The image processing apparatus according to any one of claims 1 to 5, wherein the extraction unit extracts a region where the amount of movement of the subject is small between the reference frame image and the other frame image. The extraction means comprises coefficient calculation means and processing means,
The coefficient calculation means calculates a geometric transformation coefficient for correcting the other frame image based on the position of the reference frame image.
The processing means performs geometric deformation processing on the other frame image using the geometric conversion coefficient, and the motion amount of the subject between the reference frame image and the other frame image after the geometric deformation processing is The image processing apparatus according to any one of claims 1 to 6, wherein a small area is extracted as the background area. The extraction means comprises coefficient calculation means
The coefficient calculation means calculates a geometric transformation coefficient for correcting the other frame image to the position of the reference frame image.
7. The image processing apparatus according to claim 1, wherein an area having a movement amount of an object between the reference frame image having a high correlation with the geometric transformation coefficient and the other frame image is extracted as the background area. The image processing apparatus according to claim 1. An acquisition step of acquiring a reference frame image and a plurality of other frame images;
A change amount calculation step of obtaining change amounts of signals of the plurality of other frame images with respect to the reference frame image;
Extracting the background area by comparing the reference frame image and the other frame image;
A ratio calculating step of calculating a synthesis ratio of the plurality of other frame images based on a change amount of a signal with respect to the reference frame image of the plurality of other frame images;
And combining the two images of interest at the combining ratio calculated in the ratio calculating step;
The image processing method characterized in that, in the ratio calculating step, the combining ratio is calculated by giving larger weight to the other frame image having a smaller amount of change.   A computer-executable program in which the procedure of the image processing method according to claim 9 is described.   A computer readable storage medium storing a program for causing a computer to execute the steps of the image processing method according to claim 9.

Images