JP6063680B2 - Image generation apparatus, image generation method, imaging apparatus, and imaging method - Google Patents

Description

  The present invention relates to an image generation device, an image generation method, an imaging device, and an imaging method.

  Conventionally, among pixel values located at the same coordinates included in a plurality of images (hereinafter referred to as “pixel values”), pixel values (hereinafter referred to as “singular”) that are significantly different from the pixel values of many other images. By removing “value”), there is a function of moving object removal that generates one image from a plurality of images. Moving object removal refers to a desired object that removes unnecessary moving objects such as passers-by and vehicles that cross between the camera and the subject when a subject such as a building is photographed with a still image camera (hereinafter referred to as “camera”). This is a function for generating an image.

  In moving object removal, when a moving object crosses the front of the camera, the number of moving objects that appear in the same position in multiple images taken consecutively is small, and many images have a background. Based on this idea, an image is generated by removing a subject that moved during the shooting time from a plurality of continuously shot images. More specifically, a moving object is obtained by substituting a pixel value of a moving object with a small number of times appearing at the same position in a plurality of images as a singular value and a background pixel value with a large number of images appearing at the same position. An image of only the background from which is removed is generated.

  As one method for removing moving objects, for example, a technique as disclosed in Patent Document 1 is disclosed. In the technique disclosed in Patent Document 1, a histogram is created for pixel values at the same position in a plurality of captured images that are temporally continuous, and the number of times the same pixel value appears (hereinafter referred to as “frequency”) is the highest. Output the pixel value. Then, by making a pixel value having a high frequency at each position of the captured image a final pixel value, one image from which the moving body is removed is generated.

JP-A-9-149364

  However, even in the case of a pixel in which the background is actually shown, the frequency of the histogram is reduced for a pixel in which the number of images shown is small. In such a case, as in the technique disclosed in Patent Document 1, a method of simply creating a histogram of pixel values at the same position and selecting a background pixel cannot select a background pixel correctly. The problem is that moving objects cannot be removed correctly.

  The present invention has been made based on the above problem recognition, and an image that can remove moving objects such as a moving object by correctly selecting even background pixels with a small number of captured images. An object is to provide a generation device, an image generation method, an imaging device, and an imaging method.

In order to solve the above problem, an image generation apparatus of the present invention is an image generation apparatus that generates an output image including only a background from which a moving object is removed from a plurality of input images having the same angle of view. A pixel to be processed at present, each pixel arranged at the same position in each input image as a target pixel, and a pixel arranged at the same position as the target pixel in the output image is a moving object removal pixel Then, a reference value for determining information on the moving object removal pixel is generated based on information on a plurality of surrounding pixels which are pixels arranged around each of the target pixels currently being processed. Based on the reference value generation unit and the information on each of the target pixels that are currently processed based on the reference value, the information on the target pixel that is likely to be the background is selected. Comprising a body removed pixel generator for outputting the selected information of the pixel of interest as the information of the moving object removed pixel in the output image, the said reference value generation unit, and the pixel values of each of said pixel of interest and elements Based on the reliability, a reliability generation unit that generates a first histogram, sets a maximum frequency value in the generated first histogram as a reliability, and generates a pixel value of the maximum frequency as a mode value. A reference range setting unit for setting a reference range that is a range including the surrounding pixels used for generating the reference value corresponding to the pixel of interest currently being processed, and within the set reference range A reference value calculation unit that calculates the reference value based on the reliability and the mode value generated by the reliability generation unit when each of the surrounding pixels is processed as the target pixel. Characterized by Rukoto equipped with.

  Further, the reference range setting unit of the present invention determines in advance the reliability generated by the reliability generation unit when each of the surrounding pixels included in the set reference range is processed as the target pixel. The number of the surrounding pixels having a reliability equal to or higher than a first threshold value of the reliability is counted, and the reference range is set so that the counted number of the surrounding pixels is equal to or more than a threshold value of a predetermined number of pixels. It is characterized by spreading sequentially around.

  Further, the reference value calculation unit according to the present invention is configured such that the reliability when each of the surrounding pixels included in the reference range is processed as the target pixel is equal to or more than a first threshold value of a predetermined reliability. Generating a second histogram having the mode value of each of the surrounding pixels as reliability as an element, and setting the mode value of the maximum frequency in the generated second histogram as the reference value; Features.

  Further, the reference value calculation unit of the present invention is configured so that the distance is based on a distance based on the position of the target pixel currently being processed and the positions of the surrounding pixels included in the reference range. A weighting coefficient that is larger as the distance is closer and smaller as the distance is farther is set, and the set weighting coefficient is multiplied by each mode value that is an element of the second histogram to generate the second histogram. It is characterized by.

  In addition, the reference value calculation unit of the present invention, according to the variation of the mode value of each of the surrounding pixels included in the reference range, the smaller the variation, the smaller the difference according to the distance, The weighting coefficient is set such that the difference corresponding to the distance increases as the variation increases.

  In addition, the moving object removal pixel generation unit of the present invention may be configured such that, based on the reference value, the pixel value of the target pixel that is highly likely to be the background is selected from the pixel values of the target pixel currently being processed. A pixel value detection unit that detects a pixel value, and a pixel value of the pixel of interest detected by the pixel value detection unit based on the reliability, or a mode value generated by the reference value generation unit A moving object removal pixel selecting unit that selects one of the values as a pixel value of the moving object removal pixel.

  In addition, the pixel value detection unit of the present invention has a high possibility that the pixel value closest to the reference value is the background among the pixel values of the pixel of interest currently being processed. The pixel value of the target pixel is detected.

  The moving object removal pixel selection unit of the present invention selects the mode value as the pixel value of the moving object removal pixel when the reliability is equal to or higher than a second threshold value of the predetermined reliability. When the reliability is smaller than a second threshold value of a predetermined reliability, the pixel value of the target pixel detected by the target pixel value detection unit is selected as a pixel value of the moving object removal pixel. Features.

  In addition, the reference value generation unit of the present invention is arranged such that one or both of luminance information indicating the luminance of the input image and color information indicating the color of the input image, which are arranged at each position in the input image. The reference value is generated based on the data.

  Further, the reference value generation unit of the present invention is arranged based on data of a value corresponding to a signal output from the image sensor when acquiring the input image, which is arranged at each position in the input image. A reference value is generated.

  The image pickup apparatus of the present invention is an image pickup apparatus that takes an image of a subject, and includes an image pickup element in which a plurality of pixels that output signals according to the amount of incident light are arranged in a two-dimensional matrix, An imaging unit that outputs an image based on a signal when an optical image of a subject formed on the element is exposed, a storage unit that stores a plurality of the images output from the imaging unit, and a storage unit A positional shift detection unit that detects a shift of a shooting position caused by a shake when shooting with the imaging device included in each of the stored images, and a shift of the shooting position detected by the positional shift detection unit are corrected. The image generation apparatus according to any one of the above inventions, wherein a plurality of the images are input as input images, and an output image including only a background from which a moving object is removed is generated from the input plurality of input images. With And wherein the door.

  Further, the image generation device of the present invention generates the output image by removing a moving body portion having a small number of images appearing at the same position in a plurality of input images as the moving body. It is characterized by.

The image generation method of the present invention is an image generation method for generating an output image including only a background from which a moving object is removed from a plurality of input images having the same angle of view, and is a pixel to be currently processed. And when each pixel arranged at the same position in each input image is a target pixel and a pixel arranged at the same position as the target pixel in the output image is a moving object removal pixel, A reference value generating step for generating a reference value for determining information of the moving object removal pixel based on information of a plurality of surrounding pixels which are pixels arranged around each of the target pixels performing Based on the reference value, information on the target pixel that is highly likely to be the background is selected from among the information on the target pixel currently being processed, and the selected Seen containing a moving object removed pixel generating step for outputting information eye pixel as information of the moving object removed pixel in the output image, the said reference value generating step, first and the pixel value of each of the target pixel as Element 1 A reliability generation step for generating a maximum frequency value in the generated first histogram as a reliability, and generating a pixel value of the maximum frequency as a mode value, and based on the reliability, A reference range setting step for setting a reference range, which is a range including the surrounding pixels used for generating the reference value corresponding to the target pixel being processed, and each of the set reference ranges Based on the reliability and the mode value generated by the reliability generation step when the surrounding pixels are processed as the target pixel, the reference value A reference value calculating step of calculating, the characterized by containing Mukoto.

  In the moving object removal pixel generation step of the present invention, based on the reference value, the pixel value of the target pixel that is likely to be the background is selected from the pixel values of the target pixel that are currently processed. A target pixel value detecting step for detecting a pixel value, and a pixel value of the target pixel detected by the target pixel value detecting step or a mode value generated by the reference value generating step based on the reliability. A moving object removal pixel selecting step of selecting one of the values as a pixel value of the moving object removal pixel.

  Further, the imaging method of the present invention is an imaging method by an imaging device that photographs a subject, and includes an imaging device in which a plurality of pixels that output signals according to the amount of incident light are arranged in a two-dimensional matrix. An imaging step for causing the imaging unit to output an image based on a signal when the optical image of the subject imaged on the imaging element is exposed; and a plurality of the images output from the imaging unit to the storage unit A storage step for storing, and a position shift detection unit for detecting a shift of a shooting position caused by a shake when shooting with the imaging device included in each of the images stored in the storage unit, In the image generation apparatus according to any one of the above inventions, a plurality of the images, in which the photographing position deviation detected by the positional deviation detection unit is corrected, are input as input images, and the plurality of the inputted plurality of the images From the force image, characterized in that it comprises an image generating step of generating an output image including only the background removal of the moving object, the.

  According to the present invention, it is possible to remove a moving object such as a moving object by correctly selecting even background pixels with a small number of captured images.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus to which an image generation apparatus according to an embodiment of the present invention is applied. It is the block diagram which showed schematic structure of the image generation apparatus of this embodiment. It is the flowchart which showed the procedure of the moving body removal process in the image generation apparatus of this embodiment. It is a figure explaining an example of the process which produces | generates a reliability and a mode value in the moving body removal process by the image generation apparatus of this embodiment. It is a figure explaining an example of the process which sets a reference range in the moving body removal process by the image generation apparatus of this embodiment. It is a figure explaining an example of the coefficient used in the moving body removal process by the image generation apparatus of this embodiment. It is a figure explaining an example of the process which determines a reference value in the moving body removal process by the image generation apparatus of this embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, an example in which the image generation apparatus of the present embodiment is applied to an imaging apparatus such as a digital still camera will be described. FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus to which the image generation apparatus according to the present embodiment is applied. The imaging device 10 illustrated in FIG. 1 includes a control unit 11, an imaging unit 12, a storage unit 13, a position shift detection unit 14, an image processing unit 15, a display unit 16, a recording unit 17, and moving object removal. Part 20.

  The control unit 11 controls the entire imaging device 10 by controlling each component in the imaging device 10. The imaging unit 12 includes a lens, an imaging device, and the like, and exposes an optical image of a subject formed on the imaging device via the lens in accordance with control from the control unit 11. Then, the imaging unit 12 outputs image data converted into an image signal according to the amount of light exposed by the imaging element to the storage unit 13.

  The storage unit 13 is a memory such as a DRAM (Dynamic Random Access Memory) that temporarily stores image data input from the imaging unit 12 in accordance with control from the control unit 11. The storage unit 13 stores image data for the number of sheets used by the moving body removing unit 20 for the process of removing the moving body. Each image data stored in the storage unit 13 is output to the moving body removing unit 20 in accordance with control from the control unit 11. Further, the image data stored in the storage unit 13 is output to the position shift detection unit 14 according to the control from the control unit 11, and the position shift information input from the position shift detection unit 14 is stored.

  In accordance with control from the control unit 11, the position shift detection unit 14 calculates a shift amount for matching the angle of view with respect to all the image data stored in the storage unit 13. Then, the positional deviation detection unit 14 outputs the calculated deviation amount to the storage unit 13 as positional deviation information. When the moving body removing unit 20 reads out the image data stored in the storage unit 13, the control unit 11 determines the position of each pixel in the image data based on the positional shift information stored in the storage unit 13. The reading of the image data from the storage unit 13 is controlled so as to be the position when the same place of the image is exposed. Thereby, the pixel value of each pixel arranged at the same position in all the image data input to the moving body removing unit 20 becomes a value when the same place of the optical image of the subject is exposed.

  The moving object removing unit 20 performs moving object removal processing on all image data in which the positional deviation input from the storage unit 13 is detected in accordance with the control from the control unit 11, and removes the moving object included in the image data. It is the image generation device of this embodiment which produces | generates the image data which carried out. The moving object removing unit 20 removes, as a moving object, pixel values of unnecessary moving objects such as passers-by and vehicles that cross the front of a fixed subject such as a building (hereinafter referred to as “fixed object”), for example, by moving object removal processing. Generate image data.

  The image data after moving object generation generated by the moving object removing unit 20 is image data equivalent to image data in which only the background is copied. In the following description, the image data of the output image after moving object removal generated by the moving object removing unit 20 is referred to as “moving object removal image”. The moving object removal unit 20 outputs the generated moving object removal image to the image processing unit 15. A detailed description of the moving body removing process in the moving body removing unit 20 will be described later.

  The image processing unit 15 generates image data obtained by performing various predetermined image processing (development) on the moving object removal image input from the moving object removal unit 20 in accordance with control from the control unit 11. . Then, the image processing unit 15 outputs the generated image data after image processing to a corresponding component in the imaging device 10. For example, the image processing unit 15 outputs image data for display generated by performing image processing for displaying image data to the display unit 16. Further, for example, the image processing unit 15 stores image data for recording generated by performing image processing for converting into an image format for recording image data (for example, JPEG: Joint Photographic Experts Group) in the recording unit 17. Output.

  The display unit 16 includes a display device such as an LCD (Liquid Crystal Display) that displays image data, and displays image data for display input from the image processing unit 15 in accordance with control from the control unit 11. indicate. The recording unit 17 records the recording image data input from the image processing unit 15 on an image recording medium such as a memory card in accordance with the control from the control unit 11.

  Next, the moving body removal unit 20 that is the image generation apparatus of the present embodiment provided in the imaging apparatus 10 will be described. FIG. 2 is a block diagram showing a schematic configuration of the moving object removing unit 20 which is the image generating apparatus of the present embodiment. As illustrated in FIG. 2, the moving object removal unit 20 includes a reference value generation unit 21 and a moving object removal pixel generation unit 22. In FIG. 2, the connection between the moving body removing unit 20, the storage unit 13, and the image processing unit 15 in the imaging apparatus 10 is also shown.

  The moving body removing unit 20 adjusts the reading position of the image data so as to correct the positional deviation based on the positional deviation information stored in the storage unit 13 in accordance with the control from the control unit 11, and stores it in the storage unit 13. Each of the plurality of stored image data is read out as an input image. Then, the moving body removing unit 20 outputs one piece of image data generated by performing the moving body removing process to the image processing unit 15 as a moving body removed image. In the moving object removing process by the moving object removing unit 20, in each input image used for the moving object removing process, the pixel value of each pixel of the moving object removed image is selected from the pixel values of the pixels arranged at the same coordinate position. Select. At this time, the moving object removing unit 20 determines the pixel value of the pixel currently being processed based on the value calculated by referring to the pixel value of the pixel arranged around the coordinate position of the pixel currently being processed. The pixel value of the pixel of the moving object removal image may be selected from among them. Then, each component in the moving object removing unit 20 repeats the moving object removing process for each pixel the same number of times as the number of pixels of the input image.

  In the following description, the target pixel that is currently subjected to moving object removal processing that is arranged at the same coordinate position in each input image is referred to as a “target pixel”, and the moving object removal that is generated by performing the moving object removal processing. A pixel arranged at the same coordinate position as the target pixel in the image is referred to as a “moving object removal pixel”. In addition, a pixel arranged around the coordinate position of the pixel of interest currently being processed is referred to as a “peripheral pixel”, and a value calculated by referring to the pixel value of the peripheral pixel is referred to as a “reference value”.

  Note that the pixel value of the target pixel is a value (for example, a value of RAW data) based on an image signal corresponding to the amount of light exposed by the imaging element in the imaging unit 12. The pixel value of the target pixel may be a value of luminance information representing the luminance of the image or a value of color information representing the color of the image obtained by performing image processing on the value based on the above-described image signal. . In the following description, a value based on an image signal, or one or both of a luminance information value and a color information value will not be distinguished, and will be described simply as a “pixel value”.

  The reference value generation unit 21 generates a histogram of pixel values of each target pixel arranged at the same coordinate position in a plurality of input images input from the storage unit 13. Then, the reference value generation unit 21 outputs the maximum frequency value in the histogram of the pixel value of the generated target pixel as “reliability” and outputs the maximum frequency pixel value as the “mode” to the moving object removal pixel generation unit 22. To do. Further, the reference value generation unit 21 is reference information that is reference information for determining a moving object removal pixel based on the reliability and the mode value of surrounding pixels arranged around the pixel of interest currently processed. A value is calculated, and the calculated reference value is output to the moving object removal pixel generation unit 22. The reference value generation unit 21 includes a reliability generation unit 211, a reference range setting unit 212, and a reference value calculation unit 213.

  The reliability generation unit 211 generates a histogram of the pixel value of the target pixel (hereinafter referred to as “histogram Hist1”) based on the pixel value of each target pixel of the plurality of input images input from the storage unit 13. For example, when generating a histogram Hist1 corresponding to a pixel of interest in the input image, the reliability generation unit 211 of each pixel of interest arranged at the same coordinate position in the first to Nth input images. The histogram Hist1 is generated by counting the number of pixels having the same pixel value (frequency) for each pixel value. Then, the reliability generation unit 211 outputs the reliability and the mode value of the target pixel obtained from the generated histogram Hist1 to the moving object removal pixel generation unit 22. Further, the reliability generation unit 211 outputs the reliability obtained from the histogram Hist1 to the reference range setting unit 212, and outputs the reliability and the mode value to the reference value calculation unit 213.

  Note that the method of generating the reliability and the mode value by the reliability generation unit 211 is not limited to the method of generating the histogram Hist1 and obtaining the reliability and the mode value as in the example described above. Another method may be used to generate the reliability and mode value of each pixel of interest of a plurality of input images input from the storage unit 13.

  The reference range setting unit 212 is a range of surrounding pixels arranged around the target pixel that is currently being processed, which is used to calculate a reference value based on the reliability input from the reliability generation unit 211. (Hereinafter referred to as “reference range”). The reference range is a rectangular area centered on the coordinate position of the target pixel. When the reference range setting unit 212 determines the reference range, the range is sequentially expanded by one pixel around the coordinate position of the target pixel. Then, the reference range setting unit 212 determines an appropriate reference range using the reliability obtained from the histogram Hist1 generated by the reliability generation unit 211 when each of the surrounding pixels in the reference range is the target pixel. .

  More specifically, the reference range setting unit 212 sets a rectangular area (9 pixel area including the target pixel) for each pixel around the target pixel (up, down, left, and right) as the first reference range. Then, the reference range setting unit 212 compares the reliability of each of the surrounding pixels included in the reference range with a first threshold value of the predetermined reliability (hereinafter referred to as “reliability threshold value thrT1”). The number of surrounding pixels having a reliability equal to or greater than the degree threshold thrT1 is counted. Subsequently, the reference range setting unit 212 compares the number of surrounding pixels having a reliability equal to or higher than the reliability threshold value thrT1 with a predetermined threshold value for the number of pixels (hereinafter referred to as “pixel number threshold value thrC”). When the number of surrounding pixels having reliability equal to or greater than the threshold value thrT1 is equal to or greater than the pixel number threshold value thrC, the current reference range is determined to be a range used for calculating the reference value. On the other hand, when the number of surrounding pixels having the reliability equal to or higher than the reliability threshold value thrT1 is smaller than the pixel number threshold value thrC, a rectangular area (target pixel) extended by one pixel around the current reference range (up, down, left, and right) Is determined as a range to be used for calculating a reference value. Such setting and determination of the reference range is repeated, and the reference range is sequentially expanded by one pixel around the periphery until the number of surrounding pixels having a reliability equal to or higher than the reliability threshold thrT1 becomes equal to or higher than the pixel number threshold thrC. .

  In this way, when the reference range including the target pixel is sequentially expanded and the reference range used to calculate the reference value is determined, the reference range setting unit 212 displays information indicating the size of the determined reference range. It outputs to the reference value calculation part 213.

  The reference value calculation unit 213 selects a moving object removal pixel based on the reference range input from the reference range setting unit 212 and the reliability and mode value input from the reliability generation unit 211. Is calculated. Then, the reference value calculation unit 213 outputs the calculated reference value to the moving object removal pixel generation unit 22.

  In the calculation of the reference value by the reference value calculation unit 213, the number (frequency) of the surrounding pixels having the reliability equal to or higher than the reliability threshold thrT1 among the surrounding pixels in the reference range input from the reference range setting unit 212 is set. A histogram of the mode value (hereinafter referred to as “histogram Hist2”) is generated by counting for each mode value. At this time, the reference value calculation unit 213 has a larger frequency as the distance between each surrounding pixel and the target pixel is closer, that is, the mode value where the coordinate position of the surrounding pixel and the coordinate position of the target pixel are closer. Thus, the histogram Hist2 is generated with weights. In the weighting for the frequency of each mode value, the variance (variation) of the mode value of surrounding pixels having the reliability equal to or higher than the reliability threshold value thrT1 is calculated, and the larger the calculated variance value, the higher the weighting. In this manner, the characteristic of the weighting coefficient with respect to the frequency of the mode value is inclined. Then, the reference value calculation unit 213 outputs the mode value of the maximum frequency in the generated histogram Hist2 to the moving object removal pixel generation unit 22 as a reference value.

  The moving object removal pixel generation unit 22 is the same in a plurality of input images input from the storage unit 13 based on the reliability of the target pixel input from the reference value generation unit 21 and the reference value calculated from the surrounding pixels. One value (pixel value) is selected from the pixel value of each pixel of interest arranged at the coordinate position and the mode value input from the reference value generation unit 21. Then, the moving object removal pixel generating unit 22 uses the selected one value (pixel value) as the pixel value of the moving object removal pixel arranged at the same coordinate position as the target pixel in the moving object removal image. Output. The moving object removal pixel generation unit 22 includes a target pixel value detection unit 221 and a moving object removal pixel selection unit 222.

  The target pixel value detection unit 221 selects a target pixel having a pixel value closest to the reference value input from the reference value generation unit 21 among the target pixels of the plurality of input images input from the storage unit 13. To do. For example, when selecting the pixel value of the target pixel, the target pixel value detection unit 221 calculates the difference between the pixel value of each target pixel and the reference value of the first to Nth input images by the number of input images ( N pixels) is calculated, and the target pixel having the smallest difference value for the calculated N sheets is selected. Then, the target pixel value detection unit 221 outputs the pixel value of the selected target pixel to the moving object removal pixel selection unit 222.

  Based on the reliability input from the reference value generation unit 21, the moving object removal pixel selection unit 222 receives the pixel value of the target pixel input from the target pixel value detection unit 221 or the maximum value input from the reference value generation unit 21. One of the frequent values (pixel value) is output to the image processing unit 15 as the pixel value of the moving object removal pixel. More specifically, when the reliability input from the reference value generation unit 21 is equal to or higher than a predetermined second threshold value of reliability (hereinafter referred to as “reliability threshold value thrT2”), a reference value generation is performed. When the mode value input from the unit 21 is selected and the reliability input from the reference value generation unit 21 is smaller than the reliability threshold thrT2, the pixel of the target pixel input from the target pixel value detection unit 221 Select a value. Then, the moving object removal pixel selection unit 222 outputs the selected value (pixel value) to the image processing unit 15 as the pixel value of the final moving object removal pixel that has been subjected to the moving object removal processing by the moving object removal unit 20.

  As described above, the moving object removing unit 20 moves the moving object based on the pixel values of the respective target pixels arranged at the same coordinate position and the pixel values of the surrounding pixels in all the input images input from the storage unit 13. The removal process is sequentially performed, and the pixel value of the pixel of the moving object removal image is output to the image processing unit 15.

  The reliability threshold value thrT1, the pixel number threshold value thrC, and the reliability threshold value thrT2 are determined according to the number of input images. At this time, since the reliability threshold value thrT1 is a threshold value for surrounding pixels arranged at different peripheral coordinate positions from the target pixel, in order to further increase the reliability, the reliability threshold value thrT1 is higher than the reliability threshold value thrT2 that is a threshold value for the target pixel. It is conceivable to increase the value. Thereby, the reliability of the surrounding pixels in the reference range for generating the histogram Hist2 is increased, and the pixel values of the pixels of the moving object removal image are set based on the surrounding pixels having a higher reliability input from the reference value generation unit 21. The pixel value of a pixel that can be selected and is highly likely to be the background can be selected with higher accuracy. In addition, when the number of input images is large, it may be possible to perform the moving body removal process in a state where the difference between the reliability threshold value thrT1 and the reliability threshold value thrT2 is increased and the reliability with respect to surrounding pixels is further increased.

  Note that, for example, since the number of input images is small, the reliability of the target pixel is a low value, and if a difference is provided between the reliability threshold value thrT1 and the reliability threshold value thrT2, the reliability cannot be correctly determined. If there is a possibility, the reliability threshold value thrT1 and the reliability threshold value thrT2 can be set to the same value. In this case, in the reference range setting unit 212 and the reference value calculation unit 213, the surrounding pixels having the reliability higher than the reliability threshold thrT1 (= thrT2) are set as the surroundings of the target in the determination of the reference range and the generation of the histogram Hist2. By setting it as a pixel, the pixel value of the pixel of the moving object removal image can be selected based on surrounding pixels with higher reliability.

  Next, the procedure of the moving body removal process in the moving body removal unit 20 that is the image generation apparatus of the present embodiment provided in the imaging device 10 will be described. In the moving object removal processing by the moving object removal unit 20, the reference value generation unit 21 sets the reference range and calculates the reference value while sequentially generating the reliability and the mode value of each pixel of interest, and the moving object removal pixel generation unit 22 selects and outputs the pixel value of the moving object removal pixel arranged at the same coordinate position as the target pixel based on the reliability and the reference value of each target pixel. In the following description, for ease of explanation, first, the reliability generation unit 211 in the reference value generation unit 21 generates the reliability and mode value of all the target pixels, and then the reference value generation unit The reference range setting unit 212 and the reference value calculation unit 213 in FIG. 21 set the reference range and calculate the reference value, and the moving object removal pixel generation unit 22 selects and outputs the pixel value of the moving object removal pixel. I do. At this time, the reliability generation unit 211 temporarily stores the reliability and mode value of all generated target pixels in a temporary storage unit (not shown), and the reference range setting unit 212 and the reference value calculation unit 213. Each of the moving object removal pixel generation unit 22 reads the reliability and the mode value stored in a temporary storage unit (not shown), and executes each process.

  FIG. 3 is a flowchart showing the procedure of the moving object removing process in the moving object removing unit 20 which is the image generating apparatus of the present embodiment. 4 to 7 are diagrams for explaining an example of each process in the moving object removing process by the moving object removing unit 20 which is the image generating apparatus according to the present embodiment. In the description of the flowchart of the moving object removal process illustrated in FIG. 3, it is unnecessary to refer to an example of each process in the moving object removal process illustrated in FIGS. 4 to 7 from five (frame) input images. An example in the case of generating one moving object removal image in which the pixel value of a moving object is removed as a moving object will be described.

  When the moving object removing unit 20 starts the moving object removing process in accordance with the control from the control unit 11, the reference value generating unit 21 determines the reliability and mode value of the first pixel (target pixel) from step S1. Start the generation process. When the process of generating the reliability and mode value of the first target pixel is started, a plurality of reliability generation units 211 in the reference value generation unit 21 are stored from the storage unit 13 (in this description, in step S1). Acquires the pixel values In1 (x, y) to In5 (x, y) of the first pixel of interest arranged at the same coordinate position (x, y) in the input image of (5 frames).

  Subsequently, in step S2, the reliability generation unit 211 generates the histogram Hist1 of the five pixel values In1 (x, y) to In5 (x, y) of the acquired first target pixel.

  Subsequently, in step S3, the reliability generation unit 211 stores the reliability MaxFreq (x, y), which is the maximum value of the frequency in the generated histogram Hist1, in a temporary storage unit (not illustrated).

  Subsequently, in step S4, the reliability generation unit 211 stores the mode value MaxFreqValue (x, y), which is the maximum frequency pixel value in the generated histogram Hist1, in a temporary storage unit (not shown).

  The process so far is the process of generating the reliability and mode value of the first target pixel of the input image. Thereafter, the reliability generation unit 211 performs the process of generating the reliability and mode value of one target pixel of the input image in steps S1 to S4, and generates the reliability and mode value for all the pixels of the input image. Repeat until processing is complete. After that, when the process from step S1 to step S4 is repeated as many times as the number of pixels of the input image, that is, when the process of generating the reliability and mode value for all the pixels of the input image is completed, in step S5 Exit the processing loop.

  Here, an example of the reliability and mode value generation processing for one target pixel of the input image from step S1 to step S4 performed by the reliability generation unit 211 will be described. FIG. 4 is a diagram for explaining an example of a process for generating the reliability and the mode value in the moving object removal process by the image generation apparatus of the present embodiment.

  In step S1, the reliability generation unit 211 selects five pixel values In1 (x) of the target pixel arranged at the same coordinate position (x, y) in the five-frame input image illustrated in FIG. , Y) to In5 (x, y). Then, in step S2, the reliability generation unit 211 obtains the histogram Hist1 of the acquired five pixel values In1 (x, y) to In5 (x, y) of the target pixel as illustrated in FIG. 4B. Generate. Then, as shown in FIG. 4C, the reliability generation unit 211 calculates the reliability MaxFreq (x, y) from the generated histogram Hist1 (in FIG. 4C, the reliability MaxFreq (x, y)). = 3) is stored in a temporary storage unit (not shown) (step S3), and the pixel value of the pixel of interest with the reliability MaxFreq (x, y) = 3 is set as the mode value MaxFreqValue (x, y) (not shown). Store in the temporary storage unit (step S4).

  When the generation processing of the reliability and the mode value for all the pixels of the input image is completed, the reference range setting unit 212 and the reference value calculation unit 213 in the reference value generation unit 21 and the moving object removal pixel generation unit 22 Each starts the selection processing of the pixel value of the first moving object removal pixel arranged at the same coordinate position (x, y) as the first pixel (target pixel) in the moving object removal image from step S6. . When the selection process of the pixel value of the first moving object removal pixel is started, in step S6, each of the reference range setting unit 212, the reference value calculation unit 213, and the moving object removal pixel generation unit 22 is not illustrated. The reliability MaxFreq (x, y) and the mode value MaxFreqValue (x, y) of the first target pixel stored in the temporary storage unit are read. In addition, the moving object removal pixel generation unit 22 has pixel values In1 (x, y) to In5 (x, y) of the first target pixel of a plurality of (in this description, five frames) input images from the storage unit 13. ) Get again.

  Subsequently, in step S7, the moving object removal pixel selection unit 222 in the moving object removal pixel generation unit 22 compares the reliability MaxFreq (x, y) of the first target pixel with the reliability threshold value thrT2. It is determined whether or not the reliability MaxFreq (x, y) of the pixel of interest of the eye is equal to or higher than the reliability threshold thrT2 (in this description, the reliability threshold thrT2 = 3).

  As a result of the determination in step S7, when the reliability MaxFreq (x, y) of the first pixel of interest is equal to or higher than the reliability threshold thrT2, in step S8, the moving object removal pixel selection unit 222 selects the first Is the pixel value of the first moving object removal pixel arranged at the same coordinate position (x, y) as the first attention pixel in the moving object removal image. To the image processing unit 15. That is, it is possible to select a pixel value (mode) of a pixel that is likely to be the background from only the five pixel values In1 (x, y) to In5 (x, y) of the first target pixel. If possible, the moving object removal pixel generation unit 22 does not use the reference value calculated based on the reliability and the mode value of the pixels around the first pixel of interest, and does not use the final moving object removal pixel. Select a pixel value.

  On the other hand, if the reliability MaxFreq (x, y) of the first pixel of interest is smaller than the reliability threshold thrT2 as a result of the determination in step S7, the final value is obtained using the reference value in step S9 and subsequent steps. It progresses to the process which determines the pixel value of a moving body removal pixel. That is, when only the five pixel values In1 (x, y) to In5 (x, y) of the first pixel of interest cannot select the pixel value of the pixel that is likely to be the background, The final pixel value of the moving object removal pixel is selected based on the reliability and the mode value of the surrounding pixels arranged around the first target pixel.

  In the process of determining the final pixel value of the moving object removal pixel using the reference value (hereinafter referred to as “pixel value determination process”), first, the reference range setting unit 212 performs a reference range setting process. In the reference range setting process, in step S9, the reference range setting unit 212 initializes a reference range size SizeR that is the number of pixels set as a reference range around the first pixel of interest currently being processed. Here, the reference range size SizeR = 1 is initialized. Thus, the reference range centered on the coordinate position (x, y) of the first target pixel is x-SizeR to x + SizeR in the x direction (horizontal direction), and y-SizeR to y + SizeR in the y direction (vertical direction). Is set in the range. That is, 9 pixels centered on the target pixel, which is a 3 pixel × 3 pixel rectangular area of one pixel (reference range size SizeR = 1) around the first target pixel (up / down / left / right). Set to range. In the following description, the coordinate position in the x direction (horizontal direction) of each surrounding pixel in the reference range is represented as “i”, and the coordinate position in the y direction (vertical direction) is represented as “j”. To do.

  Subsequently, in step S10, the reference range setting unit 212 reads the reliability MaxFreq (i, j) of the surrounding pixels for nine pixels in the reference range, which is stored in a temporary storage unit (not shown). Then, the reference range setting unit 212 compares the reliability MaxFreq (i, j) of the nine read pixels and the reliability threshold value thrT1, and determines the reliability MaxFreq (i, j) equal to or higher than the reliability threshold value thrT1. Count the number of surrounding pixels. In this description, since the reliability threshold value thrT1 is set to a value larger than the reliability threshold value thrT2 (reliability threshold value thrT1 = 4), the number of surrounding pixels with the reliability MaxFreq (i, j) ≧ 4 is counted. In the following description, the number of surrounding pixels counted by the reference range setting unit 212 is referred to as a pixel number count value CntT.

  Subsequently, in step S11, the reference range setting unit 212 compares the pixel count value CntT with the pixel count threshold thrC, and the pixel count value CntT, that is, the reliability MaxFreq (i, greater than or equal to the reliability threshold thrT1). It is determined whether or not the number of surrounding pixels having j) is equal to or greater than the pixel number threshold thrC (in this description, the pixel number threshold thrC = 5).

  If the result of determination in step S11 is that the pixel count value CntT is smaller than the pixel count threshold thrC, in step S12, the reference range setting unit 212 extends the reference range by one pixel around the periphery. That is, “1” is added to the reference range size SizeR so that the reference range size SizeR = 2. Thus, the reference range centered on the coordinate position (x, y) of the first target pixel is two pixels (reference range size SizeR = 2) around the first target pixel (up, down, left, and right). This is a range of 25 pixels centered on the pixel of interest, which is a range of a rectangular area of 5 pixels × 5 pixels. Then, returning to step S10, the reading of the reliability MaxFreq (i, j) of the surrounding pixels for 25 pixels within the reference range and the surrounding pixels having the reliability MaxFreq (i, j) equal to or higher than the reliability threshold thrT1. Count the number again.

  On the other hand, if the result of determination in step S11 is that the pixel count value CntT is greater than or equal to the pixel count threshold thrC, in step S13, the reference range setting unit 212 sets the rectangular area set by the current reference range size SizeR. Is determined as a reference range used for calculating a reference value. Then, the reference range setting unit 212 outputs information indicating the size of the determined reference range (for example, the size of the rectangular area determined as the reference range and the reference range size SizeR) to the reference value calculation unit 213.

  The processing from step S9 to step S13 so far is reference range setting processing by the reference range setting unit 212, which is first performed in the pixel value determination processing. Here, an example of reference range setting processing from step S9 to step S13 performed by the reference range setting unit 212 will be described. FIG. 5 is a diagram illustrating an example of processing for setting a reference range in the moving object removal processing by the image generation apparatus according to the present embodiment.

  In step S9, the reference range setting unit 212 initializes the reference range size SizeR = 1 as shown in FIG. Accordingly, the reference range centered on the target pixel indicated by “★” in FIG. 5A is a range of a rectangular area of 3 pixels × 3 pixels, one pixel around the target pixel (up, down, left and right). (9 pixel range centered on the target pixel). Here, it is assumed that the surrounding pixels having the reliability MaxFreq (i, j) equal to or higher than the reliability threshold value thrT1 are the two surrounding pixels indicated by “shaded” in FIG. For this reason, in step S10, the pixel count value CntT counted by the reference range setting unit 212 is the pixel count count CntT = 2. Therefore, the reference range setting unit 212 determines in step S11 that the reference range is expanded by one pixel around the periphery (No in step S11).

  Then, in step S12, the reference range setting unit 212 sets the reference range size SizeR = 2 as shown in FIG. 5B. As a result, the reference range centered on the target pixel indicated by “★” in FIG. 5B is a range of a rectangular area of 5 pixels × 5 pixels of two pixels around the target pixel (up, down, left and right). (Range of 25 pixels centered on the target pixel). In this reference range, in step S10, the pixel count value CntT obtained by counting the surrounding pixels indicated by “shaded” in FIG. 5B having the reliability MaxFreq (i, j) equal to or higher than the reliability threshold thrT1 is obtained. The pixel count value CntT = 6. Since this pixel count value CntT = 6 satisfies the condition of step S11 (Yes in step S11), the reference range setting unit 212 in step S13 is a rectangular area set by the current reference range size SizeR = 2. Is determined as a reference range used for calculating a reference value.

  When the reference range setting process by the reference range setting unit 212 is completed, the reference value calculation process by the reference value calculation unit 213 is then performed in the final pixel value determination process of the moving object removal pixel using the reference value. I do. In the reference value calculation process, in step S14, the reference value calculation unit 213, among the surrounding pixels in the reference range, the mode value MaxFreqValue of the surrounding pixels having the reliability MaxFreq (i, j) equal to or higher than the reliability threshold thrT1. The variance (variation) of (i, j) is calculated. As a result, the difference between the mode values MaxFreqValue (i, j) of each surrounding pixel that is highly likely to be the background within the reference range is known, and the background (for example, the background of a fine pattern) with a large change in the pixel value is known. It can be determined whether there is a background with little change in pixel value (for example, a background with little pattern). In the following description, the variance value calculated by the reference range setting unit 212 is referred to as a variance value Vari.

  Subsequently, in step S15, the reference value calculation unit 213 applies the weighting coefficient K1 (i) corresponding to the reliability MaxFreq (i, j) to the coordinate position of each surrounding pixel in the reference range for generating the histogram Hist2. , J). At this time, the reference value calculation unit 213 determines the weighting coefficient K1 (i, j) as the coordinate position (i, j) of the surrounding pixels in the reference range is closer to the center, that is, the coordinate position (x, y) of the target pixel. The weighting coefficient K1 (i, j) is decreased as the distance from the center increases. Further, the reference value calculation unit 213 increases the variance value Vari calculated by the reference range setting unit 212 with respect to the coordinate positions of surrounding pixels having the same distance from the coordinate position (x, y) of the target pixel. The weighting coefficient K1 (i, j) is decreased as the weighting coefficient K1 (i, j) is increased and the variance value Vari is decreased. That is, the larger the variance value Vari, the larger the difference in the weighting coefficient K1 (i, j) corresponding to the distance from the coordinate position (x, y) of the pixel of interest, and the smaller the variance value Vari, the coordinate position (x , Y), the difference in the weighting coefficient K1 (i, j) corresponding to the distance from is reduced.

  More specifically, the reference value calculation unit 213 calculates the weighting coefficient K1 (i, j) by the following equation (1) based on the variance value Vari calculated by the reference range setting unit 212, and the calculated weighting coefficient K1 (i, j) is set for the coordinate position of each surrounding pixel in the reference range.

  In the above equation (1), kVari represents a constant (hereinafter referred to as “constant kVari”) corresponding to the variance value Vari. Further, thrL represents a lower threshold value of the variance value Vari (hereinafter referred to as “lower threshold value thrL”), and thrU represents an upper threshold value of the variance value Vari (hereinafter referred to as “upper threshold value thrU”). The relationship between the lower threshold value thrL and the upper threshold value thrU is 0 <lower threshold value thrL <upper threshold value thrU.

  In the above equation (1), the weighting coefficient K1 (i, j) for the coordinate position of the surrounding pixel whose reliability MaxFreq (i, j) is smaller than the reliability threshold value thrT1 in each surrounding pixel in the reference range is expressed as the weighting coefficient. K1 (i, j) = 0. Thereby, the weighting coefficient K1 (i, j) is set for the coordinate position of each surrounding pixel in the reference range having the reliability MaxFreq (i, j) equal to or higher than the reliability threshold thrT1. Further, the weighting coefficient K1 (i, j) having the above-described relationship is set for the coordinate position of each surrounding pixel in the reference range where the reliability MaxFreq (i, j) is equal to or greater than the reliability threshold thrT1. The

  Here, the weighting coefficient K1 (i, j) represented by the above equation (1) set by the reference value calculation unit 213 will be described. FIG. 6 is a diagram for explaining an example of coefficients used in the moving object removal process by the image generation apparatus of the present embodiment. FIG. 6 shows the relationship between the variance value Vari and the distance from the target pixel, and the weighting coefficient K1.

  As can be seen from FIG. 6, the weighting coefficient K1 calculated by the above equation (1) is larger as the coordinate position of the surrounding pixel closer to the target pixel, and the coordinate position of the surrounding pixel farther from the target pixel. The relationship is inclined so that it becomes smaller. Further, as can be seen from FIG. 6, the weighting coefficient K1 has a larger slope as the variance value Vari is larger, and the slope is smaller as the variance value Vari is smaller. In the above equation (1), the characteristics of the three types of weighting coefficients K1 are set as shown in FIG. 6 by dividing the range of the variance value Vari by the lower threshold value thrL and the upper threshold value thrU. That is, in the above equation (1), the constant kVari represents the magnitude of the slope of the characteristic of the weighting coefficient K1.

  As described above, the reference value calculation unit 213 sets the weighting coefficient K1 (i, j), thereby allowing the variance (variation) of the mode value MaxFreqValue (i, j) of the surrounding pixels in the reference range and the reference range. The histogram Hist2 of the mode MaxFreqValue (i, j) in each surrounding pixel in the reference range can be generated in consideration of the reliability MaxFreq (i, j) of each surrounding pixel.

  Subsequently, in step S16, the reference value calculation unit 213 generates a histogram Hist2 of the mode value MaxFreqValue (i, j) of surrounding pixels in the reference range weighted by the weighting coefficient K1 (i, j). As described above, in each surrounding pixel within the reference range, the weighting coefficient K1 (i, j) of the surrounding pixels whose reliability MaxFreq (i, j) is smaller than the reliability threshold thrT1 is the weighting coefficient K1 ( i, j) = 0. Accordingly, the histogram Hist2 generated by the reference value calculation unit 213 is a histogram having, as elements, the mode value MaxFreqValue (i, j) of surrounding pixels having the reliability MaxFreq (i, j) equal to or higher than the reliability threshold thrT1.

  Subsequently, in step S17, the reference value calculation unit 213 determines whether or not the maximum frequency in the generated histogram Hist2 is one. As a result of the determination in step S17, if the maximum frequency in the generated histogram Hist2 is not one, the process returns to step S12, the reference range size SizeR is reset by the reference range setting unit 212, and steps S10 to S13. The reference range setting process and the reference value calculation process from step S14 to step S17 by the reference value calculation unit 213 are performed again.

  On the other hand, if the result of determination in step S17 is that the maximum frequency in the generated histogram Hist2 is 1, in step S18, the reference value calculation unit 213 determines the maximum frequency MaxFreqValue (i) in the generated histogram Hist2. , J) is output to the moving object removal pixel generation unit 22 as a reference value.

  The processing from step S14 to step S18 so far is reference value calculation processing by the reference value calculation unit 213 to be performed next in the pixel value determination processing. Here, an example of reference value calculation processing from step S16 to step S18 performed by the reference value calculation unit 213 will be described. FIG. 7 is a diagram illustrating an example of processing for determining a reference value in moving object removal processing by the image generation apparatus of the present embodiment. FIG. 7 shows the mode value MaxFreqValue (i, j) of surrounding pixels having a reliability MaxFreq (i, j) equal to or higher than the reliability threshold thrT1 among the surrounding pixels in the reference range shown in FIG. 5B. The histogram Hist2 is generated, and the maximum frequency mode MaxFreqValue (i, j) in the generated histogram Hist2 is determined as a reference value. In this description, six surrounding pixels (surrounding pixel A to surrounding pixel F) having a reliability MaxFreq (i, j) equal to or higher than the reliability threshold thrT1 are included in the reference range shown in FIG. The surrounding pixels A to C have the same mode value, and the surrounding pixels D to F have the same mode value.

  In step S16, the reference value calculation unit 213 generates a histogram Hist2 having the elements MaxFreqValue (i, j) of the surrounding pixels A to F in the reference range shown in FIG. To do. At this time, the reference value calculation unit 213 weights the frequency of the mode value MaxFreqValue (i, j) of the surrounding pixels A to F with the weighting coefficient K1 (i, j) set in step S15, and the histogram Hist2 Is generated, the histogram Hist2 shown in FIG. 7B is generated. In FIG. 7A, the weighting coefficients K1 (i, j) of the surrounding pixel A and the surrounding pixel B that are closer to the target pixel indicated by “★” are the peripheral pixels C to It is larger than the weighting coefficient K1 (i, j) of the pixel F.

  In step S17, the reference value calculation unit 213 determines whether the maximum frequency in the generated histogram Hist2 is one. In addition, as shown in FIG. 7B, the histogram Hist2 generated by the reference value calculation unit 213 in step S16 includes three surrounding pixels having the same mode value (peripheral pixels A to C and surrounding pixels). The pixel D to the surrounding pixels F) are weighted by the weighting coefficient K1 (i, j), and the maximum frequency is one.

  Therefore, in step S18, the reference value calculation unit 213 determines the mode value of the maximum frequency in the generated histogram Hist2, that is, the mode value MaxFreqValue (i) of the surrounding pixels A to C shown in FIG. , J) is output to the moving object removal pixel generation unit 22 as a reference value.

  Thus, by setting the weighting coefficient K1 (i, j) for the coordinate position of each surrounding pixel in the reference range, even when there are multiple types of background surrounding pixels in the reference range, A value closer to a pixel value that is highly likely to be the background at the coordinate position of the target pixel can be used as the reference value. Furthermore, for example, even if the pixel values of surrounding pixels arranged around the pixel of interest vary due to a fine change in the background pattern, the reference value is obtained from the surrounding pixels close to the coordinate position of the pixel of interest. Since it can be determined, it is possible to prevent a pixel value greatly different from the original background pixel value from being used as a reference value.

  In this description, the reference value calculation unit 213 uses the mode value MaxFreqValue (i, j) of the surrounding pixels in the reference range as an index representing the variation in the pixel values of the surrounding pixels arranged around the target pixel. The case where the variance (variation) of) is calculated has been described. However, the method for determining the variation in the pixel values of the surrounding pixels is not limited to dispersion. For example, a histogram of the mode MaxFreqValue (i, j) of the surrounding pixels having the reliability MaxFreq (i, j) equal to or higher than the reliability threshold thrT1 in the surrounding pixels in the reference range is generated, and the frequency is 1 in the generated histogram. The configuration may be such that the variation in pixel values is determined from the number of mode values as described above.

  When the pixel value determination process for the final moving object removal pixel using the reference value is completed, the moving object removal pixel generation unit 22 subsequently performs the final moving object removal process in the pixel value selection process for the moving object removal pixel. The pixel value of the pixel is selected and output to the image processing unit 15.

  More specifically, in step S <b> 19, the target pixel value detection unit 221 in the moving object removal pixel generation unit 22 is the first of a plurality of (5 frames in this description) input images acquired from the storage unit 13. Among the pixel values In1 (x, y) to In5 (x, y) of the target pixel of the target pixel, the target pixel value having the closest pixel value to the reference value input from the reference value calculation unit 213 in the reference value generation unit 21 The pixel is selected, and the pixel value of the selected target pixel is output to the moving object removal pixel selection unit 222. Then, the moving object removal pixel selection unit 222 arranges the pixel value of the attention pixel input from the attention pixel value detection unit 221 at the same coordinate position (x, y) as the first attention pixel in the moving object removal image. The pixel value of the first moving object removal pixel is output to the image processing unit 15.

  As described above, when the reliability MaxFreq (x, y) of the first target pixel is smaller than the reliability threshold thrT2 as a result of the determination in step S7, that is, the five pixel values of the first target pixel. If the pixel value of a pixel that is highly likely to be a background cannot be selected with only In1 (x, y) to In5 (x, y), the reference value input from the reference value generation unit 21 is the largest. The pixel value of the target pixel having a close pixel value is output as the final pixel value of the moving object removal pixel.

  The processing so far is the selection processing of the pixel value of the first moving object removal pixel arranged at the same coordinate position (x, y) as the first target pixel in the moving object removal image. Thereafter, the reference range setting unit 212, the reference value calculation unit 213, and the moving object removal pixel generation unit 22 in the reference value generation unit 21, respectively, the reliability MaxFreq (x, y) and the reliability of the target pixel in step S7. The comparison with the threshold value thrT2 and the selection of the final pixel value of the moving object removal pixel in accordance with the comparison results in step S8 or step S9 to step S19 are repeated until the processing for all the target pixels is completed. After that, when the process from step S7 to step S19 is repeated as many times as the number of pixels of the target pixel, that is, when the pixel value selection processing of moving object removal pixels corresponding to all the pixels of the input image is completed, step In S20, the process loop is exited, and the moving object removing process by the moving object removing unit 20 is completed.

  In this way, the moving object removing unit 20 performs the moving object removing process on each pixel of a plurality of (frame) input images, and sets the pixel value of each pixel of one moving object removed image to the image processing unit. 15 is output. As a result, the imaging apparatus 10 generates a single moving object removal image in which unnecessary moving object pixel values are removed as a moving object, and displays the image on the display device via the display unit 16 or image recording via the recording unit 17. Can be recorded on a medium.

  Each input image input to the moving body removing unit 20 is different for each input image even if the same subject is captured if the imaging conditions, image processing parameters, and the like are changed for each capturing. The pixel values are greatly different from each other. In such a case, the pixel value of the background pixel cannot be correctly selected in the moving object removal process by the moving object removal unit 20. For this reason, in shooting for obtaining a moving object-removed image, it is desirable that the imaging conditions such as the in-focus position and exposure, and the image processing parameters such as gradation conversion and white balance are the same for all input images.

  As described above, according to the embodiment for carrying out the present invention, based on the pixel value of the pixel (“target pixel” in the present embodiment) arranged at the same coordinate position in a plurality of input images. Based on the reliability generated in this way, the pixel value of the moving object removal pixel arranged at the same coordinate position as the target pixel in the moving object removal image is selected. At this time, in the embodiment for carrying out the present invention, when the reliability is high, the mode value of the target pixel is selected as the pixel value of the moving object removal pixel, and when the reliability is low, the surroundings of the target pixel are selected. The pixel value of the target pixel of any one input image is selected as the pixel value of the moving object removal pixel based on the reference value calculated from the pixel value of the pixel arranged in the pixel (“surrounding pixel” in the present embodiment) . Thus, in the embodiment for carrying out the present invention, when selecting the same subject appearing in the input image as the background, even if it is difficult to correctly specify the pixel value of the background only by the information of the target pixel, By using information on surrounding pixels arranged around, the probability that a background pixel value can be correctly selected can be improved and output as a pixel value of a moving object removal image generated by performing moving object removal processing. As a result, in the embodiment for carrying out the present invention, the moving object shown in the same position of the input image is removed, and only a fixed subject or background such as a building shown in the same position of the input image is removed. An extracted desired image (moving object removal image) can be generated.

  Further, according to the embodiment for carrying out the present invention, among the surrounding pixels arranged around the pixel of interest, the priority of the surrounding pixels arranged near the coordinate position of the pixel of interest is increased for reference. Calculate the value. Thereby, in the form for carrying out the present invention, for example, even when the pixel value of the surrounding pixels arranged around the pixel of interest varies due to a fine change in the background pattern, the pixel of interest An appropriate reference value for selecting the background pixel value of the target pixel can be calculated from the surrounding pixels arranged near the coordinate position. As a result, in the form for carrying out the present invention, for example, a fixed subject or background such as a building to be extracted as a moving object removal image is obtained when an input image is obtained because many unnecessary moving objects cross the front of the camera. Even in an area where the number of times the image is shown in the image is small, it is possible to generate a moving object removal image in which a fixed subject or background is extracted with higher accuracy.

  Further, according to the embodiment for carrying out the present invention, since the pixel value of the moving object removal pixel is not always selected based on the reference value calculated from the pixel values of the surrounding pixels, the processing speed of the moving object removal process is reduced. Can be reduced.

  In the present embodiment, the case where the reliability generation unit 211 obtains the reliability and the mode value of the target pixel by generating the histogram of the pixel value of the target pixel (histogram Hist1) has been described. However, the method of obtaining the reliability and mode value of the target pixel is not limited to the present embodiment, and the configuration may be such that the reliability and mode value of the target pixel are obtained by other methods. .

  In the present embodiment, an image generation apparatus that performs moving object removal processing using image data (captured image) captured by the imaging unit 12 included in the imaging apparatus 10 as an input image has been described. The input image on which the generation device performs the moving object removal process is not limited to the form for carrying out the present invention. For example, a plurality of image data obtained by photographing the same subject recorded on the image recording medium can be read out via the recording unit 17, and the moving object removal process can be performed using the read image data as an input image.

  The embodiment of the present invention has been described above with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes various modifications within the scope of the present invention. It is.

DESCRIPTION OF SYMBOLS 10 ... Imaging device 11 ... Control part 12 ... Imaging part 13 ... Memory | storage part 14 ... Position shift detection part 15 ... Image processing part 16 ... Display part 17 ... Recording Unit 20 ... Moving object removal unit (image generation device)
21 ... Reference value generation unit 211 ... Reliability generation unit 212 ... Reference range setting unit 213 ... Reference value calculation unit 22 ... Moving object removal pixel generation unit 221 ... Target pixel value detection unit 222 ... Moving object removal pixel selection unit

Claims (15)

An image generation device that generates an output image including only a background from which a moving object is removed from a plurality of input images having the same angle of view,
A pixel to be processed at present, each pixel arranged at the same position in each input image as a target pixel, and a pixel arranged at the same position as the target pixel in the output image is a moving object removal pixel When
A reference value generation unit that generates a reference value for determining information of the moving object removal pixel based on information of a plurality of surrounding pixels that are pixels arranged around each of the target pixels currently being processed. When,
Based on the reference value, information on the target pixel that is highly likely to be the background is selected from information on each target pixel currently being processed, and information on the selected target pixel is obtained. A moving object removal pixel generation unit that outputs the moving object removal pixel information in the output image;
Equipped with a,
The reference value generation unit
A first histogram having the pixel value of each pixel of interest as an element is generated, the maximum frequency value in the generated first histogram is set as a reliability, and the pixel value of the maximum frequency is generated as a mode value. A reliability generation unit;
A reference range setting unit that sets a reference range that is a range including the surrounding pixels used to generate the reference value corresponding to the pixel of interest currently being processed based on the reliability;
Reference for calculating the reference value based on the reliability and the mode value generated by the reliability generation unit when each of the surrounding pixels in the set reference range is processed as the target pixel A value calculator,
Image generating apparatus according to claim Rukoto equipped with. The reference range setting unit includes:
The reliability generated by the reliability generation unit when each of the surrounding pixels included in the set reference range is processed as the pixel of interest is a reliability equal to or higher than a first threshold value of a predetermined reliability. The number of the surrounding pixels is counted, and the reference range is sequentially expanded to the surroundings so that the counted number of the surrounding pixels is equal to or greater than a predetermined threshold number of pixels.
The image generating apparatus according to claim 1 . The reference value calculation unit
The reliability when each of the surrounding pixels included in the reference range is processed as the pixel of interest is a reliability equal to or higher than a first threshold value of a predetermined reliability. A second histogram having a frequency value as an element is generated, and the mode value of the maximum frequency in the generated second histogram is set as the reference value;
The image generating apparatus according to claim 1 or claim 2, characterized in that. The reference value calculation unit
According to the distance based on the position of the target pixel currently being processed and the positions of the surrounding pixels included in the reference range, the distance is larger and the distance is smaller. A weighting factor is set, and the set weighting factor is multiplied by each mode value that is an element of the second histogram to generate the second histogram.
The image generating apparatus according to claim 3 . The reference value calculation unit
According to the variation of the mode value of each of the surrounding pixels included in the reference range, the smaller the variation, the smaller the difference according to the distance, and the larger the variation, the larger the difference according to the distance. Set the weighting factor
The image generation apparatus according to claim 4 . The moving object removal pixel generation unit includes:
Based on the reference value, a pixel value detection unit that detects a pixel value of the pixel of interest that is highly likely to be the background from among pixel values of the pixel of interest currently being processed,
Based on the reliability, the pixel value of the target pixel value detected by the target pixel value detection unit or the mode value generated by the reference value generation unit is used as the pixel of the moving object removal pixel. A moving object removal pixel selection unit to select as a value;
The image generating apparatus according to any one of claims claims 1-5, characterized in that it comprises a. The target pixel value detection unit
A pixel value closest to the reference value is detected as a pixel value of the pixel of interest that is likely to be the background from among the pixel values of the pixel of interest currently being processed;
The image generation apparatus according to claim 6 . The moving object removal pixel selection unit includes:
When the reliability is equal to or higher than a second threshold value of a predetermined reliability, the mode value is selected as a pixel value of the moving object removal pixel,
When the reliability is smaller than a second threshold value of the predetermined reliability, the pixel value of the target pixel detected by the target pixel value detection unit is selected as the pixel value of the moving object removal pixel.
The image generating apparatus according to claim 6 or claim 7, characterized in that. The reference value generation unit
The reference value is generated based on either or both of luminance information representing the luminance of the input image and color information representing the color of the input image arranged at each position in the input image. ,
That the image generation apparatus according to any one of claims 8 from claim 1. The reference value generation unit
The reference value is generated on the basis of data of a value corresponding to a signal output from the image sensor when acquiring the input image, which is arranged at each position in the input image.
That the image generation apparatus according to any one of claims 8 from claim 1. An imaging device for photographing a subject,
A pixel that outputs a signal corresponding to the amount of incident light has a plurality of image sensors arranged in a two-dimensional matrix, and is based on a signal when an optical image of a subject formed on the image sensor is exposed. An imaging unit for outputting an image;
A storage unit for storing the plurality of images output from the imaging unit;
A position shift detection unit that detects a shift of a shooting position caused by a shake at the time of shooting with the imaging device included in each of the images stored in the storage unit;
A plurality of the images in which the deviation of the photographing position detected by the positional deviation detection unit is corrected are input as input images, and an output image including only the background from which the moving object is removed from the plurality of input images that are input. generating a, an image generating apparatus as claimed in any one of claims 1 0,
An imaging apparatus comprising: The image generation device includes:
Generating the output image by removing a moving body part having a small number of images appearing at the same position in a plurality of input images as the moving body;
The imaging apparatus according to claim 1 1, wherein the. An image generation method for generating an output image including only a background from which a moving object is removed from a plurality of input images having the same angle of view,
A pixel to be processed at present, each pixel arranged at the same position in each input image as a target pixel, and a pixel arranged at the same position as the target pixel in the output image is a moving object removal pixel When
A reference value generation step for generating a reference value for determining information of the moving object removal pixel based on information of a plurality of surrounding pixels which are pixels arranged around each of the target pixels currently being processed. When,
Based on the reference value, information on the target pixel that is highly likely to be the background is selected from information on each target pixel currently being processed, and information on the selected target pixel is obtained. A moving object removal pixel generating step to output as information of the moving object removal pixel in the output image;
Only including,
The reference value generation step includes:
A first histogram having the pixel value of each pixel of interest as an element is generated, the maximum frequency value in the generated first histogram is set as a reliability, and the pixel value of the maximum frequency is generated as a mode value. A confidence generation step;
A reference range setting step for setting a reference range that is a range including the surrounding pixels used to generate the reference value corresponding to the pixel of interest currently being processed based on the reliability;
A reference for calculating the reference value based on the reliability and the mode value generated by the reliability generation step when each of the surrounding pixels in the set reference range is processed as the target pixel. A value calculation step;
Image generation method comprising including Mukoto a. The moving object removal pixel generation step includes:
Based on the reference value, a pixel value detection step of detecting a pixel value of the pixel of interest that is highly likely to be the background from among pixel values of the pixel of interest currently being processed;
Based on the reliability, the pixel value of the target pixel detected by the target pixel value detection step or the mode value generated by the reference value generation step is used as the pixel of the moving object removal pixel. A moving object removal pixel selection step to select as a value;
Image generation method according to claim 1 3, characterized in that it comprises a. An imaging method by an imaging device for photographing a subject,
A signal when a pixel that outputs a signal corresponding to the amount of incident light is exposed to an optical image of a subject imaged on the imaging element on an imaging unit having a plurality of imaging elements arranged in a two-dimensional matrix. An imaging step for outputting an image based on
A storage step of storing a plurality of the images output from the imaging unit in a storage unit;
A position shift detection step of causing a position shift detection unit to detect a shift of a shooting position caused by a shake at the time of shooting with the imaging device included in each of the images stored in the storage unit;
The image generating apparatus according to any one of claims 1 0 to claim 1, the image shift is corrected in the detected shot position by the positional shift detection unit, and multiple-input as an input image, An image generation step of generating an output image including only a background from which a moving object is removed from the plurality of input images input;
An imaging method comprising:

Images