JP2012060309A - Imaging apparatus, imaging method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus, an imaging method, and a program for effective utilization at a later time by recording a plurality of definition image data having a partially duplicating subject which is acquired for generating one unit of image through the collection of a plurality of adjacent images.SOLUTION: An imaging apparatus comprises: image systems 11-13 for shooting a plurality of adjacent images; a DRAM 14 for holding a plurality of shot images; a feature-amount calculation part 15b and a block-matching part 15c for detecting the duplication portion of the subject in the plurality of images held in the DRAM 14 to calculate a position correction information for each of the plurality of images according to the detection result; an image deformation/composition addition part 15d for generating one unit of image from the plurality of images held in the DRAM 14 according to the above detection result and the calculated position correction information; and a CPU 21 for allowing a memory card 31 to record the generated composite image, the plurality of images held in the DRAM 14, and the position correction information of each image in association.

Description

  The present invention relates to an imaging device having a panorama shooting function, an imaging method, and a program, for example.

A technique for generating a data file of each image so that the positional relationship of images to be joined at the time of panoramic photographing with an electronic camera can be considered. (For example, Patent Document 1)
In the technique described in the patent document, main image data and sub-image data having a data size smaller than that of the main image data are recorded in the image data file in the shooting mode. The sub image at the time of panorama shooting is a thumbnail image corresponding to the frame and a sub image corresponding to the previous frame. The direction to arrange is the moving direction from the previous frame. In playback mode, when the image data file of a frame for displaying a sub image contains information indicating a panoramic image, the electronic camera displays a plurality of thumbnail images constituting the sub image on the LCD monitor and supports the corresponding frame The thumbnail image to be displayed is displayed so as to be surrounded by a thick frame.

Japanese Patent Laid-Open No. 2004-007168

  In general, when a panoramic image is generated from a plurality of images including the technique described in the above-mentioned patent document, the pan operation by the user is performed by an accurate turning operation with the image sensor as the center position. It is difficult. For this reason, in the acquired adjacent images, minute distortion is generated such that the angle of view in the direction orthogonal to the turning direction differs for each turning angle. Therefore, even in overlapping subject areas in a plurality of images taken at temporally consecutive timings, the angle of view in the direction orthogonal to the turning direction is different, so the overlapping areas are synthesized with the same image size. Therefore, it is necessary to perform an interpolation process by matching the image size to at least one of them.

  In general, when a single panoramic image is generated from a plurality of adjacent images partially overlapped, the image size is usually converted to a smaller size by resizing processing than that at the time of shooting. After the panorama image is generated, the individual image data used to generate the panorama image is deleted or recorded as a thumbnail image with the image size greatly reduced as in the above patent document. .

  As described above, in panoramic image shooting, even if the individual image data acquired at the time of shooting is fine, the original image data is not recorded in high definition after the panoramic image is generated.

  The present invention has been made in view of the above-described circumstances, and the object of the present invention is to overlap a plurality of adjacent images and obtain a single image by partially overlapping the subject. Another object of the present invention is to provide an imaging apparatus, an imaging method, and a program for recording a plurality of fine image data in a state that can be effectively used later.

  According to a first aspect of the present invention, there is provided photographing means for photographing a plurality of adjacent images in which at least a part of a subject overlaps, holding means for holding a plurality of images taken by the photographing means, and holding by the holding means. Detection means for detecting overlapping portions of the subject between the plurality of images, calculation means for calculating position correction information for each of the plurality of images based on the detection results by the detection means, and detection results by the detection means And an image compositing means for compositing a plurality of images held by the holding means based on the position correction information calculated by the calculating means, a composite image synthesized by the image synthesizing means, and held by the holding means And a recording unit that records the position correction information calculated for each of the plurality of images by the calculation unit in association with each other. It is characterized in.

  According to a second aspect of the present invention, in the first aspect of the invention, the calculation means is based on an image positioned at the head in the shooting order among a plurality of images held by the holding means, and other than that, The position correction information for the reference image is calculated for each image.

  According to a third aspect of the present invention, there is a holding step for holding a plurality of adjacent images in which at least a part of a subject is photographed by the photographing unit, and a subject overlap between the plurality of images held in the holding step. A detection step for detecting a part, a calculation step for calculating position correction information for each of the plurality of images based on a detection result in the detection step, a detection result in the detection step, and a calculation step Based on the calculated position correction information, an image combining step for combining one image from a plurality of images held in the holding step, a combined image synthesized in the image combining step, and in the holding step A recording step in which a plurality of held images and the position correction information calculated for each of the plurality of images in the calculation step are associated with each other and recorded on a recording medium. Characterized in that it comprises and.

  According to a fourth aspect of the present invention, there is provided a program executed by a computer incorporated in an imaging apparatus, wherein the computer is photographed by photographing means for photographing a plurality of adjacent images in which at least a part of a subject overlaps, and the photographing means. A holding unit that holds the plurality of images, a detection unit that detects an overlapping portion of the subject between the plurality of images held by the holding unit, and a position for each of the plurality of images based on a detection result by the detection unit Calculation means for calculating correction information, detection result by the detection means, and image composition means for combining a plurality of images held by the holding means based on the position correction information calculated by the calculation means, and the image A composite image synthesized by the synthesis means, a plurality of images held by the holding means, and a composite image by the calculation means. Characterized in that to function as a recording means for recording in association with position correction information calculated for each image.

  According to the present invention, a plurality of fine image data obtained by collecting a plurality of adjacent images and generating a single image are recorded in a state where they can be effectively used later. be able to.

1 is a block diagram showing a schematic configuration of a functional circuit of a digital camera according to an embodiment of the present invention. 6 is a flowchart showing a series of processing contents from image capturing to composition and recording according to the embodiment. 6 is a flowchart showing a series of processing contents from image capturing to composition and recording according to the embodiment. FIG. 6 is a view for explaining correction conversion processing for alignment according to the embodiment. FIG. 6 is a view for explaining correction conversion processing for alignment according to the embodiment.

  An embodiment in which the present invention is applied to a digital camera having a panoramic shooting function will be described below with reference to the drawings.

  FIG. 1 shows a circuit configuration of a digital camera 10 according to the present embodiment. In the figure, a solid-state imaging device (IS) 12 composed of, for example, a CCD (Charge Coupled Device), a CMOS image sensor, or the like is passed through an optical lens unit 11 disposed on the front surface of the camera housing. An optical image of a subject is incident on the imaging surface to form an image.

  In a monitor state, also referred to as through image display or live view image display, an image signal obtained by imaging with the solid-state imaging device 12 is sent to the AGC / A / D conversion unit 13 to perform correlation square sampling, automatic gain adjustment, A / D conversion processing is executed and digitized. The digital image data is held in the DRAM 14 which is a buffer memory via the system bus SB.

  The image processing unit 15 appropriately performs necessary image processing on the image data held in the DRAM 14. The image processing unit 15 includes a demosaic unit 15a, a feature amount calculation unit 15b, a block matching unit 15c, and an image deformation synthesis addition unit 15d. An SRAM 16 is connected to the image processing unit 15.

  The image processing unit 15 performs demosaic processing on matrix data, pixel interpolation processing, gamma correction processing, and the like on image data (hereinafter referred to as “Bayer data”) according to the configuration of a Bayer array color filter provided in the solid-state imaging device 12. By processing, the Bayer data, which is RAW (raw) data, is developed and converted into luminance color difference (YUV) image data.

  The image processing unit 15 creates image data in which the number of pixels and the gradation bit are greatly reduced for display from the image data, sends the image data to the display unit 17, and displays it as a through image.

  Similarly to the optical lens unit 11, a pair of microphones 18L and 18R are provided on the front surface of the camera casing, and the sound in the subject direction is input in stereo. The microphones 18 </ b> L and 18 </ b> R convert the input sound into electrical signals and output them to the sound processing unit 19.

  The sound processing unit 19 converts the sound signals input from the microphones 18L and 18R into digital data when recording a single sound, capturing a still image with sound, and capturing a moving image. Further, the sound processing unit 19 detects the sound pressure level of the digitized sound data, and compresses the sound data in a predetermined data file format, for example, AAC (moving picture experts group-4 Advanced Audio Coding) format. An audio data file is created and sent to a recording medium to be described later.

  In addition, the sound processing unit 19 includes a sound source circuit such as a PCM sound source, uncompresses and converts the sound data file sent during sound reproduction into an analog signal, and a speaker 20 provided on the rear side of the housing of the digital camera 10. To sound a loud sound.

  The CPU 21 performs overall control of the above circuit. The CPU 21 is directly connected to the main memory 22 and the program memory 23. The main memory 22 is composed of, for example, an SRAM and functions as a work memory. The program memory 23 is composed of an electrically rewritable nonvolatile memory such as a flash memory, for example, and fixedly stores an operation program, data including control of panorama image generation, which will be described later.

  The CPU 21 reads the necessary program, data, etc. from the program memory 23 and executes the control operation of the entire digital camera 10 while temporarily developing and storing it in the main memory 22.

  Further, the CPU 21 executes a control operation in response to various key operation signals input directly from the operation unit 24. The operation unit 24 includes, for example, a power key, a shutter release key, a zoom up / down key, a shooting mode key, a playback mode key, a menu key, a cursor (“↑” “→” “↓” “←”) key, a set key, Release key, display key, etc.

  In addition to the AGC / A / D conversion unit 13, DRAM 14, image processing unit 15, display unit 17, and audio processing unit 19, the CPU 21 further includes a lens driving unit 25, a flash driving unit 26, an image via the system bus SB. The sensor (IS) drive unit 27 and the memory card controller 28 are connected.

  The lens driving unit 25 controls the rotation of the lens DC motor (M) 29 in response to a control signal from the CPU 21, and a part of a plurality of lens groups constituting the optical lens unit 11, specifically zoom. The positions of the lens and the focus lens are individually moved along the optical axis direction.

  The flash drive unit 26 receives a control signal from the CPU 21 during still image shooting, and drives the flash unit 30 including a plurality of white high-intensity LEDs to be lit in synchronization with the shooting timing.

  The image sensor driving unit 27 performs scanning driving of the solid-state imaging device 12 in accordance with imaging conditions set at that time.

  The image processing unit 15 performs demosaic processing on the image data sent from the AGC / A / D conversion unit 13 and held in the DRAM 14 at the time of image capturing associated with the operation of the shutter release key of the operation unit 24, and further performs predetermined mosaic processing. In the case of a data file format, for example, JPEG (Joint Photographic Experts Group), data compression processing such as DCT (Discrete Cosine Transform) or Huffman coding is performed to create an image data file with a greatly reduced data amount. The created image data file is recorded on the memory card 31 via the system bus SB and the memory card controller 26.

Further, the image processing unit 15 receives the image data read from the memory card 31 via the memory card controller 28 in the reproduction mode via the system bus SB, holds it in the DRAM 14, and then holds it in the DRAM 14. Image data of the original size is obtained by decompression processing that decompresses the image data in the reverse procedure of recording, and this is output to the display unit 17 via the system bus SB and displayed.
The memory card controller 28 is connected to the memory card 31 via the card connector 32. A memory card 31 is a memory for recording image data and the like that is detachably attached to the digital camera 10 and serves as a recording medium for the digital camera 10, and has a nonvolatile memory that can be electrically rewritten in units of blocks. A flash memory and a driving circuit thereof are provided.

Next, the operation of the above embodiment will be described.
In the following operation, the digital camera 10 is panned (turned) in the panoramic image shooting mode, and the shutter release key is pressed for an arbitrary time to shoot a plurality of still images that are temporally continuous. At this time, the CPU 21 reads out the operation program and data stored in the program memory 23 and develops and stores them in the main memory 22 for execution.

  For example, when the digital camera 10 is upgraded, the operation program stored in the program memory 23 is stored in the program memory 23 when the digital camera 10 is shipped from the factory. To download and store new operation programs, data, and the like from the outside by connecting to.

  2 and 3 show a series of processing contents related to panoramic image shooting and subsequent synthesis processing. Initially, the CPU 21 initializes the value of the variable n, which is stored in the main memory 22 and counts the number of captured images, to “1” (step S101).

  Thereafter, the CPU 21 determines whether or not the shutter release key of the operation unit 24 has been operated (step S102). If it is determined that the shutter release key has not been operated, the CPU 21 then determines whether or not the release key of the operation unit 24 has been operated (step S103). If it is determined that the release key has not been operated, the process returns to step S102.

By repeatedly executing the processes of steps S102 and S103 in this way, the CPU 21 waits for either the shutter release key or the release key of the operation unit 24 to be operated.
If the release key is operated, the CPU 21 determines this in step S103, and the above ends the series of processing contents related to the panoramic image shooting and subsequent composition processing in FIG. 2 and FIG. Return to still image shooting.

  If it is determined that the shutter release key has been operated, the CPU 21 immediately captures a still image (step S104), and then updates the value of the variable n for counting the number of captured images by “+1” (step S104). S105). Bayer data obtained by photographing is held in the DRAM 14.

  Next, the CPU 21 confirms that the value of the updated variable n has not exceeded the maximum number of images that can be continuously shot set in advance in the digital camera 10 (step S106), and the shutter release key has not yet been pressed. It is determined whether or not the operation is continued (step S107).

  Here, if it is determined that the shutter release key is still being operated, the process returns to the processing from step S104 again, and further still images that are temporally continuous are continued in accordance with the operation of the shutter release key. The Bayer data obtained by this photographing is also sequentially held and accumulated in the DRAM 14.

  While the shutter release key is being operated in this way, the processing of steps S104 to S107 is repeatedly executed, and the photographing and holding of the image data are continued until the maximum number of continuous shots is exceeded.

  Although the shutter release key pressing operation continues, if the updated variable n exceeds the maximum number of images that can be continuously shot, or if the shutter release key pressing operation is stopped, the CPU 21 performs step S106 or S107. This is determined, the photographing operation is stopped, and the process proceeds to a process of combining images based on the Bayer data accumulated so far.

First, the demosaic unit 15a of the image processing unit 15 performs demosaicing in order to use the first Bayer data as a reference for alignment (step S108). Here, the demosaicing process is to develop the Bayer data obtained from the solid-state imaging device 12 as described above and convert the image data into luminance color difference (YUV) image data.
At this time, the image data may be converted into image data with a reduced image size, depending on the alignment accuracy.

  Next, the value of the variable m for counting the number of processed images held in the main memory 22 by the CPU 21 is initialized to “2” (step S109). In this process, the first image data positioned at the head in time is used as a reference. Therefore, the following correction process is not performed on the first image data, and therefore the above-described correction process in step S109 is performed. An initial value “2” is set to select image data to be subjected to the first position correction process.

  After confirming that the value “2” of the variable m is equal to or less than the number n of continuously captured images (step S110), the Bayer data of the “2” th image is obtained by the demosaic unit 15a in the same manner as in step S108. The demosaic process is performed (step S111).

  A correction coefficient is calculated by the block matching unit 15c and the feature amount calculation unit 15b in order to align the demosaiced “2” image data with the immediately preceding “1” image data (step S112). .

Here, as a method for calculating a correction coefficient for alignment, for example, the feature amount calculation unit 15b extracts feature points on the image edge, and a correction conversion matrix in which the most feature points match between two target images is used. There are methods to find. As the correction conversion matrix, for example, a conversion matrix shown by the following equation is used. That is,

The correction conversion process for the alignment will be described with reference to FIGS.
FIG. 4A illustrates “a night view of an urban area” as a subject to be photographed.

  Assume that such a subject is photographed while panning the digital camera 10 from left to right in the horizontal direction as shown in FIGS. 4 (B) to 4 (D).

  Here, in order to simplify the explanation, it is assumed that shooting is performed twice during the pan operation as shown in FIG. The image data FL1 obtained by the first shooting and the image data FL2 obtained by the second shooting include the shared area C1.

  As shown in FIG. 5B, the image data FL1 obtained in the first shooting is a reference image as described above. Therefore, it is unconditionally assumed that the amount of movement by the pan operation is “0 (zero)”. M1 is given.

  On the other hand, as shown in FIG. 5C, the image data FL2 obtained by the second shooting is an image whose position has been moved by X in the horizontal direction from the image data FL1, which is the reference image, as described above. The correction matrix M2 using the horizontal movement amount X by the pan operation is used.

  The block matching unit 15c executes the matching process of the shared portion C1, and the feature amount calculation unit 15b extracts the feature points on the image edge as described above, and the most feature points match between the two target images. Thus, by calculating | requiring a correction | amendment conversion matrix, the correction coefficient according to the moving amount of a position, specifically, the moving amount X in the said correction matrix M2 is computable.

  By calculating the correction coefficient in this way and accumulating it in the previous conversion matrix, each image data can be converted according to the correction conversion matrix with the first image data as a reference (step S113). .

  Thereafter, after updating the value of the variable m by “+1” (step S114), the process returns to step S110 again.

  In this way, by sequentially repeating the processes of steps S110 to S114, the correction conversion matrix shows a cumulative result based on the first image data.

  Therefore, when the value of the updated variable m exceeds the number n of continuously captured images, the calculation of the cumulative correction conversion matrix for all the continuously captured image data has been completed. It is determined in S110.

  Next, an image size that can be taken by the composite image is calculated from the parallel movement component of the acquired cumulative correction conversion matrix (step S115). Based on the calculated image size, a composite image area is secured on the DRAM 14 (step S116).

Next, the captured image and the correction conversion matrix are stored and combined.
First, the CPU 21 initializes the value of the variable m, which is stored in the main memory 22 and counts the number of processed images, to “1” (step S117). Next, after confirming that the value “1” of the variable m is equal to or less than the number n of continuously captured images (step S118), the “1” -th Bayer data is set in the same manner as a normal captured image. The demosaic process is performed by the demosaic unit 15a (step S119).

  The demosaiced “1” image data is converted into a data file based on, for example, the JPEG format, and a correction conversion matrix (here, m = 1) representing the amount of movement of the position corresponding to the image data. Therefore, the unit matrix M1) is added to the option information area of the data file and recorded on the memory card 31 which is the recording medium of the digital camera 10 (step S120).

  The option information area of the data file is assumed to be an information area defined by, for example, JFIF (JPEG File Interchange Format) or Exif (Exchangeable image file format).

  Thereafter, the “1” -th image data is converted by the cumulative correction conversion matrix M1, and the converted image data is matched with the combined image area secured in the DRAM 14 by the image deformation / synthesis adding unit 15d in the process of step S116. The pixel position range to be written is sequentially written (step S121).

  Thereafter, after updating the value of the variable m by “+1” (step S122), the process returns to step S118 again.

  In this way, the processing of steps S118 to S122 is repeatedly executed in sequence for the number of image data, so that information of a correction conversion matrix indicating the amount of movement of the position of the continuous individual image data relative to the reference image is added to the memory card 31. While recording, the data is written in the corresponding pixel position range of the composite image area secured in the DRAM 14.

  Then, after the same processing relating to the last image data is finished, if the value of the variable m is further updated by “+1” in step S122, the value of the variable m is set to the number n of continuously captured images in step S118. It is determined that all the image data has been processed.

At this time, since the panoramic image data is written in the completed image area secured in the DRAM 14, this single image data is read out and converted into a data file. It records on the memory card 31 which is a recording medium (step S123).
The series of processes in FIGS. 2 and 3 is thus completed.

  As described above in detail, according to the present embodiment, a plurality of fine image data obtained by generating a panoramic image and partially overlapping a subject are separated from the panoramic image data by using the first image. It was recorded together with information indicating the reference movement amount.

  As a result, the original image data can be automatically recorded in a state where it can be effectively used later without losing information lost in the panoramic image whose resolution is inevitably lowered by the synthesis process.

  In addition, in the above embodiment, the correction conversion matrix information indicating the amount of movement accumulated with respect to the position of the image data located at the head of the plurality of image data obtained by shooting for the panoramic image. Therefore, the movement amount of each image data can be easily calculated, and the positional relationship with respect to the reference image data can be easily grasped from the movement amount.

  In the above embodiment, the case where the present invention is applied to a digital camera that captures a panoramic image that captures a plurality of temporally continuous images by one pan operation has been described. The same processing can be applied to not only a plurality of images according to scanning but also a case where a plurality of images are photographed in a matrix while zigzag scanning or raster scanning of a subject having a two-dimensional spread. It becomes possible.

  The above embodiment has been described with respect to a case where the present invention is applied to a digital camera. However, the present invention is not limited to this, and any other electronic device having a camera function may be a mobile phone terminal or a PDA (Personal Digital Assistants). : Personal information portable terminal), portable media player, electronic book, mobile computer, and other devices.

  In addition, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention in the implementation stage. Further, the functions executed in the above-described embodiments may be combined as appropriate as possible. The above-described embodiment includes various stages, and various inventions can be extracted by an appropriate combination of a plurality of disclosed constituent elements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, if the effect is obtained, a configuration from which the constituent requirements are deleted can be extracted as an invention.

  DESCRIPTION OF SYMBOLS 10 ... Digital camera, 11 ... Optical lens unit, 12 ... Solid-state image sensor (IS), 13 ... AGC / A / D conversion part, 14 ... DRAM, 15 ... Image processing part, 15a ... Demosaic part, 15b ... Feature amount calculation , 15c: Block matching unit, 15d: Image deformation / synthesis adding unit, 16: SRAM, 17: Display unit, 18L, 18R ... Microphone, 19 ... Audio processing unit, 20 ... Speaker, 21 ... CPU, 22 ... Main memory, DESCRIPTION OF SYMBOLS 23 ... Program memory, 24 ... Operation part, 25 ... Lens drive part, 26 ... Flash drive part, 27 ... Image sensor (IS) drive part, 28 ... Memory card controller, 29 ... DC motor (M) for lenses, 30 ... Flash unit, 31 ... memory card, 32 ... card connector, SB ... system bus.

Claims (4)

Photographing means for photographing a plurality of adjacent images in which at least a part of the subject overlaps;
Holding means for holding a plurality of images photographed by the photographing means;
Detecting means for detecting an overlapping portion of a subject between a plurality of images held by the holding means;
Calculation means for calculating position correction information for each of the plurality of images based on a detection result by the detection means;
Image combining means for combining a plurality of images held by the holding means based on the detection result by the detecting means and the position correction information calculated by the calculating means;
A synthesized image synthesized by the image synthesizing unit, a plurality of images held by the holding unit, and a recording unit that records the position correction information calculated for each of the plurality of images by the calculating unit in association with each other. An imaging apparatus comprising the imaging device.   The calculation means calculates the position correction information for the reference image for each of the other images based on the first image in the shooting order among the plurality of images held by the holding means. The imaging apparatus according to claim 1, wherein: A holding step for holding a plurality of adjacent images that are photographed by the photographing unit and at least a part of the subject overlap;
A detection step of detecting overlapping portions of the subject between the plurality of images held in the holding step;
A calculation step for calculating position correction information for each of the plurality of images based on a detection result in the detection step;
An image synthesis step for synthesizing one image from a plurality of images held in the holding step based on the detection result in the detection step and the position correction information calculated in the calculation step;
The composite image synthesized in the image synthesis step, the plurality of images held in the holding step, and the position correction information calculated for each of the plurality of images in the calculation step are associated with the recording medium. An imaging method comprising: a recording step for recording. A program executed by a computer built in the imaging apparatus,
The above computer
Photographing means for photographing a plurality of adjacent images in which at least a part of the subject overlaps;
Holding means for holding a plurality of images photographed by the photographing means;
Detecting means for detecting an overlapping portion of a subject between a plurality of images held by the holding means;
Calculation means for calculating position correction information for each of the plurality of images based on a detection result by the detection means;
Based on the detection result by the detection means and the position correction information calculated by the calculation means, an image synthesis means for synthesizing a plurality of images held by the holding means, and a synthesized image synthesized by the image synthesis means A program that functions as a recording unit that records a plurality of images held by the holding unit and position correction information calculated for each of the plurality of images by the calculation unit in association with each other.

Images