JP6399120B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus.

  Conventionally, a panoramic image is generated by capturing a plurality of images while panning a digital camera or the like, and sequentially connecting the images. For example, Patent Document 1 discloses a technique for performing panorama synthesis processing using a frame image of a moving image.

JP 2000-244814 A

  However, in the related art, when the movement amount by panning is too large to join the images, the images must be picked up again from the beginning.

  The subject of this embodiment is providing the imaging device which can image suitably.

  An aspect of an imaging apparatus that exemplifies the present invention includes an imaging unit that captures an image, an imaging control unit that captures a plurality of images when the imaging direction of the imaging unit is changed, and image processing that combines the plurality of images A display unit configured to display an image captured by the imaging unit, an image instructing a change in the imaging direction, and an image synthesized by the image processing unit.

  According to the present embodiment, it is possible to provide an imaging device capable of suitable imaging.

The block diagram which shows the structure of the digital camera of one Embodiment The flowchart which shows the operation | movement of the image processing of the digital camera of one Embodiment. FIG. 11 is a diagram illustrating an example of a monitor display in the panorama mode (part 1); FIG. 2 is a diagram illustrating an example of a monitor display in the panorama mode (part 2); The figure which shows another example of the display of the monitor in panorama mode The figure which shows the other example of the display of the monitor in panorama mode

  FIG. 1 is a block diagram showing a configuration of a digital camera according to an embodiment of the present invention.

  The digital camera according to this embodiment includes an imaging optical system 11, an imaging device 12, an AFE 13, a CPU 14, a memory 15, a monitor 16, a recording interface (I / F) 17, an operation member 19, a release button 20, a touch panel 21, and a bus. Is done. The CPU 14, the memory 15, the monitor 16, and the recording I / F 17 are connected to each other via a bus so as to be able to transmit information. The operation member 19, the release button 20, and the touch panel 21 are connected to the CPU 14, respectively. Note that the digital camera of the present embodiment has a panorama mode that generates one panoramic image from a plurality of images obtained by dividing and capturing an object scene as one function of the shooting mode.

  The imaging optical system 11 includes a plurality of lens groups including a zoom lens and a focusing lens. The lens position of the imaging optical system 11 is adjusted in the optical axis direction by a lens driving unit (not shown). For simplicity, the imaging optical system 11 is illustrated as a single lens in FIG.

  The image pickup device 12 is a device that picks up an image of an object scene formed by a light beam that has passed through the image pickup optical system 11. The output of the image sensor 12 is input to the AFE 13. Note that the image sensor 12 of the present embodiment may be a progressive scan type solid-state image sensor (CCD or the like) or an XY address type solid-state image sensor (CMOS or the like).

  A plurality of light receiving elements are arranged in a matrix on the light receiving surface of the image sensor 12. In each light receiving element of the image sensor 12, red (R), green (G), and blue (B) color filters are arranged according to a known Bayer array. Therefore, each light receiving element of the imaging element 12 outputs an image signal corresponding to each color by color separation in the color filter. Thereby, the image sensor 12 can acquire a color image.

  Here, in the photographing mode of the digital camera, the image pickup device 12 picks up a recorded image (main image) in response to a full pressing operation of the release button 20. Further, the imaging device 12 in the shooting mode captures a low-resolution image (through image) for composition confirmation at a predetermined time interval even during standby for shooting. The through image data is output from the image sensor 12 by thinning-out readout. The through image data is used for image display on the monitor 16 and various arithmetic processes by the CPU 14. In addition, when the shooting mode is the moving image mode or the panorama mode, the image sensor 12 captures a moving image as a main image at a predetermined frame rate (30 fps, 60 fps, etc.).

  The AFE 13 is an analog front end circuit that performs analog signal processing on the output of the image sensor 12. The AFE 13 performs correlated double sampling, image signal gain adjustment, and image signal A / D conversion. The output of the AFE 13 is connected to the CPU 14. In the present embodiment, the imaging device 12 and the AFE 13 constitute an imaging unit.

  The CPU 14 is a processor that comprehensively controls the operation of the digital camera. The CPU 14 executes a control program to perform known autofocus (AF) control, automatic exposure (AE) calculation, auto white balance calculation, and the like based on the phase difference detection method and the contrast detection method based on the through image data. Do. The CPU 14 also performs a setting screen, display processing of a captured main image and a through image, and the like. Further, the CPU 14 operates as an image processing unit 22 that performs image processing on the digital image signal output from the AFE 13 by executing the image processing program stored in the memory 15.

  The image processing unit 22 performs various types of image processing (for example, color interpolation processing, gradation conversion processing, contour enhancement processing, white balance adjustment) on the digital image signal output from the AFE 13. In addition, as will be described later, the image processing unit 22 according to the present embodiment, when the shooting mode is set to the panorama mode, generates a frame (target image) for capturing a moving image based on a known panorama synthesis process. Panorama images (composite images) are generated by combining panoramic images by joining them in sequence order.

  The memory 15 stores image data captured by the digital camera, a control program executed by the CPU 14, an image processing program, and the like. The memory 15 can be a non-volatile semiconductor memory or the like.

  The monitor 16 is a monitor such as a liquid crystal monitor, and displays various setting screens and images in accordance with instructions from the CPU 14.

  A connector for connecting the storage medium 18 is formed in the recording I / F 17. The recording I / F 17 executes data writing / reading with respect to the storage medium 18 connected to the connector. The storage medium 18 is composed of a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is shown as an example of the storage medium 18.

  The operation member 19 includes, for example, a command dial, a cross-shaped cursor key, a determination button, and the like. The operation member 19 receives various inputs of the digital camera from the user. For example, the operation member 19 is used for a switching operation of an operation mode of a digital camera, an input operation on a setting screen, and the like.

  The release button 20 receives from the user an instruction input for starting an AF operation before imaging by a half-press operation and an instruction input for starting an imaging operation by a full-press operation.

  The touch panel 21 detects the position of a stylus (or fingertip or the like) in contact with the panel surface, and outputs the detected position information to the CPU 14 to accept an instruction input from the user. The configuration of the touch panel 21 may be either a capacitance type or a pressure sensitive type. The touch panel 21 of the present embodiment is configured by a transparent panel having the same shape as the monitor 16, and is stacked on the entire surface of the monitor 16. The user inputs an instruction to the CPU 14 via the touch panel 21 while visually recognizing the GUI button displayed on the monitor 16.

  Next, the image processing operation by the digital camera of this embodiment will be described with reference to the flowchart of FIG. In the following, it is assumed that the shooting mode is set in advance to the panorama mode by the user via the operation member 19.

  When the power button on the operation member 19 is pressed by the user, the CPU 14 turns on the digital camera and initializes each unit. The CPU 14 causes the imaging device 12 to start capturing a through image when the imaging mode is in an imaging standby state, and displays the through image, for example, over the entire display range 30 of the monitor 16 (FIG. 3a). The CPU 14 causes the image sensor 12 to start moving image capturing of the object scene in the panorama mode in response to an imaging instruction for a full press operation of the release button 20 by the user. At the same time, the user starts panning the digital camera. In this embodiment, the user is described as panning the digital camera in the horizontal direction, but the same applies to the case of the vertical direction, and the description thereof is omitted.

  Further, when starting the moving image capturing in the panorama mode, the CPU 14 switches the display on the monitor 16 from the display illustrated in FIG. 3A to the display illustrated in FIG. 3B. That is, as shown in FIG. 3 b, the CPU 14 reduces the moving image frame captured by the moving image instead of the through image to the display area 40 arranged on the right side of the display range 30 of the monitor 16 and displays it in real time. On the other hand, the CPU 14 displays in real time a part of a panoramic image that is sequentially generated by a panorama synthesizing process, which will be described later, in a display area 50 arranged on the left side of the display range 30. However, as shown in FIG. 3b, nothing is displayed in the display area 50 because there is no panorama image subjected to the panorama synthesis process at the start of moving image capturing.

  Step S101: The CPU 14 reads the first frame output from the image sensor 12 via the AEF 13, and sets it as a composite image that serves as a reference for panorama composition processing. Then, the CPU 14 reads the second frame and sets it as a target image to be joined to the composite image.

  Step S102: The image processing unit 22 extracts image feature amounts such as edge amounts and color information of the target image and the composite image by known image processing. The image processing unit 22 can detect the movement amount and the pan direction of the target image with respect to the composite image by the pan operation of the user's digital camera based on the extracted image feature amount and the correlation processing between the target image and the composite image. It is determined whether or not. When the movement amount and the pan direction can be detected, the image processing unit 22 proceeds to step S103 (YES side), and when the movement amount and the pan direction cannot be detected, the image processing unit 22 proceeds to step S106 (NO side).

  Step S103: The image processing unit 22 performs a panoramic composition process for joining the image area newly captured by the target image to the composite image based on the movement amount and pan direction detected in Step S102.

  Step S104: The image processing unit 22 displays the obtained pan direction on the monitor 16 with the arrow 60, and at the same time, slides the composite image panoramicly synthesized in Step S103 on the display area 50 in the direction opposite to the pan direction. Display in real time (FIG. 4a). Here, FIG. 4A shows a display state of the monitor 16 immediately after the start of the panorama synthesis process as an example.

  In the case where the target image is an image that is captured vertically or obliquely with respect to the composite image, the CPU 14 displays an image region that protrudes from the region 50 when displaying the composite image in which the target image is panorama combined. Is preferably trimmed and displayed. Alternatively, the CPU 14 may reduce and display the composite image so as to display the protruding image area.

  Step S105: The CPU 14 determines whether or not an imaging end instruction for the full pressing operation of the release button 20 by the user is received again. When the CPU 14 receives an imaging end instruction, the CPU 14 ends the panoramic mode moving image imaging. The image processing unit 22 generates a composite image as a panoramic image file and records it in the memory 15 or the recording medium 18. The CPU 14 ends the series of processes. In the present embodiment, the CPU 14 preferably generates a panoramic image file by trimming the vertical size of the composite image with the vertical width of the image of the first frame, for example. Further, when the user pans the digital camera in the vertical direction, the CPU 14 preferably trims the horizontal size of the composite image with the horizontal width of the image of the first frame, for example.

  On the other hand, when the imaging end instruction is not received, the CPU 14 proceeds to step S101 (NO side) and reads the next captured frame as a target image. CPU14 performs the process of step S101-step S104 with respect to the object image.

  Step S106: The image processing unit 22 moves the target image with respect to the composite image, such as when the panning speed of the digital camera by the user is too fast and the amount of movement is too large, or the posture of the digital camera changes too much. If the pan direction cannot be detected, the user is notified of the detection failure. That is, the image processing unit 22 displays a warning display as shown in FIG. For example, the image processing unit 22 displays the direction in which the digital camera should be panned with the arrow 70 instead of the arrow 60 in order to capture the target image that can be detected in the movement amount and stitched to the composite image. Then instruct. Further, the image processing unit 22 may display a warning such as the symbol 71 or the sentence 72 on the monitor 16 together with the arrow 70. Alternatively, as another example of the warning display, the image processing unit 22 may indicate a region where the target image is to be stitched in the composite image as a shaded region 73.

  Note that the CPU 14 may display the above warning to the user even when the user stops the digital camera for a certain period of time without panning or pans in a direction different from the arrow 60, for example. preferable.

  Step S107: The image processing unit 22 temporarily stores the image feature amount of the composite image extracted in step S102, for example, in an internal memory (not shown) of the CPU 14. It should be noted that the image feature amount of the stored composite image is the same as that of the target image from the end of the composite image to which the target image is to be joined, or an image feature of an image area having a size of about 70 to 80%. Just the amount. As a result, the memory capacity can be reduced, and the circuit scale can be reduced.

  Step S108: The image processing unit 22 causes the user to refer to the frame of the display area 40, performs a pan operation in the direction of the arrow 70, for example, based on the alarm display, and captures the next frame (for example, the image captured by the image sensor 12 (for example, 3rd frame) is read as a new target image.

  Step S109: The image processing unit 22 extracts the image feature amount of the target image read in step S108. The image processing unit 22 can detect the movement amount and pan direction of the target image for the synthesis process based on the extracted image feature quantity of the target image and the image feature quantity of the synthesized image stored in the internal memory (not shown). It is determined whether or not. When the movement amount and the pan direction can be detected, the image processing unit 22 proceeds to step S110 (YES side). On the other hand, when the movement amount and the pan direction cannot be detected, the image processing unit 22 proceeds to step S108 (NO side). The image processing unit 22 performs the processing from step S108 to step S109 until the movement amount and pan direction of the target image with respect to the composition processing are detected.

  Step S110: The image processing unit 22 cancels the warning display shown in FIG. 4b and switches to the display shown in FIG. 4c. The image processing unit 22 proceeds to step S103, and resumes the panorama synthesis process for joining the target image newly read in step S108 to the synthesized image.

As described above, in this embodiment, even if the digital camera is greatly panned with respect to the synthesized image, the panorama synthesis is performed by displaying the moving image and the synthesized image panorama synthesized so far in real time. It is possible to easily determine where the panorama synthesis has failed and easily return the imaging direction of the digital camera to the place where the panorama synthesis has failed. Thereby, it is possible to generate a panoramic image with high accuracy without performing imaging in the panoramic mode again from the beginning.
<< Additional items of embodiment >>
(1) In the above-described embodiment, the example in which the processing of the image processing unit 22 is realized by software in the CPU 14 has been described. However, the processing may be realized by hardware using an ASIC.

  (2) In the above-described embodiment, the image processing unit 22 detects the movement amount and pan direction of the target image based on the image feature amount of the entire image area of the target image and the composite image. It is not limited. For example, a margin portion having a predetermined width is set around each of the target image and the synthesized image, and the image processing unit 22 performs synthesis processing on the target image based on the image feature amount in the image area other than the margin portion. The amount of movement with respect to and the pan direction may be detected.

  In addition, the digital camera includes an acceleration sensor, an electronic gyro, and the like. Based on the movement amount and pan direction detected by the image processing unit 22 and the movement amount and pan direction detected by the acceleration sensor, the image processing unit 22 The target image may be connected to the image. Thereby, more accurate panorama composition is possible.

  (3) In the above embodiment, when the user starts panning the digital camera, the CPU 14 displays the moving image frame in real time in the display area 40 on the right side of the display range 30 of the monitor 16, and displays the display area on the left side. Although the composite image that has been subjected to the panoramic composition process 50 is displayed as a slide, the present invention is not limited to this. For example, as shown in FIG. 5a, the CPU 14 displays the frame of the moving image in real time in the display area 40 ′ above the center of the monitor 16, and the synthesized image is displayed in the lower display area 50 ′ as the panorama synthesis process progresses. Accordingly, real-time display may be performed. FIG. 5b shows an example of a warning display when detection of the movement amount and pan direction of the target image with respect to the composite image fails.

  Prior to the start of imaging in the panorama mode, the CPU 14 receives designation of an imaging range for panoramic imaging such as 180 degrees or 360 degrees from the user via the operation member 19 and the like, and a display area 50 ′ corresponding to the imaging range. May be set. Thereby, it is possible for the user to easily recognize to what extent imaging has been completed. Further, when the imaging is completed up to the set imaging range, the CPU 14 may automatically end the panorama mode imaging even if the release button 20 is not fully pressed again.

  Further, while the panorama synthesis process is being performed smoothly, the CPU 14 displays the entire frame of the moving image in the display range 30 shown in FIG. 3A to detect the movement amount and pan direction of the target image with respect to the synthesis process. If it fails, the warning display shown in FIG. 4b or 5b may be switched.

  FIG. 6 shows an example of display on the monitor 16 when the user pans the digital camera in the vertical direction and images a tower, a high-rise building, or the like.

  (4) In the above embodiment, in step S105, the CPU 14 ends the imaging in the panoramic mode when the release button 20 is fully pressed again by the user. However, the present invention is not limited to this. It is preferable that the user can fully press the release button 20 at any time during the imaging in the panorama mode, and the CPU 14 can accept the interruption of the imaging end.

  (5) In the above embodiment, panoramic composition is performed by capturing a moving image. However, the present invention is not limited to this, and the present invention can also be applied to continuous shooting or capturing still images one by one and performing panorama composition. .

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 11 ... Imaging optical system, 12 ... Imaging device, 13 ... AFE, 14 ... CPU, 15 ... Memory, 16 ... Monitor, 17 ... Recording I / F, 18 ... Storage medium, 19 ... Operation member, 20 ... Release button, 21 ... Touch panel, 22 ... Image processing unit

Claims (4)

An imaging unit that captures an image;
An imaging control unit that captures a plurality of images when the imaging direction of the imaging unit is changed;
An image processing unit that combines the plurality of images;
A display unit for displaying an image captured by the imaging unit, an image for instructing a change in the imaging direction, and an image synthesized by the image processing unit;
Equipped with a,
The photographing control unit, when not able to synthesize an image by the image processing unit depending on the direction of change of the imaging, the imaging device that displays a warning on the display unit. The photographing control unit, when not able to synthesize an image by the image processing unit depending on the direction of the change rate of the image pickup, the image pickup apparatus according to claim 1 for displaying a warning on the display unit.   The imaging apparatus according to claim 1, wherein the plurality of images are images obtained by moving image shooting.   The imaging device according to any one of claims 1 to 3, wherein the display unit displays the warning by a symbol or a sentence.

Images