JP7001087B2 - Imaging device - Google Patents

Description

The present invention relates to an image pickup apparatus.

Conventionally, a panoramic image is generated by capturing a plurality of images while panning a digital camera or the like and sequentially stitching the images together. For example, Patent Document 1 discloses a technique for performing panoramic composition processing using a frame image of a moving image.

Japanese Unexamined Patent Publication No. 2000-244814

However, in the prior art, if the amount of movement by the pan is too large to connect the images, the image must be re-imaged from the beginning.

An object of the present embodiment is to provide an image pickup apparatus capable of suitable image pickup.

One aspect of the image pickup apparatus illustrating the present invention is an image pickup element that captures a subject and outputs an image data, a display unit that displays an image generated based on the image data output from the image pickup element, and an image pickup direction. The display unit includes an image processing unit that generates a composite image based on a plurality of images that are captured and generated while changing the image, and a display unit is an image and an image that is generated based on the image data output from the image pickup element. A composite image generated by the processing unit is displayed, a directional image indicating the direction in which the imaging direction is changed is displayed, and the directional image is switched to a warning display to the user based on the changing direction in the imaging direction.

According to this embodiment, it is possible to provide an image pickup apparatus capable of suitable image pickup.

A block diagram showing the configuration of a digital camera according to an embodiment. A flowchart showing the operation of image processing of the digital camera of one embodiment. The figure which shows an example of the display of a monitor in a panorama mode (the 1) Figure showing an example of monitor display in panorama mode (Part 2) Diagram showing another example of monitor display in panoramic mode Diagram showing another example of a monitor display in panoramic mode

FIG. 1 is a block diagram showing a configuration of a digital camera according to an embodiment of the present invention.

The digital camera of this embodiment is composed of an image pickup optical system 11, an image pickup element 12, an AFE 13, a CPU 14, a memory 15, a monitor 16, a recording interface (I / F) 17, an operation member 19, a release button 20, a touch panel 21, and a bus. Will be done. The CPU 14, the memory 15, the monitor 16, and the recording I / F 17 are connected to each other so that information can be transmitted via the bus. Further, the operation member 19, the release button 20, and the touch panel 21 are each connected to the CPU 14. It should be noted that the digital camera of the present embodiment has a panoramic mode as one function of the shooting mode, which generates one panoramic image from a plurality of images obtained by dividing and capturing the field.

The image pickup optical system 11 is composed of a plurality of lens groups including a zoom lens and a focusing lens. The lens position of the imaging optical system 11 is adjusted in the optical axis direction by a lens driving unit (not shown). For simplicity, FIG. 1 shows the imaging optical system 11 as a single lens.

The image pickup device 12 is a device that captures an image of the field formed by the light flux passing through the image pickup optical system 11. The output of the image sensor 12 is input to the AFE 13. The image pickup device 12 of the present embodiment may be a sequential scanning type solid-state image pickup device (CCD or the like) or an XY address type solid-state image pickup device (CMOS or the like).

Further, a plurality of light receiving elements are arranged in a matrix on the light receiving surface of the image pickup element 12. A red (R), green (G), and blue (B) color filter is arranged in each light receiving element of the image pickup element 12 according to a known Bayer arrangement. Therefore, each light receiving element of the image pickup element 12 outputs an image signal corresponding to each color by color separation by the color filter. As a result, the image sensor 12 can acquire a color image.

Here, in the shooting mode of the digital camera, the image sensor 12 captures a recorded image (this image) in response to a full press operation of the release button 20. Further, the image pickup device 12 in the shooting mode captures a low-resolution image (through image) for composition confirmation at predetermined time intervals even during shooting standby. The data of this through image is output from the image sensor 12 by thinning out reading. The through image data is used for image display on the monitor 16 and various arithmetic processes by the CPU 14. Further, when the shooting mode is a moving image mode or a panoramic mode, the image pickup element 12 shall capture a moving image at a predetermined frame rate (30 fps, 60 fps, etc.) as the main image.

The AFE 13 is an analog front-end circuit that performs analog signal processing on the output of the image sensor 12. The AFE 13 performs correlated double sampling, adjustment of the gain of the image signal, and A / D conversion of the image signal. The output of AFE 13 is connected to CPU 14. In this embodiment, the image pickup device 12 and the AFE 13 form an image pickup unit.

The CPU 14 is a processor that comprehensively controls the operation of the digital camera. By executing the control program, the CPU 14 performs known autofocus (AF) control, automatic exposure (AE) calculation, auto white balance calculation, etc. by a phase difference detection method or a contrast detection method based on the data of the through image. conduct. The CPU 14 also performs setting screens, display processing of captured main images and through images, and the like. Further, the CPU 14 operates as an image processing unit 22 that performs image processing on the digital image signal output from the AFE 13 by executing the image processing program stored in the memory 15.

The image processing unit 22 performs various image processing (for example, color interpolation processing, gradation conversion processing, contour enhancement processing, white balance adjustment, etc.) on the digital image signal output from the AFE 13. Further, as will be described later, the image processing unit 22 of the present embodiment sets the frame (target image) to be imaged as a moving image based on the known panoramic composition processing when the shooting mode is set to the panoramic mode. A panoramic image (composite image) is generated by connecting them in a series order and performing panoramic composition.

The memory 15 stores image data captured by a digital camera, a control program executed by the CPU 14, an image processing program, and the like. As the memory 15, a non-volatile semiconductor memory or the like can be used.

The monitor 16 is a monitor such as a liquid crystal monitor, and displays various setting screens and images in response to instructions from the CPU 14.

The recording I / F 17 is formed with a connector for connecting the storage medium 18. Then, the recording I / F 17 executes writing / reading of data to the storage medium 18 connected to the connector. The storage medium 18 is composed of a hard disk, a memory card containing a semiconductor memory, or the like. Note that FIG. 1 shows a memory card as an example of the storage medium 18.

The operation member 19 is composed of, for example, a command dial, a cross-shaped cursor key, a determination button, and the like. Then, the operation member 19 receives various inputs of the digital camera from the user. For example, the operation member 19 is used for a switching operation of an operation mode of a digital camera, an input operation on a setting screen, and the like.

The release button 20 receives from the user an instruction input for starting the AF operation before imaging by a half-press operation and an instruction input for starting an imaging operation by a full-press operation.

The touch panel 21 detects the position of the stylus (or fingertip or the like) in contact with the panel surface, outputs the detected position information to the CPU 14, and receives an instruction input from the user. The structure of the touch panel 21 may be either a capacitance type or a pressure sensitive type. The touch panel 21 of the present embodiment is composed of a transparent panel having the same shape as the monitor 16, and is laminated and arranged on the entire surface of the monitor 16. The user directly inputs a command to the CPU 14 via the touch panel 21 while visually recognizing the GUI-type button displayed on the monitor 16.

Next, the operation of image processing by the digital camera of the present embodiment will be described with reference to the flowchart of FIG. In the following, it is assumed that the shooting mode is preset to the panorama mode by the user via the operation member 19.

When the power button of the operation member 19 is pressed by the user, the CPU 14 turns on the power to the digital camera and initializes each part. When the CPU 14 is in the shooting mode and is in standby for imaging, the image sensor 12 starts imaging a through image, and for example, the through image is displayed on the entire display range 30 of the monitor 16 (FIG. 3a). The CPU 14 causes the image sensor 12 to start image pickup of the field of view in the panoramic mode in response to the image pickup instruction of the full press operation of the release button 20 by the user. At the same time, the user starts the panning operation of the digital camera. In the present embodiment, the user will explain that the digital camera is panned in the horizontal direction, but the same applies to the case of the vertical direction and the like, and the description thereof will be omitted.

Further, when the CPU 14 starts the panoramic mode moving image imaging, the display of the monitor 16 is switched from the display shown in FIG. 3a to the display shown in FIG. 3b. That is, as shown in FIG. 3b, the CPU 14 reduces the frame of the moving image captured by the moving image to the display area 40 arranged on the right side of the display range 30 of the monitor 16 and displays it in real time instead of the through image. On the other hand, the CPU 14 displays a part of the panoramic images sequentially generated by the panoramic composition process described later in the display area 50 arranged on the left side of the display range 30 in real time. However, as shown in FIG. 3b, since there is no panoramic image that has undergone panoramic composition processing at the start of moving image imaging, nothing is displayed in the display area 50.

Step S101: The CPU 14 reads the first frame output from the image sensor 12 via the AEF 13 and sets it as a composite image as a reference for the panoramic composite processing. Then, the CPU 14 reads the second frame and sets it as a target image to be joined to the composite image.

Step S102: The image processing unit 22 extracts an image feature amount such as an edge amount and color information of the target image and the composite image by a known image processing. Based on the extracted image feature amount, the image processing unit 22 can detect the movement amount and pan direction of the target image with respect to the composite image by the pan operation of the user's digital camera by correlation processing between the target image and the composite image. Determine if it is. If the movement amount and the pan direction can be detected, the image processing unit 22 proceeds to step S103 (YES side), and if the movement amount and the pan direction cannot be detected, the image processing unit 22 proceeds to step S106 (NO side).

Step S103: The image processing unit 22 performs a panoramic composition process of connecting the image area newly captured by the target image to the composite image based on the movement amount and the pan direction detected in step S102.

Step S104: The image processing unit 22 displays the obtained pan direction on the monitor 16 with an arrow 60, and at the same time, slides the panoramic composite image in step S103 into the display area 50 in the direction opposite to the pan direction. Display in real time (Fig. 4a). Here, FIG. 4a shows, as an example, the display state of the monitor 16 immediately after the start of the panoramic composition process.

When the target image is vertically displaced or obliquely captured with respect to the composite image, the CPU 14 displays the composite image in which the target image is panoramicly composited, and the image protrudes from the display area 50. It is preferable to trim the area for display. Alternatively, the CPU 14 may reduce the composite image so as to display the protruding image area as well.

Step S105: The CPU 14 determines whether or not the user has accepted the image pickup end instruction of the full press operation of the release button 20 again. When the CPU 14 receives the imaging end instruction, the CPU 14 ends the panoramic mode moving image imaging. The image processing unit 22 generates a composite image as a panoramic image file and records it in the memory 15 or the storage medium 18. The CPU 14 ends a series of processes. In the present embodiment, it is preferable that the CPU 14 generates a panoramic image file by trimming the vertical size of the composite image with the vertical width of the image of the first frame, for example. Further, when the user pans the digital camera in the vertical direction, it is preferable that the CPU 14 trims the horizontal size of the composite image by, for example, the horizontal width of the image of the first frame.

On the other hand, when the CPU 14 has not received the image pickup end instruction, the process proceeds to step S101 (NO side), and the next captured frame is read as the target image. The CPU 14 performs the processes of steps S101 to S104 on the target image.

Step S106: The image processing unit 22 moves the target image with respect to the composite image, such as the panning operation speed of the digital camera by the user is too fast and the movement amount is too large, or the posture of the digital camera changes too much. And if the pan direction cannot be detected, the user is notified of the detection failure. That is, the image processing unit 22 displays a warning display as shown in FIG. 4b on the monitor 16 and informs the user. For example, the image processing unit 22 indicates to the user the direction in which the digital camera should be panned by the arrow 70 instead of the arrow 60 in order to capture the target image in which the movement amount can be detected and is joined to the composite image. And instruct. Further, the image processing unit 22 may display a warning such as the symbol 71 or the sentence 72 on the monitor 16 together with the arrow 70. Alternatively, as another example of the warning display, the image processing unit 22 may indicate a region in which the target images should be joined in the composite image, such as a shaded region 73.

The CPU 14 may display the above warning to the user even when the user stops the digital camera for a certain period of time without panning, or pans the digital camera in a direction different from the arrow 60. preferable.

Step S107: The image processing unit 22 temporarily stores the image feature amount of the composite image extracted in step S102 in, for example, an internal memory (not shown) of the CPU 14. The amount of image features of the stored composite image is the image feature of the image region having the same size as the target image or about 70 to 80% of the size of the target image from the end of the composite image on the side where the target images should be joined. It may be just the amount. As a result, the memory capacity can be reduced, and the circuit scale can be reduced.

Step S108: The image processing unit 22 pans the user in the direction of the arrow 70, for example, based on the warning display while referring to the frame of the display area 40, and the next frame imaged by the image sensor 12 (for example, the image processing unit 22). The third frame) is read as a new target image.

Step S109: The image processing unit 22 extracts the image feature amount of the target image read in step S108. The image processing unit 22 can detect the movement amount and pan direction of the target image with respect to the composition processing based on the image feature amount of the extracted target image and the image feature amount of the composite image stored in the internal memory (not shown). Determine if it is. When the movement amount and the pan direction can be detected, the image processing unit 22 proceeds to step S110 (YES side). On the other hand, when the movement amount and the pan direction cannot be detected, the image processing unit 22 shifts to step S108 (NO side). The image processing unit 22 performs the processes of steps S108 to S109 until the movement amount and the pan direction of the target image with respect to the compositing process are detected.

Step S110: The image processing unit 22 cancels the warning display shown in FIG. 4b and switches to the display shown in FIG. 4c. The image processing unit 22 proceeds to step S103 and restarts the panoramic composition process of joining the target image newly read in step S108 to the composite image.

As described above, in the present embodiment, even if the digital camera is greatly panned to the composite image, the panoramic composition is performed to what extent by displaying the moving image and the composite image that has been panoramicly synthesized up to that point in real time. , It is possible to easily determine where the panoramic composition failed, and the image pickup direction of the digital camera can be easily returned to the point where the panoramic composition failed. As a result, it is possible to generate a panoramic image with high accuracy without performing imaging in the panoramic mode again from the beginning.
<< Supplementary matters of the embodiment >>
(1) In the above embodiment, an example in which the processing of the image processing unit 22 is realized by software in the CPU 14 has been described, but the processing may be realized by hardware using an ASIC.

(2) In the above embodiment, the image processing unit 22 detects the movement amount and the pan direction of the target image based on the image feature amount of the entire image area of the target image and the composite image. Not limited. For example, a glue margin portion having a predetermined width is set around the target image and the composite image, respectively, and the image processing unit 22 synthesizes the target image based on the amount of image features in the image area other than the glue margin portion. The amount of movement and the pan direction with respect to the above may be detected.

Further, the digital camera is provided with an acceleration sensor, an electronic gyro, and the like, and the image processing unit 22 synthesizes the movement amount and the pan direction detected by the image processing unit 22 and the movement amount and the pan direction detected by the acceleration sensor and the like. The target image may be stitched to the image. This enables more accurate panoramic composition.

(3) In the above embodiment, when the user starts panning of the digital camera, the CPU 14 displays the frame of the moving image in real time in the display area 40 on the right side of the display range 30 of the monitor 16, and displays the frame on the left side. Although it is assumed that the composite image processed by panoramic composition is displayed as a slide in 50, the present invention is not limited to this. For example, as shown in FIG. 5a, the CPU 14 displays the frame of the moving image in real time in the display area 40'upper than the center of the monitor 16, and displays the composite image in the lower display area 50'in the progress of the panoramic composite processing. It may be displayed in real time accordingly. FIG. 5b shows an example of warning display when the movement amount of the target image and the detection of the pan direction with respect to the composite image fail.

Before starting the image pickup in the panorama mode, the CPU 14 accepts the user to specify the image pickup range for panoramic image pickup such as 180 degrees and 360 degrees via the operation member 19, and the display area 50'corresponding to the image pickup range. You may set the size of. As a result, the user can easily recognize to what extent the imaging is completed. Further, when the imaging is completed up to the set imaging range, the CPU 14 may automatically end the imaging in the panoramic mode even if the release button 20 is not fully pressed again.

Further, the CPU 14 displays the frame of the moving image in full screen in the display range 30 shown in FIG. 3a while the panoramic composition processing is smoothly performed, and detects the movement amount of the target image and the pan direction for the composition processing. If it fails, the warning display may be switched to that shown in FIGS. 4b and 5b.

Further, FIG. 6 shows an example of the display of the monitor 16 when the user pans the digital camera in the vertical direction to take an image of a tower, a skyscraper, or the like.

(4) In the above embodiment, in step S105, the CPU 14 terminates the image pickup in the panoramic mode when the release button 20 is fully pressed again by the user, but the present invention is not limited to this, and the present invention is not limited to this, for example. It is preferable that the user can fully press the release button 20 at any time during the imaging in the panoramic mode, and the CPU 14 can receive the interrupt of the imaging end.

(5) In the above embodiment, it is assumed that a moving image is captured and panoramic composition is performed, but the present invention is not limited to this, and can be applied to continuous shooting or a case where still images are captured one by one and panoramic composition is performed. ..

The above detailed description will clarify the features and advantages of the embodiments. It is intended that the claims extend to the features and advantages of the embodiments as described above, without departing from their spirit and scope of rights. Further, a person having ordinary knowledge in the technical field should be able to easily come up with any improvements and changes, and there is no intention of limiting the scope of the embodiments having the invention to the above-mentioned embodiments. It is also possible to use suitable improvements and equivalents within the scope disclosed in.

11 ... image pickup optical system, 12 ... image sensor, 13 ... AFE, 14 ... CPU, 15 ... memory, 16 ... monitor, 17 ... recording I / F, 18 ... storage medium, 19 ... operation member, 20 ... release button, 21. ... Touch panel, 22 ... Image processing unit

Claims (7)

An image sensor that captures the subject and outputs image data,
A display unit that displays an image generated based on the image data output from the image sensor, and a display unit.
It is provided with an image processing unit that generates a composite image based on the plurality of images generated by being imaged while changing the image pickup direction.
The display unit displays an image generated based on the image data output from the image sensor and the composite image generated by the image processing unit, and displays a direction image indicating a direction for changing the image pickup direction. An image pickup device that switches the direction image to a warning display to the user based on the change direction of the image pickup direction. The imaging device according to claim 1, wherein the warning display is a display indicating an imaging direction to be changed by the user. The image pickup apparatus according to claim 1, wherein the warning display indicates a region in which images captured by the image pickup element should be joined in the composite image. The imaging device according to any one of claims 1 to 3, wherein the warning display is a symbol or text display. An image sensor that captures the subject and outputs image data,
A display unit that displays an image generated based on the image data output from the image sensor, and a display unit.
It is provided with an image processing unit that generates a composite image based on the plurality of images generated by being imaged while changing the image pickup direction.
The display unit displays an image generated based on the image data output from the image pickup element and the composite image generated by the image processing unit, and displays a first image indicating a direction for changing the image pickup direction. The image is displayed, and the image generated based on the image data output from the image pickup element, the composite image, and the second image different from the first image are displayed based on the change direction of the image pickup direction. , Imaging device. It ’s an image pickup device.
An image sensor that captures the subject and outputs image data,
A display unit that displays an image generated based on the image data output from the image sensor, and a display unit.
It is provided with an image processing unit that generates a composite image based on the plurality of images generated by being imaged while changing the image pickup direction.
The display unit displays an image generated based on the image data output from the image pickup element and the composite image generated by the image processing unit, and displays a first image indicating a direction for changing the image pickup direction. The image is displayed, and the image generated based on the image data output from the image pickup element based on the movement amount of the image pickup device, the composite image, and the second image different from the first image are displayed. Image pickup device. The imaging device according to claim 5 or 6, wherein the second image is a symbol or text display.

Images