JP5847570B2 - Image processing apparatus and control method thereof - Google Patents

Description

The present invention relates to images processing device, in particular, to read technical data from the image sensor in the imaging apparatus.

  Conventionally, many methods for reading an omnidirectional image from a camera have been proposed. Patent Document 1 proposes a camera device that captures a 360-degree panoramic image and a display system that displays a panoramic image captured from the camera device. In particular, it has been proposed to convert an image obtained by cutting a part of a 360-degree panoramic annular image called a viewport into a rectangular projection image, and send it to a local receiver (or computer) for display.

  In Patent Document 2, in an image processing system, a part of a distortion image captured by a camera device using a lens for wide-angle imaging is cut out and transmitted to a display device, and the display device performs distortion corresponding to the distortion characteristics of the lens. Correction is performed. The camera device includes a receiving unit that receives angle-of-view information corresponding to the size of the display image after distortion correction displayed on the display device from the display device. In addition, the camera device includes an image cutout unit that cuts out a cutout image including a distortion region corresponding to the display image from the distortion image based on the view angle information received from the display device. The camera device further includes an image reduction unit that reduces the cut-out image at a ratio calculated from the angle-of-view information received from the display device based on the distortion characteristics of the lens.

  In Patent Document 3, which is referred to in Patent Document 2, there is a method in which the server side has a distortion correction unit and notifies the display device side (client side) of setting information used for the angle-of-view setting unit accompanying a change in the line-of-sight direction. Proposed.

Japanese translation of PCT publication No. 2002-515984 JP 2010-219872 A JP 2006-148767 A

  However, in the configuration of Patent Document 1, when a high pixel sensor is used, there is a problem that it takes a lot of time to capture all the pixels of the annular image every time a partial image is viewed.

  In the configuration of Patent Document 2, there is a problem that the size of the image to be cut out cannot be determined unless the size of the display image is determined on the display device side and the angle of view information is not transmitted to the camera by communication. In addition, when matching with the view angle information on the display side, there is a problem that a desired image is reduced and resolution is lowered. Further, when the image sensor is directly cut out, there is a problem that an image that does not match the characteristics of the image sensor is cut out if it matches the angle of view and position information on the display side.

  In the configuration of Patent Document 3, when a high pixel sensor is used, there is a problem that the data size in the capture unit increases and the image processing load increases.

The present invention, images of the partial area image for subject confirmation cut from captured on the image sensor through a wide-angle optical system image, with less processing load, and high image quality, to enable you to present to the user A processing device is provided.

According to one aspect of the present invention, in the images of the light focused through the imaging optical system Ru made by Ri production to an image sensor for photoelectrically converting a partial area corresponding to the object, the imaging optical system determined based on the optical properties, and cut-out means for cutting out an image of the partial area on the image of the partial region extracted by said cut means, a geometric conversion based on optical characteristics of the imaging optical system When the image processing means to perform and the size of the image of the partial area geometrically transformed by the image processing means is larger than the display angle of view in the display means for displaying the image of the partial area, the image is included in the display angle of view. the geometric converted images processor you; and a trimming unit for trimming the image of the partial region is provided by the image processing means.

  According to the present invention, it is possible to present a subject-confirming partial region image cut out from an image captured by an image sensor via a wide-angle optical system to a user with a small processing load and high image quality.

FIG. 2 is a block diagram illustrating the imaging apparatus according to the first embodiment. 5 is a flowchart illustrating the operation of the imaging apparatus according to the first embodiment. The figure which shows the relationship between the user designation | designated area | region in the whole view image data in Embodiment 1, and the cut-out rectangular shape containing the said user designation | designated area | region. The figure which shows the example which geometrically transformed and displayed the cut-out rectangular partial image data in Embodiment 1. FIG. FIG. 6 is a diagram illustrating a change on a cut-out rectangular image sensor when an omnidirectional mirror and lens are used in the wide-angle optical system according to the first embodiment. FIG. 6 is a block diagram illustrating an imaging apparatus according to a second embodiment. 9 is a flowchart illustrating the operation of the imaging apparatus according to the second embodiment. The figure which shows the relationship between the moving body detection area | region and readout rectangular shape in Embodiment 2. FIG. The figure which shows the relationship between several moving body detection area | region and several read-out rectangle shape in Embodiment 2. FIG.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In addition, this invention is not limited to the following embodiment, It shows only the specific example advantageous for implementation of this invention. Moreover, not all combinations of features described in the following embodiments are indispensable for solving the problems of the present invention.

<Embodiment 1>
Hereinafter, Embodiment 1 which concerns on this invention is demonstrated based on FIGS.

  FIG. 1 is a block diagram of an imaging apparatus according to the first embodiment. In this embodiment, an imaging optical system that collects scenes from the outside world and forms an image on a sensor is constituted by an omnidirectional mirror and lens 10. An image sensor 11 that photoelectrically converts a subject image formed through an omnidirectional mirror & lens 10 is composed of a semiconductor sensor such as a CMOS sensor or a CCD sensor. The image sensor 11 in the present embodiment is equipped with a function for imaging a 360 degree scene from the omnidirectional mirror & lens 10 and a partial readout function for cutting out a part from the 360 degree scene. The sensor control unit 12 performs data read control for reading image data generated by photoelectric conversion by the image sensor 11, temporary storage of the read data, control of image processing such as development processing on the data, and the like. . The geometric conversion processing unit 13 performs geometric correction on the supplied image data based on the optical characteristics of the omnidirectional mirror & lens 10 to convert the data into data with no distortion. The drawing display processing unit 14 performs display processing so that the whole-view image data temporarily stored by the sensor control unit 12 and the data subjected to geometric correction by the geometric conversion processing unit 13 and free from distortion are matched with the display 15. The display 15 is a display unit that displays data subjected to display processing by the drawing display processing unit 14. The operation unit 16 designates a partial area in the whole view data displayed on the display 15.

  The omnidirectional mirror & lens 10 is composed of a combination of an omnidirectional mirror and a lens using a curved mirror, thereby realizing an omnidirectional imaging optical system. Note that an omnidirectional imaging optical system can be realized by using an omnidirectional fisheye lens instead of a combination of an omnidirectional mirror and a lens. In the present embodiment, an example in which the omnidirectional mirror & lens 10 is configured by a combination of the former omnidirectional mirror and a lens will be taken.

  FIG. 2 is a flowchart illustrating the operation of the imaging apparatus according to the present embodiment. FIG. 3 is a diagram showing a relationship between a user-specified area in the whole-view image data and a cut-out rectangular shape including the user-specified area when the omnidirectional mirror & lens 10 is used in the wide-angle optical system. FIG. 4 is a diagram in which rectangular partial image data cut out from the user-designated area is geometrically transformed and displayed as an orthographic shape. FIG. 5 is a diagram illustrating a change on the cut-out rectangular image sensor when the omnidirectional mirror & lens 10 is used in the wide-angle optical system. Specifically, FIG. 5 illustrates a cutout including a shape region in which the user-specified region is changed in the optical parameter when the user-specified region is changed in the circumferential direction on the annular image displayed on the image sensor 11. The change of a rectangular shape is shown.

  First, the sensor control unit 12 reads out the whole-view image data (distortion data) from the image sensor 11 that has received the light from the omnidirectional mirror & lens 10 (S100). Here, the whole-view image data refers to data representing an image of the entire area where the entire scene reflected on the omnidirectional mirror can be confirmed. Thereafter, the sensor control unit 12 performs development processing or the like on the read whole-view image data and sends it to the drawing display processing unit 14.

  The drawing display processing unit 14 that has received the whole image data performs a drawing process on the whole image data so as to match the display on the display 15. At this time, if the angle of view of the whole-view image data is larger than the angle of view of the display 15, a reduction process is performed. Even if the reduction process is performed, there is no inconvenience because the partial area to be described later is designated at the relative position. Thereafter, the whole-view image data subjected to the drawing process is displayed on the display 15 (S101).

  The user designates a partial area in which the object is reflected by the operation unit 16 based on the whole-view image data subjected to the drawing process displayed on the display 15 (S102). In the present embodiment, the partial area is specified as a rectangular shape. Therefore, this designation may be performed by dragging a rectangular area with the mouse cursor in the operation unit 16, or may be designated with the touch pen. A frame of a partial area designated by the operation unit 16 (hereinafter referred to as “partial view data area”) is displayed on the overall view image data (a1 in FIG. 3). The area information of the partial vision data area is sent to the sensor control unit 12.

  The sensor control unit 12 calculates a cut-out rectangular shape as control information based on the received partial vision data area information (S103). In the present embodiment, the shape and position of the outer peripheral rectangle (+ α) including the partial vision data area are calculated based on the optical parameters of the omnidirectional mirror & lens 10 (broken line frame in FIG. 3). As an example, a shape having a vertical and horizontal size and an aspect ratio (aspect ratio) different from that of the partial view data area is calculated.

  This will be specifically described. For example, consider the case of an omnidirectional imaging device using a hyperboloidal mirror that is one of the omnidirectional mirrors. In this case, the image distortion increases as the place where the object appears on the image sensor 11 is closer to the center of the annular image (FIG. 3), that is, as the radial distance from the center of the annular image becomes shorter. Exhibits optical properties. Conversely, the image distortion becomes smaller as the place where the object appears on the image sensor 11 is farther from the center of the annular image.

  Therefore, the shape obtained by inversely transforming the partial vision data area specified in S102 by calculation using the optical parameters (hereinafter referred to as “shape before sector conversion”) and the position are located at the center of the annular image (FIG. 3). The closer it is, the more the image on the inner diameter side of the shape before fan conversion is distorted.

  As sensor characteristics, in many cases, the partial cutout shape can be cut out only in a rectangular shape. Therefore, when the center angle of the circular image (FIG. 5) is close to vertical or horizontal as a place where the object is reflected on the image sensor 11, the cutout circumscribed rectangle fitted to the shape before the sector conversion and the shape before the sector conversion. The difference between the shape is reduced. As a result, it is possible to specify a small margin (+ α) between the partial cutout shape and the shape before sector conversion (a′11 and a′14 in FIG. 5).

  However, when the center angle of the circular image (FIG. 5) is about 30 degrees to about 80 degrees as a place where the object is reflected on the image sensor 11, the circumscribing rectangle for cutting applied to the shape before fan conversion is used. The difference from the shape before the sector conversion becomes large. As a result, the partial cut-out shape needs to be specified as a margin (+ α) between the pre-fan shape conversion shape and a vertically long shape when close to the vertical direction, and a horizontal shape when close to the horizontal direction. Therefore, it is necessary to change the aspect ratio of the partially cut out shape (a′12 and a′13 in FIG. 5).

  Furthermore, it is necessary to specify a large margin (+ α) particularly in the radial direction in order to prevent the loss of the image when zooming the circumscribed rectangular image of the target object after the image distortion correction.

  As described above, the shape and position of the outer rectangle + α, which is the rectangular readout shape, are calculated.

  In the present embodiment, a rectangular region in which a′1 in FIG. 3 is larger in the circumferential direction than a1 is required.

  The sensor control unit 12 performs the following processing using the partial reading function of the image sensor 11 with the calculated shape and position of the outer peripheral rectangle as control information. Cutout rectangular shape data (hereinafter, referred to as “rectangular partial image data”) obtained by cutting out a part from the whole-view image data is read (S106). At this time, the sensor control unit 12 determines whether or not to set the imaging position as the partial cutout position according to the characteristics of the image sensor 11 (S104). If it is determined that the setting is necessary, for example, when the pixels of the image sensor 11 are in a Bayer array, the imaging position is determined so that the unit in which the RGB of the Bayer is arranged is the same as the unit in which the whole-view data is arranged ( S105).

  For example, as sensor characteristics, when the sensor pixels are arranged in the RGB Bayer arrangement as described above, the color changes unless the cutout position is shifted in units of four pixels. Therefore, it is necessary to correct the cut-out position calculated by the above optical characteristics to the position of the unit of 4 pixels that is the sensor characteristic.

  The sensor control unit 12 performs development processing or the like on the read rectangular partial image data, and sends the processed rectangular partial image data to the geometric transformation processing unit 13. The geometric conversion processing unit 13 performs image data geometric conversion (also referred to as distortion correction) on the rectangular partial image data based on the optical parameters of the omnidirectional mirror & lens 10 (S107). The geometric transformation processing unit 13 sends the rectangular partial image data subjected to the geometric transformation to the drawing display processing unit 14.

  The drawing display processing unit 14 performs drawing processing on the geometrically converted rectangular partial image data so as to match the display on the display 15. Specifically, a color conversion process or the like that matches the color of the display 15 is performed. At this time, the drawing display processing unit 14 determines whether or not the angle of view (size) of the geometrically converted rectangular partial image data is larger than the display angle of view of the display 15 (S108). When it is determined that the angle of view of the geometrically converted rectangular partial image data is larger than the display angle of view of the display 15, the drawing display processing unit 14 performs trimming on the rectangular partial image data and displays the display image of the display 15. It fits within the corner (S109). The rectangular partial image data processed in this way is displayed on the display 15 (S110) (FIG. 4).

<Embodiment 2>
Next, a second embodiment will be described.

  FIG. 6 is a block diagram of the imaging apparatus according to this embodiment. 6 is the same as FIG. 1 except that a moving object detection processing unit 17 that detects a moving object is added to FIG. The same processing blocks as those in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted.

  FIG. 7 is a flowchart illustrating the operation of the imaging apparatus according to the present embodiment. FIG. 8 shows an example of the whole-view image data (distortion data) in the present embodiment. In FIG. 8, a solid line frame indicated by b <b> 1 is a moving object detection frame obtained by the moving object detection processing unit 17. A broken-line frame indicated by b'1 is an effective rectangular range (hereinafter referred to as "moving object viewing area") calculated from the position of the moving object detection frame b1 and the optical characteristics of the omnidirectional mirror & lens 10.

  First, the sensor control unit 12 reads out the whole-view image data (distortion data) from the image sensor 11 that has received the light from the omnidirectional mirror & lens 10 (S200). The sensor control unit 12 performs development processing or the like on the read entire image data, and sends this to the moving object detection processing unit 17. The moving object detection processing unit 17 performs a moving object detection process on the received whole-view image data (S201). The sensor control unit 12 determines whether or not a moving object is detected by the moving object detection process (S202). If no moving object is detected by the moving object detection process, the process proceeds to S208. If a moving object is detected by the moving object detection process, the sensor control unit 12 determines whether the number of detected moving objects is one or more in S203. If the number of detected moving objects is one, the process proceeds to S204.

  In S204, the moving object detection frame (b1) obtained by the moving object detection process, the position of the moving object, the optical characteristics of the omnidirectional mirror & lens 10, etc. b′1) is calculated. For example, rectangular shape data having a vertical and horizontal size and an aspect ratio (aspect ratio) different from those of the moving object detection frame are calculated. Note that when calculating the moving object visual region, the moving object visual region may be set with more margins in the moving direction of the moving object.

  The calculated moving object view area is set as a cut-out rectangular area (S205). Next, it is determined whether or not the position corresponding to the characteristics of the image sensor 11 is the imaging position (S206). If it is determined that the image pickup position does not meet the characteristics, a cut-out rectangular position as the image pickup position is determined so as to be the same as the RGB pixel arrangement of the whole-view image data (S207).

  Thereafter, the cut-out rectangular area and the imaging position are designated as control information, and the following processing is performed using the partial cut-out function that is a function of the image sensor 11. That is, the sensor control unit 12 reads out cut-out rectangular shape data (hereinafter referred to as “rectangular partial image data”) obtained by cutting out a part from the whole-view image data (S208). The rectangular partial image data is sent to the geometric transformation processing unit 13. The geometric conversion processing unit 13 performs image data geometric conversion on the received rectangular partial image data based on the optical parameters of the omnidirectional mirror & lens 10 (S209). The rectangular partial image data after the geometric transformation is sent to the drawing display processing unit 14.

  The drawing display processing unit 14 that has received the received rectangular partial image data after geometric transformation performs drawing processing so as to match the display on the display 15. At this time, when the display aspect ratio and the display size of the display 15 are not matched, a process of trimming or resizing the rectangular partial image data is performed. Thereafter, the rectangular display image data after geometric transformation is displayed on the display 15 by the drawing display processing unit 14 (FIG. 4) (S210).

  Next, processing when a plurality of moving objects are detected in S203 will be described. FIG. 9 shows a plurality of moving object detection frames C1 and C2 on the overall image data. C′1 and C′2 indicated by the broken line frames are moving object view areas calculated from the positions of the moving object detection frames C1 and C2, the optical characteristics of the omnidirectional mirror & lens 10, and the like.

  When a plurality of moving objects are detected in S203, a gaze moving object can be specified (moving object specifying process) on the whole-view image data by a user operation using the operation unit 16 (S211). Here, it is assumed that C1 in FIG. 9 is designated. The sensor control unit 12 uses the moving object detection frame C′1 as control information to be specified for the partial reading function based on the moving object detection frame C1 specified by the user, the position of the moving object, the optical characteristics of the omnidirectional mirror & lens 10, and the like. Is calculated (S212). Thereafter, the process proceeds to S205.

  The embodiments of the present invention have been described above. In addition, this invention is not limited to the above-mentioned embodiment. For example, the following modifications are possible.

  In Embodiments 1 and 2, a combination of an omnidirectional mirror and a lens is used as the wide-angle optical system. However, as described above, an all-around fisheye lens or the like may be used instead. In the case of an omnidirectional imaging optical system using an omnidirectional fisheye lens, the optical characteristics are opposite to those of an omnidirectional imaging optical system using an omnidirectional mirror. As the object is reflected, the closer to the outer periphery of the annular image, that is, the longer the radial distance from the center of the annular image, the greater the image distortion.

  In the first and second embodiments, finally, the rectangular partial image data is displayed on the display 15. Instead of or in addition to this, the rectangular partial image data may be distributed to a network destination terminal or stored in a storage device.

  In the first and second embodiments, the whole-view image data may be all-pixel read data from the image sensor 11 or thinned-out read data.

  In the first embodiment, the cutout area is designated by a user operation via the operation unit 16, but the cutout area may be automatically designated by specifying the feature (eg, human body) of the cutout image data.

  In the second embodiment, when a plurality of moving objects are detected, the gaze moving object is designated by a user operation via the operation unit 16. Instead, it may be automatically specified under a predetermined condition, or may be processed by switching a gaze moving body from a certain moving body at regular intervals. As the predetermined conditions, conditions related to the position of the moving object, the size of the moving object, the moving speed of the moving object, and the like can be considered.

  In the second embodiment, the moving object detection processing unit 17 is configured as a block different from the image sensor 11. However, the image sensor 11 may serve as the moving object detection processing unit 17.

  In the second embodiment, the processing of S201 to S210 is repeatedly executed in accordance with the movement of the moving object, and the display is updated to display the rectangular partial image data tracking the moving object and confirm the moving subject. Good.

  According to the embodiment described above, the shape to be read can be dynamically changed depending on the imaging position of the wide-angle optical system. In addition, by setting the cutout position according to the image sensor characteristics, it is possible to partially read the data range for confirming the subject, and to reduce the amount of data to be processed. Furthermore, since the partially read data has the maximum resolution and the color does not change from the original data, there is an effect that the subject can be easily confirmed.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

Claims (11)

In images of the light focused through the imaging optical system Ru made by Ri production to an image sensor for photoelectrically converting a partial area corresponding to the object is determined based on optical characteristics of the imaging optical system, Clipping means for cutting out the image of the partial area;
Image processing means for performing geometric transformation on the image of the partial area cut out by the cutting means based on optical characteristics of the imaging optical system;
When the image size of the partial area geometrically transformed by the image processing means is larger than the display angle of view in the display means for displaying the image of the partial area, the image processing means is set so as to fall within the display angle of view. images processor you; and a trimming unit for trimming the image of the geometric transformation is the partial regions by. The imaging optical system, images processing device according to claim 1, characterized in that the omnidirectional imaging optical system using a combination of the omnidirectional mirrors and lenses. The imaging optical system, images processing device according to claim 1, characterized in that the omnidirectional imaging optical system using the entire circumference fisheye lens. Images it is processing device according to any one of claims 1 to 3, characterized by further comprising a designation means for designating by a user manipulating the object. The cutting means is configured to identify the partial region as a rectangular shape,
It said cutout means includes an optical characteristic of the imaging optical system, based on the position of the image of the object before Kiga image, claims and determines the size and aspect ratio of the aspect of the rectangular shape images processing device according to any one of claim 1 to 4. A motion detection means for detecting a moving object from the previous Kiga image,
Setting means for setting a moving object view area of the moving object detected by the moving object detection means as the partial area;
Further images processing device according to any one of claims 1 to 5, characterized in that it has a. In the case where a plurality of moving objects are detected by the moving object detecting means, the moving object specifying means for specifying any one moving object from the plurality of moving objects by a user operation,
The setting means, images processing device according to claim 6, characterized in that to set the moving object vision area of the designated moving body by the moving object specifying means as the partial region. It said motion detection means, images processing device according to claim 6, characterized in that it is configured to track the detected moving object. Said cutout means further in accordance with the arrangement of pixels in the image sensor, images processing device according to any one of claims 1 to 8, characterized in that to determine the position of the partial area. In images of the light focused through the imaging optical system Ru made by Ri raw Louis image sensor to photoelectrically converts the partial area corresponding to the object determined based on the optical characteristics of the imaging optical system and the cut-out step of cutting out the image of the partial region,
The image of the partial region cut out by the previous SL cutout step, an image processing step of performing geometric transformation on the basis of the optical characteristics of the imaging optical system,
If the size of the image of the partial area geometrically transformed by the image processing step is larger than the display angle of view in the display means for displaying the image of the partial area, the image processing step so as to fall within the display angle of view. geometric converted control method of image picture processor you; and a trimming step for trimming the image of the partial region by. The computer program for executing the steps of the control method according to claim 10.