JP6827302B2 - Image compositing device and image compositing method - Google Patents

Description

The present invention relates to an image compositing apparatus and an image compositing method. More specifically, the present invention relates to an image compositing device and an image compositing method for synthesizing images obtained by a plurality of cameras on the rear and side of the vehicle.

Conventionally, instead of the rear-view mirror of a vehicle, various rear-view monitors have been put into practical use in which the rear view of the vehicle is photographed by a camera and the road surface, an approaching object, or the like is displayed on the screen. Normally, since the camera used for the rear monitor captures a wide range, there is a problem that the captured image is greatly distorted and it is difficult to see it as it is. Therefore, a method of eliminating distortion of an image taken by using a correction table or the like is known (see, for example, Patent Document 1). Further, in recent years, an image processing device having a plurality of cameras in a vehicle and having improved visibility by using a method of synthesizing two-dimensional image data obtained by using each camera into one three-dimensional projection plane shape has been introduced. It has been developed (see, for example, Patent Document 2).
When synthesizing images taken by multiple cameras, if the same object is located near the boundary of each image, the shooting position will be different, so there will be problems such as objects with different appearances being mixed and displayed in the composite image. Occurs. In order to solve such a problem, for example, the image processing apparatus described in Patent Document 2 has a three-dimensional projection surface shape as a plane extending from a rectangular arrangement surface on which a vehicle is arranged and extending from the plane. It is composed of a curved surface provided, and a plurality of two-dimensional image data obtained by imaging with each camera are combined into one three-dimensional projection plane shape.

Japanese Unexamined Patent Publication No. 10-271490 Japanese Unexamined Patent Publication No. 2010-128951

Blind spots can be reduced by photographing the rear of the vehicle with multiple cameras. However, as described above, when images taken by a plurality of cameras are combined and displayed, all the objects located near the boundary of the images taken by each camera are seamless and there is little distortion as a whole. It is difficult to display it so that it looks natural. For example, even if a plurality of 2D images obtained by capturing images with each camera are combined on one 3D projection plane, if the two-dimensional images are combined so that there is no seam of one object, the distance from the camera is different. There are problems such as unnatural seams and distortions in the object, and the object is out of the projection range and disappears.
Further, in an application such as displaying an image behind a vehicle like a rear-view mirror, it is necessary to process images taken by a plurality of cameras and combine them at high speed. Further, it is required to easily display the image behind the vehicle including the objects behind the vehicle that the driver should pay attention to.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image synthesizing device and an image synthesizing method for synthesizing images obtained by a plurality of cameras at the rear and sides of a vehicle.

1. 1. An image synthesizer that synthesizes images on the road surface and on the road behind the vehicle, on the right side, and on the left side.
Images that acquire a rear image, a right rear image, and a left rear image from images taken by a rear camera that captures the rear of the vehicle, a right rear camera that captures the right rear, and a left rear camera that captures the left rear, respectively. Acquisition method and
A coordinate conversion means for converting the rear image, the right rear image, and the left rear image into a projected rear image projected onto a predetermined projection surface, a projected right rear image, and a projected left rear image, respectively.
An image synthesizing means for obtaining a rear composite image by synthesizing the projected rear image, the projected right rear image, and the projected left rear image.
With
The predetermined projection surface is a tubular elliptical cylinder projection surface whose horizontal cross-sectional shape is an ellipse centered on a predetermined virtual viewpoint in a three-dimensional coordinate space whose origin is a predetermined position, and the ground corresponding to the ground. Consists of a projection plane
The coordinate conversion means displays the rear image, the right rear image, and the left rear image from the positions of the rear camera, the right rear camera, and the left rear camera in the three-dimensional coordinate space, respectively. Project on the ground projection surface ,
The image synthesizing means generates the rear composite image obtained by synthesizing the projection rear image, the projection right rear image, and the projection left rear image, respectively, with the transition region in between.
The coordinate conversion means is a line segment connecting the rear camera and the right rear camera or the left rear camera when the projected coordinates on the elliptical cylinder projection surface are located below the ground projection surface in the transition region. A base point is set on the top, and conversion is performed so that the point where the line segment connecting the base point and the projection coordinates on the elliptical cylinder projection plane intersects the ground projection plane is the projection coordinates on the ground projection plane. A featured image synthesizer.
2. 2. The virtual viewpoint is in front of the vehicle in the three-dimensional coordinate space.
The elliptical cylinder projection surface is within a predetermined angle with respect to the rear of the vehicle from the virtual viewpoint in a horizontal cross section.
The ground projection surface is within a range in contact with the elliptical cylinder projection surface. The image synthesizer described.
3. 3. The coordinate conversion means generates an upper region in the projected rear image, the projected right rear image, and the projected left rear image from the projected image on the elliptical cylinder projection plane, and the lower region is a projected image on the ground projection plane. Generated from the above 1. Or 2. The image synthesizer according to.
4. The image synthesizing means arranges the projected rear image so as to spread downward in the lower region of the rear composite image. To 3. The image synthesizer according to any one of .
5 . It is an image composition method that synthesizes images on the road surface and the road surface behind the vehicle, on the right side, and on the left side.
Images that acquire a rear image, a right rear image, and a left rear image from images taken by a rear camera that captures the rear of the vehicle, a right rear camera that captures the right rear, and a left rear camera that captures the left rear, respectively. Acquisition process and
A coordinate conversion step of converting the rear image, the right rear image, and the left rear image into a projected rear image projected onto a predetermined projection surface, a projected right rear image, and a projected left rear image, respectively.
An image synthesizing step of obtaining a rear composite image by synthesizing the projected rear image, the projected right rear image, and the projected left rear image.
With
The predetermined projection surface is a tubular elliptical cylinder projection surface whose horizontal cross-sectional shape is an ellipse centered on a predetermined virtual viewpoint in a three-dimensional coordinate space whose origin is a predetermined position, and the ground corresponding to the ground. Consists of a projection plane
In the coordinate conversion step, the rear image, the right rear image, and the left rear image are converted from the positions of the rear camera, the right rear camera, and the left rear camera in the three-dimensional coordinate space to the elliptical cylinder projection surface and the left rear image, respectively. Project on the ground projection surface ,
In the image synthesizing step, the rearward composite image is generated by synthesizing the projection rear image, the projection right rear image, and the projection left rear image, respectively, with the transition region in between.
In the coordinate conversion step, when the projected coordinates on the elliptical cylinder projection plane are located below the ground projection plane in the transition region, the line segment connecting the rear camera with the right rear camera or the left rear camera. A base point is set on the top, and conversion is performed so that the point where the line segment connecting the base point and the projection coordinates on the elliptical cylinder projection plane intersects the ground projection plane is the projection coordinates on the ground projection plane. Characteristic image composition method.
6 . The virtual viewpoint is in front of the vehicle in the three-dimensional coordinate space.
The elliptical cylinder projection surface is within a predetermined angle with respect to the rear of the vehicle from the virtual viewpoint in a horizontal cross section.
Wherein said ground plane of projection is in the range in contact with the elliptic cylindrical projection surface 5. The described image composition method.
7 . In the coordinate conversion step, the upper region in the projection rear image, the projection right rear image and the projection left rear image is generated from the projection image on the elliptical cylinder projection surface, and the lower region is the projection image on the ground projection surface. Generated from the above 5 . Or 6 . The image composition method described in.
8 . The 5 wherein the image synthesis process, which in the lower region of the rear composite image, the projection rearward image disposed to spread downward. To 7. The image composition method according to any one of .

According to the present invention, the rear image, the right rear image and the left side are taken from the images taken by the rear camera for photographing the rear of the vehicle, the right rear camera for photographing the right rear, and the left rear camera for photographing the left rear, respectively. An image acquisition means for acquiring a rear image, and coordinate conversion for converting the rear image, the right rear image, and the left rear image into a projected rear image projected on a predetermined projection surface, a projected right rear image, and a projected left rear image, respectively. A means and an image synthesizing means for obtaining a rear composite image by synthesizing the projected rear image, the projected right rear image, and the projected left rear image are provided, so that images taken by three cameras are used to rearward the vehicle. A wide field of view can be secured and blind spots can be reduced. The predetermined projection plane is composed of a tubular elliptical cylinder projection plane whose horizontal cross-sectional shape is an ellipse centered on a predetermined virtual viewpoint in a three-dimensional coordinate space, and a ground projection plane corresponding to the ground. The conversion means converts the rear image, the right rear image, and the left rear image from the positions of the rear camera, the right rear camera, and the left rear camera in the three-dimensional coordinate space, respectively, to the elliptical cylinder projection surface and the ground. Since the projection is performed on the projection surface, each projected image that looks most natural can be obtained by appropriately setting the position of the virtual viewpoint and the shape of the elliptical cylinder projection surface. Further, since each projected image is combined, it is possible to generate a composite image without a sense of discomfort by processing the boundary portion of each image at the time of composition.
Further, since the projection plane is provided with an elliptical cylinder projection plane and a ground projection plane, the ratio between the area occupied by the image behind the vehicle and the area occupied by the image behind the left and right sides of the vehicle is changed according to the ellipticity of the ellipse of the ellipse cylinder. This makes it possible to improve visibility by increasing the proportion of the area occupied by the image to be focused on according to the purpose.

The virtual viewpoint is in front of the vehicle in the three-dimensional coordinate space, and the elliptical cylinder projection surface is within a predetermined angle from the virtual viewpoint to the rear of the vehicle in a horizontal cross section, and is the ground. When the projection surface is within the range in contact with the elliptical cylinder projection surface, the distortion of each projected image is reduced and the visibility is excellent as compared with the case where the virtual viewpoint is at the position of the vehicle or the position behind the vehicle. You can get the image.

The coordinate conversion means generates an upper region in the projected rear image, the projected right rear image, and the projected left rear image from the projected image on the elliptical cylinder projection plane, and the lower region is a projected image on the ground projection plane. In the case of generating from, the upper region and the lower region in each projected image can be formed by the respective transformations.

When the image synthesizing means generates the rear composite image obtained by synthesizing the projection rear image, the projection right rear image, and the projection left rear image with the transition region interposed therebetween, each projected image occupying the rear composite image. The distribution of the images can be optimized, and the boundary between each projected image can be smoothed.
The coordinate conversion means is a line segment connecting the rear camera and the right rear camera or the left rear camera when the projection coordinates on the elliptical cylinder projection plane are located below the ground projection plane in the transition region. When the base point is set on the top and the point where the line connecting the base point and the projection coordinates on the elliptical cylinder projection plane intersects the ground projection plane is converted to be the projection coordinates on the ground projection plane. , The position where the projected image of the rear camera is projected on the ground projection surface of the transition region and the position where the projected image of the right rear camera or the left rear camera is projected can be aligned. As a result, it is possible to obtain an overlapping composite image without shifting the positions of the objects on the road in the ground portion of the transition region.
When the projected rear image is arranged so as to spread downward in the lower region of the rear composite image, the image compositing means synthesizes the rear composite image so that the ground projection image near the rear end of the vehicle becomes wider. The situation near the rear of the vehicle can be displayed widely. As a result, it is possible to improve the distinctiveness in the vicinity of the rear part of the vehicle, which is necessary when the vehicle is retracted or the like.

The case where the present invention is realized as an apparatus has been described above, but the invention can also be realized as a method or program for realizing such an apparatus, or as a medium on which the program is recorded. In addition, the image synthesizer as described above may be realized independently, may be applied to a certain method, or the same method may be used in a state of being incorporated in another device. The present invention is not limited to this, and includes various aspects. Therefore, it can be changed as appropriate, such as software or hardware.

It is a block diagram for demonstrating the structure of an image composition apparatus. It is a schematic plan view which shows the position and the photographing range provided in the vehicle of each camera. It is a schematic side view which shows the position provided in the vehicle of each camera. It is an example of the image obtained by each camera. It is a schematic perspective view which shows the 3D projection plane. (A) is a diagram showing the positional relationship between the three-dimensional projection surface and the vehicle, and (b) is a diagram showing the range of images taken by the right rear camera and the left rear camera in the composite image in (a). (A) A diagram showing the positional relationship between the cylindrical three-dimensional projection surface and the vehicle as a comparative example, and (b) the range of images taken by the right rear camera and the left rear camera in the composite image in (a). It is a figure which shows. This is an example of synthesizing the images of the rear cameras on each side. FIG. 8 is a diagram showing a range captured by each camera and a boundary thereof. It is a figure which shows the range photographed by each camera, and the boundary thereof when the area of the rear image of a road surface is rectangular. It is a flowchart for demonstrating the processing of an image synthesizer.

The matters shown here are for exemplifying and exemplifying embodiments of the present invention, and are considered to be the most effective and effortless explanations for understanding the principles and conceptual features of the present invention. It is stated for the purpose of providing what seems to be. In this regard, it is not intended to show structural details of the invention beyond a certain degree necessary for a fundamental understanding of the invention, and some embodiments of the invention are provided by description in conjunction with the drawings. It is intended to clarify to those skilled in the art how it is actually realized.

FIG. 1 shows a configuration example of the image synthesizer (1) according to the present embodiment. The image compositing device (1) is an image compositing device that synthesizes images on the road surface and the road surface at the rear, right rear, and left rear of the own vehicle, and is a rear camera (21) for photographing the rear of the vehicle (91) and a right rear. Image acquisition means (3) for acquiring an image from the image taken by the right rear camera (22) for photographing and the left rear camera (23) for photographing the left rear, coordinate conversion means (4), and image composition. The means (7) is provided.
The type of vehicle to which the image synthesizer (1) is applied is not particularly limited, and examples thereof include automobiles, trains, bicycles, self-propelled robots, and remote-controlled robots. In addition to tires and wheels, endless tracks and the like can also be included as the traveling method of the vehicle.

The image acquisition means (3) captures images (a, b, c) taken by the rear camera (21), the right rear camera (22), and the left rear camera (23), and each captures the rear image (a'. ), Right rear image (b') and left rear image (c').
The coordinate conversion means (4) projects a rear image (a'), a right rear image (b'), and a left rear image (c') onto a predetermined projection plane (Q), respectively. It is a means for converting into a projection right rear image (b ") and a projection left rear image (c").
The image synthesizing means (7) is a means for synthesizing a projection rear image (a "), a projection right rear image (b"), and a projection left rear image (c ") to create one rear composite image.
The rear composite image can be displayed on the display device by the display means (8) provided separately.

The rear camera (21), the right rear camera (22), and the left rear camera (23) need only be able to acquire images of the road surface and the road surface behind the vehicle (91), the right rear, and the left rear, respectively. And installation position does not matter. For example, as shown in FIGS. 2 and 3, the rear camera (21) is fixedly provided on the rear surface of the vehicle (91). Further, the right rear camera (22) and the left rear camera (23) (hereinafter, abbreviated as the side rear camera) are fixedly arranged on the right side surface and the left side surface of the vehicle (91). Specifically, it can be arranged at a portion such as a door mirror that protrudes from the side of the vehicle (91).
Further, each camera (21, 22, 23) is arranged so that a part of the shooting range (92a, 93a, 94a) overlaps with a part of the shooting range of the adjacent camera. It is preferable that a fisheye lens or the like is used for each camera (21, 22, 23) in order to obtain a wide range of images.

FIG. 2 schematically shows the position and shooting range of each camera provided in the vehicle. The rear camera (21) is provided on the rear surface of the vehicle (91), and the side rear cameras (22, 23) are provided on the door mirror of the vehicle (91). In this example, the shooting range (93a) of the right rear camera (22) and the shooting range (94a) of the left rear camera (23) are configured to overlap at the rear of the vehicle. However, when only the left and right rear rear cameras are used, a blind spot is generated behind the vehicle, so that the rear of the vehicle is covered by the shooting range (92a) of the rear camera (21). It is sufficient that the rear and left / right rear of the vehicle are covered by the three cameras (21, 22, 23), and the shooting range (93a) of the right rear camera (22) and the shooting range (94a) of the left rear camera (23). ) May not overlap.

The image compositing device (1) synthesizes images on the road surface behind the vehicle, on the left and right sides, and on the road, but the "road surface" is not intended to be the actual road surface of the road, and the vehicle touches the ground. Refers to the peripheral part of the vehicle on a flat surface. The ratio of the road surface area included in the image captured by each camera (21, 22, 23) (that is, the vertical orientation of each camera) may be appropriately selected. Further, the image of the lower region (road surface region) in the rear composite image may be configured to be extracted mainly from the image (a) obtained by the rear camera (21).

The image acquisition means (3) captures images taken by each camera (21, 22, 23) and acquires each image (a, b, c) (see FIG. 4).
Correction of image distortion caused by the lens can then be performed at an appropriate stage. For example, the image acquisition means (3) can be configured to convert an image (a, b, c) into a spherical image (a', b', c'). Further, the coordinate conversion means (4) can be used to correct the distortion together with the conversion to each projected image (a ", b", c ").

The coordinate conversion means (4) uses the rear image (a), the right rear image (b), and the left rear image (c) as the rear camera (21), the right rear camera (22), and the left rear image (c) in the three-dimensional coordinate space, respectively. Coordinate conversion is performed so as to project from the position of the camera (23) onto a predetermined projection surface (Q).
The reference three-dimensional coordinate system can be set arbitrarily, but in the following description, the three-dimensional orthogonality with the left-right direction of the vehicle (91) as the X-axis, the height direction as the Y-axis, and the front-rear direction as the Z-axis. Let the coordinates (X, Y, Z) (see FIGS. 2 and 3). Further, the origin of the three-dimensional coordinates can be set arbitrarily, but in the following description, the origin (0, 0, 0) is set as the position of the rear camera (21).
In these three-dimensional Cartesian coordinates, the vehicle (91) is in front of the vehicle (91) (for example, it can be in front of the position of the side-rear camera, in front of the position where the rear composite image is displayed, etc.). A virtual viewpoint (65) is set in the height (Px, Py, Pz). Then, a tubular elliptical cylinder projection surface (63) whose horizontal (XZ) cross section has an elliptical shape centered on the virtual viewpoint (65) and a ground projection surface (62) corresponding to the ground are set. The predetermined projection plane (Q) is composed of an elliptical cylinder projection plane (63) and a ground projection plane (62) (see FIGS. 5 and 6). The straight line in the Y-axis direction passing through the virtual viewpoint (65) is called the central axis (66) of the elliptical cylinder projection plane (63).

By making the projection surface elliptical, objects other than the road surface region in the image obtained by each camera are smoothly joined on the projection surface. The length of the major axis and the minor axis of the elliptical cylinder projection plane (63) constituting the projection plane (Q), and the height of the cylinder depend on the relationship with the ground projection plane (62), the size of the image to be combined, and the like. It is set appropriately according to it.
Further, the long axis of the ellipse on the projection plane (63) of the ellipse cylinder can be in the front-rear direction of the vehicle (91). Then, since the projection plane (r) on the side of the own vehicle is closer than the projection plane (t) on the rear side, the track of another vehicle that approaches the own vehicle from the rear and overtakes the side of the own vehicle. It becomes a projection plane suitable for the above (see FIGS. 6 and 7).

The elliptical cylinder projection plane (63) is within a predetermined angle (φ) from the virtual viewpoint (65) to the rear of the vehicle (91) in the horizontal cross section, and the ground projection plane (62) is the elliptical cylinder projection plane. It can be configured to be within the range in contact with (63) (see FIG. 6). The predetermined angle (φ) may be appropriately determined according to the range of the required rear composite image. Further, the height of the elliptical cylinder projection surface (63) can be arbitrarily set.
As described above, the projection surface (Q) has a shape represented by a solid line in FIG. That is, the elliptical cylinder projection plane (63) constituting the projection plane (Q) can be a range of the elliptical cylinder included in a predetermined angle (φ) within a range of a certain height from the ground. Further, the ground projection surface (62) constituting the projection surface (Q) is a section centered on the virtual viewpoint (65) until the ground included in the predetermined angle (φ) comes into contact with the elliptical cylinder projection surface (63). be able to.

The coordinate conversion means (4) uses the rear image (a), the right rear image (b), and the left rear image (c) as the rear camera (21), the right rear camera (22), and the left rear image (c) in the three-dimensional coordinate space, respectively. Coordinate conversion is performed so as to project from the position of the camera (23) to the projection surface (Q). As a result, a projected rear image (a "), a projected right rear image (b"), and a projected left rear image (c ") are generated. This coordinate transformation corrects the distortion of the image by the camera at the same time as the projection. It can be configured so that the image is flipped horizontally at the same time as the projection.
The specific means for projecting each image (a, b, c) on each projection plane (62, 63) can be arbitrarily selected. For example, the coordinate conversion is calculated in advance and projected with each pixel of the image. Image conversion can be performed by referring to the lookup table (41) which is a correspondence table of each pixel of the surface. Further, the look-up table (41) may be a correspondence table of each pixel of the image and each pixel of the projection surface including the coordinate conversion calculation and the image distortion correction calculation by the camera. Further, the look-up table (41) may be a correspondence table of each pixel of the image and each pixel of the projection surface including the coordinate conversion calculation, the distortion correction calculation of the image by the camera, and the left-right inversion calculation of the image. ..

In the above coordinate transformation, the projected coordinates (U) on the elliptical cylinder projection plane (63) may be located below the ground projection plane (62). In this case, the coordinate conversion means (4) uses the coordinates (P1, P5) of the point where the line segment (L1, L5) connecting the rear camera (21) and the projected coordinate (U) intersects the ground projection surface (62). ), Or at the coordinates (P2, P6) of the point where the line segment (L2, L6) connecting the right rear camera (22) or the left rear camera (23) and the projection coordinate (U) intersects the ground projection surface (62). It can be converted (see Figure 3).
The rear image (a), the right rear image (b), and the left rear image (c) are the coordinates of the arrangement positions of the rear camera (21), the right rear camera (22), and the left rear camera (23) ((0,). By projecting from (0,0), (SRx, SRy, SRz), (SLx, SLy, SLz)), the images (a, b) are projected even if the positions and orientations between the cameras (21, 22, 23) are different. , C) Projection rear image (a "), projection right rear image (b") and projection left rear image on elliptical cylinder projection surface (63) and ground projection surface (62) with little deviation such as scale and inclination between (C ") is obtained.

The coordinate conversion means (4) transfers the upper regions in the projection rear image (a "), the projection right rear image (b"), and the projection left rear image (c ") from the projected image on the elliptical cylinder projection surface (63). It can be generated so that the lower region is generated from the projected image on the ground projection plane (62). In this way, each projected image (a ", b", c ") is then combined. In the rear composite image generated thereby, the upper region corresponds to the image projected on the elliptical cylinder projection plane (63), and the lower region corresponds to the image projected on the ground projection plane (62).

The image synthesizing means (7) combines the projected rear image (a "), the projected right rear image (b"), and the projected left rear image (c ") projected on the projection plane (Q) by the coordinate conversion means (4). By compositing, one back composite image is created.
Specifically, the projection rear image (a ") is arranged in the center (t), and the projection right rear image (b") and the projection left rear image (c ") are arranged on the left and right (r) of the center (t). From the images (a ", b", c "), the range required for composition is extracted. The ratio occupied by each projected image (a ", b", c ") in the rear composite image can be arbitrarily set.
The image synthesizing means (7) synthesizes the projection rear image (a "), the projection right rear image (b"), and the projection left rear image (c ") with the transition regions (b1 and b2) in between. A composite image can be generated (see FIG. 6B).

The transition region, which is the boundary between adjacent images, can be subjected to a process of reducing the difference so as to eliminate the discomfort at the boundary portion. For example, it can be mentioned that each image is mixed at a predetermined mixing ratio in a constant width (b1, b2) including a boundary. Further, the predetermined mixing ratio may be a uniform mixing ratio or may be gradually changed.

In the transition regions (b1, b2), there is an region in which the projection coordinates (U) on the elliptical cylinder projection plane (63) are located below the ground projection plane (62) (see FIG. 3). In this region, the rear image (a), the right rear image (b) and the left rear image (c) of the rear camera (21), the right rear camera (22) and the left rear camera (23) are converted into the coordinate conversion means (4). ), Each projected image (a ", b" in which pixels at the same ground position are projected onto different positions on the ground projection surface (62) (see P1, P5, and P2, P6 shown in FIG. 3). ", C"). Then, when each projected image (a ", b", c ") is combined by the image combining means (7), it becomes a composite image in which pixels at different ground positions are overlapped, so that it is visually recognized. The sex is reduced.
Therefore, the coordinate conversion means (4) sets a base point (P4, P8) on a line segment (L4, L8) connecting the rear camera (21) and the right rear camera (22) or the left rear camera (23). , The point (U') where the line segment (L3, L7) connecting the base point (P4, P8) and the projection coordinate (U) on the elliptical cylinder projection surface (63) intersects the ground projection surface (62) is the ground projection surface. (62) It can be converted to the above projected coordinates. In this way, by aligning the projected coordinates of the rear image (a) and the right rear image (b) and the projected coordinates of the rear image (a) and the left rear image (c) at predetermined points (U'), each It is possible to obtain a composite image in which the positions of the road objects on the image are overlapped and composited without shifting.
Further, by gradually changing the base points (P4, P8) corresponding to the circumferential direction (W direction) of the elliptical cylinder projection surface of the projection coordinates (U), a sense of discomfort in the composite image in the transition region (b1, b2) can be obtained. It can be suppressed. That is, as the projected coordinates (U) approach the right or left end side of the rear image (a), the base points (P4, P8) are gradually changed so as to approach the right rear camera (22) or the left rear camera (23). ..
The point (U') on the ground projection surface (62) is a point where the line segments (L1, L5) connecting the rear camera (21) and the projection coordinates (U) intersect with the ground projection surface (62). The point (P2, P6) where the line segment (L2, L6) connecting P1, P5) with the right rear camera (22) or the left rear camera (23) and the projection coordinates (U) intersects the ground projection surface (62). The same point (U') can be obtained even if it is obtained as a point on a line segment (P1-P2, P5-P6) connecting the two points.

The form in which each projected image (a ", b", c ") is distributed in the rear composite image is not limited. For example, the rear composite image is divided into three in the horizontal direction at an appropriate ratio, and the projected images (a", respectively. b ", c") can be arranged.
Further, the image compositing means (7) divides the rear composite image into two in the vertical direction at an appropriate ratio, and each projected image (a ", b", c is divided into an upper region (71) and a lower region (72). The distribution form of ") can be changed. For example, in the upper region (71) of the rear composite image, the upper region of each projected image (a", b ", c") is arranged at an appropriate ratio, and the rear region is arranged. In the lower region (72) of the composite image, the upper region (71) and the image are continuous, and the region of the central projection rear image (a ") can be arranged in a trapezoidal shape in which the region extends downward (FIG. 6 (b)). As a result, in the lower region (72) of the rear composite image, the left and right projected images (b ", c") are arranged in a trapezoidal shape that narrows downward. ..

As shown in FIG. 6B, when the projected rear image (a ") is arranged in a trapezoidal shape (76) that spreads downward in the lower region (72) of the rear composite image, the upper bottom thereof is projected. It is preferable that the length is the same as the lower side of the upper region (71) of the rear image (a "), and the length of the lower bottom corresponds to at least the vehicle width of the vehicle (91). This is because if the width is made larger than the vehicle width, the image of the road object behind the vehicle is less likely to be deformed beyond the boundary, and the visibility when the vehicle (91) moves backward is improved. In the lower region (72) of the rear composite image, the projected right rear image (b ")" and the projected left rear image (c ") are the left and right remaining portions (77, 78) of the projected rear image portion (76), respectively. It is arranged in.
The rear composite image is output by the display means (8) so as to be displayed on a display device such as a liquid crystal display.

The above image acquisition means (3), coordinate conversion means (4), and image composition means (7) may be realized by any of hardware and software, and the specific configuration does not matter. For example, it can be configured by providing peripheral circuits such as input / output interfaces around a microcontroller (microcomputer) including a CPU, memory (ROM, RAM, etc.), input / output circuits, etc. (not shown). As a microcontroller, a processor suitable for image processing can be used. It may also be configured using a programmable logic circuit, gate array or other logic circuit.

The image synthesizer 1 according to the present embodiment is a rear monitor used instead of the rear-view mirror, and as shown in FIG. 1, the rear camera 21, the two side rear cameras 22 and 23, the rear image, the right rear image, and the rear image, respectively. It is configured to include an image acquisition means 3, a coordinate conversion means (4), an image composition means (7), and the like for acquiring a left rear image.
FIG. 2 shows an example of each camera 21 to 23 and its angle of view (shooting range). Each of the cameras 21 to 23 is a camera using a fisheye lens. As shown in FIGS. 2 and 3, the rear camera 21 is provided near the door knob of the back door, which is the rear end of the vehicle 91, and photographs the range 92a behind the vehicle 91. The side-rear cameras 22 and 23 are provided on the door mirrors extending from the side surface of the vehicle 91, respectively, and photograph the left and right ranges 93a and 94a including the rear of the vehicle 91.

The information processing method and operation in the image synthesizer 1 will be described with reference to the flowchart shown in FIG. The present invention can also be realized as an image synthesizing method for synthesizing images on a road surface and a road surface behind a vehicle, on the right side, and on the left side. In this image composition method, the rear image, the right rear image, and the left side are taken from the images taken by the rear camera for photographing the rear of the vehicle, the right rear camera for photographing the right rear, and the left rear camera for photographing the left rear, respectively. An image acquisition step of acquiring a rear image, and coordinate conversion for converting the rear image, the right rear image, and the left rear image into a projected rear image, a projected right rear image, and a projected left rear image, respectively, on a predetermined projection plane. A step and an image synthesizing step of obtaining a rear composite image by synthesizing the projected rear image, the projected right rear image, and the projected left rear image are provided, and the predetermined projection surface has a predetermined position as an origin. In the three-dimensional coordinate space, the horizontal cross-sectional shape consists of a tubular elliptical cylinder projection surface having an elliptical shape centered on a predetermined virtual viewpoint and a ground projection surface corresponding to the ground. Projecting the image, the right rear image, and the left rear image from the positions of the rear camera, the right rear camera, and the left rear camera in the three-dimensional coordinate space onto the elliptical cylinder projection plane and the ground projection plane, respectively. It is characterized by.
The image acquisition step is mainly performed by the image acquisition means 3. Further, the coordinate conversion step is mainly performed by the coordinate conversion means 4, and the image composition step is mainly performed by the image composition means 7.

First, in the image acquisition step, images (a, b, c) are acquired by capturing the images captured by the rear camera 21, the side rear cameras 22 and 23 (step S1). Each of the acquired images (a, b, c) is, for example, a circular image with a fisheye lens as shown in FIG.
In the image acquisition process, each image (a, b, c) is converted into spherical coordinates to correct distortion, and a rear image (a'), a right rear image (b'), and a left rear image are generated. You may.

Next, in the coordinate conversion step, the rear image (a), the right rear image (b), and the left rear image (c) are projected on the projection surface Q shown in FIG. 5, respectively (a ", b", c. (Step S2). The projection surface Q is set with reference to the virtual viewpoint 65, and is composed of an elliptical cylinder projection surface 63 and a ground projection surface 62. It is also suitable as a rear monitor. The left and right reversals of each image are also performed at the same time during conversion so that the images are oriented.
Each projected image (a ", b", c ") is projected onto the elliptical cylinder projection surface 63 and the ground projection surface 62 from the positions of the rear camera 21, the right rear camera 22, and the left rear camera 23 in the three-dimensional coordinate space, respectively. It is an image to be done.
In the coordinate conversion to the projected image, as shown in FIG. 3, the projected coordinates U on the elliptical cylinder projection surface 63 may be located below the ground projection surface 62. In this case, in the coordinate conversion step, the line segments L1 and L5 connecting the rear camera 21 and the projected coordinate U, the line segment L6 connecting the right rear camera 22 and the projected coordinate U, and the left rear camera 23 and the projected coordinate U It is converted into the coordinates P1, P5, P6, P2 on the ground projection surface 62 at the point where the line segment L2 connecting with and intersects the ground projection surface 62.
Further, in the transition regions b1 and b2, when the projection coordinate U on the elliptical cylinder projection surface 63 is located below the ground projection surface 62, a line segment connecting the rear camera 21 with the right rear camera 22 or the left rear camera 23. On L4 and L8, base points P4 and P8 proportional to the positions of the transition regions b1 and b2 in the W direction are set, and a line segment L3 connecting the base points P4 and P8 and the projection coordinates U on the elliptical cylinder projection surface 63. The point U'where L7 intersects the ground projection surface 62 is converted to be the projection coordinates on the ground projection surface 62.

Next, in the image composition step, a rear composite image is generated by synthesizing the projection rear image, the projection right rear image, and the projection left rear image (a ", b", c ") (step S3). By cutting out a predetermined section used for composition from each projected image (a ", b", c ") and arranging the boundaries so as to overlap with each other by a predetermined width, as illustrated in FIG. 6 (b). Do. The boundary portions (transition regions b1 and b2) of the images cut out from each projected image gradually change the mixing ratio of adjacent images in the width direction to mix the images, thereby making the seams between the images unnatural. Reduce.
As a result, a rear composite image as shown in FIG. 8 is created.

In the creation of the rear composite image, the upper region 71 and the lower region 72 are divided in the vertical direction at an appropriate ratio, and the ratio of cutting out from each projected image can be changed between the upper region and the lower region. That is, as shown in FIG. 6B, in the lower region 72 of the rear composite image, the projected rear image a "is cut out in a trapezoidal shape that widens downward, and the projected right rear image and the projected right rear image and the remaining left and right regions are used. An image cut out from the projected left rear image (b ", c") can be arranged. FIG. 9 is an example of a rear composite image created by such a method, while FIG. 10 is a rear composite image. This is an example in which each projected image (a ", b", c ") is arranged without changing the ratio of cutting out from each projected image in the upper and lower regions of. As shown in FIG. 9, by arranging the projected rear image a "so as to spread downward in the lower region of the rear composite image, it is possible to make the road near the rear part of the vehicle widely visible.
The rear composite image can be output to a display device provided separately and displayed (step S4, display step).

It should be noted that the present invention is not limited to the embodiments shown above, and various modifications can be made within the scope of the present invention according to the purpose and application.

1; image synthesizer, 21; rear camera, 22; right rear camera, 23; left rear camera, 3; image acquisition means, 4; coordinate conversion means, 62; ground projection surface, 63; elliptical cylinder projection surface, 65; Virtual viewpoint, 66; central axis, 7; image compositing means, 8; display means, 91; vehicle.

Claims (8)

An image synthesizer that synthesizes images on the road surface and on the road behind the vehicle, on the right side, and on the left side.
Images that acquire a rear image, a right rear image, and a left rear image from images taken by a rear camera that captures the rear of the vehicle, a right rear camera that captures the right rear, and a left rear camera that captures the left rear, respectively. Acquisition method and
A coordinate conversion means for converting the rear image, the right rear image, and the left rear image into a projected rear image projected onto a predetermined projection surface, a projected right rear image, and a projected left rear image, respectively.
An image synthesizing means for obtaining a rear composite image by synthesizing the projected rear image, the projected right rear image, and the projected left rear image.
With
The predetermined projection surface is a tubular elliptical cylinder projection surface whose horizontal cross-sectional shape is an ellipse centered on a predetermined virtual viewpoint in a three-dimensional coordinate space whose origin is a predetermined position, and the ground corresponding to the ground. Consists of a projection plane
The coordinate conversion means displays the rear image, the right rear image, and the left rear image from the positions of the rear camera, the right rear camera, and the left rear camera in the three-dimensional coordinate space, respectively. Project on the ground projection surface ,
The image synthesizing means generates the rear composite image obtained by synthesizing the projection rear image, the projection right rear image, and the projection left rear image, respectively, with the transition region in between.
The coordinate conversion means is a line segment connecting the rear camera and the right rear camera or the left rear camera when the projected coordinates on the elliptical cylinder projection surface are located below the ground projection surface in the transition region. A base point is set on the top, and conversion is performed so that the point where the line segment connecting the base point and the projection coordinates on the elliptical cylinder projection plane intersects the ground projection plane is the projection coordinates on the ground projection plane. A featured image synthesizer. The virtual viewpoint is in front of the vehicle in the three-dimensional coordinate space.
The elliptical cylinder projection surface is within a predetermined angle with respect to the rear of the vehicle from the virtual viewpoint in a horizontal cross section.
The image synthesizer according to claim 1, wherein the ground projection surface is within a range in contact with the elliptical cylinder projection surface. The coordinate conversion means generates an upper region in the projected rear image, the projected right rear image, and the projected left rear image from the projected image on the elliptical cylinder projection plane, and the lower region is a projected image on the ground projection plane. The image compositing apparatus according to claim 1 or 2 generated from. The image compositing device according to any one of claims 1 to 3, wherein the image compositing means is arranged so that the projected rear image spreads downward in a lower region of the rear compositing image. It is an image composition method that synthesizes images on the road surface and the road surface behind the vehicle, on the right side, and on the left side.
Images that acquire a rear image, a right rear image, and a left rear image from images taken by a rear camera that captures the rear of the vehicle, a right rear camera that captures the right rear, and a left rear camera that captures the left rear, respectively. Acquisition process and
A coordinate conversion step of converting the rear image, the right rear image, and the left rear image into a projected rear image projected onto a predetermined projection surface, a projected right rear image, and a projected left rear image, respectively.
An image synthesizing step of obtaining a rear composite image by synthesizing the projected rear image, the projected right rear image, and the projected left rear image.
With
The predetermined projection surface is a tubular elliptical cylinder projection surface whose horizontal cross-sectional shape is an ellipse centered on a predetermined virtual viewpoint in a three-dimensional coordinate space whose origin is a predetermined position, and the ground corresponding to the ground. Consists of a projection plane
In the coordinate conversion step, the rear image, the right rear image, and the left rear image are converted from the positions of the rear camera, the right rear camera, and the left rear camera in the three-dimensional coordinate space to the elliptical cylinder projection surface and the left rear image, respectively. Project on the ground projection surface ,
In the image synthesizing step, the rearward composite image is generated by synthesizing the projection rear image, the projection right rear image, and the projection left rear image, respectively, with the transition region in between.
In the coordinate conversion step, when the projected coordinates on the elliptical cylinder projection plane are located below the ground projection plane in the transition region, the line segment connecting the rear camera with the right rear camera or the left rear camera. A base point is set on the top, and conversion is performed so that the point where the line segment connecting the base point and the projection coordinates on the elliptical cylinder projection plane intersects the ground projection plane is the projection coordinates on the ground projection plane. Characteristic image composition method. The virtual viewpoint is in front of the vehicle in the three-dimensional coordinate space.
The elliptical cylinder projection surface is within a predetermined angle with respect to the rear of the vehicle from the virtual viewpoint in a horizontal cross section.
The ground projection plane, the image synthesis method according to claim 5 Symbol mounting in the range in contact with the elliptic cylindrical projection surface. In the coordinate conversion step, the upper region in the projection rear image, the projection right rear image and the projection left rear image is generated from the projection image on the elliptical cylinder projection surface, and the lower region is the projection image on the ground projection surface. The image composition method according to claim 5 or 6, which is generated from. The image composition method according to any one of claims 5 to 7, wherein the image composition step is arranged so that the projection rear image spreads downward in a lower region of the rear composition image.