JP6169381B2 - Excavator - Google Patents

Description

  The present invention relates to a work machine periphery monitoring device having a function of determining the presence or absence of a person around a work machine.

  Conventionally, a function for generating a viewpoint conversion image by combining images captured by a plurality of imaging units attached to a shovel, a function for detecting a person existing around the shovel, and a viewpoint corresponding to the position of the detected person A surrounding video presentation device having a function of displaying a mark superimposed on a position on a converted image is known (see, for example, Patent Document 1).

  This surrounding image presentation device continuously presents a viewpoint-converted image when the shovel is viewed from directly above, regardless of whether a person is detected around the shovel. When a person is detected around the shovel, a mark indicating the presence of the person is superimposed and displayed at a position on the viewpoint conversion image corresponding to the actual position of the person.

  In this way, the surrounding video presenting apparatus conveys the position of the person existing around the shovel to the shovel operator in an easy-to-understand manner while continuously presenting the viewpoint conversion image.

JP 2002-176661 A

  However, the surrounding video presentation device described in Patent Document 1 only presents a mark representing the presence of a person on the viewpoint conversion image even when a person is detected around the excavator. Therefore, the operator of the excavator cannot know what state the detected person is in.

  In view of the above-described points, an object of the present invention is to provide a work machine periphery monitoring device that can more easily understand an operator's state around a work machine.

  In order to achieve the above object, a work machine periphery monitoring device according to an embodiment of the present invention is for a work machine that generates an output image using a plurality of input images captured by a plurality of cameras attached to the work machine. A perimeter monitoring device for use in generating an output image based on the presence / absence determination means for determining the presence / absence of a person in each of a plurality of monitoring spaces around the work machine and the determination result of the presence / absence determination means An image generation means for switching input images, wherein the plurality of cameras include two of the plurality of cameras, a first camera and a second camera having overlapping imaging spaces, and the plurality of monitoring spaces Includes two monitoring spaces corresponding to respective imaging spaces of the first camera and the second camera, which are two of the plurality of monitoring spaces.

  With the above-described means, the present invention can provide a work machine periphery monitoring device that can more easily understand the state of a person existing around the work machine to the operator.

It is a block diagram which shows roughly the structural example of the image generation apparatus which concerns on the Example of this invention. It is a figure which shows the structural example of the shovel mounted with an image generation apparatus. It is a figure which shows an example of the space model on which an input image is projected. It is a figure which shows an example of the relationship between a space model and a process target image plane. It is a figure for demonstrating matching with the coordinate on an input image plane, and the coordinate on a space model. It is a figure for demonstrating the matching between the coordinates by a coordinate matching means. It is a figure for demonstrating the effect | action of a parallel line group. It is a figure for demonstrating the effect | action of an auxiliary line group. It is a flowchart which shows the flow of a process target image generation process and an output image generation process. It is a display example (the 1) of an output image. It is a top view (the 1) of an excavator carrying an image generating device. FIG. 4 is a diagram (No. 1) illustrating input images of three cameras mounted on an excavator and output images generated using the input images. It is a figure for demonstrating the image loss prevention process which prevents the loss | disappearance of the object in the overlap part of each imaging space of two cameras. FIG. 13 is a comparison diagram showing the difference between the output image shown in FIG. 12 and the output image obtained by applying image loss prevention processing to the output image of FIG. 12. It is FIG. (2) which shows each input image of the three cameras mounted in the shovel, and the output image produced | generated using those input images. It is the corresponding | compatible table (the 1) which shows the correspondence of the determination result of a human presence determination means, and the input image used for the production | generation of an output image. It is a display example (the 2) of an output image. It is a top view (the 2) of an excavator carrying an image generating device. It is a correspondence table (part 2) showing the correspondence between the determination result of the presence / absence determination means and the input image used for generating the output image. It is a display example (the 3) of an output image. It is a display example (the 4) of an output image.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically illustrating a configuration example of an image generation apparatus 100 according to an embodiment of the present invention.

  The image generation apparatus 100 is an example of a work machine periphery monitoring device that monitors the periphery of the work machine. The image generation apparatus 100 includes a control unit 1, a camera 2, an input unit 3, a storage unit 4, a display unit 5, and a human detection sensor 6. Composed. Specifically, the image generation device 100 generates an output image based on an input image captured by the camera 2 mounted on the work machine and presents the output image to the operator. Further, the image generation device 100 switches the content of the output image to be presented based on the output of the human detection sensor 6.

  FIG. 2 is a diagram illustrating a configuration example of an excavator 60 as a work machine on which the image generating apparatus 100 is mounted. The excavator 60 is placed on a crawler-type lower traveling body 61 via a turning mechanism 62. A swing body 63 is mounted so as to be swingable around a swing axis PV.

  The upper swing body 63 includes a cab (operator's cab) 64 on the front left side, a drilling attachment E on the front center, and the camera 2 (right camera 2R, rear camera 2B) on the right and rear surfaces. ) And a person detection sensor 6 (right person detection sensor 6R, rear person detection sensor 6B). In addition, the display part 5 is installed in the position in the cab 64 where the operator is easy to visually recognize.

  Next, each component of the image generation apparatus 100 will be described.

  The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), and the like. In the present embodiment, the control unit 1 stores, for example, a program corresponding to each of a coordinate association unit 10, an image generation unit 11, and a presence / absence determination unit 12, which will be described later, in a ROM or NVRAM, and a RAM as a temporary storage area. The CPU is caused to execute processing corresponding to each means while using.

  The camera 2 is a device for acquiring an input image that reflects the surroundings of the excavator 60. In the present embodiment, the camera 2 is, for example, a right side camera 2R and a rear camera 2B that are attached to the right side and the rear side of the upper swing body 63 so as to image a blind spot of the operator in the cab 64 (see FIG. 2). The camera 2 includes an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 2 may be attached to a position other than the right side and the rear side of the upper swing body 63 (for example, the front side and the left side), and a wide-angle lens or a fisheye lens is attached so as to capture a wide range. It may be.

  In addition, the camera 2 acquires an input image according to a control signal from the control unit 1 and outputs the acquired input image to the control unit 1. In addition, when the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the corrected input image obtained by correcting apparent distortion and tilt caused by using these lenses is transmitted to the control unit 1. Output. Further, the camera 2 may output an input image without correcting its apparent distortion and tilt to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

  The input unit 3 is a device that allows an operator to input various types of information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

  The storage unit 4 is a device for storing various types of information, and is, for example, a hard disk, an optical disk, or a semiconductor memory.

  The display unit 5 is a device for displaying image information. For example, the display unit 5 is a liquid crystal display, a projector, or the like installed in the cab 64 (see FIG. 2) of the excavator 60. Display an image.

  The human detection sensor 6 is a device for detecting a person existing around the excavator 60. In the present embodiment, the human detection sensor 6 is attached to the right side surface and the rear surface of the upper swing body 63 so as to detect, for example, a person present in the blind spot of the operator in the cab 64 (see FIG. 2). .

  The person detection sensor 6 is a sensor that distinguishes and detects a person from an object other than a person. For example, the person detection sensor 6 is a sensor that detects a change in energy in a corresponding monitoring space, and includes a pyroelectric infrared sensor, a bolometer infrared sensor, Includes a moving object detection sensor using an output signal of an infrared camera or the like. In this embodiment, the human detection sensor 6 uses a pyroelectric infrared sensor, and detects a moving object (moving heat source) as a person. The monitoring space of the right person detection sensor 6R is included in the imaging space of the right camera, and the monitoring space of the rear person detection sensor 6B is included in the imaging space of the rear camera 2B.

  Note that the human detection sensor 6 may be attached to a position other than the right side and the rear side of the upper swing body 63 (for example, the front and left sides), like the camera 2. It may be attached to any one of the left side surface, the right side surface, and the rear surface, or may be attached to all surfaces.

  In addition, the image generation apparatus 100 generates a processing target image based on the input image, and performs an image conversion process on the processing target image so that the positional relationship with the surrounding objects and a sense of distance can be intuitively grasped. An output image to be generated may be generated, and the output image may be presented to the operator.

  The “processing target image” is an image that is a target of image conversion processing (for example, scale conversion processing, affine conversion processing, distortion conversion processing, viewpoint conversion processing, etc.) that is generated based on the input image. Specifically, the “processing target image” is, for example, an input image by a camera that images the ground surface from above, and includes an image in the horizontal direction (for example, an empty portion) with a wide angle of view. It is an image suitable for image conversion processing generated from the above. More specifically, the input image is projected onto a predetermined spatial model so that the horizontal image is not displayed unnaturally (for example, the sky is not treated as being on the ground surface) It is generated by reprojecting the projection image projected on the spatial model onto another two-dimensional plane. The processing target image may be used as an output image as it is without performing an image conversion process.

  The “space model” is a projection target of the input image. Specifically, the “space model” includes at least one plane or curved surface including a plane or curved surface other than the processing target image plane that is a plane on which the processing target image is located. The plane or curved surface other than the processing target image plane that is the plane on which the processing target image is located is, for example, a plane parallel to the processing target image plane or a plane or curved surface that forms an angle with the processing target image plane. .

  Note that the image generation apparatus 100 may generate an output image by performing image conversion processing on the projection image projected on the space model without generating a processing target image. Further, the projection image may be used as an output image as it is without being subjected to image conversion processing.

  FIG. 3 is a diagram illustrating an example of a spatial model MD onto which an input image is projected, and the left diagram in FIG. 3 illustrates a relationship between the excavator 60 and the spatial model MD when the excavator 60 is viewed from the side. 3 shows the relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from above.

  As shown in FIG. 3, the space model MD has a semi-cylindrical shape, and includes a planar region R1 inside the bottom surface and a curved region R2 inside the side surface.

  FIG. 4 is a diagram illustrating an example of a relationship between the space model MD and the processing target image plane, and the processing target image plane R3 is a plane including the plane area R1 of the space model MD, for example. 4 shows the space model MD not in a semi-cylindrical shape as shown in FIG. 3 but in a cylindrical shape for the sake of clarity, the space model MD may be either a semi-cylindrical shape or a cylindrical shape. It may be. The same applies to the subsequent drawings. Further, as described above, the processing target image plane R3 may be a circular area including the plane area R1 of the spatial model MD, or may be an annular area not including the plane area R1 of the spatial model MD.

  Next, various units included in the control unit 1 will be described.

  The coordinate association unit 10 is a unit for associating coordinates on the input image plane where the input image captured by the camera 2 is located, coordinates on the space model MD, and coordinates on the processing target image plane R3. In the present embodiment, the coordinate association unit 10 includes, for example, various parameters relating to the camera 2 that are set in advance or input via the input unit 3, and predetermined input image planes, space models MD, The coordinates on the input image plane, the coordinates on the space model MD, and the coordinates on the processing target image plane R3 are associated with each other based on the mutual positional relationship of the processing target image plane R3. The various parameters related to the camera 2 are, for example, the optical center of the camera 2, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, projection method, and the like. Then, the coordinate association unit 10 stores these correspondences in the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41 of the storage unit 4.

  When the processing target image is not generated, the coordinate association unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and the spatial model / processing target of the corresponding relationship. The storage in the image correspondence map 41 is omitted.

  The image generation means 11 is a means for generating an output image. In the present embodiment, the image generation unit 11 performs, for example, scale conversion, affine transformation, or distortion conversion on the processing target image, so that the coordinates on the processing target image plane R3 and the output image plane on which the output image is located. Match coordinates. Then, the image generation unit 11 stores the correspondence relationship in the processing target image / output image correspondence map 42 of the storage unit 4. The image generation unit 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41. Generate an output image. The value of each pixel is, for example, a luminance value, a hue value, a saturation value, and the like.

  Further, the image generation means 11 outputs the output image plane where the coordinates on the processing target image plane R3 and the output image are located based on various parameters relating to the virtual camera set in advance or input via the input unit 3. Correlate with the coordinates above. Various parameters relating to the virtual camera are, for example, the optical center of the virtual camera, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, the projection method, and the like. Then, the image generation unit 11 stores the correspondence relationship in the processing target image / output image correspondence map 42 of the storage unit 4. The image generation unit 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41. Generate an output image.

  Note that the image generation unit 11 may generate the output image by changing the scale of the processing target image without using the concept of the virtual camera.

  Further, when the processing target image is not generated, the image generation unit 11 associates the coordinates on the space model MD with the coordinates on the output image plane according to the applied image conversion processing. Then, the image generation unit 11 generates an output image by associating the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40. In this case, the image generation unit 11 omits the association between the coordinates on the processing target image plane R3 and the coordinates on the output image plane and the storage of the correspondence relationship in the processing target image / output image correspondence map 42. .

  Further, the image generation unit 11 switches the content of the output image based on the determination result of the presence / absence determination unit 12. Specifically, the image generation unit 11 switches the input image used for generating the output image based on the determination result of the presence / absence determination unit 12, for example. The details of the switching of the input image used for generating the output image and the output image generated based on the switched input image will be described later.

  The presence / absence determination means 12 is a means for determining the presence / absence of a person in each of a plurality of monitoring spaces set around the work machine. In the present embodiment, the presence / absence determination unit 12 determines the presence / absence of a person around the excavator 60 based on the output of the person detection sensor 6.

  Further, the presence / absence determination unit 12 may determine the presence / absence of a person in each of a plurality of monitoring spaces set around the work machine based on an input image captured by the camera 2. Specifically, the presence / absence determination unit 12 may determine the presence / absence of a person around the work machine using an image processing technique such as optical flow or pattern matching. The presence / absence determination unit 12 may determine the presence / absence of a person around the work machine based on an output of an image sensor different from the camera 2.

  Alternatively, the presence / absence determination unit 12 determines the presence / absence of a person in each of a plurality of monitoring spaces set around the work machine based on the output of the human detection sensor 6 and the output of the image sensor such as the camera 2. Also good.

  Next, an example of specific processing by the coordinate association unit 10 and the image generation unit 11 will be described.

  The coordinate association means 10 can associate the coordinates on the input image plane with the coordinates on the space model using, for example, a Hamilton quaternion.

  FIG. 5 is a diagram for explaining the association between the coordinates on the input image plane and the coordinates on the space model. The input image plane of the camera 2 is represented as one plane in the UVW orthogonal coordinate system with the optical center C of the camera 2 as the origin. The spatial model is represented as a three-dimensional surface in the XYZ orthogonal coordinate system.

  First, the coordinate matching unit 10 moves the origin of the XYZ coordinate system in parallel to the optical center C (the origin of the UVW coordinate system), then sets the X axis as the U axis, the Y axis as the V axis, and the Z axis. The XYZ coordinate system is rotated so as to match the −W axis. This is because the coordinates on the space model (the coordinates on the XYZ coordinate system) are converted to the coordinates on the input image plane (the coordinates on the UVW coordinate system). Note that the sign “-” of “−W axis” means that the direction of the Z axis is opposite to the direction of the −W axis. This is because the UVW coordinate system has the + W direction in front of the camera, and the XYZ coordinate system has the −Z direction in the vertically downward direction.

  When there are a plurality of cameras 2, each of the cameras 2 has an individual UVW coordinate system, so the coordinate association unit 10 moves the XYZ coordinate system in parallel with respect to each of the plurality of UVW coordinate systems, and Rotate.

  In the above conversion, the XYZ coordinate system is translated so that the optical center C of the camera 2 is the origin of the XYZ coordinate system, and then the Z axis is rotated so as to coincide with the −W axis. This is realized by rotating to coincide with the axis. Therefore, the coordinate matching means 10 can combine these two rotations into one rotation calculation by describing this conversion in Hamilton's quaternion.

  By the way, the rotation for making one vector A coincide with another vector B corresponds to the process of rotating the vector A and the vector B by the angle formed by using the normal line of the plane formed by the vectors A and B as an axis. . And, if the angle is θ, from the inner product of the vector A and the vector B, the angle θ is

It is represented by

  Further, the unit vector N of the normal line between the vector A and the vector B is obtained from the outer product of the vector A and the vector B.

It will be expressed as

  Note that the quaternion has i, j, and k as imaginary units,

In this embodiment, the quaternion Q is represented by t as a real component and a, b, and c as pure imaginary components.

The conjugate quaternion of the quaternion Q is

It is represented by

  The quaternion Q can represent a three-dimensional vector (a, b, c) with pure imaginary components a, b, c while setting the real component t to 0 (zero), and t, a, b , C can also be used to express a rotational motion with an arbitrary vector as an axis.

  Further, the quaternion Q can be expressed as a single rotation operation by integrating a plurality of continuous rotation operations. Specifically, the quaternion Q is obtained when, for example, an arbitrary point S (sx, sy, sz) is rotated by an angle θ with an arbitrary unit vector C (l, m, n) as an axis. The point D (ex, ey, ez) can be expressed as follows.

Here, in this embodiment, if the quaternion representing the rotation that makes the Z axis coincide with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X ′. X '

It is represented by

  In this embodiment, if the quaternion representing the rotation that matches the line connecting the point X ′ on the X axis and the origin to the U axis is Qx, “the Z axis matches the −W axis, , A quaternion R representing "rotation to make the X axis coincide with the U axis"

It is represented by

  As described above, the coordinate P ′ when an arbitrary coordinate P on the space model (XYZ coordinate system) is expressed by a coordinate on the input image plane (UVW coordinate system) is

It is represented by In addition, since the quaternion R is invariant in each of the cameras 2, the coordinate association unit 10 thereafter executes the calculation to convert the coordinates on the space model (XYZ coordinate system) to the input image plane (UVW). Can be converted to coordinates on the coordinate system).

  After the coordinates on the space model (XYZ coordinate system) are converted to the coordinates on the input image plane (UVW coordinate system), the coordinate association means 10 is formed by the line segment CP ′ and the optical axis G of the camera 2. The incident angle α is calculated. The line segment CP ′ is a line segment that connects the optical center C (coordinate on the UVW coordinate system) of the camera 2 and a coordinate P ′ that represents an arbitrary coordinate P on the space model in the UVW coordinate system.

  In addition, the coordinate association unit 10 calculates the deflection angle φ and the length of the line segment EP ′ in the plane H that is parallel to the input image plane R4 (for example, the CCD plane) of the camera 2 and includes the coordinates P ′. The line segment EP ′ is a line segment connecting the intersection point E between the plane H and the optical axis G and the coordinate P ′, and the deflection angle φ is formed by the U ′ axis and the line segment EP ′ in the plane H. It is an angle to do.

  In the camera optical system, the image height h is usually a function of the incident angle α and the focal length f. Therefore, the coordinate correlating means 10 performs normal projection (h = ftan α), orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), and equal solid angle projection (h = 2 fsin (α / 2). )), An appropriate projection method such as equidistant projection (h = fα) is selected to calculate the image height h.

  Thereafter, the coordinate matching means 10 decomposes the calculated image height h into U and V components on the UV coordinate system by the declination φ, and is a numerical value corresponding to the pixel size per pixel of the input image plane R4. Divide. As a result, the coordinate association unit 10 can associate the coordinates P (P ′) on the space model MD with the coordinates on the input image plane R4.

Incidentally, when the pixel size per one pixel in the U axis direction of the input image plane R4 and a _U, the pixel size per one pixel in the V axis direction of the input image plane R4 and a _V, coordinates P of the space model MD The coordinates (u, v) on the input image plane R4 corresponding to (P ′) are

It is represented by

  In this way, the coordinate association unit 10 associates the coordinates on the space model MD with the coordinates on one or a plurality of input image planes R4 existing for each camera, and coordinates on the space model MD, the camera identifier. And the coordinates on the input image plane R4 are stored in the input image / space model correspondence map 40 in association with each other.

  Further, since the coordinate association unit 10 calculates the coordinate conversion using the quaternion, unlike the case where the coordinate conversion is calculated using the Euler angle, there is an advantage that no gimbal lock is generated. . However, the coordinate association unit 10 is not limited to the one that calculates the coordinate conversion using the quaternion, and may perform the coordinate conversion using the Euler angle.

  Note that, when it is possible to associate the coordinates on the plurality of input image planes R4, the coordinate associating means 10 inputs the coordinates P (P ′) on the spatial model MD with respect to the camera having the smallest incident angle α. It may be associated with coordinates on the image plane R4, or may be associated with coordinates on the input image plane R4 selected by the operator.

  Next, a process of reprojecting coordinates on the curved surface area R2 (coordinates having a component in the Z-axis direction) among the coordinates on the spatial model MD onto the processing target image plane R3 on the XY plane will be described.

  FIG. 6 is a diagram for explaining the association between coordinates by the coordinate association means 10. F6A is a diagram illustrating a correspondence relationship between coordinates on the input image plane R4 of the camera 2 that adopts normal projection (h = ftanα) as an example and coordinates on the space model MD. The coordinate associating means 10 is arranged so that each line segment connecting the coordinates on the input image plane R4 of the camera 2 and the coordinates on the spatial model MD corresponding to the coordinates passes through the optical center C of the camera 2. Associate both coordinates.

  In the example of F6A, the coordinate association means 10 associates the coordinate K1 on the input image plane R4 of the camera 2 with the coordinate L1 on the plane area R1 of the space model MD, and coordinates K2 on the input image plane R4 of the camera 2 Is associated with the coordinate L2 on the curved surface region R2 of the space model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

  In addition, when the camera 2 employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the coordinate association unit 10 uses each projection method. Accordingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the space model MD.

  Specifically, the coordinate association unit 10 is configured to use a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), or equal solid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc.), the coordinates on the input image plane are associated with the coordinates on the space model MD. In this case, the line segment K1-L1 and the line segment K2-L2 do not pass through the optical center C of the camera 2.

  F6B is a diagram illustrating a correspondence relationship between coordinates on the curved surface region R2 of the space model MD and coordinates on the processing target image plane R3. The coordinate association unit 10 introduces a parallel line group PL that is located on the XZ plane and forms an angle β with the processing target image plane R3. Then, the coordinate association means 10 causes both the coordinates on the curved surface region R2 of the spatial model MD and the coordinates on the processing target image plane R3 corresponding to the coordinates to ride on one of the parallel line groups PL. Associate both coordinates.

  In the example of F6B, the coordinate association means 10 associates both coordinates on the assumption that the coordinate L2 on the curved surface region R2 of the spatial model MD and the coordinate M2 on the processing target image plane R3 are on a common parallel line.

  The coordinate association means 10 can also associate the coordinates on the plane area R1 of the space model MD with the coordinates on the processing target image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved surface area R2. is there. However, in the example of F6B, the plane area R1 and the processing target image plane R3 are a common plane. Therefore, the coordinate L1 on the plane area R1 of the space model MD and the coordinate M1 on the processing target image plane R3 have the same coordinate value.

  In this way, the coordinate association unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3. And stored in the space model / processing object image correspondence map 41.

  F6C is a diagram illustrating a correspondence relationship between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V adopting the normal projection (h = ftanα) as an example. The image generation means 11 causes each of the line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates to pass through the optical center CV of the virtual camera 2V. And correlate both coordinates.

  In the example of F6C, the image generation unit 11 associates the coordinate N1 on the output image plane R5 of the virtual camera 2V with the coordinate M1 on the processing target image plane R3 (plane area R1 of the spatial model MD), and the virtual camera 2V The coordinate N2 on the output image plane R5 is associated with the coordinate M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

  Note that when the virtual camera 2V employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the image generation unit 11 uses each projection method. Accordingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

  Specifically, the image generating unit 11 is configured to use a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), or equal solid angle projection (h = 2 fsin (α / 2)). )), Equidistant projection (h = fα), etc.), the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3 are associated with each other. In this case, the line segment M1-N1 and the line segment M2-N2 do not pass through the optical center CV of the virtual camera 2V.

  In this way, the image generation unit 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and sets the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3. The image is associated and stored in the processing image / output image correspondence map 42. The image generation unit 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41. Generate an output image.

  F6D is a diagram in which F6A to F6C are combined, and shows the positional relationship between the camera 2, the virtual camera 2V, the plane area R1 and the curved area R2 of the spatial model MD, and the processing target image plane R3.

  Next, the operation of the parallel line group PL introduced by the coordinate association unit 10 to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  The left figure of FIG. 7 is a figure in case the angle (beta) is formed between the parallel line group PL located on XZ plane, and process target image plane R3. On the other hand, the right diagram in FIG. 7 is a diagram in the case where an angle β1 (β1> β) is formed between the parallel line group PL positioned on the XZ plane and the processing target image plane R3. Each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in the left diagram of FIG. 7 and the right diagram of FIG. 7 corresponds to each of the coordinates Ma to Md on the processing target image plane R3. Further, the intervals between the coordinates La to Ld in the left diagram in FIG. 7 are equal to the respective intervals between the coordinates La to Ld in the right diagram in FIG. The parallel line group PL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from all points on the Z axis toward the processing target image plane R3. To do. In this case, the Z axis is referred to as a “reprojection axis”.

  As shown in the left diagram of FIG. 7 and the right diagram of FIG. 7, the distance between the coordinates Ma to Md on the processing target image plane R3 increases the angle between the parallel line group PL and the processing target image plane R3. Decreases linearly with time. That is, the distance decreases uniformly regardless of the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md. On the other hand, since the coordinate group on the plane region R1 of the space model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 7, the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups is such that only the image portion corresponding to the image projected on the curved surface region R2 of the spatial model MD is linearly enlarged or out of the image portion on the output image plane R5 (see FIG. 6). It means to be reduced.

  Next, an alternative example of the parallel line group PL introduced by the coordinate association unit 10 to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  The left diagram in FIG. 8 is a diagram in the case where all the auxiliary line groups AL positioned on the XZ plane extend from the start point T1 on the Z axis toward the processing target image plane R3. On the other hand, the right figure of FIG. 8 is a figure in case all the auxiliary line groups AL are extended toward the process target image plane R3 from the starting point T2 (T2> T1) on a Z-axis. Further, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in the left diagram of FIG. 8 and the right diagram of FIG. 8 corresponds to each of the coordinates Ma to Md on the processing target image plane R3. In the example of the left diagram in FIG. 8, the coordinates Mc and Md are not shown because they are outside the region of the processing target image plane R3. Further, the respective intervals between the coordinates La to Ld in the left diagram of FIG. 8 are equal to the respective intervals of the coordinates La to Ld in the right diagram of FIG. The auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from an arbitrary point on the Z axis toward the processing target image plane R3. To do. As in FIG. 7, the Z axis in this case is referred to as a “reprojection axis”.

  As shown in the left diagram of FIG. 8 and the right diagram of FIG. 8, the distance between the coordinates Ma to Md on the processing target image plane R3 is the distance (height) between the starting point of the auxiliary line group AL and the origin O. As it increases, it decreases non-linearly. That is, the greater the distance between the curved surface region R2 of the space model MD and each of the coordinates Ma to Md, the greater the reduction width of each interval. On the other hand, since the coordinate group on the plane region R1 of the spatial model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 8, the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups corresponds to the image projected on the curved surface region R2 of the spatial model MD in the image portion on the output image plane R5 (see FIG. 6), as in the case of the parallel line group PL. It means that only the image portion is enlarged or reduced nonlinearly.

  In this way, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the space model MD, and does not affect the space model MD. The image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface area R2 can be linearly or nonlinearly enlarged or reduced. Therefore, the image generation apparatus 100 does not affect the road surface image in the vicinity of the excavator 60 (virtual image when the excavator 60 is viewed from directly above), and does not affect the object (the horizontal direction from the excavator 60). The object in the image when viewing the surroundings can be expanded and contracted quickly and flexibly, and the visibility of the blind spot area of the excavator 60 can be improved.

  Next, referring to FIG. 9, the image generation apparatus 100 generates a processing target image (hereinafter referred to as “processing target image generation processing”), and an output image using the generated processing target image. Processing to be generated (hereinafter referred to as “output image generation processing”) will be described. FIG. 9 is a flowchart showing the flow of the processing target image generation process (steps S1 to S3) and the output image generation process (steps S4 to S6). Further, the arrangement of the camera 2 (input image plane R4), the space model (plane area R1 and curved surface area R2), and the processing target image plane R3 is determined in advance.

  First, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the space model MD using the coordinate association unit 10 (step S1).

  Specifically, the coordinate association unit 10 acquires an angle formed between the parallel line group PL and the processing target image plane R3. Then, the coordinate association unit 10 calculates a point where one of the parallel line groups PL extending from one coordinate on the processing target image plane R3 intersects the curved surface region R2 of the space model MD. Then, the coordinate association unit 10 derives the coordinates on the curved surface region R2 corresponding to the calculated point as one coordinate on the curved surface region R2 corresponding to the one coordinate on the processing target image plane R3, and the correspondence relationship is derived. It is stored in the spatial model / processing object image correspondence map 41. It should be noted that the angle formed between the parallel line group PL and the processing target image plane R3 may be a value stored in advance in the storage unit 4 or the like. It may be a value to be entered.

  In addition, when one coordinate on the processing target image plane R3 matches one coordinate on the plane area R1 of the space model MD, the coordinate association unit 10 uses the one coordinate on the plane area R1 as the processing target image. It is derived as one coordinate corresponding to the one coordinate on the plane R3, and the correspondence is stored in the space model / processing object image correspondence map 41.

  Thereafter, the control unit 1 causes the coordinate association unit 10 to associate one coordinate on the spatial model MD derived by the above-described processing with a coordinate on the input image plane R4 (step S2).

  Specifically, the coordinate association unit 10 acquires the coordinates of the optical center C of the camera 2 that employs normal projection (h = ftanα). And the coordinate matching means 10 is a line segment extended from one coordinate on the space model MD, and calculates the point where the line segment passing through the optical center C intersects the input image plane R4. Then, the coordinate association unit 10 derives a coordinate on the input image plane R4 corresponding to the calculated point as one coordinate on the input image plane R4 corresponding to the one coordinate on the space model MD, and the correspondence relationship is derived. The image is stored in the input image / space model correspondence map 40.

  Thereafter, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 are associated with the coordinates on the space model MD and the coordinates on the input image plane R4 (step S3). And when it determines with the control part 1 having not matched all the coordinates yet (NO of step S3), the process of step S1 and step S2 is repeated.

  On the other hand, when determining that all the coordinates are associated (YES in step S3), the control unit 1 ends the processing target image generation process and then starts the output image generation process. And the control part 1 matches the coordinate on process target image plane R3, and the coordinate on output image plane R5 by the image generation means 11 (step S4).

  Specifically, the image generation unit 11 generates an output image by performing scale conversion, affine conversion, or distortion conversion on the processing target image. Then, the image generation unit 11 determines the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5, which are determined by the contents of the scale conversion, affine transformation, or distortion conversion performed. Store in the output image correspondence map 42.

  Alternatively, when generating the output image using the virtual camera 2V, the image generation unit 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the adopted projection method, The correspondence relationship may be stored in the processing target image / output image correspondence map 42.

  Alternatively, the image generation unit 11 acquires the coordinates of the optical center CV of the virtual camera 2V when generating an output image using the virtual camera 2V that employs normal projection (h = ftanα). Then, the image generation unit 11 calculates a point that is a line segment extending from one coordinate on the output image plane R5 and that intersects the processing target image plane R3 with a line segment passing through the optical center CV. Then, the image generation unit 11 derives the coordinate on the processing target image plane R3 corresponding to the calculated point as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5. In this way, the image generation unit 11 may store the correspondence relationship in the processing target image / output image correspondence map 42.

  Thereafter, the image generation unit 11 of the control unit 1 refers to the input image / space model correspondence map 40, the space model / processing target image correspondence map 41, and the processing target image / output image correspondence map 42. The image generation unit 11 then associates the coordinates on the input image plane R4 with the coordinates on the space model MD, the correspondence between the coordinates on the space model MD and the coordinates on the processing target image plane R3, and the processing target. The correspondence between the coordinates on the image plane R3 and the coordinates on the output image plane R5 is traced. Then, the image generation unit 11 acquires values (for example, luminance values, hue values, saturation values, etc.) possessed by the coordinates on the input image plane R4 corresponding to the respective coordinates on the output image plane R5. The acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). Note that when a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the image generation unit 11 uses each of the plurality of coordinates on the plurality of input image planes R4. A statistical value based on the value may be derived, and the statistical value may be adopted as the value of the one coordinate on the output image plane R5. The statistical value is, for example, an average value, a maximum value, a minimum value, an intermediate value, or the like.

  Thereafter, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 are associated with the coordinate values on the input image plane R4 (step S6). And when it determines with the control part 1 having not matched all the values of the coordinate yet (NO of step S6), the process of step S4 and step S5 is repeated.

  On the other hand, if the control unit 1 determines that all coordinate values are associated (YES in step S6), the control unit 1 generates an output image and ends the series of processes.

  Note that the image generation apparatus 100 omits the processing target image generation processing when the processing target image is not generated. In this case, “coordinates on the processing target image plane” in step S4 in the output image generation processing is read as “coordinates on the space model”.

  With the above configuration, the image generation apparatus 100 can generate a processing target image and an output image that allow the operator to intuitively grasp the positional relationship between the object around the excavator 60 and the excavator 60.

  In addition, the image generation apparatus 100 performs coordinate association so as to go back from the processing target image plane R3 to the input image plane R4 via the spatial model MD. Thereby, the image generation device 100 can reliably correspond each coordinate on the processing target image plane R3 to one or a plurality of coordinates on the input image plane R4. Therefore, the image generation apparatus 100 can quickly generate a higher-quality processing target image as compared with the case where the coordinate association is executed in the order from the input image plane R4 to the processing target image plane R3 via the spatial model MD. Can do. When the coordinate association is executed in the order from the input image plane R4 to the processing target image plane R3 via the space model MD, each coordinate on the input image plane R4 is set to one or more on the processing target image plane R3. It is possible to reliably correspond to the coordinates. However, some of the coordinates on the processing target image plane R3 may not be associated with any of the coordinates on the input image plane R4, and in this case, some of the coordinates on the processing target image plane R3. Need to be interpolated.

  Further, when enlarging or reducing only the image corresponding to the curved surface region R2 of the space model MD, the image generating apparatus 100 changes the angle formed between the parallel line group PL and the processing target image plane R3. Thus, it is possible to realize a desired enlargement or reduction without rewriting the contents of the input image / space model correspondence map 40 only by rewriting only the portion related to the curved surface region R2 in the space model / processing object image correspondence map 41. .

  Further, when changing the appearance of the output image, the image generating apparatus 100 simply rewrites the processing target image / output image correspondence map 42 by changing the values of various parameters relating to scale conversion, affine transformation, or distortion transformation. The desired output image (scale-converted image, affine-transformed image, or distortion-converted image) can be generated without rewriting the contents of the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41.

  Similarly, when changing the viewpoint of the output image, the image generating apparatus 100 simply changes the values of various parameters of the virtual camera 2V and rewrites the processing target image / output image correspondence map 42 to change the input image / space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the space model / processing object image correspondence map 41.

  FIG. 10 is a display example when an output image generated using input images of two cameras 2 (the right side camera 2R and the rear camera 2B) mounted on the excavator 60 is displayed on the display unit 5. .

  The image generation apparatus 100 projects the input images of the two cameras 2 onto the plane region R1 and the curved surface region R2 of the space model MD, and then reprojects them onto the processing target image plane R3 to generate a processing target image. To do. Then, the image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image. In this way, the image generating apparatus 100 has an image in which the vicinity of the excavator 60 is looked down from above (an image in the plane region R1), and an image in which the periphery is viewed from the shovel 60 in the horizontal direction (an image in the processing target image plane R3) An output image that simultaneously displays is generated. Hereinafter, such an output image is referred to as a peripheral monitoring virtual viewpoint image.

  Note that the virtual viewpoint image for periphery monitoring is subjected to image conversion processing (for example, viewpoint conversion processing) on the image projected on the space model MD when the image generation apparatus 100 does not generate a processing target image. Generated by.

  Further, the periphery monitoring virtual viewpoint image is trimmed in a circle so that the image when the excavator 60 performs the turning motion can be displayed without a sense of incongruity, the center CTR of the circle is on the cylindrical center axis of the space model MD, and It is generated so as to be on the pivot axis PV of the excavator 60. Therefore, the peripheral viewpoint virtual viewpoint image is displayed so as to rotate around the center CTR in accordance with the turning operation of the excavator 60. In this case, the cylindrical central axis of the space model MD may or may not coincide with the reprojection axis.

  Note that the radius of the space model MD is, for example, 5 meters. Further, the angle formed between the parallel line group PL and the processing target image plane R3 or the starting point height of the auxiliary line group AL is the maximum reachable distance (for example, 12 meters) of the excavation attachment E from the turning center of the shovel 60. If there is an object (for example, an operator) at a position separated by a certain distance, the display unit 5 is set to display the object sufficiently large (for example, 7 mm or more). obtain.

  Further, the virtual viewpoint image for peripheral monitoring may be a CG image of the excavator 60 so that the front of the excavator 60 coincides with the upper part of the screen of the display unit 5 and the turning center thereof coincides with the center CTR. . This is to make the positional relationship between the shovel 60 and the object appearing in the output image easier to understand. It should be noted that a frame image including various information such as an orientation may be arranged around the virtual viewpoint image for periphery monitoring.

  Next, the details of the periphery monitoring virtual viewpoint image generated by the image generation apparatus 100 will be described with reference to FIGS.

  FIG. 11 is a top view of the excavator 60 on which the image generating apparatus 100 is mounted. In the embodiment shown in FIG. 11, the excavator 60 includes three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) and three person detection sensors 6 (left side person detection sensor 6L, right side). A person detection sensor 6R and a rear person detection sensor 6B). Note that regions CL, CR, and CB indicated by alternate long and short dashed lines in FIG. 11 indicate imaging spaces of the left side camera 2L, the right side camera 2R, and the rear camera 2B, respectively. Further, areas ZL, ZR, and ZB indicated by dotted lines in FIG. 11 indicate monitoring spaces for the left side person detection sensor 6L, the right side person detection sensor 6R, and the rear person detection sensor 6B, respectively.

  In the present embodiment, the monitoring space of the human detection sensor 6 is narrower than the imaging space of the camera 2, but the monitoring space of the human detection sensor 6 may be the same as the imaging space of the camera 2. It may be wide. The monitoring space of the human detection sensor 6 is located in the vicinity of the shovel 60 in the imaging space of the camera 2, but may be in a region farther from the shovel 60. In addition, the monitoring space of the human detection sensor 6 has an overlapping portion in the portion where the imaging spaces of the camera 2 overlap. For example, in the overlapping portion of the imaging space CR of the right camera 2R and the imaging space CB of the rear camera 2B, the monitoring space ZR of the right person detection sensor 6R overlaps with the monitoring space ZB of the rear person detection sensor 6B. However, the monitoring space of the human detection sensor 6 may be arranged so as not to overlap.

  FIG. 12 is a diagram illustrating input images of the three cameras 2 mounted on the excavator 60 and output images generated using the input images.

  The image generation apparatus 100 projects the input images of the three cameras 2 onto the plane area R1 and the curved surface area R2 of the space model MD, and then reprojects them onto the process target image plane R3 to generate a process target image. . The image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image. As a result, the image generating apparatus 100 includes an image in which the vicinity of the excavator 60 is looked down from above (an image in the plane region R1) and an image in which the periphery is viewed from the shovel 60 in the horizontal direction (an image in the processing target image plane R3). A virtual viewpoint image for periphery monitoring to be displayed at the same time is generated. Note that the image displayed at the center of the virtual viewpoint image for periphery monitoring is the CG image 60CG of the excavator 60.

  In FIG. 12, the input image of the right-side camera 2R and the input image of the rear camera 2B each capture a person in an overlapping portion between the imaging space of the right-side camera 2R and the imaging space of the rear camera 2B (right side). (See region R10 surrounded by a two-dot chain line in the input image of the direction camera 2R and region R11 surrounded by a two-dot chain line in the input image of the rear camera 2B.)

  However, if the coordinates on the output image plane are associated with the coordinates on the input image plane for the camera with the smallest incident angle, the output image loses the person in the overlapping portion (one point in the output image). (See region R12 surrounded by a chain line.)

  Therefore, in the output image portion corresponding to the overlapping portion, the image generating apparatus 100 associates the region on the input image plane of the rear camera 2B with the coordinate on the input image plane of the right-side camera 2R. The area is mixed to prevent the object in the overlapping portion from disappearing.

  FIG. 13 is a diagram for explaining a stripe pattern process which is an example of an image disappearance prevention process for preventing the disappearance of an object in an overlapping portion of the imaging spaces of two cameras.

  F13A is a diagram illustrating an output image portion corresponding to an overlapping portion between the imaging space of the right-side camera 2R and the imaging space of the rear camera 2B, and corresponds to a rectangular region R13 indicated by a dotted line in FIG.

  In F13A, a region PR1 filled in gray is an image region in which the input image portion of the rear camera 2B is arranged, and the input image of the rear camera 2B is at each coordinate on the output image plane corresponding to the region PR1. Coordinates on the plane are associated.

  On the other hand, a region PR2 filled with white is an image region in which the input image portion of the right side camera 2R is arranged, and each coordinate on the output image plane corresponding to the portion PR2 has an input image plane of the right side camera 2R. The upper coordinates are associated.

  In this embodiment, the region PR1 and the region PR2 are arranged so as to form a stripe pattern (stripe pattern processing), and the boundary line of the portion where the regions PR1 and the region PR2 are alternately arranged in a stripe shape is the swivel of the excavator 60. Defined by concentric circles on a horizontal plane centered on the center.

  F13B is a top view showing the situation of the space area diagonally right behind the excavator 60, and shows the current situation of the space area imaged by both the rear camera 2B and the right-side camera 2R. Further, F13B indicates that a rod-shaped three-dimensional object OB exists on the right rear side of the excavator 60.

  F13C represents a part of an output image generated based on an input image obtained by actually capturing the space area indicated by F13B with the rear camera 2B and the right-side camera 2R.

  Specifically, the image OB1 is expanded in the extension direction of the line connecting the rear camera 2B and the three-dimensional object OB by the viewpoint conversion for generating the road surface image of the three-dimensional object OB in the input image of the rear camera 2B. Represents what was done. That is, the image OB1 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the rear camera 2B.

  In addition, the image OB2 is expanded in the extension direction of the line connecting the right camera 2R and the three-dimensional object OB by the viewpoint conversion for generating the road surface image of the three-dimensional object OB in the input image of the right camera 2R. Represents a thing. That is, the image OB2 is a part of an image of the three-dimensional object OB displayed when a road surface image in the output image portion is generated using the input image of the right side camera 2R.

  As described above, the image generating apparatus 100 includes the region PR1 in which the coordinates on the input image plane of the rear camera 2B are associated with the region PR2 in which the coordinates on the input image plane of the right-side camera 2R are associated with each other in the overlapping portion. Mix. As a result, the image generation apparatus 100 displays both the two images OB1 and OB2 related to one solid object OB on the output image, and prevents the solid object OB from disappearing from the output image.

  FIG. 14 is a comparison diagram showing the difference between the output image of FIG. 12 and the output image obtained by applying image loss prevention processing (stripe pattern processing) to the output image of FIG. The output image of FIG. 12 is shown, and the lower part of FIG. 14 shows the output image after applying the image loss prevention process (stripe pattern process). The person disappears in the region R12 surrounded by the alternate long and short dash line in the upper diagram of FIG. 14, whereas the person is displayed without disappearing in the region R14 surrounded by the dashed dotted line in the lower diagram of FIG.

  Note that the image generating apparatus 100 may prevent the disappearance of the object in the overlapping portion by applying a mesh pattern process, an averaging process, or the like instead of the stripe pattern process. Specifically, the image generation apparatus 100 outputs an average value of corresponding pixel values (for example, luminance values) in the input images of the two cameras, corresponding to the overlapping portion, by averaging processing. It is adopted as the pixel value of the image portion. Alternatively, the image generation apparatus 100 performs mesh pattern processing so that, in the output image portion corresponding to the overlapping portion, the region in which the pixel value in the input image of one camera is associated with the pixel value in the input image of the other camera Are arranged so as to form a net pattern (mesh pattern). Thereby, the image generation device 100 prevents the objects in the overlapping portion from disappearing.

  Next, referring to FIGS. 15 to 17, the image generation unit 11 determines an input image to be used for generating an output image from a plurality of input images based on the determination result of the presence / absence determination unit 12 (hereinafter, referred to as “image generation unit”). The “first input image determination process” will be described. FIG. 15 is a diagram illustrating input images of the three cameras 2 mounted on the excavator 60 and output images generated using the input images, and corresponds to FIG. FIG. 16 is a correspondence table showing a correspondence relationship between the determination result of the presence / absence determination unit 12 and the input image used for generating the output image. FIG. 17 is a display example of an output image generated based on the input image determined in the first input image determination process.

  As shown in FIG. 15, the image generating apparatus 100 projects the input images of the three cameras 2 onto the plane area R1 and the curved area R2 of the space model MD, and then reprojects them onto the processing target image plane R3. To generate a processing target image. The image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image. As a result, the image generation device 100 generates a peripheral monitoring virtual viewpoint image that simultaneously displays an image of the vicinity of the excavator 60 as viewed from above and an image of the periphery of the excavator 60 viewed in the horizontal direction.

  In FIG. 15, the input images of the left camera 2L, the rear camera 2B, and the right camera 2R indicate a state where there are three workers. The output image shows a state where nine workers are present around the excavator 60.

  Here, with reference to the correspondence table of FIG. 16, the correspondence between the determination result of the presence / absence determination unit 12 and the input image used for generating the output image will be described. A circle indicates that the person is determined to exist by the presence / absence determination means 12, and a mark x indicates that it is determined that no person exists.

  Pattern 1 is determined using the input image of the left side camera 2L when it is determined that there is a person only in the left side monitoring space ZL and no person is present in the rear side monitoring space ZB and the right side monitoring space ZR. An output image is generated. This pattern 1 is employed, for example, when there are workers (three people in this example) only on the left side of the excavator 60. As shown by the output image D1 in FIG. 17, the image generation means 11 outputs the input image of the left-side camera 2L capturing three workers as an output image as it is. Hereinafter, an output image using the input image as it is is referred to as a “through image”.

  The pattern 2 is output using the input image of the rear camera 2B when it is determined that there is a person only in the rear monitoring space ZB and it is determined that there is no person in the left side monitoring space ZL and the right side monitoring space ZR. Indicates that an image is generated. This pattern 2 is employed when there are workers (three people in this example) only behind the excavator 60, for example. As shown by the output image D2 in FIG. 17, the image generating means 11 outputs the input image of the rear camera 2B capturing three workers as an output image as it is.

  Pattern 3 is determined by using the input image of the right side camera 2R when it is determined that there is a person only in the right side monitoring space ZR, and when it is determined that no person exists in the left side monitoring space ZL and the rear monitoring space ZB. An output image is generated. This pattern 3 is employed, for example, when there are workers (three people in this example) only on the right side of the excavator 60. As shown by the output image D3 in FIG. 17, the image generating means 11 outputs the input image of the right-side camera 2R that captures three workers as an output image as it is.

  Pattern 4 is an output image using all three input images when it is determined that there is a person in the left monitoring space ZL and the rear monitoring space ZB, and it is determined that there is no person in the right monitoring space ZR. Is generated. This pattern 4 is employed, for example, when there are workers on the left side and the rear side of the excavator 60 (three people in this example, a total of six people). As shown by an output image D4 in FIG. 17, the image generation unit 11 outputs, as an output image, a peripheral monitoring virtual viewpoint image that captures six workers generated based on three input images.

  The pattern 5 is an output image using all three input images when it is determined that there is a person in the rear monitoring space ZB and the right side monitoring space ZR and no person is present in the left side monitoring space ZL. Is generated. This pattern 5 is employed, for example, when there are workers (three people in this example, six people in total in this example) on the rear and right sides of the excavator 60. As shown by an output image D5 in FIG. 17, the image generation means 11 outputs a peripheral monitoring virtual viewpoint image that captures six workers generated based on the three input images as an output image.

  Pattern 6 is an output image using all three input images when it is determined that there is a person in the left side monitoring space ZL and the right side monitoring space ZR and no person is present in the rear monitoring space ZB. Is generated. This pattern 6 is employed, for example, when there are workers on the left side and the right side of the excavator 60 (in this example, three persons in total, six persons). As shown by an output image D6 in FIG. 17, the image generation means 11 outputs a peripheral monitoring virtual viewpoint image that captures six workers generated based on three input images as an output image.

  In pattern 7, when it is determined that there is a person in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR, an output image is generated using all three input images. Represents. This pattern 7 is employed, for example, when there are workers on the left side, rear side, and right side of the excavator 60 (three people in this example, a total of nine people). As shown in the output image of FIG. 15, the image generation unit 11 outputs a peripheral monitoring virtual viewpoint image that captures nine workers generated based on three input images as an output image.

  In pattern 8, when it is determined that no person is present in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR, an output image is generated using all three input images. Represents that. This pattern 8 is employed, for example, when there is no worker on the left side, rear side, and right side of the excavator 60. The image generation means 11 outputs, as an output image, a peripheral monitoring virtual viewpoint image that is generated based on the three input images and reflects a state in which no worker is present in the vicinity, as indicated by an output image D7 in FIG.

  As described above, when it is determined that a person exists in only one of the three monitoring spaces, the image generation unit 11 outputs a through image of the corresponding camera as an output image. This is because a person existing in the monitoring space is displayed on the display unit 5 as large as possible. On the other hand, when it is determined that there is a person in two or more of the three monitoring spaces, the image generation unit 11 outputs a virtual viewpoint image for peripheral monitoring without outputting a through image. This is because it is not possible to display all the people existing around the excavator 60 on the display unit 5 with only one through image. Also, if a virtual viewpoint image for peripheral monitoring is output, it exists around the excavator 60. This is because all persons can be displayed on the display unit 5. In addition, when it is determined that no person is present in any of the three monitoring spaces, the image generation unit 11 outputs the periphery monitoring virtual viewpoint image without outputting the through image. This is because there is no person to be magnified and other objects other than the person around the excavator 60 can be monitored widely.

  In addition, when displaying the through image, the image generation unit 11 may display a text message so that it can be understood which input image is used.

  Next, referring to FIGS. 18 to 20, another example of processing for determining an input image to be used for generating an output image from a plurality of input images based on the determination result of the presence / absence determination unit 12 (hereinafter referred to as “No. 2 input image determination process). FIG. 18 is a top view of the excavator 60 showing another arrangement example of the human detection sensor 6 and corresponds to FIG. 11. FIG. 19 is a correspondence table showing the correspondence between the determination result of the presence / absence determination unit 12 and the input image used for generating the output image, and corresponds to FIG. FIG. 20 is a display example of an output image generated based on the input image determined in the second input image determination process.

  In the embodiment shown in FIG. 18, the excavator 60 includes two cameras 2 (right side camera 2R and rear camera 2B) and three person detection sensors 6 (right side person detection sensor 6R, right rear person detection sensor 6BR, And a rear person detection sensor 6B). Note that regions CR and CB indicated by alternate long and short dashed lines in FIG. 18 indicate imaging spaces of the right side camera 2R and the rear camera 2B, respectively. In addition, areas ZR, ZBR, and ZB indicated by dotted lines in FIG. 18 indicate monitoring spaces for the right person detection sensor 6R, the right rear person detection sensor 6BR, and the rear person detection sensor 6B, respectively. In addition, a region X indicated by hatching in FIG. 18 indicates an overlapping portion of the imaging space CR and the imaging space CB (hereinafter referred to as “overlapping imaging space X”).

  The arrangement example of FIG. 18 is the arrangement example of FIG. 11 in that the monitoring space ZR and the monitoring space ZB do not have an overlapping portion and the right rear person detection sensor 6BR having the monitoring space ZBR including the overlapping imaging space X is provided. Is different.

  With this arrangement of the human detection sensor 6, the image generation apparatus 100 can determine whether or not a person is present in the overlapping imaging space X. Then, the image generation apparatus 100 can switch the contents of the output image more appropriately by using the determination result in determining the input image used for generating the output image.

  Here, with reference to the correspondence table of FIG. 19, the correspondence relationship between the determination result of the presence / absence determination unit 12 and the input image used to generate the output image will be described.

  The pattern A is output using the input images of the rear camera 2B and the right camera 2R when it is determined that no person is present in all of the rear monitor space ZB, the right monitor space ZBR, and the right monitor space ZR. Indicates that an image is generated. This pattern A is employed, for example, when there is no worker around the excavator 60. The image generation unit 11 generates and outputs a peripheral monitoring virtual viewpoint image that reflects a state in which no worker is present in the vicinity based on the two input images, as shown by an output image D7 in FIG.

  The pattern B is output using the input image of the rear camera 2B when it is determined that there is a person only in the rear monitoring space ZB and it is determined that there is no person in the right rear monitoring space ZBR and the right side monitoring space ZR. Indicates that an image is generated. This pattern B is employed, for example, when there is one worker P1 behind the excavator 60. As shown by the output image D10 in FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B capturing the worker P1 as an output image as it is.

  In the pattern C, when it is determined that a person exists only in the right side monitoring space ZR and it is determined that no person exists in the rear monitoring space ZB and the right rear monitoring space ZBR, the input image of the right side camera 2R is used. An output image is generated. This pattern C is employed when, for example, one worker P2 is present on the right side of the excavator 60. As shown by the output image D11 in FIG. 20, the image generation means 11 outputs the input image of the right-side camera 2R capturing the worker P2 as it is as the output image.

  In the pattern D, when it is determined that there is a person only in the right rear monitoring space ZBR and it is determined that no person exists in the rear monitoring space ZB and the right side monitoring space ZR, the input of the rear camera 2B and the right side camera 2R is performed. It represents that an output image is generated using an image. This pattern D is employed, for example, when one worker P3 exists in the overlapping imaging space X on the right rear side of the excavator 60. As shown by the output image D12 in FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B capturing the worker P3 as it is as the first output image (left side in the figure), and captures the same worker P3. The input image of the right side camera 2R is output as it is as the second output image (right side in the figure).

  The pattern E is an input image of the rear camera 2B and the right camera 2R when it is determined that there is a person in the rear monitoring space ZB and the right rear monitoring space ZBR and no person is present in the right monitoring space ZR. Represents that an output image is generated. This pattern E is employed, for example, when there is one worker P4 behind the excavator 60 and one worker P5 in the overlapping imaging space X on the right rear side of the excavator 60. . As shown by the output image D13 in FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B capturing the worker P4 and the worker P5 as it is as the first output image (left side in the figure), and the worker The input image of the right-side camera 2R that captures only P4 is output as it is as the second output image (right side in the figure).

  The pattern F is an input image of the rear camera 2B and the right side camera 2R when it is determined that there is a person in the rear monitoring space ZB and the right side monitoring space ZR and it is determined that no person exists in the right rear monitoring space ZBR. Represents that an output image is generated. This pattern F is employed when, for example, one worker P6 exists behind the excavator 60 and another worker P7 exists on the right side of the excavator 60. As shown by the output image D14 in FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B capturing the worker P6 as it is as the first output image (left side in the figure), and captures the worker P7. The input image of the right side camera 2R is output as it is as the second output image (right side in the figure).

  The pattern G is an input image of the rear camera 2B and the right side camera 2R when it is determined that there is a person in the right rear monitoring space ZBR and the right side monitoring space ZR and no person is present in the rear monitoring space ZB. Represents that an output image is generated. This pattern G is obtained when, for example, one worker P8 exists in the overlapping imaging space X on the right rear side of the excavator 60, and another worker P9 exists on the right side of the excavator 60. Adopted. As shown by the output image D15 in FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B capturing the worker P8 as it is as the first output image (left side in the figure), and the worker P8 and the worker The input image of the right-side camera 2R that captures P9 is output as it is as the second output image (right side in the figure).

  The pattern H is an output image using input images of the rear camera 2B and the right side camera 2R when it is determined that a person exists in all of the rear monitoring space ZB, the right rear monitoring space ZBR, and the right side monitoring space ZR. Is generated. In this pattern H, for example, one worker P10 exists in the overlapping imaging space X on the right rear side of the excavator 60, another worker P11 exists behind the excavator 60, and the excavator 60 This is employed when another worker P12 is present on the right side of. As shown by the output image D16 in FIG. 20, the image generation means 11 outputs the input image of the rear camera 2B that captures the worker P10 and the worker P11 as a first output image (left side in the figure) as it is. The input image of the right-side camera 2R that captures P10 and the worker P12 is output as it is as the second output image (right side in the figure).

  In the second input image determination process, when a person is seen only in the input image of one camera, the image generation means 11 outputs a through image of the corresponding camera as an output image. This is because a person existing in the monitoring space is displayed on the display unit 5 as large as possible. On the other hand, when a person is shown in any of the input images of the two cameras, the image generation means 11 outputs the through images of the two cameras simultaneously and individually. This is because it is not possible to display all persons existing around the excavator 60 on the display unit 5 with only one through image, and when a through image is output by outputting a virtual viewpoint image for peripheral monitoring. This is because it makes it difficult to visually recognize the worker. In addition, when it is determined that no person is present in any of the three monitoring spaces, the image generation unit 11 outputs the periphery monitoring virtual viewpoint image without outputting the through image. This is because there is no person to be magnified and other objects other than the person around the excavator 60 can be monitored widely.

  In addition, when displaying the through image, the image generation unit 11 may display a text message so that it can be understood which input image is used, as in the first input image determination process.

  Next, another display example of the output image generated based on the input image determined by the first input image determination process or the second input image determination process will be described with reference to FIG.

  As illustrated in FIG. 21, the image generation unit 11 may simultaneously display the periphery monitoring virtual viewpoint image and the through image. For example, when there is an operator behind the excavator 60, the image generation unit 11 uses the periphery monitoring virtual viewpoint image as the first output image as shown in the output image D20 of FIG. 21, and the through image of the rear camera 2B. May be displayed as the second output image. In this case, the through image is displayed below the periphery monitoring virtual viewpoint image. This is to make the operator of the excavator 60 intuitively understand that there is an operator behind the excavator 60. In addition, when an operator is present on the left side of the excavator 60, the image generation unit 11 uses the periphery monitoring virtual viewpoint image as the first output image as shown by the output image D21 in FIG. The through image may be displayed as the second output image. In this case, the through image is displayed on the left side of the periphery monitoring virtual viewpoint image. Similarly, when an operator is present on the right side of the excavator 60, the image generation unit 11 sets the periphery monitoring virtual viewpoint image as the first output image as shown by the output image D22 in FIG. 21, and the right-side camera 2R. The through image may be displayed as the second output image. In this case, the through image is displayed on the right side of the peripheral viewpoint virtual viewpoint image.

  Alternatively, the image generation unit 11 may simultaneously display the periphery monitoring virtual viewpoint image and the plurality of through images. For example, when there are workers behind and on the right side of the excavator 60, the image generation unit 11 uses the periphery monitoring virtual viewpoint image as the first output image, as shown by the output image D23 in FIG. 21, and the rear camera 2B. May be displayed as the second output image, and the through image of the right side camera 2R may be displayed as the third output image. In this case, the through image of the rear camera 2B is displayed below the periphery monitoring virtual viewpoint image, and the through image of the right side camera 2R is displayed on the right side of the periphery monitoring virtual viewpoint image.

  Note that the image generation means 11 may display information representing the contents of the output image when displaying a plurality of output images simultaneously. For example, the image generation means 11 may display text such as “rear camera through image” on or around the through image of the rear camera 2B.

  Further, when displaying a plurality of output images at the same time, the image generating means 11 may pop-up one or a plurality of through images on the periphery monitoring virtual viewpoint image.

  With the above configuration, the image generation apparatus 100 switches the content of the output image based on the determination result of the presence / absence determination unit 12. Specifically, the image generation apparatus 100, for example, includes a peripheral monitoring virtual viewpoint image that reduces and changes the contents of individual input images and a through image that displays the contents of the individual input images as they are. Switch. As described above, the image generating apparatus 100 displays the through image when an operator is detected around the excavator 60, so that the operator of the excavator 60 can display the through-viewing virtual viewpoint image only. It is possible to more reliably prevent the operator from being overlooked. This is because the worker is displayed large and easy to understand.

  In the above-described embodiment, when the image generation unit 11 generates an output image based on an input image of one camera, the through image of that camera is used as the output image. However, the present invention is not limited to this configuration. For example, the image generation means 11 may generate a rear monitoring virtual viewpoint image as an output image based on the input image of the rear camera 2B.

  In the above-described embodiment, the image generation apparatus 100 associates the monitoring space of one human detection sensor with the imaging space of one camera. However, the monitoring space of one human detection sensor is set in the imaging space of a plurality of cameras. You may make it respond | correspond, and you may make the monitoring space of several human detection sensor respond | correspond to the imaging space of one camera.

  Further, in the above-described embodiment, the image generation apparatus 100 overlaps a part of the monitoring space of two adjacent human detection sensors, but the monitoring space may not be overlapped. Further, the image generation apparatus 100 may be configured such that the monitoring space of one human detection sensor is completely included in the monitoring space of another human detection sensor.

  In the above-described embodiment, the image generation apparatus 100 switches the content of the output image at the moment when the determination result of the presence / absence determination unit 12 changes. However, the present invention is not limited to this configuration. For example, the image generation apparatus 100 may set a predetermined delay time from when the determination result of the presence / absence determination unit 12 changes until the content of the output image is switched. This is to suppress frequent switching of the contents of the output image.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, the image generation apparatus 100 employs the cylindrical spatial model MD as the spatial model, but may employ a spatial model having other columnar shapes such as a polygonal column, A spatial model composed of two side surfaces may be adopted, or a spatial model having only side surfaces may be adopted.

  The image generating apparatus 100 is mounted on a self-propelled excavator with movable members such as a bucket, an arm, a boom, and a turning mechanism together with a camera and a human detection sensor. The image generating apparatus 100 constitutes an operation support system that supports the movement of the shovel and the operation of these movable members while presenting the surrounding image to the operator. However, the image generation apparatus 100 may be mounted together with a camera and a human detection sensor on a work machine that does not have a turning mechanism such as a forklift or an asphalt finisher. Alternatively, the image generation apparatus 100 may be mounted together with a camera and a human detection sensor on a work machine that has a movable member but does not self-propel, such as an industrial machine or a fixed crane. The image generation apparatus 100 may constitute an operation support system that supports the operation of these work machines.

  DESCRIPTION OF SYMBOLS 1 ... Control part 2 ... Camera 2L ... Left side camera 2R ... Right side camera 2B ... Rear camera 3 ... Input part 4 ... Memory | storage part 5 ... Display part 6 .... -Human detection sensor 6L ... Left side person detection sensor 6R ... Right side person detection sensor 6B ... Back person detection sensor 10 ... Coordinate matching means 11 ... Image generation means 12 ... Human Presence / absence determination means 40 ... Input image / spatial model correspondence map 41 ... Spatial model / processing target image correspondence map 42 ... Processing target image / output image correspondence map 60 ... Excavator 61 ... Undercarriage 62 ... turning mechanism 63 ... upper turning body 64 ... cab 100 ... image generating device

Claims (8)

A lower traveling body,
A revolving structure mounted on the lower traveling body;
A plurality of cameras mounted on the revolving body,
An image generating means for said plurality of cameras to generate an output image for display using the input image you captured,
And determining who existence judgment means your Keru human presence around the shovel,
Bei example and an image generating means for switching the contents of the output image based on the determination result of the person existence judgment unit,
A non-composite image that is an input image captured by each of the plurality of cameras, and a composite image generated from the plurality of input images, a CG image of an excavator being arranged at the center, and around the CG image A composite image in which a road surface image is arranged can be displayed on the display screen as an output image, and the composite image and the non-composite image are selectively displayed on the display screen based on a determination result of the presence or absence of a person.
Excavator . The image generating means generates an output image using an input image corresponding to a monitoring space determined to have a person;
The excavator according to claim 1. Simultaneously displaying a plurality of images corresponding to each of a plurality of monitoring spaces determined to have a person,
The shovel according to claim 1 or 2. The plurality of cameras includes a first camera and a second camera,
The plurality of monitoring spaces include a first monitoring space that can be imaged by the first camera and a second monitoring space that can be imaged by the second camera,
The image generation means generates an output image using the input image of the first camera and the input image of the second camera when it is determined that a person exists in the first monitoring space and the second monitoring space. To
The excavator according to claim 3 . The image generation means generates a first output image using an input image of the first camera and a second output image using an input image of the second camera.
The excavator according to claim 4. The non-synthetic image includes an image of an end portion of the revolving structure.
The excavator according to any one of claims 1 to 5 . Based on the determination result of the presence or absence of a person, display either the composite image or the through image as the non-composite image on the display screen.
The excavator according to any one of claims 1 to 6. The plurality of cameras are a rear camera, a left side camera, and a right side camera,
The plurality of monitoring spaces include a rear monitoring space that can be imaged by the rear camera, a left side monitoring space that can be imaged by the left side camera, and a right side monitoring space that can be imaged by the right side camera,
The image generating means includes
When it is determined that a person exists only in the rear monitoring space, an output image is generated using an input image of the rear camera,
When it is determined that a person exists only in the left side monitoring space, an output image is generated using an input image of the left side camera,
When it is determined that a person exists only in the right side monitoring space, an output image is generated using an input image of the right side camera,
When it is determined that no person is present in any of the rear monitoring space, the left side monitoring space, and the right side monitoring space, a virtual viewpoint image for peripheral monitoring using the plurality of input images is output as an output image Generate as
The excavator according to claim 3 .