JP7131779B2 - Image display system for working machine, remote control system for working machine, working machine, and image display method for working machine - Google Patents

Description

The present invention relates to an image display system for a working machine, a remote control system for the working machine, a working machine, and an image display method for the working machine.

2. Description of the Related Art As described in Patent Literature 1, there is known a technique for remotely controlling a working machine such as a hydraulic excavator.

JP 2004-294067 A

When operating a work machine remotely, an operation using an image of the operator's viewpoint of the work machine lacks a sense of perspective because the displayed image is two-dimensional. Therefore, it becomes difficult to grasp the distance between the work target and the work machine, and work efficiency may decrease. Also, when an operator on a work machine operates the work machine, depending on the skill level of the operator, it may be difficult to grasp the distance between the work machine and the work target, which may reduce work efficiency. .

SUMMARY OF THE INVENTION An object of the present invention is to suppress a decrease in work efficiency when working with a work machine having a work implement having a work tool.

The present invention provides an imaging device attached to a work machine having a work machine having a work tool, an attitude detection device for detecting the attitude of the work machine, and a distance detector for obtaining information on the distance from the work machine to a work target. using a device, information on the position of the work implement obtained using the posture of the work machine, and information on the position of the work target obtained from the information on the distance obtained by the distance detection device, a processing device for generating an image of a portion corresponding to the work tool on the work object facing the work tool, synthesizing the image with the image of the work object captured by the imaging device, and displaying the image on a display device; An image display system for a work machine, comprising: It is preferable that the processing device generates an image of a portion corresponding to the work implement with reference to the imaging device.

Preferably, the line image is a grid comprising a plurality of first line images and a plurality of second line images intersecting the plurality of first line images.

Preferably, the processing device obtains a region occupied by the work tool in the image of the work object using the posture of the work machine, and removes the obtained region from the shape information of the work object.

The work tool is a bucket, and the processing device generates a line of a portion corresponding to the cutting edge of the bucket on the work target as an image of a portion corresponding to the work tool on the work target facing the work tool. An image is preferably generated.

The processing device generates an image of a straight line connecting one end in the width direction of the cutting edge of the bucket and the work target, and a straight line connecting the other end in the width direction of the cutting edge of the bucket and the work target. is preferably generated, combined with the image of the work target captured by the imaging device, and displayed on a display device.

Preferably, the processing device obtains spatial position information regarding the work tool or the work object and causes the display device to display the information.

The processing device controls the position of the work tool, the attitude of the work tool, the position of the work target, the relative attitude of the work target, the relative distance between the work tool and the work target, and the work tool. It is preferable that at least one relative posture with respect to the work object is obtained and displayed on the display device.

Preferably, the processing device generates a line image along the surface of the work target using information on the position of the work target, synthesizes it with the image of the work target, and displays it on the display device.

It is preferable that the imaging device, the attitude detection device, and the distance detection device are provided in the working machine, and the processing device and the display device are provided in a facility equipped with an operation device for remotely operating the working machine.

According to the present invention, in remote-controlling a working machine having a working tool, the working machine includes a display device, information on the position of the working tool obtained using the attitude of the working machine, and the information on the position of the working tool. Using information on the position of the work object obtained from information on the distance of the work machine to the work object obtained by the distance detection device, the work implement is positioned on the work object facing the work implement. an image display system for a working machine, comprising: a processing device that generates an image of a part of the work machine based on the imaging device, synthesizes the image with the image of the work target captured by the imaging device, and causes the display device to display the image. is.

The present invention is a remote control system for a work machine including the image display system for the work machine described above and an operation device for operating the work machine provided in the work machine.

The present invention is a work machine equipped with the image display system for the work vehicle described above.

ADVANTAGE OF THE INVENTION This invention can suppress the fall of the working efficiency at the time of working using the working machine provided with the working machine which has a working tool.

FIG. 1 is a diagram showing an image display system for a work machine and a remote control system for a work machine according to an embodiment. FIG. 2 is a perspective view showing a hydraulic excavator that is a working machine according to the embodiment. FIG. 3 is a diagram showing a control system of the hydraulic excavator, which is the work machine according to the embodiment. FIG. 4 is a diagram for explaining a coordinate system in the image display system and the remote control system according to the embodiment; FIG. 5 is a rear view of the hydraulic excavator. FIG. 6 is a diagram for explaining the coordinate system of the imaging device and the distance detection device. FIG. 7 is a flow chart of an example of control executed by the image display system and the remote control system. FIG. 8 is a diagram showing an imaging device, a distance detection device, and a work target. FIG. 9 is a diagram for explaining occupied areas. FIG. 10 is a diagram showing information on the shape of the work target from which the occupied area has been removed. FIG. 11 is a diagram for explaining an image showing the position of the bucket on the work target. FIG. 12 is a diagram for explaining an image showing the position of the bucket on the work target. FIG. 13 is a diagram for explaining an image showing the position of the bucket on the work target. FIG. 14 is a diagram showing a lattice image that is a reference image. FIG. 15 is a diagram showing a lattice image that is a reference image. FIG. 16 is a diagram showing a working image. FIG. 17 is a diagram for explaining a cutting edge position image when using a loading shovel type work machine. FIG. 18 is a diagram showing a first modified example of processing for obtaining a cutting edge position image. FIG. 19 is a diagram for explaining a second modification of the processing for obtaining the cutting edge position image. FIG. 20 is a diagram for explaining a second modification of the processing for obtaining the cutting edge position image. FIG. 21 is a diagram showing a control system of a hydraulic excavator according to a modification.

A form (embodiment) for carrying out the present invention will be described in detail with reference to the drawings.

<Overview of image display system for working machine and remote control system for working machine>
FIG. 1 is a diagram showing an image display system 100 for a work machine and a remote control system 101 for a work machine according to an embodiment. An image display system 100 for a working machine (hereinafter referred to as the image display system 100 as appropriate) is designed to allow an operator to remotely operate the hydraulic excavator 1, which is a working machine. The imaging device 19 captures images of the topographical surface to be worked by the work machine 2 of the hydraulic excavator 1 , that is, the work target WA and the bucket 8 that is the working tool, and the obtained image is displayed on the display device 52 . At this time, the image display system 100 displays an image 68 of the work target WA captured by the imaging device 19, a grid image 65, and an image 60 for indicating the position of the bucket 8 on the work target WA. is displayed on the display device 52 .

The image display system 100 includes an imaging device 19 , an orientation detection device 32 , a distance detection device 20 and a processing device 51 . A work machine remote control system 101 (hereinafter referred to as a remote control system 101 as appropriate) includes an imaging device 19, an attitude detection device 32, a distance detection device 20, a work machine control device 27, a display device 52, a processing It includes a device 51 and an operating device 53 . In the embodiment, the imaging device 19 , the posture detection device 32 and the distance detection device 20 of the image display system 100 are provided in the excavator 1 , and the processing device 51 is provided in the facility 50 . The facility 50 is a facility for remotely controlling the hydraulic excavator 1 and managing the hydraulic excavator 1 . In the embodiment, the imaging device 19 , the posture detection device 32 , the distance detection device 20 and the work machine control device 27 of the remote control system 101 are provided in the hydraulic excavator 1 , and the display device 52 , the processing device 51 and the operation device 53 are provided in the facility 50 . provided in

The processing device 51 of the image display system 100 includes a processing section 51P, a storage section 51M, and an input/output section 51IO. The processing unit 51P is a processor such as a CPU (Central Processing Unit), for example. The storage unit 51M is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk drive, a storage device, or a combination thereof. The input/output unit 51IO is an interface circuit for connecting the processing device 51 and an external device. In the embodiment, a display device 52, an operation device 53, and a communication device 54 are connected to the input/output unit 51IO as external devices. External devices connected to the input/output unit 51IO are not limited to these.

The processing device 51 receives positional information of the bucket 8, which is a working tool, obtained using the attitude of the working machine 2, and positional information of the work target WA obtained from the distance information obtained by the distance detection device 20. is used to generate an image of a portion corresponding to the bucket 8 on the work target WA facing the bucket 8 with the imaging device 19 as a reference. Then, the processing device 51 synthesizes it with the image of the work target WA captured by the imaging device 19 and causes the display device 52 to display it. The work target WA is a surface on which the working machine 2 of the hydraulic excavator 1 performs work such as excavation or leveling.

The display device 52 is exemplified by a liquid crystal display or a projector, but is not limited to these. The communication device 54 has an antenna 54A. The communication device 54 communicates with the communication device 25 provided in the excavator 1 to acquire information about the excavator 1 and transmit information to the excavator 1 .

The operating device 53 has a left operating lever 53L arranged on the operator's left side and a right operating lever 53R arranged on the operator's right side. The left operating lever 53L and the right operating lever 53R correspond to front, rear, left, and right motions on two axes. For example, the operation of the right operating lever 53R in the front-rear direction corresponds to the operation of the boom 6 of the working machine 2 of the hydraulic excavator 1 . The operation of the right operating lever 53R in the left-right direction corresponds to the operation of the bucket 8 of the working machine 2. As shown in FIG. Operation of the left operating lever 53</b>L in the front-rear direction corresponds to operation of the arm 7 of the working machine 2 . Operation of the left control lever 53L in the left-right direction corresponds to turning of the upper turning body 3 of the hydraulic excavator 1 .

The amount of operation of the left control lever 53L and the right control lever 53R is detected by, for example, a potentiometer and a Hall IC, and the processing device 51 generates control signals for controlling the electromagnetic control valves based on these detected values. . This signal is sent to the work implement control device 27 via the communication device 54 of the facility 50 and the communication device 25 of the hydraulic excavator 1 . The work machine control device 27 controls the work machine 2 by controlling the electromagnetic control valve based on the control signal. The electromagnetic control valve will be described later.

The processing device 51 acquires an input to at least one of the left operating lever 53L and the right operating lever 53R and generates a command for operating at least one of the work implement 2 and the upper revolving body 3 . The processing device 51 transmits the generated command to the communication device 25 of the hydraulic excavator 1 via the communication device 54 . A work machine control device 27 provided in the hydraulic excavator 1 acquires a command from the processing device 51 via the communication device 25 and operates at least one of the work machine 2 and the upper revolving body 3 according to the command.

The hydraulic excavator 1 includes a communication device 25 , a work machine control device 27 , an attitude detection device 32 , an imaging device 19 , a distance detection device 20 , antennas 21 and 22 , and a global position calculation device 23 . The work machine control device 27 controls the work machine 2 . The communication device 25 is connected to the antenna 24 and communicates with a communication device 54 provided at the facility 50 . The work machine control device 27 controls the work machine 2 and the upper revolving body 3 . The posture detection device 32 detects the posture of at least one of the work implement 2 and the hydraulic excavator 1 . The imaging device 19 is attached to the hydraulic excavator 1 and images the work target WA. The distance detection device 20 obtains information on the distance from a predetermined position of the hydraulic excavator 1 to the work target WA. Antennas 21 and 22 receive radio waves from positioning satellite 200 . The global position calculation device 23 uses the radio waves received by the antennas 21 and 22 to obtain the global positions of the antennas 21 and 22, that is, the positions in global coordinates.

<Overall Configuration of Hydraulic Excavator 1>
FIG. 2 is a perspective view showing a hydraulic excavator 1 that is a working machine according to the embodiment. The hydraulic excavator 1 has a vehicle main body 1B and a working machine 2 as a main body. The vehicle body 1B has an upper revolving body 3 as a revolving body and a traveling device 5 as a traveling body. The upper revolving body 3 accommodates devices such as an engine, which is a power generation device, and a hydraulic pump, in an engine room 3EG. In the embodiment, the hydraulic excavator 1 uses, for example, an internal combustion engine such as a diesel engine as the engine that is the power generator, but the power generator is not limited to the internal combustion engine. The power generation device of the hydraulic excavator 1 may be, for example, a so-called hybrid device that combines an internal combustion engine, a generator motor, and a power storage device. Further, the power generation device of the hydraulic excavator 1 may be a device that does not have an internal combustion engine and that is a combination of a power storage device and a generator motor.

The upper revolving body 3 has an operator's cab 4 . The operator's cab 4 is installed on the other end side of the upper revolving body 3 . That is, the driver's cab 4 is installed on the side opposite to the side where the engine room 3EG is arranged. A handrail 9 is attached above the upper revolving body 3 .

The travel device 5 has the upper revolving body 3 mounted thereon. The travel device 5 has crawler belts 5a and 5b. The travel device 5 is driven by one or both of hydraulic motors 5c provided on the left and right sides. The crawler belts 5a and 5b of the traveling device 5 rotate to cause the hydraulic excavator 1 to travel. The working machine 2 is attached to the side of the operator's cab 4 of the upper revolving body 3 .

The hydraulic excavator 1 may include tires instead of the crawler belts 5a and 5b, and may include a traveling device capable of traveling by transmitting the driving force of the engine to the tires via a transmission. As such a hydraulic excavator 1, for example, there is a wheel hydraulic excavator. The hydraulic excavator 1 includes a travel device having such tires, a work machine attached to the vehicle body (main body), and an upper revolving body 3 and its revolving mechanism as shown in FIG. It may also be a backhoe loader, for example, having a structure that does not That is, the backhoe loader has a working machine attached to a vehicle body, and includes a travel device that constitutes a part of the vehicle body.

The upper revolving body 3 has a front side on which the working machine 2 and the operator's cab 4 are arranged, and a rear side on which an engine room 3EG is arranged. The front-rear direction of the upper revolving body 3 is the x direction. The left side of the upper revolving body 3 is the left side when viewed forward, and the right side of the upper revolving body 3 is the right side when viewed forward. The left-right direction of the upper revolving body 3 is also referred to as the width direction or the y direction. The hydraulic excavator 1 or the vehicle body 1B has the travel device 5 side with the upper revolving body 3 as a reference, and the upper revolving body 3 side with the travel device 5 as a reference. The vertical direction of the upper revolving body 3 is the z direction. When the hydraulic excavator 1 is installed on a horizontal plane, the bottom is the vertical direction, that is, the side in which gravity acts, and the top is the side opposite to the vertical direction.

The working machine 2 has a boom 6 , an arm 7 , a bucket 8 as a working tool, a boom cylinder 10 , an arm cylinder 11 and a bucket cylinder 12 . A base end portion of the boom 6 is rotatably attached to the front portion of the vehicle body 1B via a boom pin 13 . The base end of the arm 7 is rotatably attached to the tip of the boom 6 via an arm pin 14 . A bucket 8 is attached to the tip of the arm 7 via a bucket pin 15 . Bucket 8 rotates around bucket pin 15 . A plurality of blades 8B are attached to the bucket 8 on the side opposite to the bucket pin 15 . The cutting edge 8T is the tip of the cutting edge 8B.

The bucket 8 may not have multiple blades 8B. In other words, a bucket that does not have the blade 8B as shown in FIG. 2 and has a blade tip formed in a straight shape from a steel plate may be used. The work implement 2 may include, for example, a tilt bucket having a single blade. A tilt bucket is equipped with a bucket tilt cylinder, and the bucket tilts to the left and right, so even if the hydraulic excavator is on a slope, it is possible to form slopes and flat ground into a free shape and to level the ground. This bucket can also perform rolling compaction work with a plate. In addition, instead of the bucket 8, the work machine 2 may include a slope bucket or a rock-drilling attachment having a chip for rock-drilling as a work tool.

A boom cylinder 10, an arm cylinder 11, and a bucket cylinder 12 shown in FIG. 2 are hydraulic cylinders driven by the pressure of hydraulic oil discharged from hydraulic pumps. A boom cylinder 10 drives the boom 6 to raise and lower it. Arm cylinder 11 drives arm 7 to rotate around arm pin 14 . Bucket cylinder 12 drives bucket 8 to rotate around bucket pin 15 .

Antennas 21 , 22 and an antenna 24 are attached to the upper portion of the upper revolving body 3 . Antennas 21 and 22 are used to detect the current position of hydraulic excavator 1 . Antennas 21 and 22 are electrically connected to a global position calculator 23 shown in FIG. The global position calculation device 23 is a position detection device that detects the position of the hydraulic excavator 1 . The global position calculation device 23 detects the current position of the hydraulic excavator 1 using RTK-GNSS (Real Time Kinematic-Global Navigation Satellite Systems, GNSS means global navigation satellite system). In the following description, the antennas 21, 22 are appropriately referred to as GNSS antennas 21, 22. Signals corresponding to GNSS radio waves received by the GNSS antennas 21 and 22 are input to the global position calculation device 23 . The global position calculation device 23 obtains the installation positions of the GNSS antennas 21 and 22 in the global coordinate system. An example of the global navigation satellite system is GPS (Global Positioning System), but the global navigation satellite system is not limited to this.

As shown in FIG. 2, the GNSS antennas 21 and 22 are preferably installed on the upper revolving body 3 at both ends of the hydraulic excavator 1 that are separated from each other in the left-right direction, that is, in the width direction. In the embodiment, the GNSS antennas 21 and 22 are attached to handrails 9 attached to both sides of the upper swing body 3 in the width direction. Although the positions at which the GNSS antennas 21 and 22 are attached to the upper revolving body 3 are not limited to the handrail 9, the GNSS antennas 21 and 22 are preferably installed at positions as far apart as possible from the hydraulic excavator 1. This is preferable because it improves the detection accuracy of the current position. Also, the GNSS antennas 21 and 22 are preferably installed at positions that do not interfere with the operator's field of view as much as possible.

The imaging device 19 captures an image of the work target WA shown in FIG. 1, and the distance detection device 20 obtains the distance from itself (predetermined position of the hydraulic excavator 1) to the work target WA. It is preferable to obtain information from Therefore, in the embodiment, the antenna 24 , the imaging device 19 and the distance detection device 20 are installed above the driver's cab 4 of the upper swing body 3 . The place where the imaging device 19 and the distance detection device 20 are installed is not limited to above the driver's seat 4 . For example, the imaging device 19 and the distance detection device 20 may be installed inside and above the driver's cab 4 .

An imaging surface 19L of the imaging device 19 faces the front of the upper rotating body 3 . The detection surface 20L of the distance detection device 20 faces the front of the upper revolving body 3 . In an embodiment, the imaging device 19 is a monocular camera with an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. In embodiments, the distance detection device 20 is a three-dimensional laser range finder or distance sensor. The imaging device 19 and the distance detection device 20 are not limited to these. For example, instead of the imaging device 19 and the distance detection device 20, a device having both a function of acquiring an image of the work target WA and a function of obtaining the distance to the work target WA may be used. An example of such a device is a stereo camera.

<Control system of hydraulic excavator 1>
FIG. 3 is a diagram showing the control system 1S of the hydraulic excavator 1, which is the work machine according to the embodiment. The control system 1S includes a communication device 25, a sensor controller 26, a work machine control device 27, an imaging device 19, a distance detection device 20, a global position calculation device 23, an attitude detection device 32, and an IMU (Inertial Measurement Device). Unit: inertial measurement device) 33 and a hydraulic system 36 . Communication device 25 , sensor controller 26 , and working machine control device 27 are connected by signal line 35 . With such a structure, the communication device 25 , the sensor controller 26 , and the working machine control device 27 can mutually exchange information via the signal line 35 . An in-vehicle signal line such as a CAN (Controller Area Network) is exemplified as a signal line for transmitting information within the control system 1S.

The sensor controller 26 has a processor such as a CPU (Central Processing Unit) and storage devices such as RAM and ROM. The sensor controller 26 receives the detection values of the global position calculation device 23, the information of the image captured by the imaging device 19, the detection values of the distance detection device 20, the detection values of the orientation detection device 32, and the detection values of the IMU 33. . The sensor controller 26 transmits the input detection values and image information to the processing device 51 of the facility 50 shown in FIG. 1 via the signal line 35 and the communication device 25 .

The work machine control device 27 has a processor such as a CPU (Central Processing Unit) and storage devices such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The work machine control device 27 acquires a command for operating at least one of the work machine 2 and the upper swing body 3 generated by the processing device 51 of the facility 50 via the communication device 25 . The work implement control device 27 controls the electromagnetic control valve 28 of the hydraulic system 36 based on the acquired command.

The hydraulic system 36 includes an electromagnetic control valve 28, a hydraulic pump 29, and hydraulic actuators such as the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12 and the swing motor 30. The hydraulic pump 29 is driven by the engine 31 and discharges hydraulic oil for operating the hydraulic actuators. The work machine control device 27 controls the flow rate of hydraulic oil supplied to the boom cylinder 10 , the arm cylinder 11 , the bucket cylinder 12 and the swing motor 30 by controlling the electromagnetic control valve 28 . In this manner, the work implement control device 27 controls the operations of the boom cylinder 10 , the arm cylinder 11 , the bucket cylinder 12 and the swing motor 30 .

The sensor controller 26 acquires detection values of the first stroke sensor 16 , the second stroke sensor 17 and the third stroke sensor 18 . The first stroke sensor 16 is provided on the boom cylinder 10, the second stroke sensor 17 is provided on the arm cylinder 11, and the third stroke sensor 18 is provided on the bucket cylinder 12, respectively.

The first stroke sensor 16 detects the boom cylinder length, which is the length of the boom cylinder 10 , and outputs it to the sensor controller 26 . The second stroke sensor 17 detects the arm cylinder length, which is the length of the arm cylinder 11 , and outputs it to the sensor controller 26 . The third stroke sensor 18 detects the bucket cylinder length, which is the length of the bucket cylinder 12 , and outputs it to the sensor controller 26 .

Once the boom cylinder length, the arm cylinder length, and the bucket cylinder length are determined, the attitude of the work implement 2 is determined. Therefore, the first stroke sensor 16 , the second stroke sensor 17 and the third stroke sensor 18 that detect these correspond to the attitude detection device 32 that detects the attitude of the working machine 2 . The posture detection device 32 is not limited to the first stroke sensor 16, the second stroke sensor 17 and the third stroke sensor 18, and may be an angle detector.

Based on the boom cylinder length detected by the first stroke sensor 16 , the sensor controller 26 calculates the tilt angle of the boom 6 with respect to the direction perpendicular to the horizontal plane (z-axis direction) in the local coordinate system, which is the coordinate system of the hydraulic excavator 1 . The work implement control device 27 calculates the tilt angle of the arm 7 with respect to the boom 6 from the arm cylinder length detected by the second stroke sensor 17 . The work implement control device 27 calculates the tilt angle of the bucket 8 with respect to the arm 7 from the bucket cylinder length detected by the third stroke sensor 18 . The tilt angles of the boom 6 , the arm 7 and the bucket 8 are information indicating the posture of the work implement 2 . That is, the sensor controller 26 obtains information indicating the attitude of the work implement 2 . The sensor controller 26 transmits the calculated tilt angle to the processing device 51 of the facility 50 shown in FIG. 1 via the signal line 35 and the communication device 25 .

The GNSS antenna 21 receives a position P1 indicating its own position from positioning satellites. The GNSS antenna 22 receives a position P2 indicating its own position from positioning satellites. The GNSS antennas 21, 22 receive positions P1, P2 with a period of, for example, 10 Hz. Positions P1 and P2 are information on the positions where the GNSS antennas are installed in the global coordinate system. Signals corresponding to GNSS radio waves received by the GNSS antennas 21 and 22 , that is, the positions P<b>1 and P<b>2 are input to the global position calculation device 23 . The GNSS antennas 21 and 22 output to the global position calculator 23 each time they receive the positions P1 and P2.

The global position calculation device 23 has a processor such as a CPU and storage devices such as RAM and ROM. The global position calculation device 23 detects positions P1 and P2 of the GNSS antennas 21 and 22 in the global coordinate system at a frequency of 10 Hz, for example, and outputs them to the sensor controller 26 as reference position information Pga1 and Pga2. In the embodiment, the global position calculation device 23 obtains the azimuth angle of the hydraulic excavator 1, more specifically, the yaw angle, which is the azimuth angle of the upper revolving structure 3, from the two acquired positions P1 and P2, and the sensor controller 26 output to The sensor controller 26 transmits the acquired reference position information Pga1, Pga2 and yaw angle to the processing device 51 of the facility 50 shown in FIG.

The IMU 33 detects the motion and posture of the hydraulic excavator 1 . The operation of the hydraulic excavator 1 includes at least one of the operation of the upper revolving body 3 and the operation of the travel device 5 . The posture of the hydraulic excavator 1 can be represented by the roll angle, pitch angle, and yaw angle of the hydraulic excavator 1 . In the embodiment, the IMU 33 detects and outputs the angular velocity and acceleration of the excavator 1 .

<About the coordinate system>
FIG. 4 is a diagram for explaining coordinate systems in the image display system 100 and the remote control system 101 according to the embodiment. FIG. 5 is a rear view of the hydraulic excavator 1. FIG. FIG. 6 is a diagram for explaining the coordinate system of the imaging device and the distance detection device. In the image display system 100 and the remote control system 101, a global coordinate system, a local coordinate system, a coordinate system of the imaging device 19, and a coordinate system of the distance detection device 20 exist. In embodiments, the global coordinate system is, for example, the coordinate system in GNSS. The global coordinate system is a three-dimensional coordinate system indicated by (X, Y, Z) with reference to, for example, the reference position PG of the reference pile 80, which is the reference installed in the work area GA of the hydraulic excavator 1. As shown in FIG. 5, the reference position PG is positioned, for example, at the tip 80T of the reference pile 80 installed in the work section GA.

The local coordinate system is a three-dimensional coordinate system represented by (x, y, z) with the excavator 1 as a reference. In the embodiment, the origin position PL of the local coordinate system is the intersection of the z-axis, which is the central axis of rotation of the upper revolving body 3, and the plane perpendicular to the z-axis within the swing circle of the upper revolving body 3. It is not limited. The plane perpendicular to the z-axis within the swing circle can be a plane passing through the center of the swing circle in the z-axis direction.

In the embodiment, the coordinate system of the imaging device 19 is a three-dimensional coordinate system represented by (Xc, Yc, Zc) with the center of the light receiving surface 19P of the imaging element 19RC as the origin PC, as shown in FIG. be. In the embodiment, the coordinate system of the distance detection device 20 is three-dimensional coordinates indicated by (Xd, Yd, Zd) with the center of the light receiving surface 20P of the distance detection element 20RC as the origin PD, as shown in FIG. It is a system.

<Posture of hydraulic excavator 1>
As shown in FIG. 5, the inclination angle θ4 of the upper revolving structure 3 with respect to the lateral direction, that is, the width direction is the roll angle of the hydraulic excavator 1, and the inclination angle θ5 of the upper revolving structure 3 with respect to the longitudinal direction is the pitch of the hydraulic excavator 1. The angle of the upper revolving body 3 around the z-axis is the yaw angle of the hydraulic excavator 1 . The roll angle is obtained by integrating the angular velocity around the x-axis detected by the IMU 33 over time, the pitch angle is obtained by integrating the angular velocity around the y-axis detected by the IMU 33 over time, and the yaw angle is detected by the IMU 33. It is obtained by integrating the angular velocity around the z-axis over time. The angular velocity around the z-axis is the turning angular velocity ω of the hydraulic excavator 1 . That is, the yaw angle of the hydraulic excavator 1, more specifically, the upper swing body 3 can be obtained by integrating the swing angular velocity ω with respect to time.

The acceleration and angular velocity detected by the IMU 33 are output to the sensor controller 26 as motion information. The sensor controller 26 performs processing such as filtering and integration on the motion information acquired from the IMU 33 to obtain an inclination angle θ4 that is a roll angle, an inclination angle θ5 that is a pitch angle, and a yaw angle. The sensor controller 26 transmits the calculated inclination angle θ4, inclination angle θ5, and yaw angle as information related to the attitude of the hydraulic excavator 1 shown in FIG. It is transmitted to the processing device 51 of the facility 50 where the data is sent.

The sensor controller 26 obtains information indicating the attitude of the work implement 2 as described above. Specifically, the information indicating the posture of the work implement 2 is the tilt angle θ1 of the boom 6 with respect to the direction (z-axis direction) orthogonal to the horizontal plane in the local coordinate system, the tilt angle θ2 of the arm 7 with respect to the boom 6, and the angle θ2 with respect to the arm 7. It is the inclination angle θ3 of the bucket 8 . The processing device 51 of the facility 50 shown in FIG. , appropriately referred to as the cutting edge position) P4.

The storage unit 51M of the processing device 51 stores data of the working machine 2 (hereinafter referred to as working machine data as appropriate). The work machine data includes the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8. As shown in FIG. 4 , the length L1 of boom 6 corresponds to the length from boom pin 13 to arm pin 14 . The length L2 of arm 7 corresponds to the length from arm pin 14 to bucket pin 15 . The length L3 of the bucket 8 corresponds to the length from the bucket pin 15 to the cutting edge 8T of the bucket 8. As shown in FIG. The cutting edge 8T is the tip of the cutting edge 8B shown in FIG. The work machine data also includes information on the position up to the boom pin 13 with respect to the origin position PL of the local coordinate system. Using the lengths L1, L2, L3, the inclination angles θ1, θ2, θ3, and the origin position PL, the processing device 51 can obtain the cutting edge position P4 with respect to the origin position PL. In the embodiment, the processing device 51 of the facility 50 obtains the cutting edge position P4, but the sensor controller 26 of the hydraulic excavator 1 may obtain the cutting edge position P4 and transmit it to the processing device 51 of the facility 50. FIG.

<Example of Control Executed by Image Display System 100 and Remote Control System 101>
FIG. 7 is a flow chart of an example of control executed by the image display system 100 and the remote control system 101. As shown in FIG. FIG. 8 is a diagram showing the imaging device 19, the distance detection device 20, and the work target WA.

At step S101, the sensor controller 26 shown in FIG. The information of the hydraulic excavator 1 is information obtained from the imaging device 19 , the distance detection device 20 , the global position calculation device 23 , the attitude detection device 32 and the IMU 33 . As shown in FIG. 8, the imaging device 19 images the work target WA within the imaging range TA to obtain an image of the work target WA. The distance detection device 20 detects the distance Ld from the distance detection device 20 to the work target WA and other objects present within the detection range MA. The global position calculator 23 obtains reference position information Pga1 and Pga2 corresponding to the positions P1 and P2 of the GNSS antennas 21 and 22 in the global coordinate system. The posture detection device 32 detects the boom cylinder length, the arm cylinder length, and the bucket cylinder length. The IMU 33 detects the attitude of the hydraulic excavator 1 , more specifically, the roll angle θ4, the pitch angle θ5 and the yaw angle of the upper swing body 3 .

In step S<b>102 , the image display system 100 and the processing device 51 of the remote control system 101 receive the hydraulic pressure from the sensor controller 26 of the hydraulic excavator 1 via the communication device 25 of the hydraulic excavator 1 and the communication device 54 connected to the processing device 51 . Get the information of Excavator 1.

The information of the hydraulic excavator 1 acquired by the processing device 51 from the sensor controller 26 includes the image of the work target WA captured by the imaging device 19 and the distance from the distance detection device 20 to the work target WA detected by the distance detection device 20. , information on the posture of the work implement 2 included in the hydraulic excavator 1 detected by the posture detection device 32 , reference position information Pga1 and Pga2, and information on the posture of the hydraulic excavator 1 .

The information on the distance from the distance detection device 20 to the work target WA includes the distance Ld to the work target WA or the object OB existing within the detection range MA and information on the direction of the position Pd corresponding to the distance Ld. FIG. 8 shows the distance Ld as the distance to the work target WA. The azimuth information of the position Pd is the azimuth of the position Pd with respect to the distance detection device 20 and the angles with respect to the axes Xd, Yd, and Zd of the coordinate system of the distance detection device 20 . The information on the posture of the work implement 2 acquired by the processing device 51 is the inclination angles θ1, θ2, and θ3 of the work implement 2 obtained by the sensor controller 26 using the boom cylinder length, the arm cylinder length, and the bucket cylinder length. The posture information of the hydraulic excavator 1 is the hydraulic excavator 1, more specifically, the roll angle θ4, the pitch angle θ5, and the yaw angle of the upper revolving body 3. FIG.

The processing device 51 acquires the inclination angles θ1, θ2, and θ3 of the working machine 2 obtained from the sensor controller 26, the length L1 of the boom 6, the length L2 of the arm 7, and the length of the bucket 8 stored in the storage unit 51M. The cutting edge position P4 of the bucket 8 is obtained using the height L3. A cutting edge position P4 of the bucket 8 is a set of coordinates in the local coordinate system (x, y, z) of the excavator 1 .

Proceeding to step S103, the processing device 51 converts the distance Ld to the work target WA into position information using the distance information to the work target WA. The positional information is the coordinates of the position Pd in the coordinate system (Xd, Yd, Zd) of the distance detection device 20 . In step S103, all distances Ld detected by distance detection device 20 within detection range MA are converted into position information. The processing device 51 converts the distance Ld into positional information using the distance Ld and the azimuth information of the position Pd corresponding to the distance Ld. In step S103, the distance to the object OB existing within the detection range MA is also converted into position information in the same manner as the distance Ld of the work target WA. Information on the position of the work target WA within the detection range MA is obtained by the processing of step S103. Information on the shape of the work target WA can be obtained from information on the position information of the work target WA.

The position information and shape information of the work target WA are a set of coordinates of the position Pd in the coordinate system (Xd, Yd, Zd) of the distance detection device 20 . After converting the shape information of the work target WA into values of the coordinate system (Xc, Yc, Zc) of the imaging device 19, the processing device 51 converts the values of the local coordinate system (x, y, z) of the hydraulic excavator 1 Convert to

In step S104, the processing device 51 converts the position information of the work target WA, the cutting edge position P4 of the bucket 8, and the reference position information Pga1 and Pga2 acquired from the sensor controller 26 of the hydraulic excavator 1 into the global coordinate system (X, Y, Z). For conversion to the global coordinate system (X, Y, Z), the processing device 51 generates a rotation matrix using the roll angle θ4, pitch angle θ5 and yaw angle of the hydraulic excavator 1 acquired from the sensor controller 26 . Using the generated rotation matrix, the processing device 51 converts the position information of the work target WA, the blade edge position P4 of the bucket 8, and the reference position information Pga1 and Pga2 into the global coordinate system (X, Y, Z). Next, proceeding to step S105, the processing device 51 obtains an occupied area.

FIG. 9 is a diagram for explaining the occupied area SA. The occupied area SA is an area occupied by the work implement 2 within the information on the shape of the work target WA. In the example shown in FIG. 9, a portion of the bucket 8 of the working machine 2 is within the detection range MA of the distance detection device 20 and between the distance detection device 20 and the work target WA. Therefore, in the occupied area SA, the distance to the bucket 8 is detected by the distance detection device 20 instead of the distance to the work target WA. In the embodiment, the processing device 51 removes the occupied area SA from the shape information of the work target WA obtained in step S103.

The processing device 51 causes the storage unit 51M, for example, to store information on at least one of the position and orientation detected by the distance detection device 20 according to at least one of the position and orientation of the bucket 8 . Such information is included in the attitude of the work implement 2 of the hydraulic excavator 1 in this embodiment. The posture of the work implement 2 is determined using the inclination angles θ1, θ2, and θ3 of the work implement 2, the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8. It can be obtained using posture. Then, the processing device 51 compares the data detected by the distance detection device 20 and the information stored in the storage unit 51M, and if the two match, it can be assumed that the bucket 8 has been detected. . With such processing using the posture of work implement 2, processing device 51 does not use information on buckets 8 in occupied area SA when generating grid image 65 shown in FIG. can be generated accurately.

In order to remove the portion of the occupied area SA, the processing using the attitude of the work implement 2 may be performed by the following method. Information on at least one of the position and attitude of the bucket 8 in the global coordinate system, which is included in the attitude of the work implement 2, includes the tilt angles θ1, θ2, and θ3 of the work implement 2, the length L1 of the boom 6, the length of the arm 7, and the It is obtained from L2 and the length L3 of the bucket 8. In steps S103 and S104, information on the shape of the work target WA in the global coordinate system is obtained. In step S106, the processing device 51 removes the area obtained by projecting the position of the bucket 8 onto the shape information of the work target WA as the occupied area SA from the shape of the work target WA.

FIG. 10 is a diagram showing information on the shape of the work target WA from which the occupied area has been removed. The shape information IMWA of the work target WA is a set of coordinates Pgd (X, Y, Z) in the global coordinate system (X, Y, Z). The occupied area IMBA does not have coordinate information due to the processing in step S106. Next, proceeding to step S<b>107 , the processing device 51 generates an image showing the position of the bucket 8 . The image showing the position of the bucket 8 is the image of the portion corresponding to the bucket 8 on the work target WA.

11 to 13 are diagrams for explaining images showing the position of the bucket 8 on the work target WA. In the embodiment, the image indicating the position of the bucket 8 is an image indicating the position of the cutting edge 8T of the bucket 8 on the work target WA. Hereinafter, an image showing the position of the cutting edge 8T of the bucket 8 is appropriately referred to as a cutting edge position image. As shown in FIG. 11, the cutting edge position image is a position Pgt (X, Y, Z ) is an image defined by The vertical direction is the Z direction in the global coordinate system (X, Y, Z) and is a direction orthogonal to the X and Y directions.

As shown in FIG. 12, between the first position Pgt1 (X1, Y1, Z1) and the second position Pgt2 (X2, Y2, Z2) of the surface WAP of the work target WA, the surface WAP of the work target WA is A line image formed along the edge is the cutting edge position image 61 . A first position Pgt1 (X1, Y1, Z1) is defined by a straight line LV1 extending in the vertical direction from a position Pgb1 outside the blade 8B on one end 8Wt1 side in the width direction Wb of the bucket 8 and the surface of the work target WA. It is the intersection with WAP. A second position Pgt2 (X2, Y2, Z2) is defined by a straight line LV2 extending in the vertical direction from a position Pgb2 outside the blade 8B on the other end 8Wt2 side in the width direction Wb of the bucket 8 and the surface of the work target WA. It is the intersection with WAP. A width direction Wb of the bucket 8 is a direction in which the plurality of blades 8B are arranged.

The processing device 51 obtains straight lines LV1 and LV2 extending vertically from the positions Pgb1 and Pgb2 of the bucket 8 . Next, the processing device 51 calculates the first position Pgt1 (X1, Y1, Z1) and the second position Pgt2 (X2, Y2, Z2) from the obtained straight lines LV1 and LV2 and the shape information of the work target WA. ). Then, the processing device 51 defines a blade edge position image 61 as a set of positions Pgt of the surface WAP of the work target WA when a straight line connecting the first position Pgt1 and the second position Pgt2 is projected onto the surface WAP of the work target WA.

In the embodiment, the processing device 51 generates a first straight line image 62 which is an image of a straight line LV1 connecting the position Pgb1 and the first position Pgt1 (X1, Y1, Z1), a position Pgb2 and a second position Pgt2 (X2, Y2). , Z2), and a second straight line image 63, which is an image of a straight line LV2 connecting . Next, the processing device 51 converts the cutting edge position image 61, the first straight line image 62 and the second straight line image 63 into images with the imaging device 19 as a reference, that is, images at the viewpoint of the imaging device 19. FIG.

As shown in FIG. 13, the image of the viewpoint of the imaging device 19 is obtained from the origin Pgc (Xc, Yc, Zc) of the imaging device in the global coordinate system (X, Y, Z), the cutting edge position image 61, the first straight line image 62 and the second linear image 63 are viewed. The origin Pgc (Xc, Yc, Zc) of the imaging device is coordinates obtained by converting the center of the light receiving surface 19P of the imaging device 19RC included in the imaging device 19, that is, the origin PC, into the global coordinate system (X, Y, Z). .

The cutting edge position image 61, the first straight line image 62, and the second straight line image 63 are images in a three-dimensional space, but the image at the viewpoint of the imaging device 19 is a two-dimensional image. Therefore, the processing device 51 converts the cutting edge position image 61, the first straight line image 62 and the second straight line image 63 defined in the three-dimensional space, that is, the global coordinate system (X, Y, Z), onto a two-dimensional plane. Performs a perspective projection transformation that projects. The cutting edge position image 61, the first straight line image 62, and the second straight line image 63, which have been converted into images from the viewpoint of the imaging device 19, are hereinafter appropriately referred to as a work implement guide image 60. FIG.

14 and 15 are diagrams showing a lattice image 65, which is a reference image. After the work implement guide image 60 is generated, the process advances to step S108, and the processing device 51 generates a lattice image 65 as a reference image. The lattice image 65 is a line image along the surface WAP of the work target WA using the positional information of the work target WA. The grid image 65 is a grid including a plurality of first line images 66 and a plurality of second line images 67 intersecting the plurality of first line images 66 . In the embodiment, the first line image 66 is, for example, a line image extending parallel to the X direction in the global coordinate system (X, Y, Z) and arranged in the Y direction. The first line image 66 is a line image extending parallel to the longitudinal direction of the upper revolving body 3 of the hydraulic excavator 1 and arranged in the width direction of the upper revolving body 3 in the global coordinate system (X, Y, Z). may be

The lattice image 65 is generated using the position information of the work target WA, more specifically, the position Pgg (X, Y, Z) of the surface WAP. The intersection of the first line image 66 and the second line image 67 is the position Pgg(X, Y, Z). The first line image 66 and the second line image 67 are defined in a global coordinate system (X, Y, Z), as shown in FIG. 15, and therefore contain three-dimensional information. In the embodiment, the plurality of first line images 66 are arranged at regular intervals, and the plurality of second line images 67 are arranged at regular intervals. The interval between adjacent first line images 66 and the interval between adjacent second line images 67 are equal.

The lattice image 65 is an image obtained by converting the first line image 66 and the second line image 67 generated using the position Pgg (X, Y, Z) of the surface WAP into the image of the viewpoint of the imaging device 19. is. After generating the first line image 66 and the second line image 67 , the processing device 51 converts them into images of the viewpoint of the imaging device 19 to generate the grid image 65 . By converting the first line image 66 and the second line image 67 into the image of the viewpoint of the imaging device 19, in order to assist the absolute distance of the work target WA, the grid image 65 with equal intervals on the horizontal plane is It can be deformed and displayed according to the shape of the work target WA.

Next, in step S109, the processing device 51 removes the above-described occupation area SA from the generated work tool guide image 60 and the lattice image 65, which is the reference image. In step S109, the processing device 51 converts the occupied area SA into an image of the viewpoint of the imaging device 19, and removes it from the work implement guide image 60 and the grid image 65, which is the reference image. In the embodiment, the processing device 51 converts the cutting edge position image 61, the first straight line image 62 and the second straight line image 63 before being converted into the image of the viewpoint of the imaging device 19, and the image of the viewpoint of the imaging device 19. The occupied area SA before being converted into the image of the viewpoint of the imaging device 19 may be removed from the first line image 66 and the second line image 67 before being captured.

FIG. 16 is a diagram showing an image 69 for work. In step S110, the processing device 51 synthesizes the work tool guide image 60 from which the occupied area SA has been removed, the lattice image 65, and the image 68 of the work target WA imaged by the image pickup device 19 to obtain a working image. Image 69 is generated. In step S<b>111 , the processing device 51 displays the generated work image 68 on the display device 52 . The work image 69 is an image in which the grid image 65 and the work implement guide image 60 are displayed on the image 68 of the work target WA.

Since the grid image 65 is a grid along the surface WAP of the work target WA, the operator of the hydraulic excavator 1 can grasp the position of the work target WA by referring to the grid image 65 . For example, the operator can grasp the depth, that is, the longitudinal position of the upper revolving body 3 of the hydraulic excavator 1 from the second line image 67 , and can grasp the width direction position of the bucket 8 from the first line image 66 . .

In the work tool guide image 60, the cutting edge position image 61 is displayed along the surface WAP of the work target WA and the lattice image 65. As shown in FIG. Therefore, the operator can grasp the positional relationship between the bucket 8 and the work target WA from the grid image 65 and the cutting edge position image 61, thereby improving work efficiency and work accuracy. In the embodiment, the first linear image 62 and the second linear image 63 connect both ends of the cutting edge position image 61 from both sides in the width direction Wb of the bucket 8 . The operator can more easily grasp the positional relationship between the bucket 8 and the work target WA from the first linear image 62 and the second linear image 63 . Since the grid image 65 and the cutting edge position image 61 are displayed along the shape of the work target terrain (work target WA), the relative positional relationship between the two on the terrain surface (two-dimensional) can be easily grasped. . Furthermore, since the first line image 66 and the second line image 67 that make up the grid image 65 are arranged at equal intervals in the global coordinate system, it is easy to grasp the sense of distance on the topographical surface, and the perspective Easier to grasp feelings.

In the embodiment, the work image 69 includes information 64 indicating the distance between the cutting edge 8T of the bucket 8 and the work target WA. By doing so, there is an advantage that the operator can grasp the actual distance between the cutting edge 8T of the bucket 8 and the work target WA. The distance between the cutting edge 8T of the bucket 8 and the work target WA can be the distance from the cutting edge 8T at the center of the bucket 8 in the width direction Wb to the surface WAP of the work target WA.

Note that the information 64 includes, in place of or in addition to the distance between the cutting edge 8T of the bucket 8 and the work target WA, information about the posture such as the angle of the bucket 8, and information indicating the relative distance between the bucket 8 and the work target WA. , information indicating the relationship between, for example, the orientation of the cutting edge 8T of the bucket 8 and the orientation of the surface of the work target WA, information indicating the position of the bucket 8 in coordinates, information indicating the orientation of the surface of the work target WA, and from the imaging device 19 Spatial position information about the work tool or work object W including information indicating the distance in the x direction in the local coordinate system to the cutting edge 8T of the bucket 8 may be used.

That is, the processing device 51 can determine the position of the bucket 8 which is a working tool, the attitude of the bucket 8, the position of the work object WA, the relative attitude of the work object WA, the relative distance between the bucket 8 and the work object WA, the bucket At least one of the relative postures between 8 and the work target WA may be obtained and displayed on the display device 52 .

As described above, the image display system 100 and the remote control system 101 superimpose the work implement guide image 60 and the lattice image 65 generated from the viewpoint of the imaging device 19 on the image 68 of the actual work target WA captured by the imaging device 19. Together, they are displayed on the display device 52 . Through such processing, the image display system 100 and the remote control system 101 display the position of the bucket 8 and the work target WA to the operator who remotely operates the hydraulic excavator 1 using the image of the work target WA displayed on the display device 52 . Since it is possible to make it easier to grasp the positional relationship between and, it is possible to improve work efficiency and work accuracy. Even an inexperienced operator can easily grasp the positional relationship between the position of the bucket 8 and the work target WA by using the image display system 100 and the remote control system 101 . As a result, deterioration in work efficiency and work accuracy is suppressed. In addition, the image display system 100 and the remote control system 101 superimpose the work tool guide image 60, the lattice image 65, and the image 68 of the actual work target WA on the display device 52, thereby enabling the operator to The work efficiency can be improved by making the screen that the operator pays attention to a single screen.

In the grid image 65, the interval between adjacent first line images 66 and the interval between adjacent second line images 67 are equal. Therefore, by superimposing and displaying the grid image 65 and the image 68 of the actual work target WA captured by the imaging device 19, it becomes easier to grasp the work point on the work target WA. Also, by superimposing the cutting edge position image 61 of the work implement guide image 60 and the grid image 65, the operator can easily grasp the distance that the bucket 8 has moved, thereby improving work efficiency.

Since the occupied area SA, which is the area of the work implement 2 , is removed from the work implement guide image 60 and the grid image 65 , the work implement guide image 60 and the grid image 65 are distorted by the occupied area SA and the work implement is placed on the work implement 2 . It is possible to prevent the guide image 60 and the grid image 65 from being superimposed and displayed. As a result, the image display system 100 and the remote control system 101 can display the working image 69 on the display device 52 in a form that is easy for the operator to see.

In the embodiment, the work implement guide image 60 may include at least the cutting edge position image 61 . The grid image 65 may include at least a plurality of second line images 67 , that is, a plurality of line images indicating a direction perpendicular to the front-rear direction of the upper revolving body 3 of the hydraulic excavator 1 . Further, the processing device 51 may change the color of, for example, the cutting edge position image 61 in the work implement guide image 60 according to the distance between the cutting edge 8T of the bucket 8 and the work target WA. By doing so, the operator can easily grasp the distance between the position of the bucket 8 and the work target WA.

In the embodiment, the processing device 51 converts the information on the shape of the work target WA into the global coordinate system (X, Y, Z) to generate the work tool guide image 60 and the grid image 65, but the shape of the work target WA information to the global coordinate system (X, Y, Z). In this case, the processing device 51 handles the shape information of the work target WA in the local coordinate system (x, y, z) of the hydraulic excavator 1 and generates the work implement guide image 60 and the grid image 65 . When handling the shape information of the work target WA in the local coordinate system (x, y, z) of the hydraulic excavator 1, the GNSS antennas 21 and 22 and the global position calculation device 23 are unnecessary.

In the above-described embodiment, a portion of the hydraulic excavator 1 (for example, the bucket 8 as described above) detected by the distance detection device 20 is removed to obtain shape information (three-dimensional terrain data) of the work target WA. However, the three-dimensional terrain data acquired in the past (for example, several seconds ago) is stored in the storage unit 51M of the processing device 51, and the processing unit 51P of the processing device 51 stores the current work target WA and the stored data. It is also possible to determine whether the grid image 65 is the same position as the three-dimensional landform data, and if it is the same position, display the grid image 65 using the past three-dimensional landform data. In other words, the processing device 51 can display the grid image 65 if there is past three-dimensional landform data, even if there is a landform hidden by part of the hydraulic excavator 1 when viewed from the imaging device 19 .

Further, instead of displaying the lattice image 65 using a lattice, for example, the lattice image 65 may be displayed using the local coordinate system as a polar coordinate system. Specifically, concentric circles at equal intervals corresponding to the distance from the center of the hydraulic excavator 1 (for example, the center of rotation of the upper revolving body 3) are drawn as a line image (second line image), and the upper revolving body 3 is drawn. Radial line images (first line images) may be drawn at equal intervals from the turning center according to the turning angle. In this case, the second line image, which is a line image of concentric circles, and the first line image, which is a line image radially from the turning center, intersect. By displaying such a lattice image, it is possible to easily grasp the positional relationship between the position of the bucket 8 and the work target WA during turning or excavation.

<Blade edge position image 61 of loading shovel type working machine 2a>
FIG. 17 is a diagram for explaining the cutting edge position image 61 when using the loading shovel type work machine 2a. In the loading shovel, since the bucket 8 rotates from the rear to the front of the hydraulic excavator 1, earth and sand can be scooped up. The loading shovel-type work machine 2 a excavates a work target WA in front of the upper revolving body 3 with the cutting edge 8 T of the bucket 8 facing forward of the upper revolving body 3 . In this case, as shown in FIG. 17, the cutting edge position image 61 is the surface WAP of the work target WA when the cutting edge 8T is projected onto the work target WA in a horizontal direction, that is, in a direction orthogonal to the direction in which gravity acts. is an image defined by the position Pgt (X, Y, Z) of . The horizontal direction is the X direction or the Y direction in the global coordinate system (X, Y, Z), and is the direction orthogonal to Z. Using the information of the position Pgt (X, Y, Z) of the surface WAP of the work target WA, the processing device 51 generates the cutting edge position image 61, the first straight line image 62 and the second line image 62 using the same method as described above. A linear image 63 is generated. The processing device 51 converts the generated cutting edge position image 61 , the first straight line image 62 and the second straight line image 63 into images from the viewpoint of the imaging device 19 to obtain the work implement guide image 60 .

<Regarding the processing for obtaining the cutting edge position image 61>
FIG. 18 is a diagram showing a first modified example of processing for obtaining a cutting edge position image. In the first modification, the processing device 51 obtains a straight line 72 that is perpendicular to the line of intersection 71 between the virtual plane 70 and the work target WA and that passes through the cutting edge 8T of the bucket 8 . The virtual plane 70 is the xz plane in the local coordinate system (x, y, z) of the hydraulic excavator 1 shown in FIGS. The xz plane passes through the center of the bucket 8 in the width direction Wb.

Next, the processing device 51 draws a straight line LV1 passing through the outer position Pgb1 of the blade 8B on one end 8Wt1 side in the width direction Wb of the bucket 8 and parallel to the straight line 72, and the other end in the width direction Wb. A straight line LV2 passing through the outer position Pgb2 of the blade 8B on the 8Wt2 side and parallel to the straight line 72 is obtained. The intersection of the straight line LV1 and the surface WAP of the work target WA is the first position Pgt1, and the intersection of the straight line LV2 and the surface WAP of the work target WA is the second position Pgt2. The processing device 51 obtains a first position Pgt1 and a second position Pgt2, and calculates a set of positions Pgt of the surface WAP when a straight line connecting the first position Pgt1 and the second position Pgt2 is projected onto the surface WAP of the work target WA. A cutting edge position image 61 is assumed.

The first straight line image 62 and the second straight line image 63 are images of straight lines LV1 and LV2. The processing device 51 converts the generated cutting edge position image 61 , the first straight line image 62 and the second straight line image 63 into images from the viewpoint of the imaging device 19 to obtain the work implement guide image 60 . Since the bucket 8 moves parallel to the virtual plane 70, the cutting edge position image 61 obtained by the processing of the first modification shows the position where the cutting edge 8T of the bucket 8 faces the work target WA.

19 and 20 are diagrams for explaining a second modification of the processing for obtaining the cutting edge position image. When the upper revolving body 3 of the hydraulic excavator 1 is tilted with respect to the horizontal plane, that is, the XY plane of the global coordinate system (X, Y, Z), as shown in FIG. The surface of WA may be tilted with respect to WAP. Since the bucket 8 moves in parallel with the virtual plane 70 described above, when the cutting edge position image 61 is obtained by projecting the cutting edge 8T onto the surface WAP of the work target WA that exists in the vertical direction of the cutting edge 8T, the moving direction of the bucket 8 is and the cutting edge position image 61 .

In the second modified example, the processing device 51 obtains a straight line LV1 and a straight line LV2 extending vertically from the cutting edge position P4 of the bucket 8 . Next, the processing device 51 rotates the obtained straight lines LV1 and LV2 by the angle at which the upper revolving body 3 of the hydraulic excavator 1 is inclined with respect to the horizontal plane, that is, by the roll angle θ4. The direction in which the straight lines LV1 and LV2 are rotated is the direction in which they are parallel to the virtual plane 70 . In this case, the processing device 51 rotates the straight lines LV1 and LV2 about the positions Pgb1 and Pgb2 of the bucket 8 by θ4 on the plane PV12 formed by the straight lines LV1 and LV2. Thus, the processing device 51 obtains the straight lines LV1a and LV2a after rotation.

Next, the processing device 51 obtains the points of intersection between the straight lines LV1a and LV2a after the rotation and the surface WAP of the work target WA, and sets the obtained two points of intersection as a first position Pgt1a and a second position Pgt2a, respectively. . Then, the processing device 51 defines a blade edge position image 61 as a set of positions Pgt of the surface WAP of the work target WA when a straight line connecting the first position Pgt1a and the second position Pgt2a is projected onto the surface WAP of the work target WA. The first straight line image 62 and the second straight line image 63 are images of the straight line LV1a and the straight line LV2a. The processing device 51 converts the generated cutting edge position image 61 , the first straight line image 62 and the second straight line image 63 into images from the viewpoint of the imaging device 19 to obtain the work implement guide image 60 . The cutting edge position image 61 obtained by the process of the second modification shows the position where the cutting edge 8T of the bucket 8 faces the work target WA.

<Modified Example of Control System of Hydraulic Excavator 1>
FIG. 21 is a diagram showing a control system 1Sa of the hydraulic excavator 1 according to the modification. The image display system 100 and the remote control system 101 described above remotely controlled the hydraulic excavator 1 using the control device 53 of the facility 50 shown in FIG. In this modified example, a display device 52 is provided in the operator's cab 4 shown in FIG. .

Therefore, in the control system 1Sa, the processing device 51 and the operation device 53a are connected to the signal line 35 of the control system 1S. A display device 52 is connected to the processing device 51 . The processing device 51 provided in the control system 1Sa has the same function as the processing device 51 provided in the facility 50 shown in FIG. 1 in the image display system 100 and the remote control system 101 described above. The display device 52 of the control system 1Sa may be a dedicated display device for displaying the work image 69, or may be a display device included in the hydraulic excavator 1. FIG. The operating device 53a is a device for operating the hydraulic excavator 1, and includes a left operating lever 53La and a right operating lever 53Ra. The operating device 53a may be of a pilot hydraulic type or an electric type.

The hydraulic excavator 1 equipped with the control system 1Sa displays the work tool guide image 60 and the grid image 65 generated from the viewpoint of the imaging device 19 together with the image 68 of the actual work target WA captured by the imaging device 19 in the cab 4. is displayed on the display device 52 inside. Through such processing, the excavator 1 allows the operator who operates the excavator 1 to grasp the positional relationship between the position of the bucket 8 and the work target WA using the image of the work target WA displayed on the display device 52. can be made easier. As a result, work efficiency and work accuracy can be improved. Also, even an inexperienced operator can easily grasp the positional relationship between the position of the bucket 8 and the work target WA by using the hydraulic excavator 1 having the control system 1Sa. As a result, deterioration in work efficiency and work accuracy is suppressed. Furthermore, in the case of nighttime work or the like, even if it is difficult for the operator to see the actual work target WA, the operator can work while looking at the work tool guide image 60 and the lattice image 65 displayed on the display device 52. , a decrease in work efficiency is suppressed.

Although the embodiment has been described above, the embodiment is not limited by the contents described above. In addition, the components described above include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. Furthermore, the components described above can be combined as appropriate. Furthermore, at least one of various omissions, replacements, and modifications of components can be made without departing from the gist of the embodiments. The working machine is not limited to the hydraulic excavator 1, but may be other working machines such as wheel loaders or bulldozers.

1 Hydraulic Excavator 1B Vehicle Body 1S, 1Sa Control System 2, 2a Work Machine 3 Upper Revolving Body 4 Driver's Seat 6 Boom 7 Arm 8 Bucket 8B Blade 8T Cutting Edge 16 First Stroke Sensor 17 Second Stroke Sensor 18 Third Stroke Sensor 19 Imaging Device 20 Distance detection device 21, 22 Antenna (GNSS antenna)
23 global position calculation device 26 sensor controller 27 work machine control device 32 attitude detection device 33 IMU
50 facility 51 processing device 52 display device 53, 53a operating device 60 work implement guide image (image)
61 cutting edge position image 62 first straight line image 63 second straight line image 65 grid image 66 first line image 67 second line image 68 image 69 image for work 100 image display system for work machine (image display system)
101 Remote control system for working machine (remote control system)

Claims (21)

An image of a work target of a work machine having a revolving body that revolves around a revolving center, and a work machine that has a work tool that moves parallel to the virtual plane with the front-rear direction of the revolving body as a virtual plane, and the work tool. an imaging device that acquires an image of
a measuring device that acquires the shape of the work target;
a first image that is parallel to the virtual plane and connects one end in the width direction of the work tool and the surface of the work target, based on the information on the position of the work tool and the information on the work target; and a second image that is parallel to the virtual plane and connects the other end in the width direction of the work tool and the surface of the work object, and the first image and the second image, and the imaging a processing device that synthesizes the image of the work target and the image of the work tool acquired by the device and displays the result on a display device;
Image display system for working machines. Parallelism between the virtual plane and the first image and parallelism between the virtual plane and the second image are maintained even when the work machine is tilted;
The image display system for a working machine according to claim 1. The processing device is
Extending in the width direction of the work implement between a first intersection point where the first image and the surface of the work object intersect and a second intersection point where the second image and the surface of the work object intersect A line image corresponding to a part of the work tool and along the surface of the work object is generated, the line image and the image of the work object acquired by the imaging device are synthesized , and displayed on the display device to display
The image display system for a work machine according to claim 1 or 2 . an imaging device that acquires an image of a work target of a work machine that includes a work machine having a work tool and an image of the work tool;
a measuring device that acquires the shape of the work target;
A line image corresponding to a portion of the work tool extending in the width direction of the work tool and along the surface of the work object is generated based on the information on the position of the work tool and the information on the work object. and a processing device that synthesizes the line image with the image of the work target and the image of the work tool acquired by the imaging device and displays the result on a display device;
Image display system for working machines. The work tool rotates around a pin,
The processing device is
generating the line image by projecting a portion of the work tool onto the work target in a direction perpendicular to the rotation axis of the work tool;
The image display system for a working machine according to claim 4. The processing device is
changing the color of the line image according to the distance between the work tool and the work target;
The image display system for a work machine according to claim 4 or 5. The processing device generates a line image from the end of the work tool to the surface of the work object, and causes the display device to display the line image.
The image display system for a work machine according to any one of claims 1 to 6. an imaging device that acquires an image of a work target of a work machine that includes a work machine having a work tool and an image of the work tool;
a measuring device that acquires the shape of the work target;
Based on the information on the position of the work tool and the information on the work object, the work tool passes through one end in the width direction of the work tool and intersects with the surface of the work object below the end. and a second image passing through the other end of the width direction of the working tool and extending to intersect with the surface of the work object below the other end. a processing device for synthesizing the first image and the second image with the image of the work target and the image of the work tool acquired by the imaging device and displaying the result on a display device; comprising a
Image display system for working machines. The processing device generates a line image extending from the end of the work tool in the vertical direction and connected to the surface of the work object, and causes the display device to display the line image.
The image display system for a working machine according to claim 8. The processing device is
Extending in the width direction of the work implement between a first intersection point where the first image and the surface of the work object intersect and a second intersection point where the second image and the surface of the work object intersect A line image corresponding to a part of the work tool and along the surface of the work object is generated, the line image and the image of the work object acquired by the imaging device are synthesized, and displayed on the display device to display
The image display system for a work machine according to claim 9. The processing device is
changing the color of the line image according to the distance between the work tool and the work target;
The image display system for a work machine according to claim 10. an imaging device for acquiring an image of a work target and an image of the work tool of a work machine equipped with a work tool having a work tool that rotates around a pin;
a measuring device that acquires the shape of the work target;
Based on the information on the position of the work tool and the information on the work object, the position of one end of the work tool in the width direction parallel to the direction perpendicular to the rotation axis of the work tool and the work object is determined. A first image that connects the surface and a second image that is parallel to the direction orthogonal to the rotation axis of the work tool and connects the other end in the width direction of the work tool and the surface of the work target is generated. and a processing device for synthesizing the first image and the second image with the image of the work target and the image of the work tool acquired by the imaging device and displaying the result on a display device;
Image display system for working machines. The processing device is
Extending in the width direction of the work implement between a first intersection point where the first image and the surface of the work object intersect and a second intersection point where the second image and the surface of the work object intersect A line image corresponding to a part of the work tool and along the surface of the work object is generated, the line image and the image of the work object acquired by the imaging device are synthesized, and displayed on the display device to display
The image display system for a working machine according to claim 12. the work tool is a bucket,
A portion of the work implement is a cutting edge of the bucket,
The image display system for a work machine according to any one of claims 1 to 13 . The processing device is
generating a line image along the surface of the work target using information on the shape of the work target, synthesizing it with the image of the work target, and displaying the line image on the display device;
The image display system for a work machine according to any one of claims 1 to 14 . The line image is
A grid comprising a plurality of first line images and a plurality of second line images intersecting the plurality of first line images,
The image display system for a work machine according to claim 15 . The processing device is
Obtaining spatial position information about the work implement or the work object and causing the display device to display the information;
The image display system for a work machine according to any one of claims 1 to 16 . an image display system for a work machine according to any one of claims 1 to 17 ;
an operation device that operates the work machine provided in the work machine,
A remote control system for work machines. Equipped with the image display system for a work machine according to any one of claims 1 to 17 ,
working machine. An image of a work target of a work machine having a revolving body that revolves around a revolving center, and a work machine that has a work tool that moves parallel to the virtual plane with the front-rear direction of the revolving body as a virtual plane, and the work tool. obtaining an image of
obtaining a shape of the work target;
a first image that is parallel to the virtual plane and that connects one end of the work tool in the width direction with the surface of the work target, based on the information on the position of the work tool and the information on the work target; generating a second image parallel to the virtual plane and connecting the other widthwise end of the work tool and the surface of the work object, and capturing the first image and the second image by an imaging device; synthesizing the image of the work target and the image of the work tool thus obtained, and displaying the image on a display device;
Image display method of working machine. Acquiring an image of a work target of a work machine having a work machine having a work tool and an image of the work tool;
obtaining a shape of the work target;
Based on the information on the position of the work tool and the information on the work object, the work tool passes through one end in the width direction of the work tool and intersects with the surface of the work object below the end. and a second image passing through the other end of the width direction of the working tool and extending to intersect with the surface of the work object below the other end. and synthesizing the first image and the second image with the image of the work target and the image of the work tool, and displaying the result on a display device;
Image display method of working machine.

Images