JP7581995B2 - Imaging function control system and imaging function control method - Google Patents

Description

The present invention relates to a technology for controlling the imaging function of an actual machine imaging device mounted on a work machine such as a hydraulic excavator.

Technology has been proposed in which an operator uses a remote control device to operate a work machine while referring to a remote image showing the working status of the work machine (see, for example, Patent Document 1). When an operator remotely operates a work machine, the operator feels a different way about the situation around the work machine compared to when the operator is on board the work machine (actual machine) and operates the actual machine. For this reason, the system is configured to change the direction of the actual machine imaging device mounted on the work machine so that the operator can check the situation around the work machine by changing the direction of their line of sight in various ways, just as when operating the actual machine.

JP 2018-021395 A

However, if the degree of freedom of the swing function of the actual machine imaging device is excessively high or low, the change in the captured image acquired through the actual machine imaging device may cause problems in grasping the surrounding situation if the setting that increased the display magnification of the work machine's camera is forgotten to be released or the display magnification of the camera does not fully return to the minimum (initial) display magnification, resulting in an insufficient understanding of the surrounding situation necessary for carrying out work.

The present invention aims to provide a system that can improve the convenience of the operator in understanding the situation around the work machine by controlling the imaging function of the actual machine imaging device mounted on the work machine in an appropriate form in consideration of the state of the work machine.

The imaging function control system of the present invention comprises:
An imaging function control system including: a remote control device including a remote operation mechanism that is operated to operate a work mechanism of a work machine; and an actual machine imaging device that is mounted on the work machine and obtains an image including at least a part of the work mechanism and outputs the image to a remote image output device provided in the remote control device ,
a remote control device provided in the remote control device and recognizing an operation mode of a swing operation for adjusting an optical axis direction of the actual imaging device;
An imaging function control element that exerts a swing function of changing the optical axis direction of the real imaging device to an angle according to the operation mode of the swing operation ,
The imaging function control element , in a first state in which the remote control device is operated to operate the working mechanism and an operation is being performed, restricts the swing function more than in a second state in which the remote control device is not operated to operate the working mechanism and an operation is not being performed .

According to the imaging function control system having this configuration, in the first state, the swing function of the real imaging device is restricted compared to the second state. The "swing function" means at least one of the pan function and the tilt function. That is, in the first state, when the real imaging device has both the pan function and the tilt function, at least one of the two functions is restricted, and when the real imaging device has only one of the pan function and the tilt function, the one function is restricted. As a result, the amount of change in the real space area reflected in the remote image output device is suppressed, and thus, a situation in which the operator's attention is distracted in the first state and the operation of the work machine through the remote operation mechanism is hindered can be avoided.

In the imaging function control system having the above configuration,
It is preferable that, in the first state , the imaging function control element restores the attitude of the real imaging device to a predetermined initial attitude , which is the attitude of the real imaging device at the start of the swing operation , by exerting the swing function when the swing operation is stopped, while, in the second state , the imaging function control element maintains the attitude of the real imaging device as it is when the swing operation is stopped.

According to the imaging function control system having this configuration, when the swing operation is stopped in the first state, the attitude of the real machine imaging device is returned to the initial attitude, which is the attitude of the real machine imaging device at the start of the swing operation . As a result, after the operator is made to understand the state of the real space area corresponding to the attitude of the real machine imaging device when the swing operation is stopped, the operator can be made to understand the state of the real space area corresponding to the initial attitude of the real machine imaging device without the need for the operator to perform a swing operation. On the other hand, when the swing operation is stopped in the second state, the attitude of the real machine imaging device is maintained as it is. As a result, the real machine imaging device is made to perform the swing function according to any swing operation mode during the period when the operation of the work machine is stopped, and the operator can be made to understand the state of the real space area reflected in the captured image acquired through the real machine imaging device.

In the imaging function control system having the above configuration,
It is preferable that, in the first state , when the swing operation is stopped, the imaging function control element intermittently outputs the captured image obtained through the real imaging device to the remote image output device until the attitude of the real imaging device returns to a predetermined initial attitude which is the attitude of the real imaging device at the start of the swing operation , or continuously outputs the captured image at the time when the swing operation is stopped to the remote image output device .

According to the imaging function control system having this configuration , when the swing operation is stopped in the first state, the captured image when the attitude of the real imaging device is returned to the initial attitude is intermittently output through the remote image output device (or the captured image when the operation is stopped is continuously output). As a result, the discomfort felt by the operator due to the captured image changing even though the swing operation is stopped can be reduced.

FIG. 1 is an explanatory diagram regarding the configuration of a remote operation support system. FIG. 2 is an explanatory diagram relating to the configuration of a remote control device. FIG. 2 is an explanatory diagram relating to the configuration of a work machine. FIG. 2 is an explanatory diagram regarding functions of a remote operation support system (imaging function control system). FIG. FIG. 4 is an explanatory diagram regarding a control mode of a swing function of the actual imaging device. FIG. 11 is an explanatory diagram relating to functions of a remote operation support system (imaging function control system) according to another embodiment.

(Configuration of remote operation support system)
The remote operation support system shown in Fig. 1 is composed of a remote operation support server 10 and a remote operation device 20 for remotely operating a work machine 40. The remote operation support server 10, the remote operation device 20, and the work machine 40 are configured to be able to communicate with each other via a network. The mutual communication network of the remote operation support server 10 and the remote operation device 20 and the mutual communication network of the remote operation support server 10 and the work machine 40 may be the same as or different from each other.

(Configuration of remote operation support server)
The remote operation support server 10 includes a database 102, a first support processing element 121, and a second support processing element 122. The database 102 stores and holds captured image data and the like. The database 102 may be configured as a database server separate from the remote operation support server 10. Each support processing element is configured as a calculation processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes a calculation process (described later) according to the software on the data. In this embodiment, the remote operation support server 10 constitutes an "imaging function control system", and the second support processing element 122 constitutes an "imaging function control element".

(Configuration of remote control device)
The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data according to the software.

The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes a remote image output device 221, an audio output device 222, and a remote wireless communication device 224.

The remote control mechanism 211 includes a travel operation device, a slewing operation device, a boom operation device, an arm operation device, and a bucket operation device. Each operation device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the travel operation device is operated to move the lower travel body 410 of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. The operation lever (slewing lever) of the slewing operation device is operated to move the hydraulic slewing motor that constitutes the slewing mechanism 430 of the work machine 40. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder 442 of the work machine 40. The operation lever (arm lever) of the arm operation device is operated to move the arm cylinder 444 of the work machine 40. The operation lever (bucket lever) of the bucket operation device is operated to move the bucket cylinder 446 of the work machine 40.

The operating levers constituting the remote control mechanism 211 are arranged around the seat St on which the operator sits, as shown in FIG. 2, for example. The seat St is in the form of a high-back chair with armrests, but it may be in the form of a low-back chair without a headrest, or in the form of a chair without a backrest, or any other form of seating on which the operator can sit.

A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operating lever may serve as multiple operating levers. For example, the left operating lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 may function as an arm lever when operated in the forward/backward direction, and as a rotation lever when operated in the left/right direction. Similarly, the right operating lever 2112 provided in front of the right frame of the seat St shown in FIG. 2 may function as a boom lever when operated in the forward/backward direction, and as a bucket lever when operated in the left/right direction. The lever pattern may be changed as desired by the operator's operating instructions.

At the positions of the left operation lever 2111 and the right operation lever 2112 that are gripped by the operator, a pan switch SW1 and a tilt switch SW2 are provided, respectively, for adjusting the pan angle and tilt angle of the real imaging device 412, as described below. Only one of the left operation lever 2111 and the right operation lever 2112 may be provided with both the pan switch SW1 and the tilt switch SW2. Each of the pan switch SW1 and the tilt switch SW2 is configured so that its operating position is returned to the initial operating position by a biasing element such as an actuator or a spring under certain conditions. If the real imaging device 412 has only one of the pan function and the tilt function as a swing function, one of the pan switch SW1 and the tilt switch SW2 may be omitted.

2, the remote image output device 221 is composed of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each having a substantially rectangular screen arranged in front of the seat St, diagonally forward to the left, and diagonally forward to the right. The shapes and sizes of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different.

As shown in FIG. 2, the right edge of the left remote image output device 2211 is adjacent to the left edge of the central remote image output device 2210 such that the screens of the central remote image output device 2210 and the left remote image output device 2211 form an inclination angle θ1 (e.g., 120°≦θ1≦150°). As shown in FIG. 2, the left edge of the right remote image output device 2212 is adjacent to the right edge of the central remote image output device 2210 such that the screens of the central remote image output device 2210 and the right remote image output device 2212 form an inclination angle θ2 (e.g., 120°≦θ2≦150°). The inclination angles θ1 and θ2 may be the same or different.

The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or inclined to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of multiple image output devices divided into multiple parts. For example, the central remote image output device 2210 may be composed of a pair of image output devices adjacent to each other above and below, each having a roughly rectangular screen.

The sound output device 222 is composed of one or more speakers, and is composed of a central sound output device 2220, a left sound output device 2221, and a right sound output device 2222, which are arranged behind the seat St, at the rear of the left armrest, and at the rear of the right armrest, as shown in FIG. 2 for example. The specifications of the central sound output device 2220, the left sound output device 2221, and the right sound output device 2222 may be the same or different.

(Configuration of the work machine)
The work machine 40 is equipped with an actual machine control device 400, an actual machine input interface 41, an actual machine output interface 42, and a work mechanism 440. The actual machine control device 400 is equipped with an image processing device 30. The image processing device 30 is equipped with a state detection element 31, an image prediction element 32, and an image compression element 34. Each of the components of the actual machine control device 400 and the image processing device 30 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.

The work machine 40 is, for example, a crawler excavator (construction machine), and as shown in FIG. 3, includes a crawler-type lower track body 410 and an upper rotating body 420 that is rotatably mounted on the lower track body 410 via a rotating mechanism 430. A cab 424 (operator's compartment) is provided on the front left side of the upper rotating body 420. A work mechanism 440 is provided in the front center of the upper rotating body 420.

The actual machine input interface 41 includes an actual machine operation mechanism 411, an actual machine imaging device 412, and a positioning device 414. The actual machine operation mechanism 411 includes a plurality of operation levers arranged around a seat arranged inside the cab 424 in the same manner as the remote control mechanism 211. A drive mechanism or robot that receives a signal according to the operation mode of the remote control lever and moves the actual machine operation lever based on the received signal is provided in the cab 424. The actual machine imaging device 412 is installed, for example, inside the cab 424, and images the environment including at least a part of the working mechanism 440 through the front window and a pair of left and right side windows. Some or all of the front window and side windows may be omitted. The positioning device 414 is composed of a GPS and, if necessary, a gyro sensor or the like.

The actual device output interface 42 is equipped with an actual device wireless communication device 422.

As shown in FIG. 3, the working mechanism 440 includes a boom 441 that is movably attached to the upper rotating body 420, an arm 443 that is rotatably connected to the tip of the boom 441, and a bucket 445 that is rotatably connected to the tip of the arm 443. The working mechanism 440 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 that are configured as extendable hydraulic cylinders.

The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 420 so as to extend and retract when supplied with hydraulic oil, thereby rotating the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract when supplied with hydraulic oil, thereby rotating the arm 443 around a horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to extend and retract when supplied with hydraulic oil, thereby rotating the bucket 445 around a horizontal axis relative to the arm 443.

(function)
The functions of the remote operation support system and the imaging function control system having the above configuration will be described with reference to the flowchart shown in Fig. 4. In the flowchart, a block "C" is used for the sake of simplicity, and means transmission and/or reception of data, and means a conditional branch in which processing in a branching direction is executed on the condition that the data is transmitted and/or received.

The remote operation device 20 transmits an environment confirmation request to the remote operation assistance server 10 via the remote wireless communication device 224 (FIG. 4/STEP 210). For example, a requirement for initiating transmission of the environment confirmation request may be that the operator has performed a designation operation via the remote input interface 210. A "designation operation" is, for example, an operation such as tapping on the remote input interface 210 for the operator to designate the work machine 40 that he or she intends to remotely operate.

When an environment confirmation request is received by the remote operation support server 10, the first support processing element 121 transmits the environment confirmation request to the corresponding work machine 40 (Figure 4/C10).

When an environment confirmation request is received in the work machine 40 via the actual machine wireless communication device 422 (Fig. 4/C40), the actual machine control device 400 acquires an image via the actual machine imaging device 412, and transmits image data representing the image to the remote control device 10 via the actual machine wireless communication device 422 (Fig. 4/STEP 410).

When the first support processing element 121 receives captured image data in the remote operation support server 10 (FIG. 4/C11), the second support processing element 122 transmits environmental image data corresponding to the captured image to the remote operation device 20 (FIG. 4/STEP 110). The environmental image data is not only the captured image data itself, but also image data representing a simulated environmental image generated based on the captured image.

When the remote operation device 20 receives environmental image data through the remote wireless communication device 224 (Fig. 4/C21), the remote control device 200 outputs an environmental image corresponding to the environmental image data to the remote image output device 221 (Fig. 4/STEP 212).

As a result, for example, as shown in FIG. 5, an environmental image is output to the remote image output device 221 through the window frame that defines the cab 424, showing the boom 441, arm 443, which are part of the work mechanism 440, and a pile of rubble or earth and sand (the object of work performed by the bucket 445) reflected in front of the cab 424.

In the remote operation device 20, the remote control device 200 recognizes the swing operation mode of the remote operation mechanism 211, and a swing command corresponding to the swing operation mode is transmitted to the remote operation support server 10 via the remote wireless communication device 224 (FIG. 4/STEP 214). The swing operation mode is specified by the operation postures of the pan switch SW1 and tilt switch SW2 provided at the gripping positions of the left operation lever 2111 and the right operation lever 2112, respectively (see FIG. 2). The switches SW1 and SW2 are, for example, slide switches or lever-type switches that can detect the operation posture (amount of operation), detect the amount of operation to one side and the other side of the switch, and are in a neutral position when no operation is performed.

A swing command is generated to control the pan angle _φp of the actual imaging device 412 to a pan angle _fp (θp) in the pan angle range [ _φp1 , _φp2 ] in accordance with an arbitrary operation attitude _θp in the operation attitude range [ _θp1 , _θp2 ] of the pan switch SW1. The function _fp is defined by, for example, relational expression (01).

f _p (θ _p )={(φ _p2 −φ _p1 )/(θ _p2 −θ _p1 )}(θ _p −θ _p1 )+φ _p1 ‥(01).

A swing command is generated to control the tilt angle _φt of the actual imaging device 412 to a tilt angle _ft (θt) in the tilt angle range [ _φt1 , _φt2 ] according to an arbitrary operation attitude θt in the operation attitude range [ _θt1 , _θt2 ] of the tilt switch _SW2 . The function _ft is defined by, for example, relational expression (02).

f _t (θ _t )={(φ _t2 −φ _t1 )/(θ _t2 −θ _t1 )}(θ _t −θ _t1 )+φ _t1 ‥ (02).

When a swing command is received by the remote operation support server 10, the second support processing element 122 (imaging function control element) transmits the swing command to the work machine 40 (Figure 4/C12).

When a normal swing command is received in the work machine 40 via the real machine wireless communication device 422 (FIG. 4/C42), the real machine control device 400 controls the swing function of the real machine imaging device 412 (more precisely, the pan drive mechanism and/or tilt drive mechanism possessed by the real machine imaging device 412) and adjusts the pan angle φ _p and/or tilt angle φ _t (FIG. 4/STEP 412).

This changes the real space area reflected in the captured image acquired through the real machine imaging device 412, and therefore in the environmental image output by the remote image output device 221 (see Fig. 4/STEP 410->...->STEP 212, and Fig. 5). For example, when the optical axis direction changes from in front of the work machine 40 to diagonally forward and left by controlling the pan angle _φp of the real machine imaging device 412, the real space area reflected in the environmental image changes from the area in front of the work machine 40 to an area diagonally forward and left of the work machine 40. Also, when the optical axis direction changes from the forward horizontal direction to diagonally downward and forward by controlling the tilt angle _φt of the real machine imaging device 412, the real space area reflected in the environmental image changes to a lower area than before.

In the remote operation device 20, the remote control device 200 determines whether the swing operation has been stopped (FIG. 4/STEP 216). For example, whether the swing operation has been stopped is determined based on whether the amount of change in the operating attitude of each of the pan switch SW1 and the tilt switch SW2 has remained below a threshold value (a state in which neither the pan switch SW1 nor the tilt switch SW2 is being operated) for a specified period of time.

If the determination result is positive (FIG. 4/STEP 216...YES), the remote control device 200 transmits a swing stop command to the remote operation support server 100 (FIG. 4/STEP 218). On the other hand, if the determination result is negative (FIG. 4/STEP 216...NO), the swing stop command is not generated and the subsequent process is executed.

When the remote operation support server 10 receives a swing stop command (FIG. 4/C14), the second support processing element 122 determines the state of the work machine 40 based on the communication result with the work machine 40 (FIG. 4/STEP 120). The "first state" is a state in which the work machine 40 can be operated (or a state in which the work machine 40 is working). The "second state" is a state in which the work machine 40 cannot be operated (or a state in which the work machine 40 has stopped working). For example, the first state and the second state of the work machine 40 may be determined depending on whether or not the hydraulic cutoff lever constituting the actual machine operation mechanism 411 of the work machine 40 is in a predetermined position. The first state and the second state of the work machine 40 may also be determined depending on whether or not a remote hydraulic cutoff lever simulating a hydraulic cutoff lever as a remote input interface is in a predetermined position.

When it is determined that the work machine 40 is in the first state (FIG. 4/STEP 120...1), the second support processing element 122 transmits an initial attitude return command to the work machine 40 (FIG. 4/STEP 124). On the other hand, when it is determined that the work machine 40 is in the second state (FIG. 4/STEP 120...2), the initial attitude return command is not generated and the subsequent processing is executed.

When the work machine 40 receives a command to return to the initial attitude via the actual machine wireless communication device 422 (FIG. 4/C44), the actual machine control device 400 controls the swing function of the actual machine imaging device 412, and adjusts the pan angle φ _p and/or tilt angle φ _t to the initial angle or initial attitude (FIG. 4/STEP 414).

For example, as shown in FIG. 6, consider a case in which a swing operation is started at time t= _ts , stopped at time t= _t0 , resumed at time t= _t1 , and stopped at time t= _t2 .

In this case, if the state of the work machine 40 is the first state, as shown by the dashed line in Fig. 6, the attitude φ = ( _φp , _φt ) of the real machine imaging device 412 is controlled to return to the initial attitude _φ0 at time t = _t0 , change from time t = _t1 in accordance with the operation mode of the pan switch SW1 and/or the tilt switch SW2, and then return to the initial attitude _φ0 again at time t = _t2 . Therefore, the operator adjusts the attitude φ of the real machine imaging device 412 from the initial attitude _φ0 by starting to operate the pan switch SW1 and/or the tilt switch SW2 at time t = _t1 . In response to the attitude φ of the real machine imaging device 412 returning to the initial attitude _φ0 , the second support processing element 122 may transmit an initial attitude return command to the remote operation device 20 so that the operation attitudes θ of the pan switch SW1 and the tilt switch SW2 are returned to the initial operation attitude _θ0 .

On the other hand, if the state of the work machine 40 is the second state, as shown by the two-dot chain line in Figure 6, the attitude φ = ( _φp , _φt ) of the real machine imaging device 412 is controlled to be maintained at the same attitude φ( _{t1 )} at time t = _t0 , and to be maintained at the same attitude φ( _t2 ) at time t = _t2 after changing from time t = _t1 in accordance with the operation mode of the pan switch SW1 and/or tilt switch SW2. Therefore, by resuming operation of the pan switch SW1 and/or tilt switch SW2 at time t = t1, the operator adjusts the attitude φ of the real machine imaging device 412 from the attitude φ( _t0 ) at the time of the previous stop of operation.

In the remote operation device 20, the remote control device 200 recognizes the operation mode of the remote operation mechanism 211, and a remote operation command corresponding to the operation mode is transmitted to the remote operation support server 10 via the remote wireless communication device 224 (FIG. 4/STEP 220).

When the remote operation command is received by the second support processing element 122 in the remote operation support server 10, the remote operation command is transmitted to the work machine 40 by the first support processing element 121 (Figure 4/C16).

When the actual machine control device 400 receives an operation command via the actual machine wireless communication device 422 (FIG. 4/C46) in the work machine 40, the operation of the work mechanism 440 and the like is controlled (FIG. 4/STEP 420). For example, the bucket 445 is used to scoop up soil in front of the work machine 40, the upper rotating body 410 is rotated, and then the soil is dropped from the bucket 445.

(Action and Effect)
According to the imaging function control system constituting the remote operation support system of this configuration, in the first state, the swing function of the actual imaging device 412 is restricted compared to the second state (Figure 4/STEP 120..1 → STEP 124 → C44 → STEP 414, see Figure 6).

In detail, when the work machine 40 is in the first state, after the operation of the pan switch SW1 and the tilt switch SW2 is stopped, the attitude φ of the real machine imaging device 412 when the operation is resumed is limited to the initial attitude φ _0. As a result, after the operator has grasped the state of the real space area corresponding to the attitude when the swing operation of the real machine imaging device was stopped, the operator can grasp the state of the real space area corresponding to the initial attitude of the real machine imaging device without the need for the operator to perform a swing operation via the remote input interface 21 (pan switch SW1 and/or tilt switch SW2).

On the other hand, when the work machine 40 is in the second state, after the operation of the pan switch SW1 and tilt switch SW2 is stopped, the attitude φ of the real machine imaging device 412 when the operation is resumed remains the attitude φ at the time when the operation was stopped immediately before. As a result, during the period when the operation of the work machine 40 is stopped, the real machine imaging device 412 can be made to perform a swing function in response to any swing operation mode performed by the operator through the remote input interface 21 (pan switch SW1 and/or tilt switch SW2), allowing the operator to grasp the state of the real space area reflected in the captured image acquired through the real machine imaging device 412.

Other Embodiments of the Invention
In the above embodiment, the imaging function control system is constituted by the remote operation support server 10, and the imaging function control element is constituted by the second support processing element 122. However, in another embodiment, the imaging function control system may be constituted by the work machine 40 and/or the remote operation device 20, and the imaging function control element may be constituted by the actual machine control device 400 and/or the remote control device 200.

In the above embodiment, the swing function is restricted in the first state compared to the second state in that the attitude φ of the real image capturing device 412 is restricted when the swing operation of the remote control device 20 is resumed. However, in other embodiments, the swing function may be restricted in that the adjustment range and/or the change speed of the attitude φ of the real image capturing device 412 is restricted.

In this case, a series of processes may be executed according to the flowchart shown in FIG. 7 instead of the flowchart shown in FIG. 4. In the flowchart, a block "C●" is used for the sake of simplicity, and means transmission and/or reception of data, and a conditional branch in which processing in a branching direction is executed on the condition that the data is transmitted and/or received. In FIG. 7, the same reference numerals are used for processes common to FIG. 4, and their explanations are omitted.

When the remote operation support server 10 receives a swing command (FIG. 7/C13), the second support processing element 122 determines the state of the work machine 40 based on the results of communication with the work machine 40 (FIG. 7/STEP 120). When it is determined that the work machine 40 is in the "first state" (FIG. 7/STEP 120 . . . 2), the second support processing element 122 transmits a first swing command to the work machine 40 (FIG. 7/STEP 121). On the other hand, when it is determined that the work machine 40 is in the "second state" (FIG. 7/STEP 120 . . . 2), the second support processing element 122 transmits a second swing command to the work machine 40 (FIG. 7/STEP 122).

When the work machine 40 receives a second swing command through the real machine wireless communication device 422 (FIG. 7/C42), the real machine control device 400 controls the swing function of the real machine imaging device 412 in a normal manner according to the operation manner of the pan switch SW1 and/or the tilt switch SW2, and adjusts the pan angle φ _p and/or the tilt angle φ _t (FIG. 7/STEP 412). On the other hand, when the work machine 40 receives a first swing command through the real machine wireless communication device 422 (FIG. 7/C41), the real machine control device 400 controls the swing function of the real machine imaging device 412 in a manner that is more restricted than the normal manner, although it corresponds to the operation manner of the pan switch SW1 and/or the tilt switch SW2 (FIG. 7/STEP 411). At this time, the swing function is restricted in a form in which the adjustment range and/or the change speed of the attitude φ of the real machine imaging device 412 is restricted, compared to when the work machine 40 is in the first state.

In the first state, when the swing operation is stopped (see times t= _t0 and t= _t2 in FIG. 6 ), until the attitude φ of the real machine imaging device 412 returns to a predetermined initial attitude _φ0 , the captured image or environmental image acquired through the real machine imaging device 412 may be intermittently output to the remote output interface 22, or the captured image at the time of stopping the swing operation may be continuously output to the remote output interface 22. As a result, the discomfort felt by the operator due to the captured image or environmental image changing even though the swing operation has been stopped may be reduced.

In the first state, when a swing operation is input through the remote input interface 21 (pan switch SW1 and/or tilt switch SW2), the position and/or attitude of the actual machine imaging device 412 may be controlled to a predetermined usage position and/or usage attitude by exerting the swing function. This allows the operator to easily perform a swing operation in parallel with work by the work machine 40.

10: Remote operation support server, 20: Remote operation device, 40: Work machine, 102: Database, 121: First support processing element, 122: Second support processing element (imaging function control element), 200: Remote control device, 210: Remote input interface, 211: Remote operation mechanism, 220: Remote output interface, 221: Remote image output device, 222: Remote sound output device, 400: Actual machine control device, 410: Actual machine input interface, 420: Actual machine output interface, 424: Cab (operating room), 440: Work mechanism, 445: Bucket (working part), SW1: Pan switch, SW2: Tilt switch.

Claims (4)

An imaging function control system including: a remote control device having a remote operation mechanism that is operated to operate a work mechanism of a work machine; and an actual machine imaging device that is mounted on the work machine and obtains an image including at least a part of the work mechanism and outputs the image to a remote image output device provided in the remote control device ,
a remote control device provided in the remote control device and recognizing an operation mode of a swing operation for adjusting an optical axis direction of the actual imaging device;
An imaging function control element that exerts a swing function of changing the optical axis direction of the real imaging device to an angle according to the operation mode of the swing operation ,
The imaging function control element , in a first state in which the remote control device is operated to operate the working mechanism, limits the swing function more than in a second state in which the remote control device is not operated to operate the working mechanism .
Imaging function control system. 2. The imaging function control system according to claim 1,
An imaging function control system in which, in the first state , the imaging function control element restores the attitude of the real imaging device to a predetermined initial attitude , which is the attitude of the real imaging device at the start of the swing operation, by exerting a swing function when the swing operation is stopped, while, in the second state , the imaging function control element maintains the attitude of the real imaging device as it is when the swing operation is stopped. 2. The imaging function control system according to claim 1,
An imaging function control system in which, in the first state , when the swing operation is stopped, the imaging function control element intermittently outputs the captured image obtained through the real imaging device to the remote image output device until the attitude of the real imaging device returns to a predetermined initial attitude which is the attitude of the real imaging device at the start of the swing operation , or continuously outputs the captured image at the time the swing operation is stopped to the remote image output device . A method for controlling an imaging function of an actual machine imaging device that obtains an image including at least a part of a work mechanism and outputs the image to a remote image output device provided in a remote operation device that is mounted on a work machine and has a remote operation mechanism that is operated to operate a work mechanism of the work machine, the method comprising:
The process includes a process for recognizing an operation mode of a swing operation for adjusting an optical axis direction of the real imaging device by a remote control device provided in the remote operation device, and an imaging control process for exerting a swing function for changing an optical axis direction of the real imaging device to an angle corresponding to the operation mode ,
The imaging function control method includes a step of restricting the swing function more in a first state in which the imaging control process is performing work to operate the working mechanism by operating the remote control mechanism than in a second state in which the imaging control process is not performing work to operate the working mechanism by operating the remote control mechanism .

Images