JP7091880B2 - Information processing equipment, mobiles, remote control systems, information processing methods and programs - Google Patents

Description

The present invention relates to an information processing device, a mobile body, a remote control system, an information processing method and a program.

A remote control system is known in which a telepresence robot (hereinafter referred to as a robot) located at a remote base is remotely controlled by using a display terminal located at another base via a communication network. This remote control system can confirm the information of the base where the robot is located from a remote place by displaying the image taken by the photographing device provided in the robot on the display terminal.

In addition, a method of activating communication using a robot is also known. Patent Document 1 discloses a technique for detecting placement information (position, body orientation, and line-of-sight direction) between a person and a robot, and controlling the movement of the robot based on the detected placement information.

However, in the conventional method, when communicating with a plurality of people from a remote place using a robot (for example, a moving body), the moving body cannot be moved to an appropriate position based on the positional relationship of the plurality of people. There was a problem that it hindered smooth communication using moving objects.

The information processing device according to claim 1 is an information processing device that adjusts the position of a moving body including a photographing device for photographing a subject and acquiring image data, and acquires the image data acquired by the photographing device. Based on the acquisition means for specifying the attention point specifying means for specifying the attention point to be focused on by the plurality of persons based on the positions and orientations of the plurality of persons included in the acquired image data, and the attention point specifying means for specifying the attention point. The moving body is provided with a target point specifying means for specifying a target point indicating a destination of the moving body, and the moving body is moved to the specified target point.

According to the present invention, smooth communication using a mobile body can be realized.

It is a figure which shows an example of the system configuration of the remote control system which concerns on embodiment. It is a figure which shows an example of the outline of the structure of the robot which concerns on embodiment. It is a figure which shows the outline of the structure of the robot which concerns on the modification 1 of embodiment. It is a figure which shows the outline of the structure of the robot which concerns on the modification 2 of embodiment. It is a figure which shows an example of the hardware composition of the robot which concerns on embodiment. It is a figure which shows an example of the hardware composition of the display terminal which concerns on embodiment. It is a figure which shows an example of the hardware configuration of the management server which concerns on embodiment. It is a figure which shows an example of the functional structure of the remote control system which concerns on 1st Embodiment. It is a figure which shows an example of the command table which concerns on 1st Embodiment. It is a figure which shows an example of the condition table which concerns on 1st Embodiment. It is a figure which shows an example of the user command table which concerns on 1st Embodiment. (A) is a conceptual diagram showing an example of an authentication management DB, and (b) is a conceptual diagram showing an example of a terminal management DB. (A) is a conceptual diagram showing an example of a destination list management DB, and (b) is a conceptual diagram showing an example of a session management DB. It is a sequence diagram which shows an example of the process of the preparatory stage which starts the transmission / reception of data between a robot and a display terminal. It is a figure which shows an example of the destination list screen displayed on the display terminal which concerns on 1st Embodiment. It is a sequence diagram which shows an example of the process from the selection of a destination candidate to the start of transmission / reception of data. It is a flowchart for demonstrating an example of the control process of a robot based on the request command from a display terminal in the remote control system which concerns on 1st Embodiment. (A) is a screen example of a display screen displayed on the robot according to the first embodiment, and (b) is a diagram showing a screen example of a display screen displayed on the display terminal according to the first embodiment. It is a flowchart for demonstrating an example of the target point calculation process in the robot which concerns on 1st Embodiment. (A) is processing data schematically showing an image of a person's head extracted by an image processing unit, and (b) is an image of a person's head calculated by an attention point calculation unit (a1 to a4). Position and size values. (A) is coordinate information indicating the relative position specified by the attention point calculation unit, and (b) is processing data schematically showing the relative position of the person specified by the attention point calculation unit. (A) is the processing data schematically showing the images of the eyes and nose of the person extracted by the image processing unit, and (b) is the orientation of the person specified by the attention point calculation unit. (A) is information on the relative position and orientation specified by the attention point calculation unit, and (b) is processing data schematically showing the relative position and orientation of the person specified by the attention point calculation unit. (A) is a diagram for explaining an example of a process of estimating the position and direction of a person located around the robot 10 when the beacon is installed in the base. (B) is information on the relative position and orientation of the person (h1 to h4) specified by the method shown in (a). This is a calculation result obtained by calculating the intersection of straight lines extending in the direction of the person specified by the attention point calculation unit from the relative position of the person (h1 to h4) specified by the attention point calculation unit. (A) is coordinate information indicating the positions of the intersections collected by the processing of the attention point calculation unit, and (b) is processing data schematically showing the positions of the intersections collected by the processing of the attention point calculation unit. be. (A) is information on the target point specified by the target point calculation unit, and (b) is processing data schematically showing the position of the target point specified by the target point calculation unit. It is a flowchart for demonstrating an example of the movement processing in the robot which concerns on 1st Embodiment. It is a flowchart for demonstrating an example of existence notification processing in the robot which concerns on 1st Embodiment. (A) (b) It is a figure for demonstrating an example of the movement path of the robot which concerns on 1st Embodiment. (A) (b) (c) It is a flowchart for demonstrating an example of the process until the target point calculation process is started in the robot which concerns on 1st Embodiment. It is a figure which shows an example of the functional structure of the remote control system which concerns on 2nd Embodiment. It is a sequence diagram for demonstrating an example of the control processing of a robot in the remote control system which concerns on 2nd Embodiment. It is a figure which shows an example of the system configuration of the remote control system which concerns on 3rd Embodiment. It is a figure which shows an example of the functional structure of the remote control system which concerns on 3rd Embodiment. It is a sequence diagram for demonstrating an example of the control processing of a robot in the remote control system which concerns on 3rd Embodiment.

Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate description will be omitted.

● Embodiment ●
● System Configuration FIG. 1 is a diagram showing an example of a system configuration of a remote control system according to an embodiment. In the remote control system shown in FIG. 1, the display terminal 50 is used to remotely control the robot 10 located at the remote base to manage and maintain the equipment in the base, or the position of a person located in the base. It is a system that can check the flow line and so on. Further, the remote control system 1a realizes a remote conference by bidirectionally transmitting and receiving an image (video) taken at the base on the robot 10 side and an image (video) taken at the base on the display terminal 50 side. It is a system that can be done.

The remote control system 1a is a robot 10 located at each of a plurality of bases (base A, base B, base C) (10A, 10B, 10C, hereinafter referred to as a robot 10 when it is not necessary to distinguish them). It is composed of a display terminal 50 and a management server 90. The robot 10, the display terminal 50, and the management server 90 are communicably connected via the communication network 9. The communication network 9 is constructed by, for example, a LAN (Local Area Network), a dedicated line, the Internet, or the like. The communication network 9 may have a place where wireless communication such as Wi-Fi (registered trademark) is performed as well as wired communication.

The robot 10 is a mobile body that is installed at each base (base A, base B, base C) and autonomously travels by remote control from the display terminal 50. The robot 10 moves in the base while taking an image of a subject around the robot 10 by the photographing device 21 described later, and transmits the image acquired by the photographing device 21 to the display terminal 50, thereby using the display terminal 50. Information (image) in the base is provided to the operator who operates the robot 10. The robot 10 is an example of a moving body.

The display terminal 50 is a terminal device such as a PC (Personal Computer) that remotely controls the robot 10 installed at each base (base A, base B, base C). The display terminal 50 displays an image transmitted from the robot 10. The operator can remotely control the robot 10 while looking at the image displayed on the display terminal 50.

The display terminal 50 may be any one provided with a display means for displaying an image transmitted from the robot 10, and is, for example, a wearable terminal such as a tablet terminal, a mobile phone, a smartphone, or a head-mounted display (HMD). , A communication terminal provided with a wide-angle screen (cylindrical, all-sky, hemisphere screen, etc.), PDA (Personal Digital Assistant), or the like.

The management server 90 is a server computer for managing communication between the robot 10 located at each base and the display terminal 50. The management server 90 is connected to the robot 10 and the display terminal 50 via the communication network 9. The management server 90 may be configured by a plurality of server computers, and any server computer may be provided with a function.

The base where the robot 10 is installed is, for example, an office, a warehouse, a factory, a school, or the like. The operator who operates the robot 10 using the display terminal 50 confirms the position and flow line of a person in the base by confirming the image in the base transmitted from the robot 10, and the device installed in the base. Can be managed and maintained. Further, the robot 10 and the display terminal 50 can also perform two-way communication (remote conference) by transmitting and receiving images taken by both of them.

Although the configuration in which one robot 10 is installed in each base is described in FIG. 1, a plurality of robots 10 may be installed in one base. Further, the display terminal 50 may be configured to communicate with each of the robots 10 arranged at a plurality of bases, or may be configured to communicate only with the robots 10 arranged at one base.

● Configuration of Robot 10 Here, a specific configuration of the robot 10 shown in FIG. 1 will be described with reference to FIGS. 2 to 4. FIG. 2 is a diagram showing an outline example of the configuration of the robot according to the embodiment.

The robot 10 shown in FIG. 2 includes a moving mechanism 17 for moving the robot 10, a housing 15 having a control device 30 for controlling the processing or operation of the robot, a display 306a for displaying an image transmitted from the display terminal 50, and a display. It includes a support member 13 that supports the 306a, a rotating shaft 12 that rotates the support member 13, a photographing device 21 that photographs a subject around the robot 10, a microphone 308a for collecting sounds around the robot, and a speaker 308b.

The moving mechanism 17 is a unit for moving the robot 10, and is composed of wheels, a traveling motor, a traveling encoder, a steering motor, a steering encoder, and the like. Since the movement control of the robot 10 is an existing technique, detailed description thereof will be omitted, but the robot 10 receives a running instruction from the operator (display terminal 50), and the moving mechanism 17 receives the received running. The robot 10 is moved based on the instruction.

Hereinafter, an example in which the moving mechanism 17 is composed of two wheels will be described, but the moving mechanism 17 may be a bipedal foot type or a single wheel. Further, the shape of the robot 10 is not limited to the vehicle type as shown in FIG. 2, and may be, for example, a humanoid walking on two legs, a form imitating a living thing, a form imitating a specific character, or the like.

The housing 15 is located in the body portion of the robot 10, and includes a power supply unit that supplies necessary power to the entire robot 10 and a control device 30 that controls the processing or operation of the robot 10.

The photographing device 21 is a camera arranged in front of the robot 10 for photographing a subject around the robot 10 and acquiring image data 200. The photographing device 21 may be an photographing device that photographs a part of the periphery of the robot 10 to acquire a plane image, or photographs the periphery of the robot 10 to acquire a spherical (360 °) panoramic image. It may be a special photographing device. Further, the image of the image data 200 may be a moving image or a still image, and may be both a moving image and a still image. Further, the image of the image data 200 may be an image including sound data together with the image.

The support member 13 is fixed to the housing 15 and supports the display 306a. The support member 13 changes the direction of the display 306a by being rotated by the rotation shaft 12. The support member 13 may be a pole or the like fixed to the housing 15, or may be a pedestal fixed to the housing 15. Further, the display 306a displays, for example, an image taken and acquired by the display terminal 50 of the other party having a remote conference.

In addition to the above configuration, the robot 10 may be configured to have an operating means that enables additional operations other than the movement of the robot 10. The operating means is, for example, a hand that grabs an object.

〇 Modification example of the configuration of the robot 10 〇 〇 Modification example 1
Here, a modification 1 of the configuration of the robot 10 will be described with reference to FIG. In the robot 10a shown in FIG. 3, the display 306a is arranged on the movable arm 11 by changing the support member 13. The movable arm 11 is a movable member that can be deformed to direct the display 306a in a specific direction. As a result, the robot 10a can change the direction of the display 306a by the rotating shaft 12 and can change the direction of the display 306a by deforming the movable arm 11.

〇 Modification example 2
Next, a modification 2 of the configuration of the robot 10 will be described with reference to FIG. The robot 10b shown in FIG. 4 has a configuration in which the moving mechanism 17 is removed from the configuration shown in FIG. The robot 10b shown in FIG. 4 is used by placing it on a desk or the like and fixing it. The robot 10b is an example of a transmission terminal.

● Hardware Configuration Next, the hardware configurations of the robot 10, the display terminal 50, and the management server 90 in the embodiment will be described with reference to FIGS. 5 to 7. The hardware configurations shown in FIGS. 5 to 7 may have the same configuration in each embodiment, and components may be added or deleted as needed.

〇Hardware configuration of the robot 10 〇First, the hardware configuration of the robot 10 will be described with reference to FIG. FIG. 5 is a diagram showing an example of the hardware configuration of the robot according to the embodiment. In the hardware configuration shown in FIG. 5, components may be added or deleted as needed. The robot 10 includes a control device 30 that controls the processing or operation of the robot 10. As described above, the control device 30 is provided inside the housing 15 of the robot 10. The control device 30 may be provided outside the housing 15 of the robot 10, or may be provided as a device separate from the robot 10. The control device 30 is an example of an information processing device.

The control device 30 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, an HDD (Hard Disk Drive) 304, a network interface (I / F) 305, and an input / output I / O. It includes an F306, a media I / F307, a sound input / output I / F308, an external I / F309, and a bus line 310.

The CPU 301 controls the entire robot 10. The CPU 301 is an arithmetic unit that realizes each function of the robot 10 by reading a program or data stored in a ROM 302 or an HD (Hard Disk) 304a or the like onto the RAM 303 and executing processing. The remote control system 1a realizes the information processing method according to the present invention by, for example, executing a part or all of the program according to the present invention on the CPU 301.

The RAM 303 is a volatile memory used as a work area or the like of the CPU 301. The ROM 302 is a non-volatile memory that can hold a program or data even when the power is turned off. The HDD 304 controls reading or writing of various data to the HD304a according to the control of the CPU 301. The HD304a stores various data such as programs.

The network I / F 305 is a communication interface that communicates (connects) with the display terminal 50 via the communication network 9. The network I / F305 is, for example, a communication interface such as a wired or wireless LAN (Local Area Network). The network I / F305 includes 3G (3rd Generation), LTE (Long Term Evolution), 4G (4rd Generation), 5G (5rd Generation), Zigbee (registered trademark), BLE (Bluetooth (registered trademark) Low Energy), A communication interface for millimeter-wave wireless communication may be provided.

The input / output I / F 306 is an interface for inputting / outputting characters, numerical values, various instructions, etc. to / from various external devices. The input / output I / F 306 controls the display of various information such as a cursor, a menu, a window, characters, or an image on a display 306a such as an LCD (Liquid Crystal Display). The display 306a may be a touch panel display provided with input means. Further, the input / output I / F 306 may be connected to an input means such as a mouse or a keyboard in addition to the display 306a.

The media I / F 307 controls reading or writing (storage) of data to a recording medium 307a such as a USB (Universal Serial Bus) memory, a memory card, an optical disk, or a flash memory. The sound input / output I / F 308 is a circuit that processes sound signal input / output between the microphone 308a and the speaker 308b under the control of the CPU 301. The external I / F 309 is an interface for connecting another device to the control device 30.

The bus line 310 is an address bus, a data bus, or the like for electrically connecting each of the above components, and transmits an address signal, a data signal, various control signals, and the like. The CPU 301, ROM 302, RAM 303, HDD 304, network I / F 305, input / output I / F 306, media I / F 307, sound input / output I / F 308, and external I / F 309 are connected to each other via a bus line 310.

Further, the control device 30 has a mobile motor 101, an actuator 102, an acceleration / direction sensor 103, a GPS receiving unit 104, a power supply unit 105, various sensors 106, a timer 107, and the above-mentioned photographing, via an external I / F 309. The device 21 is connected.

The mobile motor 101 rotationally drives the moving mechanism 17 to move the robot 10 along the ground based on a command from the CPU 301. The actuator 102 deforms the movable arm 11 based on a command from the CPU 301. The acceleration / orientation sensor 103 is a sensor such as an electronic magnetic compass, a gyro compass, and an acceleration sensor that detects geomagnetism. The GPS (Global Positioning System) receiving unit 104 receives GPS signals from GPS satellites. The power supply unit 105 is a unit that supplies necessary power to the entire robot 10. The various sensors 106 are sensor devices capable of detecting information around the robot 10. The various sensors 106 are, for example, a barometer, a thermometer, a photometer, a motion sensor, an illuminance meter, or the like. The timer 107 is a measuring device having a time measuring function. The timer 107 may be a soft timer by a computer.

〇 Hardware configuration of the display terminal 50 〇 Next, the hardware configuration of the display terminal 50 will be described with reference to FIG. FIG. 6 is a diagram showing an example of the hardware configuration of the display terminal according to the embodiment. The display terminal 50 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM (Electrically Erasable Programmable Read-Only Memory) 504, an image pickup element I / F505, a CMOS (Complementary Metal Oxide Semiconductor) sensor 505a, an acceleration / orientation sensor 506, a media I / F507, and the like. It is equipped with a GPS receiving unit 508.

The CPU 501 controls the operation of the entire display terminal 50. The CPU 501 is an arithmetic unit that realizes each function of the display terminal 50 by reading a program or data stored in the ROM 502 or the like onto the RAM 503 and executing processing. The remote control system 1a realizes the information processing method according to the present invention by, for example, executing a part or all of the program according to the present invention on the CPU 501.

The ROM 502 stores a program used for driving the CPU 501 such as an IPL (Initial Program Loader). The RAM 503 is used as a work area of the CPU 501. The EEPROM 504 reads or writes various data such as a display terminal program according to the control of the CPU 501.

The CMOS sensor 505a captures a subject (mainly a self-image) and obtains image data according to the control of the CPU 501. The image sensor I / F 505 is a circuit that controls the drive of the CMOS sensor 505a. The acceleration / orientation sensor 506 is a variety of sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor that detect the geomagnetism. The media I / F 507 controls reading or writing (storage) of data to the recording medium 507a such as a flash memory. The GPS receiving unit 508 receives a GPS signal from a GPS satellite.

Further, the display terminal 50 includes a long-distance communication circuit 510, an antenna 510a of the long-distance communication circuit 510, a sound input / output I / F511, a microphone 511a, a speaker 511b, a display 512, an external device connection I / F513, and a short-distance communication circuit 514. , The antenna 514a of the short-range communication circuit 514, the touch panel 515, the timer 516, and the line-of-sight detection device 517.

The telecommunications circuit 510 is a circuit that communicates with other devices via the communication network 9. The microphone 511a is a kind of built-in sound collecting means for inputting voice. The sound input / output I / F 511 is a circuit that processes sound signal input / output between the microphone 511a and the speaker 511b under the control of the CPU 501.

The display 512 is a kind of display means such as a liquid crystal display or an organic EL for displaying an image of a subject, various icons, and the like. The external device connection I / F 513 is an interface for connecting various external devices. The short-range communication circuit 514 is a communication circuit such as Wi-Fi, NFC, or Bluetooth. The touch panel 515 is a kind of input means for operating the display terminal 50 by the user pressing the display 512. The timer 516 is a measuring device having a time measuring function. The timer 516 may be a soft timer by a computer. The line-of-sight detection device 517 continuously detects the position of the user's line of sight as line-of-sight information. The line-of-sight detection device 517 includes, for example, an image processing device that analyzes an image captured by the CMOS sensor 505a. The line-of-sight detection device 517 detects the direction of the line of sight from the positional relationship between the inner corner of the eye and the iris, for example, with the reference point as the inner corner of the eye.

Further, the display terminal 50 includes a bus line 509. The bus line 509 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

〇 Hardware configuration of the management server 90 〇 Next, the hardware configuration of the management server 90 will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of the hardware configuration of the management server according to the embodiment. The management server 90 is constructed by a general computer. The management server 90 includes CPU901, ROM902, RAM903, HDD905, media I / F907, display 908, network I / F909, keyboard 911, mouse 912, CD-RW (Compact Disc-ReWritable) drive 914, timer 915, and bus line 910. It is equipped with. Since the management server 90 functions as a server, it does not have to be provided with an input device such as a keyboard 911 or a mouse 912, or an output device such as a display 908.

The CPU 901 controls the operation of the entire management server 90. The ROM 902 stores the program used to drive the CPU 901. The RAM 903 is used as a work area for the CPU 901. The HDD 905 controls reading or writing of various data to the HD 904 according to the control of the CPU 901. The HD904 stores various data such as programs. The media I / F 907 controls reading or writing (storage) of data to a recording medium 906 such as a flash memory.

The display 908 displays various information such as cursors, menus, windows, characters, or images. The network I / F909 is an interface for performing data communication using the communication network 9. The keyboard 911 is a kind of input means including a plurality of keys for inputting characters, numerical values, various instructions, and the like. The mouse 912 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The CD-RW drive 914 controls reading of various data to the CD-RW 913 as an example of a detachable recording medium. The timer 915 is a measuring device having a time measuring function. The timer 915 may be a soft timer by a computer.

Further, the management server 90 includes a bus line 910. The bus line 910 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 901 shown in FIG. 7.

● First embodiment ●
Subsequently, the configuration of the remote control system according to the first embodiment will be described with reference to FIGS. 8 to 30.

● Functional configuration First, the functional configuration of the remote control system according to the first embodiment will be described with reference to FIG. FIG. 8 is a diagram showing an example of the functional configuration of the remote control system according to the first embodiment.

〇 Functional configuration of the control device 30 〇 First, the functional configuration of the control device 30 that controls the processing or operation of the robot 10 will be described with reference to FIG. The functions realized by the control device 30 are the transmission / reception unit 31, the operation input reception unit 32, the display control unit 33, the judgment unit 34, the shooting instruction unit 35, the image acquisition unit 36, the movement control unit 37, the orientation adjustment unit 38, and the position. It includes a specific unit 39, an image processing unit 41, an attention point calculation unit 42, a target point calculation unit 43, a peripheral information detection unit 44, an existence notification unit 45, a storage / reading unit 46, and a storage unit 3000.

The transmission / reception unit 31 is a function for transmitting / receiving various data or information to / from other devices via the communication network 9. The transmission / reception unit 31 transmits, for example, the image data 200 acquired by the image acquisition unit 36 to the display terminal 50 via the communication network 9. Further, the transmission / reception unit 31 receives, for example, a request command (request information) transmitted from the display terminal 50 via the communication network 9. Further, the transmission / reception unit 31 receives, for example, image data 250, which is an image acquired by the display terminal 50, from the display terminal 50 via the communication network 9. Further, the transmission / reception unit 31 transmits the image data 200 acquired by the photographing device 21 at the target point calculated by the target point calculation unit 43 to the display terminal 50 via the communication network 9. The transmission / reception unit 31 is mainly realized by the processing of the CPU 301 shown in FIG. 5 and the network I / F 305. The transmission / reception unit 31 is an example of the first transmission means.

The operation input receiving unit 32 is a function of receiving an operation input to an input means such as a display 306a. The operation input receiving unit 32 is mainly realized by the processing of the CPU 301 shown in FIG. 5 and the input / output I / F 306. The display control unit 33 is a function of displaying various screens on the display 306a. The display control unit 33 is mainly realized by the processing of the CPU 301 shown in FIG. 5 and the input / output I / F 306.

The determination unit 34 is a function of determining a process or an operation executed by the robot 10 in response to a request command transmitted from the display terminal 50. The determination unit 34 is mainly realized by the processing of the CPU 301 shown in FIG.

The shooting instruction unit 35 is a function of instructing the shooting device 21 to perform shooting processing. The shooting instruction unit 35 transmits, for example, instruction information for instructing shooting by the shooting device 21 to the shooting device 21. The shooting instruction unit 35 is mainly realized by the processing of the CPU 301 shown in FIG. 5 and the external I / F 309.

The image acquisition unit 36 is a function of acquiring an image acquired by the photographing device 21. The image acquisition unit 36 acquires, for example, image data 200, which is an image acquired by photographing a subject by the photographing apparatus 21, from the photographing apparatus 21. The image acquisition unit 36 is mainly realized by the processing of the CPU 301 shown in FIG. 5 and the external I / F 309. The image acquisition unit 36 is an example of acquisition means.

The movement control unit 37 is a function of controlling the movement of the robot 10 by driving the movement mechanism 17. The movement control unit 37 moves the robot 10 by controlling the drive of the movement mechanism 17 in response to a request command transmitted from the display terminal 50, for example. Further, the movement control unit 37 controls the drive of the movement mechanism 17 so as to move the robot 10 to the target point calculated by the target point calculation unit 43, for example. The movement control unit 37 is mainly realized by the processing of the CPU 301 shown in FIG. 5 and the external I / F 309. The movement control unit 37 is an example of the movement control means.

The orientation adjusting unit 38 is a function of changing the orientation of the robot 10. The orientation adjusting unit 38 controls the driving of the moving mechanism 17 or the rotating shaft 12 according to the orientation of the robot 10 to which the robot 10 is moved, which is calculated by the target point calculation unit 43, so that the orientation of the robot 10 is oriented. To change. The orientation adjusting unit 38 is mainly realized by the processing of the CPU 301 shown in FIG. 5 and the external I / F 309.

The position specifying unit 39 is a function of specifying the current position of the robot 10. The position specifying unit 39 identifies the current position of the robot 10 based on the detection results such as the direction of each direction (azimuth, magnetic north) detected by the acceleration / direction sensor 103 or the GPS receiving unit 104, for example. The detection result such as the direction of each direction is the position information indicating the position and direction of the robot 10 at a predetermined time point. The position specifying unit 39 is mainly realized by the processing of the CPU 301 shown in FIG. 5 and the external I / F 309. The position specifying unit 39 is an example of the position detecting means.

The image processing unit 41 is a function of processing the image data 200 acquired by the image acquisition unit 36. The image processing unit 41 extracts, for example, an image of a person's head included in the image data 200 acquired by the image acquisition unit 36. Further, the image processing unit 41 extracts, for example, an image of the eyes and nose of a person acquired by the image acquisition unit 36. The image processing unit 41 is mainly realized by the processing of the CPU 301 shown in FIG.

The attention point calculation unit 42 is a function of specifying the attention points attracted by a plurality of people based on the positions and orientations of the plurality of persons included in the image data processed by the image processing unit 41. The point of interest is a position where a plurality of people are paying attention at the base where the robot 10 is installed, for example, an intersection in a direction in which a plurality of people around the robot 10 are facing. The point of interest is, for example, in a remote conference, the direction in which a plurality of participants at the base on the robot 10 side are facing, that is, the central position of the topic at a certain point in time. The attention point calculation unit 42 is mainly realized by the processing of the CPU 301 shown in FIG. The attention point calculation unit 42 is an example of the calculation means. Further, the attention point calculation unit 42 is an example of the estimation means. Further, the attention point calculation unit 42 is an example of the attention point specifying means.

The target point calculation unit 43 is a function of specifying a target point to be moved to the robot 10 based on the attention point calculated by the attention point calculation unit 42. The target point specified by the target point calculation unit 43 includes position information (coordinate information) indicating the target position G of the destination of the robot 10 and direction information indicating the direction of the robot 10 to which the robot 10 is moved. The target point calculation unit 43 is mainly realized by the processing of the CPU 301 shown in FIG. The target point calculation unit 43 is an example of the calculation means. Further, the target point calculation unit 43 is an example of the target point specifying means.

The peripheral information detection unit 44 is a function of detecting information around the robot 10 by using the image data 200 acquired by the photographing device 21 or the detection result detected by various sensors 106. The peripheral information detection unit 44 is mainly realized by the processing of the CPU 301 shown in FIG. 5 and the external I / F 309.

The existence notification unit 45 is a function of notifying the existence of the robot 10 based on predetermined conditions. The existence notification unit 45 notifies the existence of the robot 10 when, for example, the distance between the current position detected by the position specifying unit 39 and the target point calculated by the target point calculation unit 43 is equal to or greater than the threshold value or longer than the threshold value. Output notification information. The notification information is, for example, a sound signal, and the presence notification unit 45 outputs a sound signal from the speaker 308b. Further, the notification information is, for example, a predetermined gesture of the robot 10, and the presence notification unit 45 drives the moving mechanism 17, the movable arm 11, or the rotating shaft 12. The threshold value is stored in the storage unit 3000 in advance. The existence notification unit 45 is mainly realized by the processing of the CPU 301 shown in FIG. 5 and the external I / F 309. The existence notification unit 45 is an example of the existence notification means.

The storage / reading unit 46 is a function of storing various data in the storage unit 3000 or reading various data from the storage unit 3000. The storage / reading unit 46 is mainly realized by the processing of the CPU 301 shown in FIG. The storage unit 3000 is mainly realized by the ROM 302, HD304a and the recording medium 307a shown in FIG.

Further, the storage unit 3000 stores the image data 200 taken by the image acquisition unit 36 and the image data 250 transmitted from the display terminal 50. Further, the storage unit 3000 stores the command table 3001 and the condition table 3002. The image data 200 stored in the storage unit 3000 may be configured to be deleted when a predetermined time has elapsed since the image data 200 was acquired by the image acquisition unit 36, or the data transmitted to the display terminal 50. May be deleted. Similarly, the image data 250 stored in the storage unit 3000 may be deleted when a predetermined time has elapsed since it was transmitted from the display terminal 50.

〇 Command table 3001
Here, the details of the data stored in the storage unit 3000 will be described. FIG. 9 is a diagram showing an example of a command table according to the first embodiment. The command table 3001 shown in FIG. 9 is for specifying a process or an operation executed by the robot 10 based on a request command transmitted from the display terminal 50. The command table 3001 stores the processing contents corresponding to each request command in association with each request command. The determination unit 34 of the control device 30 uses the command table 3001 to specify the process corresponding to the request command transmitted from the display terminal 50.

For example, in the command table 3001, the process corresponding to the request command "FORWARD" is a process of advancing the robot 10 by a certain amount. The content of the process corresponding to each command is not limited to this, and can be appropriately modified or changed by an operator who operates the robot 10 using the display terminal 50, an administrator of the robot 10, or the like.

〇 Condition table 3002
FIG. 10 is a diagram showing an example of imaging parameters according to the first embodiment. The condition table 3002 shown in FIG. 10 shows the conditions for starting the position adjustment process for the point of interest. The condition table 3002 stores the condition information in association with each base indicated by the base name. The condition information includes the elapsed time from the previous position adjustment process, the distance to a person, and the adjustment cycle of the position adjustment process. The items included in the condition information are not limited to this, and the conditions of one item may be included for each base, or the conditions of a plurality of items may be included for each base.

By storing the condition information for each base, the condition table 3002 can make the frequency and conditions for performing the position adjustment process different for each base where the robot 10 is located. Although the condition table 3002 has described an example in which the condition information is associated with each base, the condition information may be associated and stored for each robot 10.

〇 Functional configuration of the display terminal 50 〇 Next, the functional configuration of the display terminal 50 will be described with reference to FIG. The functions realized by the display terminal 50 include a transmission / reception unit 51, an operation input reception unit 52, a display control unit 53, a judgment unit 54, a request command generation unit 55, a line-of-sight detection unit 56, a storage / reading unit 57, and a storage unit 5000. include. The display terminal 50 has a dedicated application program for remotely controlling the robot 10 installed. The display terminal 50 realizes each function by, for example, the CPU 501 executing the installed application program.

The transmission / reception unit 51 is a function for transmitting / receiving various data or information to / from other devices via the communication network 9. The transmission / reception unit 51 receives, for example, the image data 200 acquired by the photographing device 21 included in the robot 10 from the robot 10 via the communication network 9. Further, the transmission / reception unit 51 transmits, for example, image data 250, which is an image captured and acquired by the CMOS sensor 505a, to the robot 10 via the communication network 9. The transmission / reception unit 51 is mainly realized by the processing of the CPU 501 shown in FIG. 6 and the telecommunications circuit 510. The transmission / reception unit 51 is an example of a reception means.

The operation input receiving unit 52 is a function of receiving various selections or operation inputs to the display terminal 50. The operation input receiving unit 52 is mainly realized by the processing of the CPU 501 shown in FIG. 6 and the touch panel 515. The touch panel 515 may be shared with the display 512. Further, the operation input receiving unit 52 may be realized by an input means other than the touch panel.

The display control unit 53 is a function of displaying various images on the display 512 of the display terminal 50. The display control unit 53 displays, for example, the image data 200 received by the transmission / reception unit 51 on the display 512, which is an example of the display means. The display control unit 53 is mainly realized by the processing of the CPU 501 shown in FIG. 6 and the display 512. The display control unit 53 is an example of the display control means.

The determination unit 54 is a function of determining a specific process requested to the robot 10. The determination unit 54 determines, for example, a specific process requested to the robot 10 based on the operation input received by the operation input reception unit 52. The determination unit 54 is mainly realized by the processing of the CPU 301 shown in FIG.

The request command generation unit 55 is a function of generating a request command, which is an execution request for the robot 10 to execute a specific process. The request command generation unit 55 is mainly realized by the processing of the CPU 501 shown in FIG.

The line-of-sight detection unit 56 is a function of detecting the line of sight of an operator who operates the robot 10 by using the display terminal 50. The line-of-sight detection unit 56 is mainly realized by the processing of the CPU 501 shown in FIG. 6 and the line-of-sight detection device 517.

The storage / reading unit 57 is a function of storing various data in the storage unit 5000 or reading various data from the storage unit 5000. The storage / reading unit 57 is mainly realized by the processing of the CPU 501 shown in FIG. The storage unit 5000 is mainly realized by the ROM 502, the EEPROM 504 and the recording medium 507 shown in FIG.

Further, the storage unit 5000 stores the image data 200 received by the transmission / reception unit 51 and the image data 250 captured and acquired by the CMOS sensor 505a. Further, the storage unit 5000 stores the user command table 5001. The image data 200 stored in the storage unit 5000 may be deleted when a predetermined time has elapsed after being received by the transmission / reception unit 51, or may be received by the operation input reception unit 52. , May be configured to be deleted based on a deletion command from the user. Similarly, the image data 250 stored in the storage unit 5000 may be deleted when a predetermined time has elapsed since it was acquired by the CMOS sensor 505a, or may be received by the operation input reception unit 52. In addition, it may be configured to be deleted based on a deletion command from the user.

〇 User command table 5001
FIG. 11 is a diagram showing an example of a user command table according to the first embodiment. The user command table 5001 shown in FIG. 11 is used to specify the content of the process or operation requested from the robot 10 based on the operation input received by the operation input receiving unit 52. The user command table 5001 stores the contents of the corresponding command in association with the input command corresponding to the operation input received by the operation input receiving unit 52.

〇 Functional configuration of the management server 90 〇 Next, the functional configuration of the management server 90 will be described with reference to FIG. The functions realized by the management server 90 include a transmission / reception unit 91, an authentication unit 92, a judgment unit 93, a creation unit 94, a storage / reading unit 95, and a storage unit 9000.

The transmission / reception unit 91 is a function for transmitting / receiving various data or information to / from other devices via the communication network 9. The transmission / reception unit 91 is mainly realized by the processing of the CPU 901 shown in FIG. 7 and the network I / F 909.

The authentication unit 92 is a function for authenticating the login request source based on the login request received by the transmission / reception unit 91. For example, the authentication unit 92 uses the terminal ID and password included in the login request received by the transmission / reception unit 91 as a search key to search the authentication management DB 9001 of the storage unit 9000. Then, the authentication unit 92 performs terminal authentication by determining whether the same set of terminal IDs and passwords are managed in the authentication management DB 9001. The authentication unit 92 is mainly realized by the processing of the CPU 901 shown in FIG. 7.

The determination unit 93 is a function of determining whether or not the terminal ID of the display terminal 50 is managed in the session management table described later. The determination unit 93 is mainly realized by the processing of the CPU 901 shown in FIG. 7.

The creation unit 94 is a function for creating a session ID used for communication. The creating unit 94 is mainly realized by the processing of the CPU 901 shown in FIG. 7.

The storage / reading unit 95 is a function of storing various data in the storage unit 9000 or reading various data from the storage unit 9000. The storage / reading unit 95 is mainly realized by the processing of the CPU 901 shown in FIG. 7. The storage unit 9000 is mainly realized by the ROM 902, HD904 or recording medium 906 shown in FIG. 7. Further, the storage unit 9000 includes destination list frame data (icon shown in FIG. 15, "rA01", "robot 10A-1", and the like in the destination list screen 500 shown in FIG. 15, which will be described later. Not) is remembered.

〇Authentication management table In the storage unit 9000, an authentication management DB 9001 configured by an authentication management table as shown in FIG. 12A is constructed. In this authentication management table, related information associated with each password is managed for each terminal ID of all the display terminals 50 managed by the management server 90. For example, in the authentication management table shown in FIG. 12A, it is shown that the terminal ID of the display terminal 50A is "o01" and the password is "aaaa".

〇 Terminal management table Further, in the storage unit 9000, a terminal management DB 9002 composed of a terminal management table as shown in FIG. 12B is constructed. In this terminal management table, for each terminal ID of each terminal (robot 10 and display terminal 50), the terminal name of each terminal, the IP address of the terminal, the operation status information indicating the current operating status of each terminal, and the terminal are displayed. Related information associated with the base name indicating the base where the robot 10 is located in the case of the robot 10 is managed.

For example, in the terminal management table shown in FIG. 12B, the display terminal 50 having the terminal ID "o01" has the terminal name "display terminal 50A" and the IP address of the display terminal 50 is "1.2.1". ".3" indicates that the operating status is "online (call is possible)". Further, the robot 10 having the terminal ID "rA01" has the terminal name "robot 10A-1", the IP address of the robot 10 is "1.3.2.3", and the operating state is "online (call is possible)". ) ”Indicates that the base name is“ base A ”.

〇 Destination list management table Further, in the storage unit 9000, a destination list management DB 9003 configured by a destination list management table as shown in FIG. 13A is constructed. In this destination list management table, destination candidates registered as candidates for the destination robot 10 for each terminal ID of the display terminal 50 as a start terminal requesting the start of communication for remote control of the robot 10. Related information associated with the terminal ID of the robot 10 of the robot 10 is managed.

For example, in the destination list management table shown in FIG. 13A, a destination candidate that can request the start of communication from a start terminal (display terminal 50A) having a terminal ID of "o01a" has a terminal ID of "rA01". It is shown that the robot 10A-1 of the above, the robot 10A-2 having a terminal ID of "rA02", the robot 10C-1 having a terminal ID of "rC01", and the like. The terminal ID of the destination candidate robot 10 is updated by being added or deleted by a request from an arbitrary start terminal (display terminal 50) to the management server 90 for addition or deletion.

* Session management table Further, in the storage unit 9000, a session management DB 9004 composed of a session management table as shown in FIG. 13B is constructed. In this session management table, for each session ID for identifying the session used when communicating between the robot 10 and the display terminal 50, the robot 10 using the session specified by this session ID and the robot 10 Related information associated with the terminal ID of the display terminal 50 is managed. For example, in the session management table shown in FIG. 13 (b), the terminal using the session executed using the session ID "se1" is a display terminal 50A having a terminal ID of "o01" and a terminal ID of "rA01". The robot 10A-1 and the robot 10C-1 having a terminal ID of "rC01" are shown.

● Processing or operation of the first embodiment 〇 Session establishment processing 〇 Next, the operation or processing of the remote control system according to the first embodiment will be described with reference to FIGS. 14 to 30. In the following description, the process executed by the control device 30 included in the robot 10 will be described as the process executed by the robot 10.

First, the process of establishing a communication session between the robot 10 and the display terminal 50 will be described with reference to FIGS. 14 to 16. FIG. 14 is a sequence diagram showing an example of processing in the preparatory stage for starting transmission / reception of data between the robot and the display terminal. Here, the transmission / reception processing of each management information in the preparatory stage before starting the transmission / reception of data between the display terminal 50A as the start terminal and the robot 10A-1 as the destination terminal will be described.

First, in step S11, the transmission / reception unit 51 of the display terminal 50A transmits a login request to the management server 90 via the communication network 9. Specifically, when the user of the display terminal 50A turns on the power switch in the display terminal 50A, the power is turned on. Then, the transmission / reception unit 51 of the display terminal 50A transmits a login request from the transmission / reception unit 51 to the management server 90 via the communication network 9 when the power is turned on. As a result, the transmission / reception unit 91 of the management server 90 receives the login request transmitted from the display terminal 50A.

This login request includes a terminal ID for identifying the starting terminal as the display terminal 50A, and a password. These terminal IDs and passwords are data read from the storage unit 5000 by the storage / reading unit 57 and sent to the transmission / reception unit 51. The terminal ID and password are not limited to this, and the terminal ID and password input by the user by an input means such as a touch panel 515 may be transmitted. Further, a terminal ID or password read from a storage medium such as a SIM (Subscriber Identity Module Card) card or SD card connected to the display terminal 50A may be transmitted.

Further, when the login request is transmitted from the display terminal 50A to the management server 90, the management server 90 on the receiving side can acquire the IP address of the display terminal 50A on the transmitting side. Further, the start of the login request does not necessarily have to be triggered by turning on the power switch, and may be transmitted according to the input to the input means such as the touch panel 515 by the user.

Next, in step S12, the authentication unit 92 of the management server 90 uses the terminal ID and password included in the login request received via the transmission / reception unit 91 as search keys, and the authentication management table of the storage unit 9000 (FIG. 12). (See) is searched, and authentication is performed by determining whether the same terminal ID and the same password are managed in the authentication management DB 9001. Hereinafter, a case where the display terminal 50A is determined by the authentication unit 92 to be a terminal having a legitimate usage authority will be described.

In step S13, when the storage / reading unit 95 is determined to be a login request from a starting terminal having a legitimate usage authority because the same terminal ID and the same password are managed by the authentication unit 92. , Read the destination list frame data from the storage unit 9000.

In step S14, the transmission / reception unit 91 transmits the authentication result information showing the authentication result obtained by the authentication unit 92 to the display terminal 50 that has requested the login via the communication network 9. As a result, the transmission / reception unit 51 of the display terminal 50A receives the authentication result information. The authentication result information includes the destination list frame data read by step S13. Then, in step S15, the storage / reading unit 57 of the display terminal 50A stores the destination list frame data received in step S14 in the storage unit 5000.

Next, in step S16, when the transmission / reception unit 51 of the display terminal 50A receives the authentication result information indicating the authentication result determined to be a terminal having a legitimate usage authority, the transmission / reception unit 51 manages the display terminal 50A via the communication network 9. Request the contents of the destination list from the server 90. As a result, the transmission / reception unit 91 of the management server 90 receives the request for the contents of the destination list. This request includes the terminal ID of the display terminal 50A.

Next, in step S17, the storage / reading unit 95 of the management server 90 uses the terminal ID “o01” of the display terminal 50A received in step S16 as a search key, and the destination list management table (FIG. 13A). The reference is searched and the terminal IDs of all the corresponding destination candidates are read out. Further, in step S18, the storage / reading unit 95 searches the terminal management table (see FIG. 12B) using each terminal ID read in step S17 as a search key, and the terminal name and operation of the corresponding destination candidate. Read the status information and base name.

Next, in step S19, the transmission / reception unit 91 of the management server 90 transmits the destination list content information to the display terminal 50A via the communication network 9. As a result, the transmission / reception unit 51 of the display terminal 50A receives the destination list content information. The destination list content information includes the terminal ID of the destination candidate, the terminal name of the destination candidate, the operation status information, and the base name read out in steps S17 and S18.

Next, in step S20, the display control unit 53 of the display terminal 50A is created using the destination list frame data stored in the storage unit 5000 in step S15 and the destination list content information received in step S19. The destination list screen 500 is displayed on the display 512.

The destination list screen 500 as shown in FIG. 15 is displayed on the display 512. On the destination list screen 500, an icon showing the operating status of the destination candidate terminal (robot 10), the terminal ID of the destination candidate terminal, the destination name of the destination candidate, and the destination candidate terminal are located for each destination candidate. The name of the base to be used is shown. The "terminal name" sent from the management server 9 in step S19 is displayed as the "destination name" on the destination list screen 500 shown in FIG.

Subsequently, with reference to FIG. 16, the process from the selection of the destination candidate in the display terminal 50 to the start of transmission / reception of image data will be described. FIG. 16 is a sequence diagram showing a process from selection of a destination candidate to start of transmission / reception of image data.

First, in step S31, the operation input receiving unit 52 of the display terminal 50A accepts the selection of the destination candidate (here, the robot 10A-1) on the destination list screen shown in FIG. 15 from the user. Then, in step S32, the transmission / reception unit 51 of the display terminal 50A transmits a start request indicating that the transmission / reception of image data or the like is to be started to the management server 90. As a result, the transmission / reception unit 91 of the management server 90 receives the start request. Further, the start request includes the terminal ID of the display terminal 50A and the terminal ID of the destination candidate terminal.

Next, in step S33, the determination unit 93 of the management server 90 determines whether or not the terminal ID of the display terminal 50A received in step S32 is managed by the session management table (see FIG. 13B). do. Here, the case where the terminal ID of the destination terminal (robot 10A-1) is not managed will be described below.

In step S34, if the terminal ID of the destination candidate terminal is not managed, the creation unit 94 of the management server 90 newly creates a session ID. Then, in step S35, the storage / reading unit 95 displays the session ID created in step S34 and the terminal ID of the display terminal 50A received in step S32 in the session management table (see FIG. 13B). And a new record associated with the terminal ID of the destination candidate terminal is additionally stored. Here, as shown in FIG. 12B, by adding a new record, the session ID “se3” and the terminal IDs “o01” and “rA01” are managed in association with each other.

Next, in step S36, the transmission / reception unit 91 of the management server 90 transmits a session start instruction including the session ID created in step S34 to the display terminal 50A. As a result, the transmission / reception unit 51 of the display terminal 50A receives the session start instruction.

Next, in step S37, the storage / reading unit 95 of the management server 90 uses the terminal ID of the destination candidate terminal (robot 10A-1) sent in step S32 as a search key in the terminal management table (FIG. 12 (FIG. 12). b) Read the corresponding IP address by searching (see). Then, in step S38, the transmission / reception unit 91 of the management server 90 transmits a session start instruction including the session ID created in step S33 to the IP address read in step S37. As a result, the transmission / reception unit 31 of the destination terminal (robot 10A-1) receives the session start instruction.

As described above, the start terminal (display terminal 50A) and the destination terminal (robot 10A-1) establish a communication session with the management server 90, respectively (steps S39-1 and S39-2).

〇 Transmission and display of image data 〇 Next, using the session established with the management server 90, the data transmitted from the robot 10 to the display terminal 50, and the processing or operation of the robot 10 by the display terminal 50. The process to be controlled will be described. FIG. 17 is a flowchart for explaining an example of a robot control process based on a request command from a display terminal in the remote control system according to the first embodiment.

In step S51, the image acquisition unit 36 of the robot 10 acquires the image data 200 acquired by the photographing device 21. Here, the robot 10 starts shooting by the shooting device 21 with a shooting instruction from the shooting instruction unit 35 to the shooting device 21 as a trigger. The image acquisition unit 36 of the robot 10 acquires image data 200, which is an image acquired by the imaging device 21, from the imaging device 21.

In steps S52-1 and 52-2, the transmission / reception unit 31 of the robot 10 transmits the image data 200 acquired by the image acquisition unit 36 to the display terminal 50 using the communication session with the established management server 90. do. As a result, the transmission / reception unit 51 of the display terminal 50 receives the image data 200.

In step S53, the display control unit 53 of the display terminal 50 displays the image data 200 received by the transmission / reception unit 51 on the display screen 600 of the display 512. As a result, the operator who remotely controls the robot 10 using the display terminal 50 can confirm the situation of the base where the robot 10 is located while looking at the display screen 600 on which the image data 200 is displayed. Similarly, the display terminal 50 transmits the image data 250, which is an image taken by the base on the display terminal 50 side, to the robot 10. Then, the transmission / reception unit 31 of the robot 10 receives the image data 250, and the display control unit 33 displays the received image data 250 on the display screen 400 of the display 306a.

Here, the display screen displayed on the robot 10 and the display terminal 50 will be described. FIG. 18A is a diagram showing an example of a screen example of a display screen displayed on the robot according to the first embodiment. In the robot 10, the display control unit 33 causes the display 306a to display the display screen 400 as shown in FIG. 18A. The image data 250 transmitted from the display terminal 50 is displayed on the display screen 400. As shown in FIG. 18A, for example, the face of an operator who operates the robot 10 using the display terminal 50 is displayed on the display screen 400, and the people around the robot 10 who see the display screen 400 are displayed. , Who is operating the robot 10 can be confirmed.

FIG. 18B is a diagram showing an example of a screen example of a display screen displayed on the display terminal according to the first embodiment. In the display terminal 50, the display control unit 53 causes the display 512 to display the display screen 600 as shown in FIG. 18B. The image data 200 transmitted from the robot 10 is displayed on the display screen 600. As shown in FIG. 18B, the display screen 600 displays, for example, people around the robot 10. Further, the display screen 600 has a movement instruction key 601 and an input area 603a that are pressed when the robot 10 is requested to move in the horizontal direction (forward, backward, clockwise rotation, left rotation) next to the image of the image data 200. "Move" button 603b pressed when requesting the movement of the robot 10 to the position input to, and "Rotate" button pressed when changing the direction of the robot 10 to the direction input in the input area 605a. 605b includes a "stop" button 607 pressed when the robot 10 is stopped, and a "♪" button 609 pressed when a sound is emitted from the speaker 308b of the robot 10. An operator who operates the robot 10 using the display terminal 50 controls the movement of the robot 10 by looking at the image data 200 displayed on the display screen 600 and performing an input operation to the movement instruction key 601 or the like. ..

Although an example of operating the movement of the robot 10 by the movement instruction key 601 displayed on the display screen 600 has been described in FIG. 18B, the movement operation of the robot 10 is a game including a keyboard or a joystick. The configuration may be performed by a dedicated controller such as a pad.

Returning to FIG. 17, in step S54, the operation input receiving unit 52 of the display terminal 50 receives a specific operation input for the display screen 600 displayed on the display 512. For example, the operation input receiving unit 52 receives an operation input for any key of the movement instruction key 601 included in the display screen 600, for example.

In step S55, the request command generation unit 55 of the display terminal 50 generates a request command corresponding to the specific operation input received by the operation input reception unit 52. Specifically, when a specific operation input is received by the operation input receiving unit 52, the storage / reading unit 57 of the display terminal 50 reads out the user command table 5001 (see FIG. 11) stored in the storage unit 5000. .. The determination unit 54 of the display terminal 50 searches the user command table 5001 read by the storage / reading unit 57, and identifies the input command corresponding to the operation input received by the operation input receiving unit 52. Then, the determination unit 54 searches the user command table 5001 and extracts the command corresponding to the specified input command. Then, the request command generation unit 55 generates a request command including the extracted command. For example, when the operation input receiving unit 52 accepts the input to the "↑ (forward)" key of the movement instruction key 601, the determination unit 54 specifies "pressing the" ↑ (forward) "key" as an input command. .. Then, the determination unit 54 extracts "FORWARD" as a command corresponding to the specified input command. Then, the request command generation unit 55 generates a request command including the extracted command "FORWARD".

In step S55, an example in which the determination unit 54 extracts the process using the user command table 5001 has been described, but the display terminal 50 is predetermined based on the event given to the input key such as the movement instruction key 605. It may be configured to extract or execute the process of.

In steps S56-1 and 56-2, the transmission / reception unit 51 of the display terminal 50 transmits the request command generated by the request command generation unit 55 to the robot 10 using the communication session with the established management server 90. do. As a result, the transmission / reception unit 31 of the robot 10 receives the request command.

In step S57, the storage / reading unit 46 of the robot 10 reads out the command table 3001 (see FIG. 9) stored in the storage unit 3000. In step S58, the determination unit 34 of the robot 10 searches the command table 3001 read by the storage / reading unit 46, and extracts the process corresponding to the request command received by the transmission / reception unit 31. In step S59, the robot 10 executes the process extracted by step S58.

For example, when the request command received by the transmission / reception unit 31 is "FORWARDMOVE", the determination unit 34 searches the command table 3001 and extracts the process corresponding to the request command "FORWARD". In this case, the extracted process is "advancing the robot 10 by a certain amount". The determination unit 34 notifies the movement control unit 37 of the execution request of the extracted process. Then, the movement control unit 37 drives the movement mechanism 17 of the robot 10 based on the process extracted by the determination unit 34. The notification destination of the execution request differs depending on the process extracted by the determination unit 34. For example, when the request command received by the transmission / reception unit 31 is "SOUND", the notification destination of the execution request is the existence notification unit 45.

As a result, the remote control system 1a causes the display terminal 50 to display the image data 200 acquired by the photographing device 21 included in the robot 10 so that the operator who remotely controls the robot 10 using the display terminal 50 can operate the remote control system 1a. Therefore, the information around the robot 10 can be grasped more accurately. Further, since the remote control system 1a can move the robot 10 while checking the surrounding situation of the robot 10 by performing an operation input by the operator to the display terminal 50 on which the image data 200 is displayed. It is possible to improve the operability for the operator who operates the robot 10 by using the display terminal 50.

〇 Position adjustment process of the robot 10 〇 〇 Target point calculation process Next, the position adjustment process of the robot 10 will be described with reference to FIGS. 19 to 29. Here, it is assumed that a remote conference is being held between the base where the robot 10 is installed and the base where the display terminal 50 is installed. When multiple people are participating in a conference at a base where the robot 10 is installed, if the participants move or the topic changes during the conference, the points of interest that the multiple people (participants) pay attention to change. Sometimes. In this case, even if the attention point changes, the user of the display terminal 50 located at the remote location will see the image that continues to capture the position before the attention point changes, so that the user at the remote location can see the image from the conference. It will be left behind and hinder smooth communication. Therefore, the remote control system 1a aims at smooth communication with the user of the display terminal 50 located at a remote location by moving the robot 10 based on the points of interest of a plurality of people located around the robot 10. Can be done.

FIG. 19 is a flowchart for explaining an example of the target point calculation process in the robot according to the first embodiment. In step S71, the image acquisition unit 36 of the robot 10 acquires image data 200, which is an image acquired by the photographing device 21 (an example of the acquisition step). In step S72, the image processing unit 41 of the robot 10 extracts an image of the head of a person included in the image data 200 acquired by the image acquisition unit 36. Specifically, the image processing unit 41 collates the image data 200 acquired by the image acquisition unit 36 with the image pattern of the head of a person stored in advance, and obtains a similar image region in FIG. 20 (a). It is extracted as an image of the head of a person as shown in. The processed data 800a shown in FIG. 20A includes images of the heads of the four persons a1 to a4.

In step S73, the attention point calculation unit 42 of the robot 10 calculates the size of the head image extracted by the image processing unit 41 and the position of the person's head image in the image data 200. Specifically, the attention point calculation unit 42 calculates the size of the image of the head of the person and the position of the image of the head of the person based on the processing data 800a processed by the image processing unit 41. The position of the image of the head of a person is calculated, for example, with the coordinates of the center of the image data 200 as (0,0), the coordinates of the upper right as (100,100), and the coordinates of the lower left as (-100, -100). Will be done. For example, in the processing data 800a shown in FIG. 20A, the position and size of the images (a1 to a4) of the head of the person calculated by the attention point calculation unit 42 are the values shown in FIG. 20B. ..

In step S74, the attention point calculation unit 42 of the robot 10 specifies the relative position in space of the image of the head of the person included in the image data 200. For example, the attention point calculation unit 42 uses the coordinates (X, Y) in the image data 200 calculated in step S73 and the head size r to position the image of the head of the person in the image data 200. Is converted into a relative position (x, y) in space. As an example of the conversion method to the relative position (x, y) in space, the heuristics of (1) and (2) below are used.

(1) The smaller the head size r, the farther it is (ignoring the difference in head size between people).
(2) Ignoring perspective, if the X coordinate value is positive, it is on the right side of the robot 10 (if it is negative, it is on the left side of the robot 10), and if the Y coordinate is large, it is in the back.

The calculation formula based on the above conditions (1) and (2) is as follows (formula 1). In addition, k1 and k2 are predetermined coefficients. Here, k1 = 0.2 and k2 = 1000.

The attention point calculation unit 42 specifies the relative position (x, y) in the space of the image of the head of the person by using the above (Equation 1). The coordinate information (x, y) indicating the specified relative position is shown in FIG. 21 (a). Further, FIG. 21B is processing data 810 schematically showing the specified relative position. The processing data 810 shown in FIG. 21B visualizes the relative positions of the images a1 to a4 of the head of a person and the robot R with reference to the center position S (0,0) of the image data 200. be.

In step S75, the image processing unit 41 of the robot 10 extracts an image of the eyes and nose of a person included in the image data 200 acquired in step 71. Specifically, the image processing unit 41 collates the image data 200 acquired by the image acquisition unit 36 with the pre-stored image patterns of the eyes and nose of the person, and obtains a similar image region in FIG. 22 (FIG. 22). It is extracted as an image of the eyes and nose of a person as shown in a). The processed data 800b shown in FIG. 22A includes an image of the eyes and nose with respect to the images of the heads (a1 to a4) of the four persons extracted by step S72.

In step S76, the attention point calculation unit 42 of the robot 10 identifies the orientation of each person from the positions of the eyes and nose extracted by the image processing unit 41. As an example, the heuristics of (1) and (2) below are used to specify the orientation of the person.

(1) If there are two eyes, it is facing the front (the direction opposite to the direction of the robot 10), if it is one, it is facing either the left or right, and if it is zero, it is facing the back (the same direction as the direction of the robot 10).
(2) If there is one eye and nose, it is assumed that the nose is facing in a certain direction (the direction is 0 ° in the front and up to 360 ° clockwise).

Based on the above conditions (1) and (2), the orientation of the person is specified as shown in FIG. 22 (b). For example, since a1 has one eye and the nose points to the right, the direction (θ) of a1 is 90 °.

As a result, the attention point calculation unit 42 of the robot 10 can approximately obtain the relative position and orientation of the person corresponding to the image data 200. FIG. 23A is information on the relative position and orientation specified by the attention point calculation unit 42. FIG. 23B is processing data 820 schematically showing the relative positions (x, y) and orientations (θ) of each person. As shown in FIG. 23 (a), the relative position (x, y) of a1 is (0,60), and the orientation (θ) is 90 °. The relative position (x, y) of b1 is (90,76.7) and the orientation (θ) is 180 °. The relative position (x, y) of c1 is (-20,42), and the orientation (θ) is 0 °. The relative position (x, y) of d1 is (70,51), and the orientation (θ) is 0 °.

As a result, the robot 10 can easily estimate the position and orientation of a person located around the robot 10. The method for estimating the position and orientation of a person located around the robot 10 is not limited to this, and may be a method such as adopting a perspective method or using a stereo image taken by a stereo camera. .. Further, a method of measuring the distance using a light beam such as a laser, or a method of arranging a plurality of cameras in the base and calculating the position and orientation of a person from a plurality of images may be used.

Here, another example of a method of estimating the position and orientation of a person located around the robot 10 will be described. The robot 10 may estimate (calculate) the position and orientation of a person located around the robot 10 by using a beacon or the like installed at a base where the robot 10 is located. FIG. 24A is a diagram for explaining an example of processing for estimating the position and orientation of a person located around the robot 10 when the beacon is installed in the base.

In the example shown in FIG. 24, beacons for transmitting predetermined radio waves are installed in various places in the base (for example, a space such as a closed room) 1100. Participants (h1 to h4) of a conference or the like located in the base 1100 have a communication terminal such as a smartphone having a function of receiving radio waves transmitted from a beacon and a gyro sensor. In addition, participants (h1 to h4) of the conference or the like wear the communication terminal at a specific position such as a chest pocket or a waist belt so that the orientation of the body and the orientation of the communication terminal are linked.

As a result, the communication terminal held by the participants (h1 to h4) of the conference or the like can acquire the position of the communication terminal (participant) from the strength of the radio waves transmitted from the plurality of beacons. Further, the communication terminal held by the participants (h1 to h4) of the conference or the like can acquire the orientation of the communication terminal (participant) from the information of the gyro sensor. Then, the robot 10 located in the base 1100 receives (acquires) information on the position and orientation of the participants (h1 to h4) transmitted from each of the communication terminals held by each participant (h1 to h4). can do.

FIG. 24 (b) is information on the relative position (x, y) and orientation (θ) of each person (h1 to h4) specified by the method shown in FIG. 24 (a). As shown in FIG. 24 (b), the relative position (x, y) of h1 is (5,17), and the orientation (θ) is 45 °. The relative position (x, y) of h2 is (18,18) and the orientation (θ) is 240 °. The relative position (x, y) of h3 is (5,12) and the orientation (θ) is 60 °. The relative position (x, y) of h4 is (15,12) and the orientation (θ) is 330 °.

In the following description, the values shown in FIG. 24B are used as the values of the coordinate information (x, y) and the direction (θ) indicating the relative positions calculated by the attention point calculation unit 42.

Returning to FIG. 19, the explanation of the target point calculation process by the robot 10 will be continued. In the processing after step S77, the robot 10 calculates the points of interest of a plurality of people around the robot 10. The point of interest is a position where a plurality of people are paying attention at the base where the robot 10 is installed, for example, a direction in which a plurality of people around the robot 10 are facing. The point of interest is, for example, in a remote conference, the direction in which a plurality of participants at the base on the robot 10 side are facing, that is, the central position of the topic at a certain point in time.

In step S77, the image processing unit 41 of the robot 10 has each person (h1 to h1) specified by the attention point calculation unit 42 from the relative position of each person (h1 to h4) specified by the attention point calculation unit 42. The intersection of straight lines extended in the direction (θ) of h4) is calculated. The calculated result is shown in FIG. In FIG. 25, the combination of points and the coordinates (x, y) of the intersections are shown. Since the direction of the person is one direction, there may be no intersection like the combination of h1 and h2 and h2 and h3.

In step S78, the attention point calculation unit 42 of the robot 10 shifts the process to step S79 when there are a plurality of intersections obtained from the most directions among the calculated intersections. On the other hand, the point of interest calculation unit 42 shifts the process to step S80 when there is only one intersection obtained from the most directions among the calculated intersections.

Specifically, first, the attention point calculation unit 42 puts together the coordinates of the intersections calculated in step S77 that are close to each other. The coordinate of the combined result is, for example, the average value of the coordinates of each intersection. FIG. 26A is coordinate information indicating the positions of the intersections summarized by the above processing. FIG. 26B is the processing data 830 which schematically shows the positions of the intersections collected by the above processing. As shown in FIG. 26 (a), the intersection of the straight lines extending from h1, h3 and h4 is A (10.5,18), and the intersection of the straight lines extending from h2 and h4 is B (14,12). ).

Then, the attention point calculation unit 42 extracts the intersections obtained from the most directions among the points (intersections) in which the intersections having a short distance are put together. In the case of FIG. 26A, the intersection A is an intersection obtained from three directions (h1, h2, h3), and the intersection B is an intersection obtained from two directions (h2, h4). Therefore, the point of interest calculation unit 42 shifts the process to step S80 because the number of intersections obtained from the most directions is one (intersection point A).

In step S80, the point of interest calculation unit 42 identifies the intersection point A, which is the intersection point obtained from the most directions, as the point of interest. On the other hand, in step S79, the attention point calculation unit 42 specifies the coordinates of the center position between the plurality of intersections obtained from the most directions as the attention points.

In step 77, the attention point calculation unit 42 performed a process of collecting the coordinates of the calculated intersections having a short distance, but the calculated intersections were used as they were to pay attention to the intersections obtained from the most directions. It may be specified as a point.

As a result, the remote control system 1a can calculate the points of interest that a plurality of people located around the robot 10 pay attention to by using the image data 200 acquired by the photographing device 21 included in the robot 10. Therefore, the remote control system 1a can grasp the points of interest of a plurality of participants on the base side where the robot 10 is installed, for example, in a remote conference using the robot 10 and the display terminal 50.

Subsequently, a process of specifying the moving destination of the robot 10 will be described using the attention points calculated by the attention point calculation unit 42. In step S81, the target point calculation unit 43 of the robot 10 identifies a target point to be moved by the robot 10 based on the position of the attention point calculated by the attention point calculation unit 42. The target point includes, for example, position information indicating the target position G (X, Y) of the destination of the robot 10 and direction information indicating the direction (θ) of the robot 10 at the destination of the robot 10. Specifically, as shown in FIG. 27B, the target point calculation unit 43 sets the target position G at a position separated by the distance p from the attention point A on the line connecting the positions of the attention point A and the robot 10. Calculated as (X, Y). Here, the distance p is, for example, the average value of the distances between the positions of the respective persons (h1 to h4) and the points of interest A. As the position of the person (h1 to h4), the relative position (x, y) calculated by the attention point calculation unit 42 is used. Further, the target point calculation unit 43 calculates the direction of the attention point A as seen from the current position of the robot 10 as the direction (θ) of the robot 10. The target points (target position G and orientation) specified by the target point calculation unit 43 are, for example, the values shown in FIG. 27 (a). In this case, the target position G (X, Y) is (6,11), and the orientation (θ) of the robot 10 is 35 °.

The orientation of the robot 10 calculated by the target point calculation unit 43 is not limited to the direction of the attention point A, and may be the average direction of the orientations of a plurality of persons calculated by the attention point calculation unit 42.

As a result, the remote control system 1a can calculate the target point to which the robot 10 is moved based on the points of interest that a plurality of people located around the robot 10 pay attention to. Therefore, the remote control system 1a is based on a location where a plurality of participants on the base side where the robot 10 is installed are paying attention, for example, in a remote conference using the robot 10 and the display terminal 50. You can specify the destination.

〇Movement process to the target point Next, a process of moving the robot 10 toward the target point specified by the target point calculation unit 43 will be described. FIG. 28 is a flowchart for explaining an example of the movement process in the robot according to the first embodiment.

In step S101, the target point calculation unit 43 of the robot 10 calculates a movement route toward the calculated target position G (X, Y).

In step S102, the robot 10 moves based on the calculated movement path. First, the orientation adjusting unit 38 of the robot 10 changes the orientation of the robot 10 by rotating the movement mechanism 17 or the like with respect to the traveling direction with respect to the movement path calculated by the target point calculation unit 43. Then, the movement control unit 37 of the robot 10 moves the robot 10 by driving the movement mechanism 17 when the direction change in the traveling direction by the orientation adjustment unit 38 is completed. The movement control unit 37 issues (transmits) a movement command corresponding to the movement route calculated by the target point calculation unit 43 to the movement mechanism 17 at any time, so that the robot 10 can be moved along the calculated movement route. Move it.

In step S103, when the peripheral information detection unit 44 of the robot 10 detects an obstacle while the robot 10 is moving, the process shifts to step S104. Specifically, the peripheral information detection unit 44 detects an obstacle existing in the moving path of the robot 10 by using the image data 200 acquired by the photographing device 21 and the detection results detected by various sensors 106. The obstacle is, for example, a device or an object installed in the base, or a person located in the base. In step S104, the movement control unit 37 of the robot 10 stops the robot 10 when an obstacle is detected by the peripheral information detection unit 44.

As a result, when the robot 10 detects an obstacle while moving toward the target position G, the robot 10 stops in front of the detected obstacle, so that contact with the obstacle can be avoided. In this case, the robot 10 may resume the movement toward the target position G again when it is detected by the peripheral information detection unit 44 that the obstacle has disappeared after the robot 10 has stopped once. Further, the robot 10 is configured to change the movement route calculated by the target point calculation unit 43 when the obstacle detected by the peripheral information detection unit 44 does not disappear even after a predetermined time has elapsed after the robot 10 has stopped once. May be.

On the other hand, in step S103, if the peripheral information detection unit 44 does not detect an obstacle while the robot 10 is moving, the process shifts to step S105. In this case, the movement control unit 37 of the robot 10 continues the movement control of the robot 10 toward the target position G.

In step S105, the movement control unit 37 of the robot 10 shifts the process to step S106 when the robot 10 reaches the target position G. On the other hand, when the robot 10 has not reached the target position G, the movement control unit 37 repeats the process from step S103 and continues the movement control of the robot 10 toward the target position G.

In step S106, the movement control unit 37 of the robot 10 stops the robot 10 when the robot 10 reaches the target position G. In step S107, the orientation adjusting unit 38 of the robot 10 changes the orientation of the robot 10 to the orientation calculated by the target point calculation unit 43 (see FIG. 27A). Specifically, the orientation adjusting unit 38 changes the orientation of the robot 10 based on the direction information indicating the orientation (θ) calculated by the target point calculation unit 43 by rotating the moving mechanism 17 or the like. .. Then, the transmission / reception unit 31 of the robot 10 transmits the image data 200 photographed and acquired by the photographing device 21 at the moved target point to the display terminal 50 via the communication network 9. As a result, the display terminal 50 can display the image data 200 acquired by photographing the surroundings of the robot 10 at the target point where the robot 10 is moved.

Therefore, the remote control system 1a can move the robot 10 to the target point calculated by the target point calculation unit 43. Therefore, the remote control system 1a can move the robot 10 based on the points of interest of a plurality of participants at the base where the robot 10 is located, and at the same time, the display terminal 50 at a remote location captures the points of interest and their vicinity. Since the image can be displayed, smooth communication between remote bases can be realized.

When the robot 10b does not have the moving mechanism 17 like the robot 10b shown in FIG. 4, the robot 10b (transmission terminal) determines the direction (θ) of a person among the target points calculated by the target point calculation unit 43. The robot 10b may be configured to change the direction of the robot 10b to the direction of the change destination indicated by the direction information by using only the indicated direction information.

〇Existence notification processing Next, when the robot 10 is moving to the target point, the processing for notifying the surroundings of its existence from the robot 10 will be described. FIG. 29 is a flowchart for explaining an example of existence notification processing in the robot according to the first embodiment.

In step S121, the position specifying unit 39 of the robot 10 specifies the current position of the robot 10. Specifically, the position specifying unit 39 identifies the current position of the robot 10 based on the detection results such as the direction of each direction (azimuth, magnetic north) detected by the acceleration / direction sensor 103 or the GPS receiving unit 104. do. In step S122, the position specifying unit 39 of the robot 10 calculates the distance between the current position of the specified robot 10 and the point of interest A calculated by the point of interest calculating unit 42.

In step S123, when the distance calculated by the position specifying unit 39 is longer than or equal to a predetermined threshold value or longer than a predetermined threshold value, the robot 10 shifts the process to step S124. On the other hand, when the distance calculated by the position specifying unit 39 is shorter than the predetermined threshold value or equal to or less than the predetermined threshold value, the robot 10 repeats the process from step S121.

In step S124, the existence notification unit 45 of the robot 10 outputs notification information for notifying the existence of the robot 10. Specifically, the existence notification unit 45 outputs a sound signal from the speaker 308b as notification information. Further, the existence notification unit 45 performs a predetermined gesture by driving the movement mechanism 17, the movable arm 11, or the rotation shaft 12 as notification information. The method of outputting the notification information by the existence notification unit 45 is not limited to this.

As a result, when the robot 10 is far away from the attention point, the robot 10 approaches the attention point while notifying the surrounding people of the existence of the robot 10, and the robot 10 suddenly appears to surprise the surrounding people. Can be prevented. In addition, the robot 10 can prevent the progress of the conference or the like from being hindered by approaching the point of interest.

The existence notification unit 45 of the robot 10 may be configured to notify the existence of the robot when the distance between the current position of the robot 10 and the point of interest is equal to or less than a predetermined threshold value or shorter than a predetermined threshold value. ..

As described above, in the remote control system 1a, the robot 10 automatically moves to a suitable position based on the points of interest that a plurality of people (participants) located around the robot 10 pay attention to. In the remote communication performed using the robot 10, it is possible to reduce the discomfort and annoyance felt by the participants, and it is possible to realize the remote communication via the robot 10 in a natural manner.

Traditional fixed-position video conferencing systems do not move from the initial position, so for example, if the center of the topic (point of interest) changes, those who participate from a remote location will be left out of the conversation. I get the impression that Further, for example, a participant who is around the robot 10 may feel that the person who participates from a remote place is not interested in the topic, and may exclude the person who participates from the remote place from the conversation target. There is. As a result, the sense of unity caused by remote communication is impaired, which causes a sense of alienation.

On the other hand, in order to change the position and orientation of the robot 10 in accordance with the central position (point of interest) of the topic, the robot 10 is a person participating from a remote location (for example, the robot 10 using the display terminal 50). The operator who operates the robot 10 can get the impression that he / she is always participating in the topic through the robot 10. On the other hand, a plurality of participants located around the robot 10 also have the impression that the participants in a remote place participate in the conference and listen to the story. As a result, the remote control system 1a can realize smooth communication by a remote conference using the robot 10.

〇 Example of the movement path of the robot 10 〇 Here, the movement path of the robot 10 calculated by the target point calculation unit 43 will be described. The example shown in FIG. 30 (a) is a method in which the robot 10 (R) does not go straight to the target position G, but finds a space between people and interrupts there. This method is effective for preventing the body distance (private space) from being invaded by the robot 10 (R) getting too close to a person. In this case, the target point calculation unit 43 of the robot 10 (R) identifies a position where a certain distance or more is vacant from all combinations of positions of each participant (h1 to h4), and the robot is set at an intermediate distance. The movement route is calculated so as to move 10 (R).

The example shown in FIG. 30B is a method in which the robot 10 (R) does not go straight to the target position G, but passes by the person closest to the current position of the robot 10 (R). The target point calculation unit 43 of the robot 10 (R) identifies a participant (for example, h3) closest to the current position of the robot 10 (R) specified by the position specifying unit 39, and from the position of the specified participant. The movement route is calculated so as to move the robot 10 (R) to a position separated by a certain distance. The robot 10 (R) may, for example, pass through a position separated from the position of h3 by a certain distance and move to the target position G, or may stop at a position separated by a certain distance from the position of h3. good. In this case, the robot 10 (R) is not oriented in the direction of the point of interest A as shown in FIG. 30 (b), but is oriented in the same direction as h3 by the orientation adjusting unit 38. You may change the direction of.

〇Processing until the target point calculation process is started 〇Here, the frequency or timing of performing the target point calculation process shown in FIG. 19 will be described. In a meeting or the like, when the positions of the participants around the robot 10 change frequently, if the robot 10 moves each time, the participants may be hindered from the meeting. Therefore, the remote control system 1a performs the target point calculation process shown in FIG. 19 on the robot 10 when a predetermined condition is satisfied.

The flowchart shown in FIG. 31A is an example of performing the target point calculation process based on the elapsed time from the previous target point calculation. In step S151a, the determination unit 34 of the robot 10 measures the elapsed time from the time of the previous calculation of the target point by using the timer 107. In step S152a, when the elapsed time from the time of the previous target point calculation exceeds the condition of the set elapsed time, the determination unit 34 shifts the process to step S153a. Specifically, the storage / reading unit 46 of the robot 10 reads out the condition table 3002 (see FIG. 10) stored in the storage unit 3000. The determination unit 34 searches the condition table 3002 read by the storage / reading unit 46, and extracts the condition of the elapsed time corresponding to the base where the robot 10 is located. For example, when the robot 10 is located at the base A, the determination unit 34 extracts "3 min" as a condition of the elapsed time corresponding to the base A. Then, when the elapsed time from the time of the previous target point calculation exceeds the extracted condition, the determination unit 34 shifts the process to step S153a. On the other hand, in step S152a, if the elapsed time from the time of the previous target point calculation does not exceed the set elapsed time condition (within the set elapsed time), the determination unit 34 has step S151a. Repeat the process from.

In step S153a, when a predetermined time has elapsed from the time of the previous target point calculation, the robot 10 performs the target point calculation process shown in FIG. As a result, the robot 10 performs the calculation process of the target point indicating the movement destination of the robot 10 based on the elapsed time since the previous target point calculation process, and moves the robot 10 to the calculated target point. It is possible to prevent the smooth communication from being hindered by the frequent movement of the robot 10.

The flowchart shown in FIG. 31B is an example of performing target point calculation processing based on the distance to a person (participant) located around the robot 10. In step S151b, the robot 10 measures the distance to a person (participant) located around the robot 10. Specifically, the position specifying unit 39 of the robot 10 is based on the detection result such as the direction of each direction (azimuth, magnetic north) detected by the acceleration / direction sensor 103 or the GPS receiving unit 104. Identify the position. Then, the position specifying unit 39 calculates the distance between the current position of the specified robot 10 and the person (participant) calculated by the attention point calculation unit 42.

In step S152b, when the calculated distance to the person (participant) exceeds the condition of the distance to the set person, the determination unit 34 shifts the process to step S153b. Specifically, the storage / reading unit 46 of the robot 10 reads out the condition table 3002 (see FIG. 10) stored in the storage unit 3000. The determination unit 34 searches the condition table 3002 read by the storage / reading unit 46, and extracts the condition of the distance to the person corresponding to the base where the robot 10 is located. For example, when the robot 10 is located at the base A, the determination unit 34 extracts "8 m" as a condition of the distance from the person corresponding to the base A. Then, when at least one of the distances between the robot 10 and the person calculated by the position specifying unit 39 exceeds the extracted condition, the determination unit 34 shifts the process to step S153b. On the other hand, in step S152b, the determination unit 34 does not exceed the condition of the calculated distance to the person (participant) from the set person (the robot 10 and all the participants are predetermined). If it exists within a distance), the process from step S151b is repeated.

In step S153b, when the distance to the person (participant) is more than a predetermined distance, the robot 10 performs the target point calculation process shown in FIG. As a result, the robot 10 performs the calculation processing of the target point indicating the movement destination of the robot 10 and calculates it only when the distance from the person (participant) located around the robot 10 is more than a predetermined distance. Since the robot 10 is moved to the target point, it is possible to prevent the smooth communication from being hindered by the frequent movement of the robot 10.

The flowchart shown in FIG. 31 (c) is an example of performing the target point calculation process based on the set adjustment cycle. In step S151c, the determination unit 34 of the robot 10 measures the elapsed time from the time of the previous calculation of the target point by using the timer 107. In step S152c, when the elapsed time from the time of the previous target point calculation exceeds the condition of the set adjustment cycle, the determination unit 34 shifts the process to step S153c. Specifically, the storage / reading unit 46 of the robot 10 reads out the condition table 3002 (see FIG. 10) stored in the storage unit 3000. The determination unit 34 searches the condition table 3002 read by the storage / reading unit 46, and extracts the conditions of the adjustment cycle corresponding to the base where the robot 10 is located. For example, when the robot 10 is located at the base A, the determination unit 34 extracts "5 min" as a condition of the adjustment cycle corresponding to the base A. Then, when the elapsed time from the time of the previous target point calculation exceeds the extracted condition, the determination unit 34 shifts the process to step S153c. On the other hand, in step S152c, when the elapsed time from the time of the previous target point calculation does not exceed the condition of the set adjustment cycle (within the range of the set adjustment cycle), the determination unit 34 The process from step S151c is repeated.

In step S153c, the robot 10 performs the target point calculation process shown in FIG. 19 when a predetermined adjustment cycle has elapsed since the previous target point calculation. As a result, the robot 10 frequently calculates the target point indicating the movement destination of the robot 10 based on the adjustment cycle of the set target point, and moves the robot 10 to the calculated target point. It is possible to prevent smooth communication from being hindered by the movement of 10.

Therefore, in the remote control system 1a, as shown in FIGS. 31A to 31C, the robot 10 frequently moves in order to determine the frequency or timing at which the target point calculation process is started based on predetermined conditions. It is possible to prevent smooth communication from being hindered by this. The processes shown in FIGS. 31 (a) to 31 (c) and the conditions included in the condition table 3002 shown in FIG. 10 are merely examples, and the present invention is not limited to this, and the administrator of the robot 10 or the display terminal 50 is used. It can be modified or changed as appropriate by an operator or the like who operates the robot 10.

Further, the remote control system 1a may be configured to perform a target point calculation process when at least one person (participant) around the robot 10 moves. In this case, the robot 10 changes the position or orientation of at least one of the plurality of persons included in the image data 200 acquired by the image acquisition unit 36, for example, in the attention point calculation unit 42 and the target point calculation unit 43. If so, the target point is calculated based on the position or orientation of the changed person. Specifically, in the robot 10, the attention point calculation unit 42 and the target point calculation unit 43 set the position or orientation of the unchanged person among the plurality of persons included in the image data 200 as a fixed value, and the changed person. The target point is calculated based on the amount of change in position or orientation.

● Effects of the First Embodiment As described above, the remote control system 1a according to the first embodiment is a robot 10 (an example of a moving body) provided with a photographing device 21 for photographing a subject and acquiring image data 200. ) And a display terminal 50 capable of communicating with the robot 10 via the communication network 9 and remotely controlling the robot 10, and performing remote communication using image data transmitted and received between the robot 10 and the display terminal 50. Realize.

Here, in the remote control system 1a according to the first embodiment, the control device 30 included in the robot 10 (an example of a mobile body) is an example of the information processing device according to the present invention. The control device 30 as an information processing device includes an image acquisition unit 36 (an example of acquisition means) for acquiring image data 200 acquired by the photographing device 21, positions of a plurality of persons included in the acquired image data 200, and positions of a plurality of persons. A point of interest calculation unit 42 for calculating a target point indicating a destination of the robot 10 based on the orientation and a target point calculation unit 43 (an example of a calculation means) are provided, and the robot 10 is moved to the calculated target point. Therefore, the control device 30 as the information processing device according to the first embodiment moves the robot 10 according to the positions and orientations of the plurality of participants at one base, thereby participating in the other base. It is possible to activate remote communication with a person.

Further, the control device 30 included in the robot 10 (an example of a moving body) according to the first embodiment estimates the positions and orientations of a plurality of persons included in the acquired image data 200 (an example of an estimation means, estimation). Based on the designated position and orientation, the attention point calculation unit 42 for specifying the attention point to be noticed by a plurality of people included in the image data 200 (an example of the attention point specifying means) and the identified attention point are used. It has a target point calculation unit 43 (an example of a target point specifying means) for specifying a target point indicating a destination of the robot 10, and therefore, the control device 30 as an information processing device according to the first embodiment is the robot 10. By moving the robot 10 based on the points of interest paid by a plurality of participants located around the robot 10, it is possible to activate remote communication with the participants at the other base.

● Second embodiment ●
Next, the remote control system according to the second embodiment will be described. The same configurations and the same functions as those of the first embodiment are designated by the same reference numerals, and the description thereof will be omitted. The remote control system 1b according to the second embodiment is configured such that the display terminal 50 that displays the image data transmitted from the robot 10 performs the position adjustment process of the robot 10.

The hardware configuration of the device or terminal (robot 10, display terminal 50, and management server 90) constituting the remote control system 1b according to the second embodiment constitutes the remote control system 1a according to the first embodiment. Since it is the same as the device or terminal (robot 10, display terminal 50, and management server 90), the description thereof will be omitted.

● Functional configuration First, the functional configuration of the remote control system according to the second embodiment will be described with reference to FIG. 32. FIG. 32 is a diagram showing an example of the functional configuration of the remote control system according to the second embodiment. Since the functions realized by the management server 90 according to the second embodiment are the same as the functions realized by the management server 90 according to the first embodiment (see FIG. 8), the description thereof will be omitted.

〇 Functional configuration of the control device 30 〇 First, the functional configuration of the control device 30 for controlling the processing or operation of the robot 10 according to the second embodiment will be described with reference to FIG. 32. The control device 30 according to the second embodiment includes an image processing unit 41, an attention point calculation unit 42, and a target point calculation unit 43 from the functions (see FIG. 8) realized by the control device 30 according to the first embodiment. Realize the excluded functions. Specifically, the functions realized by the control device 30 according to the second embodiment are the transmission / reception unit 31, the operation input reception unit 32, the display control unit 33, the determination unit 34, the shooting instruction unit 35, and the image acquisition unit 36. , The movement control unit 37, the orientation adjustment unit 38, the position identification unit 39, the peripheral information detection unit 44, the existence notification unit 45, the storage / reading unit 46, and the storage unit 3000. Since each function realized by the control device 30 according to the second embodiment is the same as the content shown in FIG. 8, the description thereof will be omitted.

〇 Functional configuration of the display terminal 50 〇 Next, the functional configuration of the display terminal 50 will be described with reference to FIG. 32. The functions realized by the display terminal 50 according to the second embodiment include the image processing unit 61 and the attention point calculation unit in addition to the functions realized by the display terminal 50 according to the first embodiment (see FIG. 8). 62 and the target point calculation unit 63 are included. Specifically, the functions realized by the display terminal 50 include a transmission / reception unit 51, an operation input reception unit 52, a display control unit 53, a judgment unit 54, a request command generation unit 55, a line-of-sight detection unit 56, and a storage / reading unit 57. , Image processing unit 61, attention point calculation unit 62, target point calculation unit 63, and storage unit 5000. Further, the transmission / reception unit 51 is an example of acquisition means for receiving (acquiring) image data 200, which is an image acquired by the photographing apparatus 21. Further, the transmission / reception unit 51 is an example of a second transmission means for transmitting a request command (request information) for moving the robot 10 to the target point calculated by the target point calculation unit 63, which will be described later, to the robot 10. ..

The image processing unit 61 is a function of processing an image received (acquired) by the transmission / reception unit 51. The image processing unit 61 extracts, for example, an image of a person's head included in the image data 200 acquired by the transmission / reception unit 51. Further, the image processing unit 61 extracts, for example, an image of the eyes and nose of a person acquired by the transmission / reception unit 51. The image processing unit 61 is mainly realized by the processing of the CPU 501 shown in FIG.

The attention point calculation unit 62 is a function of specifying the attention points attracted by a plurality of people based on the positions and orientations of the plurality of persons included in the image data processed by the image processing unit 61. Since the function realized by the attention point calculation unit 62 is the same as that of the attention point calculation unit 42 shown in FIG. 8, the description thereof will be omitted. The attention point calculation unit 62 is mainly realized by the processing of the CPU 501 shown in FIG. The attention point calculation unit 62 is an example of the calculation means. Further, the attention point calculation unit 62 is an example of the estimation means. Further, the attention point calculation unit 62 is an example of the attention point specifying means.

The target point calculation unit 63 is a function of specifying a target point to be moved to the robot 10 based on the attention point calculated by the attention point calculation unit 62. Since the function realized by the target point calculation unit 63 is the same as that of the target point calculation unit 43 shown in FIG. 8, the description thereof will be omitted. The target point calculation unit 63 is mainly realized by the processing of the CPU 501 shown in FIG. The target point calculation unit 63 is an example of the calculation means. Further, the target point calculation unit 63 is an example of the target point specifying means.

● Processing or operation of the second embodiment Next, the processing or operation of the robot in the remote control system according to the second embodiment will be described with reference to FIG. 33. In FIG. 33, the process executed by the control device 30 included in the robot 10 will be described as the process executed by the robot 10. FIG. 33 is a sequence diagram for explaining an example of robot control processing in the remote control system according to the second embodiment.

In step S201, the image acquisition unit 36 of the robot 10 acquires image data 200, which is an image captured by the imaging device 21.

In steps S202-1 and S202-2, the transmission / reception unit 31 of the robot 10 transmits the image data 200 acquired by the image acquisition unit 36 to the display terminal 50 using the communication session with the established management server 90. do. As a result, the transmission / reception unit 51 of the display terminal 50 receives (acquires) the image data 200, which is an image acquired by the photographing device 21 (an example of the acquisition step).

In step S203, the display control unit 53 of the display terminal 50 displays the image data 200 received by the transmission / reception unit 51 on the display screen 600 (see FIG. 18B) of the display 512. As a result, the operator who operates the robot 10 using the display terminal 50 can confirm the situation of the base where the robot 10 is located while looking at the display screen 600 on which the image data 200 is displayed.

In step S204, the display terminal 50 executes a calculation process of a target point to which the robot 10 is moved (an example of the calculation step). Since the target point calculation process by the image processing unit 61, the attention point calculation unit 62, and the target point calculation unit 63 of the display terminal 50 is the same as the process shown in FIG. 19, the description thereof will be omitted.

In step S205, the request command generation unit 55 of the display terminal 50 generates a request command to be transmitted to the robot 10 based on the target point calculated by the target point calculation unit 63. In this case, the request command generated by the request command generation unit 55 is a "GO (X, Y)" command whose variable is the coordinate information (X, Y) of the target position G calculated by the target point calculation unit 63. include. Further, the request command generated by the request command generation unit 55 includes a "TURN (θ)" command whose variable is the direction (θ) of the robot 10 calculated by the target point calculation unit 63. The request command generated by the request command generation unit 55 is an example of request information for moving the robot 10 to the target point calculated by the target point calculation unit 63.

In steps S206-1 and S206-2, the transmission / reception unit 31 of the display terminal 50 uses the communication session with the established management server 90 to generate the request command (“GO (X,”. Y) ”,“ TURN (θ) ”) is transmitted to the robot 10. As a result, the transmission / reception unit 31 of the robot 10 receives the request command (“GO (X, Y)”, “TURN (θ)”).

In step S207, the storage / reading unit 46 of the robot 10 reads out the command table 5001 stored in the storage unit 5000. In step S208, the determination unit 34 of the robot 10 searches the command table 3001 read by the storage / reading unit 46, and extracts the process corresponding to the request command received by the transmission / reception unit 31. In this case, since the request commands received by the transmission / reception unit 31 are “GO (X, Y)” and “TURN (θ)”, the determination unit 34 searches the command table 3001 and requests the request command “GO (GO (). X, Y) ”and“ TURN (θ) ”are extracted. The extracted processes are "move the robot 10 to the position (X, Y)" and "rotate the robot 10 by θ ° to the right".

In step S209, the robot 10 executes the process extracted by step S208. Specifically, the determination unit 34 of the robot 10 notifies the movement control unit 37 of the execution request of the extracted process. Then, the movement control unit 37 moves the robot 10 based on the process extracted by the determination unit 34. Since the movement process of the robot 10 is the same as the process shown in FIG. 28, the description thereof will be omitted.

Then, the transmission / reception unit 31 of the robot 10 transmits the image data 200 captured by the photographing device 21 at the target point included in the request command transmitted from the display terminal 50 to the display terminal 50. The display terminal 50 receives the image data 200 transmitted from the robot 10. As a result, the display terminal 50 can display the image data 200 acquired by photographing the surroundings of the robot 10 at the target point included in the request command.

● Effects of the Second Embodiment As described above, the remote control system 1b according to the second embodiment performs the target point calculation process of the robot 10 (an example of a mobile body) via the communication network 9. Is performed by the display terminal 50 that remotely controls. Therefore, the remote control system 1b according to the second embodiment can reduce the processing load of the robot 10 and can remotely communicate using the robot 10 between the base where the robot 10 is located and the display terminal 50. Can be activated.

Here, in the remote control system 1b according to the second embodiment, the display terminal 50 is an example of the information processing apparatus according to the present invention. The display terminal 50 as an information processing device includes a transmission / reception unit 51 (an example of acquisition means) for acquiring image data 200 acquired by the photographing apparatus 21, and positions and orientations of a plurality of persons included in the acquired image data 200. A point of interest calculation unit 62 for calculating a target point indicating a destination of the robot 10 and a target point calculation unit 63 (an example of a calculation means) are provided, and the robot 10 is moved to the calculated target point. Therefore, the display terminal 50 as the information processing device according to the second embodiment calculates the destination of the robot 10 based on the positions and orientations of the plurality of participants at one of the bases, and the robot 10 to the calculated destination. By moving, it is possible to activate remote communication with participants at the other base.

● Third embodiment ●
Next, the remote control system according to the third embodiment will be described. The same configurations and the same functions as those of the first embodiment or the second embodiment are designated by the same reference numerals, and the description thereof will be omitted. The remote control system 1c according to the third embodiment is configured such that the position adjustment process of the robot 10 is performed by the image processing server 70 capable of communicating with the robot 10 and the display terminal 50 via the communication network 9.

● System Configuration First, the outline of the configuration of the remote control system of the third embodiment will be described with reference to FIG. 34. FIG. 34 is a diagram showing an example of the system configuration of the remote control system according to the third embodiment. As shown in FIG. 34, in the remote control system 1c of the third embodiment, the image processing server 70 is added to the configuration shown in FIG. The image processing server 70 is communicably connected to the robot 10, the display terminal 50, and the management server 90 via the communication network 9. The image processing server 70 uses the communication session established by the management server 90 to send and receive image data to and from the robot 10 or the display terminal 50.

The image processing server 70 is a server computer, and includes a case where image processing is performed in a distributed manner among a plurality of server computers. The image processing server 70 performs image processing of the image data 200 transmitted from the robot 10 according to the moving state of the robot 10, and transmits the processed data to the display terminal 50. The image processing server 70 may be configured by a plurality of server computers, and any server computer may be provided with a function. The image processing server 70 is an example of an information processing device.

Here, the image processing server 70 and the management server 90 constitute the server system 7. The image processing server 70 and the management server 90 may be configured by one server computer. Further, the display terminal 50 and the image processing server 70 constitute a display system 5. Further, the robot 10 (10A, 10B, 10C) and the image processing server 70 constitute a mobile control system (transmission control system) 3.

Since the hardware configuration of the image processing server 70 is the same as the hardware configuration of the management server 90 shown in FIG. 7, the description thereof will be omitted. Hereinafter, the image processing server 70 will be described as having the hardware configuration shown in FIG. 7.

● Functional configuration FIG. 35 is a diagram showing an example of the functional configuration of the remote control system according to the third embodiment. The functions realized by the image processing server 70 are the transmission / reception unit 71, the judgment unit 72, the data processing unit 73, the image processing unit 74, the attention point calculation unit 75, the target point calculation unit 76, the storage / reading unit 77, and the storage unit 7000. including.

The transmission / reception unit 71 is a function for transmitting / receiving various data or information to / from another device (for example, a management server 90, a display terminal 50, or a robot 10) via a communication network 9. The transmission / reception unit 71 receives (acquires) image data 200, which is an image acquired by the photographing device 21, for example. Further, the transmission / reception unit 71 transmits the request data (request information) generated by the data processing unit 73 to the robot 10 via the communication network 9. The transmission / reception unit 71 is mainly realized by the processing of the CPU 901 shown in FIG. 7 and the network I / F 909. The transmission / reception unit 71 is an example of acquisition means. Further, the transmission / reception unit 71 is an example of a third transmission means.

The determination unit 72 is a function of determining the content of processing for various data transmitted / received by the transmission / reception unit 71. The determination unit 72 is mainly realized by the processing of the CPU 901 shown in FIG. 7.

The data processing unit 73 is a function of processing various data transmitted / received by the transmission / reception unit 71. The data processing unit 73 generates request data (request information) for moving the robot 10 to the target point calculated by the target point calculation unit 76. The data processing unit 73 is mainly realized by the processing of the CPU 901 shown in FIG.

The image processing unit 74 is a function of processing an image received (acquired) by the transmission / reception unit 71. The image processing unit 74 extracts, for example, an image of a person's head included in the image data 200 acquired by the transmission / reception unit 71. Further, the image processing unit 74 extracts, for example, an image of the eyes and nose of a person acquired by the transmission / reception unit 71. The image processing unit 74 is mainly realized by the processing of the CPU 901 shown in FIG. 7.

The attention point calculation unit 75 is a function of specifying the attention points attracted by a plurality of people based on the positions and orientations of the plurality of persons included in the image data processed by the image processing unit 74. Since the function realized by the attention point calculation unit 75 is the same as that of the attention point calculation unit 42 shown in FIG. 8, the description thereof will be omitted. The attention point calculation unit 75 is mainly realized by the processing of the CPU 901 shown in FIG. 7. The attention point calculation unit 75 is an example of the calculation means. Further, the attention point calculation unit 75 is an example of the estimation means. Further, the attention point calculation unit 75 is an example of the attention point specifying means.

The target point calculation unit 76 is a function of specifying a target point to be moved to by the robot 10 based on the attention points calculated by the attention point calculation unit 75. Since the function realized by the target point calculation unit 76 is the same as that of the target point calculation unit 43 shown in FIG. 8, the description thereof will be omitted. The target point calculation unit 76 is mainly realized by the processing of the CPU 901 shown in FIG. 7. The target point calculation unit 76 is an example of the calculation means. Further, the target point calculation unit 76 is an example of the target point specifying means.

The storage / reading unit 77 is a function of storing various data in the storage unit 7000 or reading various data from the storage unit 7000. The storage / reading unit 77 is mainly realized by the processing of the CPU 901 shown in FIG. 7. Further, the storage unit 7000 is mainly realized by the ROM 902, HD904 or recording medium 906 shown in FIG. 7.

Further, the storage unit 7000 stores the image data 200 and the image data 250 received by the transmission / reception unit 71. Further, the storage unit 7000 stores the condition table 7001. Since the condition table 7001 is the same as the condition table 3002 shown in FIG. 11, the description thereof will be omitted. Further, the image data 200 and the image data 250 stored in the storage unit 7000 may be deleted when a predetermined time has elapsed since they were received by the transmission / reception unit 71, or the operation input reception unit 52 may be deleted. It may be configured to be deleted based on the deletion command from the user received by. Similarly, the image data 250 stored in the storage unit 5000 may be deleted when a predetermined time has elapsed since it was acquired by the CMOS sensor 505a, or may be transmitted to the robot 10 or the display terminal 50. The data may be deleted.

● Processing or operation of the third embodiment Next, the processing or operation of the robot in the remote control system according to the third embodiment will be described with reference to FIG. 36. In FIG. 36, the process executed by the control device 30 included in the robot 10 will be described as the process executed by the robot 10. FIG. 36 is a sequence diagram for explaining an example of robot control processing in the remote control system according to the third embodiment.

In step S301, the image acquisition unit 36 of the robot 10 acquires image data 200, which is an image acquired by the photographing device 21.

In steps S302-1, S302-2, and S302-3, the transmission / reception unit 31 of the robot 10 displays the image data 200 acquired by the image acquisition unit 36 using the communication session with the established management server 90. Send to terminal 50. Of these, the transmission / reception unit 71 of the image processing server 70 receives (acquires) image data 200, which is an image acquired by the photographing device 21 transmitted from the robot 10 (an example of an acquisition step). As a result, the transmission / reception unit 51 of the display terminal 50 receives the image data 200.

In step S303, the display control unit 53 of the display terminal 50 displays the image data 200 received by the transmission / reception unit 51 on the display screen 600 (see FIG. 18B) of the display 512. As a result, the operator who operates the robot 10 using the display terminal 50 can confirm the situation of the base where the robot 10 is located while looking at the display screen 600 on which the image data 200 is displayed.

In step S304, the image processing server 70 executes a calculation process of a target point to which the robot 10 is moved (an example of the calculation step). Since the target point calculation process by the image processing unit 74, the attention point calculation unit 75, and the target point calculation unit 76 of the image processing server 70 is the same as the process shown in FIG. 19, the description thereof will be omitted.

In step S305, the data processing unit 73 of the image processing server 70 generates request data (request information) to be transmitted to the robot 10 in order to move the robot 10 to the target point calculated by the target point calculation unit 76. In this case, the request data generated by the data processing unit 73 is the position information indicating the target position G (X, Y) calculated by the target point calculation unit 63, and the orientation (θ) of the robot 10 at the destination of the robot 10. Includes directional information indicating. The request data generated by the data processing unit 73 corresponds to the request command generated by the request command generation unit 55 of the display terminal 50.

In steps S306-1 and S306-2, the transmission / reception unit 71 of the image processing server 70 transmits the request data (request information) generated by the data processing unit 73 to the robot 10. As a result, the transmission / reception unit 31 of the robot 10 receives the request data (request information).

In step S307, the storage / reading unit 46 of the robot 10 reads out the command table 5001 stored in the storage unit 5000. In step S308, the determination unit 34 of the robot 10 searches the command table 3001 read by the storage / reading unit 46, and corresponds to the request data (request command corresponding to the request data) received by the transmission / reception unit 31. Extract the process. In this case, since the request commands corresponding to the request data received by the transmission / reception unit 31 are “GO (X, Y)” and “TURN (θ)”, the determination unit 34 searches the command table 3001 and searches for the command table 3001. Extract the processes corresponding to the request commands "GO (X, Y)" and "TURN (θ)". The extracted processes are "move the robot 10 to the position (X, Y)" and "rotate the robot 10 by θ ° to the right".

In step S309, the robot 10 executes the process extracted by step S308. Specifically, the determination unit 34 of the robot 10 notifies the movement control unit 37 of the execution request of the extracted process. Then, the movement control unit 37 moves the robot 10 based on the process extracted by the determination unit 34. Since the movement process of the robot 10 is the same as the process shown in FIG. 28, the description thereof will be omitted.

Then, the transmission / reception unit 31 of the robot 10 transmits the image data 200 captured by the imaging device 21 at the target point included in the request data transmitted from the image processing server 70 to the display terminal 50. The display terminal 50 receives the image data 200 transmitted from the robot 10. As a result, the display terminal 50 can display the image data 200 acquired by photographing the surroundings of the robot 10 at the target point included in the request data generated by the image processing server 70.

● Effects of the Third Embodiment As described above, the remote control system 1c according to the third embodiment performs the target point calculation process of the robot 10 (an example of a mobile body) via the communication network 9. The configuration is performed by the image processing server 70 capable of communicating with the display terminal 50. Therefore, the remote control system 1c according to the third embodiment can reduce the processing load of the robot 10 and the display terminal 50, and can reduce the processing load of the robot 10 and the display terminal 50 between the base where the robot 10 is located and the display terminal 50. The remote communication used can be activated.

Here, in the remote control system 1c according to the third embodiment, the image processing server 70 is an example of the information processing apparatus according to the present invention. The image processing server 70 as an information processing device includes a transmission / reception unit 71 (an example of acquisition means) for acquiring image data 200 acquired by the photographing apparatus 21, positions of a plurality of persons included in the acquired image data 200, and positions of a plurality of persons. A point of interest calculation unit 75 for calculating a target point indicating a destination of the robot 10 (an example of a moving body) and a target point calculation unit 76 (an example of a calculation means) for calculating a target point based on the orientation are provided, and the calculated target point is reached. Move the robot 10. Therefore, the image processing server 70 as the information processing device according to the third embodiment calculates the destination of the robot 10 based on the positions and orientations of a plurality of participants at one of the bases, and the robot goes to the calculated destination. By moving 10, it is possible to activate remote communication with the participants at the other base.

● Summary ●
As described above, the information processing device according to the embodiment of the present invention is information processing that adjusts the position of a robot 10 (an example of a moving body) including a photographing device 21 that photographs a subject and acquires image data 200. The device acquires the image data 200 acquired by the photographing device 21, and calculates a target point indicating the destination of the robot 10 based on the positions and orientations of a plurality of persons included in the image data 200. .. Then, the information processing apparatus according to the embodiment of the present invention moves the robot 10 to the target point by calculating the target point. Therefore, the information processing device according to the embodiment of the present invention can realize smooth communication using the robot 10.

Further, the remote control system 1a according to the embodiment of the present invention is via a robot 10 (an example of a moving body) including a photographing device 21 for photographing a subject and acquiring image data 200, the robot 10 and a communication network 9. It is equipped with a display terminal 50 that can communicate with the robot 10 and remotely control the robot 10. Then, the robot 10 is based on the position and orientation of the image acquisition unit 36 (an example of acquisition means) that acquires the image data 200 acquired by the photographing device 21 and a plurality of persons included in the acquired image data 200. , Attention point calculation unit 42 and target point calculation unit 43 (an example of calculation means) that calculate a target point indicating the destination of the robot 10, and movement control unit 37 (movement control) that moves the robot 10 to the calculated target point. An example of the means) and a transmission / reception unit 31 (an example of the first transmission means) for transmitting the image data 200 acquired by the photographing apparatus 21 at the calculated target point to the display terminal 50. Further, the display terminal 50 displays the transmission / reception unit 51 (an example of the receiving means) for receiving the image data 200 transmitted from the robot 10 and the received image data 200 on the display 512 (an example of the display means). A control unit 53 (an example of display control means) is provided. Therefore, the remote control system 1a can activate remote communication using the robot 10 between the base where the robot 10 is located and the display terminal 50.

Further, the remote control system 1b according to the embodiment of the present invention is via a robot 10 (an example of a moving body) including a photographing device 21 for photographing a subject and acquiring image data 200, and the robot 10 and a communication network 9. It is equipped with a display terminal 50 that can communicate with the robot 10 and remotely control the robot 10. Then, the display terminal 50 is based on the position and orientation of the transmission / reception unit 51 (an example of acquisition means) for acquiring the image data 200 acquired by the photographing device 21 and a plurality of persons included in the acquired image data 200. , Attention point calculation unit 62 and target point calculation unit 63 (an example of calculation means) that calculate a target point indicating the destination of the robot 10, and a request command (request information) for moving the robot 10 to the calculated target point. (Example) is provided with a transmission / reception unit 51 (an example of a second transmission means) for transmitting the robot 10. Further, the robot 10 is acquired by the movement control unit 37 (an example of the movement control means) for moving the robot 10 to the calculated target point and the shooting device 21 at the target point included in the request command transmitted from the display terminal 50. A transmission / reception unit 31 (an example of a first transmission means) for transmitting the generated image data 200 to the display terminal 50 is provided. Further, the display terminal 50 displays the transmission / reception unit 51 (an example of the receiving means) for receiving the image data 200 transmitted from the robot 10 and the received image data 200 on the display 512 (an example of the display means). A control unit 53 (an example of display control means) is provided. Therefore, the remote control system 1b can activate remote communication using the robot 10 between the base where the robot 10 is located and the display terminal 50.

Further, the remote control system 1c according to the embodiment of the present invention is via a robot 10 (an example of a moving body) including a photographing device 21 for photographing a subject and acquiring image data 200, the robot 10 and a communication network 9. A display terminal 50 that can remotely control the robot 10 and an image processing server 70 (an example of an information processing device) that can communicate with each of the robot 10 and the display terminal 50 via a communication network 9 are provided. .. Then, the image processing server 70 is based on the position and orientation of the transmission / reception unit 71 (an example of acquisition means) for acquiring the image data 200 acquired by the photographing device 21 and a plurality of persons included in the acquired image data 200. The point of interest calculation unit 75 and the target point calculation unit 76 (an example of the calculation means) for calculating the target point indicating the destination of the robot 10 and the request data (request) for moving the robot 10 to the calculated target point. An example of information) is provided with a transmission / reception unit 71 (an example of a third transmission means) for transmitting the robot 10. Further, the robot 10 is moved by the movement control unit 37 (an example of the movement control means) for moving the robot 10 to the calculated target point, and the photographing device 21 at the target point included in the request data transmitted from the image processing server 70. A transmission / reception unit 31 (an example of a first transmission means) for transmitting the acquired image data 200 to the display terminal 50 is provided. Further, the display terminal 50 displays the transmission / reception unit 51 (an example of the receiving means) for receiving the image data 200 transmitted from the robot 10 and the received image data 200 on the display 512 (an example of the display means). A control unit 53 (an example of display control means) is provided. Therefore, the remote control system 1c can activate remote communication using the robot 10 between the base where the robot 10 is located and the display terminal 50.

● Supplement ●
The functions of each embodiment can be realized by a computer-executable program described in a legacy programming language such as an assembler, C, C ++, C #, Java (registered trademark), an object-oriented programming language, or the like, and ROM, EEPROM. , EPROM (Erasable Programmable Read-Only Memory), flash memory, flexible disc, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disc, SD card, MO (Magneto-Optical disc) Can be stored in a readable recording medium such as, or distributed through a telecommunications line.

In addition, some or all of the functions of each embodiment can be implemented on a programmable device (PD) such as an FPGA (Field Programmable Gate Array), or can be implemented as an ASIC, and each embodiment can be implemented. Circuit configuration data (bitstream data) downloaded to PD to realize the functions of PD, HDL (Hardware Description Language) for generating circuit configuration data, VHDL (Very High Speed Integrated Circuits Hardware Description Language), It can be distributed by a recording medium as data described by Verilog-HDL or the like.

Although the information processing apparatus, mobile body, remote control system, information processing method and program according to one embodiment of the present invention have been described so far, the present invention is not limited to the above-described embodiment, and other embodiments are made. It can be changed within the range that can be conceived by those skilled in the art, such as addition, change or deletion of the form, and it is included in the scope of the present invention as long as the action and effect of the present invention are exhibited in any of the embodiments. be.

1a, 1b, 1c Remote control system 10 Robot (an example of a moving body)
17 Mobile mechanism 21 Imaging device 30 Control device (example of information processing device)
31 Transmission / reception unit (an example of the first transmission means)
36 Image acquisition unit (example of acquisition means)
37 Movement control unit (an example of movement control means)
39 Positioning part (an example of position detecting means)
41 Image processing unit 42 Point of interest calculation unit (an example of calculation means, an example of estimation means, an example of attention point identification means)
43 Target point calculation unit (an example of calculation means, an example of target point identification means)
45 Presence notification unit (an example of existence notification means)
50 Display terminal (example of information processing device)
51 Transmission / reception unit (example of acquisition means, example of reception means, example of second transmission means)
53 Display control unit (an example of display control means)
61 Image processing unit 62 Focus point calculation unit (example of calculation means, estimation means, focus point identification means)
63 Target point calculation unit (an example of calculation means, an example of target point identification means)
70 Image processing server (an example of information processing device)
71 Transmission / reception unit (an example of acquisition means, an example of a third transmission means)
74 Image processing unit 75 Point of interest calculation unit (an example of calculation means, an example of estimation means, an example of attention point identification means)
76 Target point calculation unit (an example of calculation means, an example of target point identification means)
90 Management server

Japanese Unexamined Patent Publication No. 2012-161851

Claims (13)

An information processing device that adjusts the position of a moving object equipped with a shooting device that shoots a subject and acquires image data.
An acquisition means for acquiring the image data acquired by the photographing apparatus, and
A point-of-focusing means for identifying a point of interest that the plurality of persons pay attention to based on the positions and orientations of the plurality of persons included in the acquired image data.
A target point specifying means for specifying a target point indicating a movement destination of the moving body based on the specified attention point is provided.
An information processing device that moves the moving body to the specified target point. The target point specifying means according to claim 1, wherein when the position or orientation of at least one of the plurality of persons changes, the target point is specified based on the changed position or orientation of the person. Information processing device. The target point includes position information indicating the target position of the destination, and the target point includes position information.
The target point specifying means is
The information processing apparatus according to claim 1 or 2 , wherein a position separated from the attention point by an average value of the distances between the attention point and the plurality of persons is specified as the target position. The target point includes direction information indicating the direction of the moving body at the moving destination.
The target point specifying means is
The information processing apparatus according to any one of claims 1 to 3 , wherein the direction of the point of interest as seen from the position of the current moving body is specified as the direction of the moving body. The information processing apparatus according to any one of claims 1 to 4 .
A moving mechanism that moves the moving body to the specified target point,
A mobile body equipped with. The moving body according to claim 5 , further
A position detecting means for detecting the current position of the moving body and
When the distance between the detected current position and the specified target point is equal to or less than the threshold value or smaller than the threshold value, the existence notification means for outputting the notification information for notifying the existence of the moving body and the presence notification means.
A mobile body equipped with. The notification information is a sound signal and is
The moving body according to claim 6 , wherein the presence notification means outputs the sound signal when the distance between the detected current position and the specified target point is equal to or greater than a threshold value or longer than the threshold value. It is a remote control system including a moving body provided with a photographing device for photographing a subject and acquiring image data, and a display terminal capable of communicating with the moving object via a communication network and remotely controlling the moving object. hand,
An acquisition means for acquiring the image data acquired by the photographing apparatus, and
A point-of-focusing means for identifying a point of interest that the plurality of persons pay attention to based on the positions and orientations of the plurality of persons included in the acquired image data.
A target point specifying means for specifying a target point indicating a destination of the moving body based on the specified attention point, and a target point specifying means .
A movement control means for moving the moving body to the specified target point, and
Remote control system with. The remote control system according to claim 8 .
The moving body is
A first transmission means for transmitting image data acquired by the photographing apparatus at the specified target point to the display terminal is provided.
The display terminal is
A receiving means for receiving the image data transmitted by the first transmitting means, and a receiving means.
A remote control system including a display control means for displaying the received image data on the display means. The remote control system according to claim 8 .
The display terminal is
A second transmission means for transmitting the request information for moving the moving body to the specified target point to the moving body is provided.
The moving body is
A first transmission means for transmitting image data acquired by the photographing apparatus at the target point included in the request information transmitted by the second transmission means to the display terminal is provided.
The display terminal further
A receiving means for receiving the image data transmitted by the first transmitting means, and a receiving means.
A remote control system including a display control means for displaying the received image data on the display means. The remote control system according to claim 10 , further
An information processing device capable of communicating with each of the mobile body and the display terminal via the communication network is provided.
The information processing device is
A third transmission means for transmitting the request information for moving the moving body to the specified target point to the moving body is provided.
The moving body is
A first transmission means for transmitting image data acquired by the photographing apparatus at the target point included in the request information transmitted by the third transmission means to the display terminal is provided.
The display terminal is
A receiving means for receiving the image data transmitted by the first transmitting means, and a receiving means.
A remote control system including a display control means for displaying the received image data on the display means. It is an information processing method executed by an information processing device that adjusts the position of a moving object equipped with a shooting device that shoots a subject and acquires image data.
The acquisition step of acquiring the image data acquired by the photographing apparatus, and
A focus point specifying step for identifying a focus point to be focused on by the plurality of persons based on the positions and orientations of the plurality of persons included in the acquired image data.
A target point specifying step of specifying a target point indicating the destination of the moving body based on the specified attention point is executed.
An information processing method for moving the moving body to the specified target point. A program that causes a computer to perform the method according to claim 12 .

Images