JP6721023B2 - Information processing device and program - Google Patents

Description

The present invention relates to an information processing device and a program.

Patent Documents 1 to 3 disclose a robot device that executes a process in cooperation with another device.

JP, 2014-188597, A JP-A-5-65766 JP, 2005-111637, A

In general, the means by which the information processing device solves the problem that has occurred is predetermined. Therefore, there are cases where the problem cannot be solved by flexibly changing the means according to the situation where the information processing apparatus is placed.

An object of the present invention is to be able to solve a problem depending on the situation.

According to the invention described in claim 1, when the problem of the situation cannot be solved only by the detection means for detecting the surrounding situation and the function of the own device, the information about the situation is provided as the problem of the situation. And a control unit that performs control for transmitting to a terminal device that is identified as the user 's terminal device that is assumed to be present, and the control unit is such that the user can understand the situation in the information regarding the situation. The information processing apparatus controls the display of a screen for the user to input additional information when the information indicating that the information cannot be received is received.

The invention according to claim 2 is the information processing apparatus according to claim 1, wherein the additional information is information related to a user request.

The invention according to claim 3 is the information processing apparatus according to claim 2, wherein the control means collects information in response to the request and controls display of the information.

In the invention according to claim 4, the control means further transmits an image of a device existing around the device itself to the terminal device, and the image of the device is displayed on the terminal device. The information processing apparatus according to any one of claims 1 to 3, which is characterized.

According to a fifth aspect of the present invention, in the control means, the image of the device is further designated by the user, and an instruction to execute a solving means for solving the problem of the situation is given by the user. The information processing apparatus according to claim 4, further comprising: transmitting an instruction to execute the execution to the device.

In the invention according to claim 6, when the problem of the situation cannot be solved only by the function of the detection means for detecting the surrounding situation of the computer and the information processing apparatus, the information about the situation is provided as the problem of the situation. Is operated as control means for performing control for transmitting to the terminal device identified as the user 's terminal device that is assumed to occur, and the control means causes the user to understand the situation in the information regarding the situation. A program that controls the display of a screen for the user to input additional information when receiving information indicating that it is not possible.

According to the invention described in claims 1 to 6 , it is possible to solve the problem depending on the situation.

It is a block diagram showing a device system concerning this embodiment. It is a figure which shows the external appearance of the robot apparatus which concerns on this embodiment. It is a block diagram which shows the robot apparatus which concerns on this embodiment. It is a block diagram which shows a terminal device. It is a figure which shows the characteristic of wireless communication technology. It is a figure which shows the characteristic of wireless communication technology. It is a figure which shows an owned function management table. It is a figure which shows a non-owned function management table. It is a figure which shows a solving means management table. It is a figure which shows a device function management table. It is a figure which shows a cooperation function management table. It is a flow chart which shows an outline of situation confirmation processing. It is a flow chart which shows the details of situation confirmation processing. It is a flowchart which shows the process in a user determination mode. It is a flow chart which shows processing in autonomous judgment mode. It is a flow chart which shows execution control of a solving means. It is a figure for demonstrating the application scene 1 of a robot apparatus. It is a figure for demonstrating the application scene 2 of a robot apparatus. It is a figure for demonstrating the application scene 3 of a robot apparatus. It is a figure for demonstrating the application scene 4 of a robot apparatus. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows the external appearance of an apparatus typically. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a flowchart which shows a connection process. It is a figure which shows a cooperation function management table. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a device function management table. It is a figure which shows a screen. It is a figure which shows a device function management table. It is a figure which shows a cooperation function management table. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen.

A device system as an information processing system according to an embodiment of the present invention will be described with reference to FIG. 1. FIG. 1 shows an example of a device system according to this embodiment.

The device system according to the present embodiment includes, for example, a robot device 10, one or a plurality of devices 12, and a terminal device 14. The robot device 10, the device 12, and the terminal device 14 may communicate with other devices via a communication path N such as a network, or may communicate with other devices via different communication paths. .. Of course, the configuration shown in FIG. 1 is merely an example, and the robot device 10, the device 12, and the terminal device 14 may not communicate with other devices.

In the example shown in FIG. 1, one device 12 is included in the device system, but a plurality of devices 12 may be included in the device system. In this case, a plurality of devices 12 having the same function may be included in the device system, or a plurality of devices 12 having different functions may be included in the device system. Further, although one terminal device 14 is included in the equipment system, a plurality of terminal devices 14 may be included in the equipment system. Other devices such as a server may be included in the equipment system.

The robot device 10 is a device having a function of detecting a situation around the robot device 10 and detecting a problem that has occurred. The robot apparatus 10 may detect the surrounding situation while moving, or may detect the surrounding situation while stopped. When a problem is detected, the robot apparatus 10 determines whether the problem can be solved by the robot apparatus 10 itself, and performs processing according to the determination result. When the problem can be solved by the robot device 10, the robot device 10 solves the problem by executing a solution for solving the problem. If the problem cannot be solved by the robot device 10, the robot device 10 performs control for executing a solving means that uses an element other than the robot device 10. The element is, for example, a person, a device other than the robot device 10, a robot device other than the robot device 10, or the like. Of course, the element may include the robot apparatus 10 itself. For example, the robot apparatus 10 may cause another device to solve the problem, may request a person to cooperate, may cause the other device and the person to solve the problem, or may You may work with others to solve the problem.

The problem detected by the robot device 10 is, for example, a problem that can be solved only by the robot device 10, a problem that can be solved only by a device other than the robot device 10, a problem that can be solved only by a person, a collaboration between another device and a person. Problems that can be solved by work, problems that can be solved by the joint work of the robot apparatus 10 and a person, problems that can be solved by the joint work of the other apparatus and the robot apparatus 10, and between the other apparatus and the other robot apparatus and the robot apparatus 10. These are problems that can be solved by collaborative work, problems that can be solved by collaborative work between other devices, people, and the robot apparatus 10. That is, the solving means is, for example, a means realized only by the robot apparatus 10, a means realized only by another device, a means realized only by a person, a means realized by a joint work between another device and a person. , Means realized by the joint work of the robot apparatus 10 and a person, means realized by the joint work of the robot apparatus 10 with another device, by the joint work of the robot apparatus 10 with another device and another robot apparatus. Means to be realized, means to be realized by collaborative work between another device, a person, and the robot apparatus 10, and the like.

The device 12 is a device having a specific function, and includes, for example, an image forming device having an image forming function, a personal computer (PC), a display device such as a liquid crystal display or a projector, an aroma device for generating a scent, a telephone, and a clock. , Cameras, surveillance cameras, vending machines, air conditioners (air conditioners, coolers, etc.), fans, humidifiers, etc. Of course, devices other than these may be included in the device system. Note that the device may be included in the category of the robot device depending on the function of the device.

The terminal device 14 is, for example, a device such as a personal computer (PC), a tablet PC, a smartphone, or a mobile phone. The terminal device 14 is used by a user, for example, at the time of executing a solution for solving a problem.

In the present embodiment, the robot apparatus 10 detects a surrounding situation, and controls for executing a solving means for solving the problem according to the problem of the situation.

Hereinafter, each device included in the device system according to the present embodiment will be described in detail.

FIG. 2 shows the appearance of the robot apparatus 10. The robot device 10 is, for example, a humanoid robot. Of course, the robot apparatus 10 may be a robot of a type other than the humanoid type. In the example illustrated in FIG. 2, the robot apparatus 10 includes a body portion 16, a head portion 18 provided on the body portion 16, a leg portion 20 provided below the body portion 16, and both side portions of the body portion 16. Each of the arm portions 22 is provided with a finger portion 24 provided at the tip of each arm portion 22.

The robot device 10 has various sensors such as a visual sensor, an auditory sensor, a tactile sensor, a taste sensor, and an olfactory sensor, for example, and has the capability of sight, hearing, touch, taste, and smell corresponding to the five senses of a person. Regarding the tactile sensation, the robot apparatus 10 has, for example, superficial sensation (tactile sensation, pain angle, temperature sensation, etc.), deep sensation (pressure sensation, position sensation, vibration sensation, etc.), cortical sensation (two-point discrimination sensation, stereoscopic discrimination capability, etc.). ) Has the ability to sort and understand. Further, the robot device 10 has a sense of balance. For example, a sensor such as a camera 26 is provided on the head 18, and vision is realized by image recognition using an image obtained by the camera 26 or the like. Further, the robot apparatus 10 is provided with a sound collecting means such as a microphone, and hearing is realized by voice recognition using sound obtained by the microphone or the like.

The robot apparatus 10 may include means for detecting a human electroencephalogram (for example, a person is provided with an electroencephalogram detection device, and means for receiving information transmitted from the electroencephalogram detection device).

The leg portion 20 corresponds to an example of a moving unit, and is driven by a driving force from a driving source such as a motor. The robot apparatus 10 can be moved by the legs 20. The leg 20 may have a shape like a human foot, a roller, a tire, or the like, or may have another shape. The leg 20 is merely an example of a moving unit, and the robot apparatus 10 may include, as a moving unit other than the leg 20, a configuration for flying (for example, a propeller, wings, an engine for flight, etc.). Alternatively, a structure for moving in water (for example, an engine for moving in water) may be provided. That is, the robot apparatus 10 may include at least one of a means for moving on land, a means for flying, and a means for moving underwater as moving means. Of course, the robot apparatus 10 does not have to include the moving means.

The robot apparatus 10 may be provided with the ability to grasp and operate an object by the arm 22 and the finger 24. The robot apparatus 10 may have the ability to move while gripping or holding an object.

The robot device 10 may have a function of emitting a voice. The robot device 10 may have a communication function and may transmit/receive data to/from another device. The robot device 10 may have the ability to communicate with a person, another device, or another robot device by producing a sound or sending a communication message.

The robot apparatus 10 may have the ability to make a judgment close to that of a person by machine learning using artificial intelligence (AI), for example. In addition, neural network type deep learning may be used, or reinforcement learning for partially strengthening the learning field may be used.

The robot apparatus 10 may have a function of searching information (for example, a solution means for solving a problem) using the Internet or the like.

The robot apparatus 10 may communicate with another device by a communication function to control the operation of the other device, operate the other device using a remote controller, or use the remote controller. Instead, other devices may be directly operated. In the direct operation, for example, the operation unit (for example, button or panel) provided in another device is operated by the robot device 10. For example, when the operation of another device cannot be controlled by communication, the robot apparatus 10 may operate the other device using a remote controller or the like, or may directly operate the other device. For example, the robot apparatus 10 analyzes the image obtained by the visual sensor to identify the operation unit and the remote controller of the other device, and operate the other device and the remote controller.

The robot device 10 may also include the display unit 28. The display unit 28 displays information regarding the problem that has occurred, information regarding the solution, various messages, and the like.

Hereinafter, the configuration of the robot apparatus 10 will be described in detail with reference to FIG. FIG. 3 is a block diagram showing the configuration of the robot apparatus 10.

The communication unit 30 is a communication interface and has a function of transmitting data to another device and a function of receiving data from another device. The communication unit 30 may be a communication interface having a wireless communication function or may be a communication interface having a wired communication function. The communication unit 30 supports, for example, one or a plurality of types of communication methods, and communicates with a communication partner according to a communication method suitable for the communication partner (that is, a communication method supported by the communication partner). The communication method is, for example, infrared communication, visible light communication, Wi-Fi (registered trademark) communication, short-range wireless communication (for example, Bluetooth (registered trademark), RFID (Radio Frequency Identifier), etc.), or the like. The communication unit 30 switches, for example, the communication method according to the communication partner, the surrounding environment (for example, the distance between the robot device 10 and the communication partner, and the obstacle between the robot device 10 and the communication partner). The communication method is switched according to the presence or absence, etc.). Further, as the communication frequency band, for example, a short wavelength band such as 800 MHz to 920 MHz (LPWA (Low Power Wide Area)) and a long wavelength band such as 2.4 GHz and 5 GHz (MuLTEfire) may be used. The communication unit 30 switches the frequency band according to the communication partner, or switches the communication system according to the surrounding environment, for example.

The storage unit 32 is a storage device such as a hard disk or a memory (for example, SSD). In the storage unit 32, for example, owned function management information 34, non-owned function management information 36, solution management information 38, device function management information 40, cooperation function management information 42, other various data, various programs, etc. Is remembered. In addition, device address information indicating addresses of other devices may be stored in the storage unit 32. The above information may be stored in different storage devices, or may be stored in the same storage device.

The owned function management information 34 is information indicating the function of the robot apparatus 10. In the owned function management information 34, for example, information indicating a function of the robot apparatus 10 and information indicating that the function can be executed (operation, processing, operation, etc.) are associated with each other. By referring to the owned function management information 34, it is specified (identified) that the robot apparatus 10 can execute the operation. Since the functions not shown in the owned function management information 34 are functions that the robot apparatus 10 does not have, by referring to the owned function management information 34, it is specified that the robot apparatus 10 cannot execute (identify). ) May be done.

The unowned function management information 36 is information indicating a function that the robot apparatus 10 does not have. In the non-owned function management information 36, for example, information indicating a function that the robot apparatus 10 does not have and information indicating that the robot apparatus 10 cannot execute because the robot apparatus 10 does not have the function correspond to each other. It is attached. By referring to the non-owned function management information 36, it is specified (identified) that the robot device 10 cannot execute. Since the function not shown in the unowned function management information 36 may be a function that the robot apparatus 10 has, it can be executed by the robot apparatus 10 by referring to the unowned function management information 36. May be specified (identified).

The storage unit 32 stores both information of the owned function management information 34 and the unowned function management information 36, and it is determined based on these information that the robot device 10 can execute or cannot execute. Alternatively, one of the pieces of information may be stored, and it may be specified that the robot apparatus 10 can execute or cannot execute based on the information.

The solution management information 38 is information indicating a solution (solution, solution action) for solving the problem that has occurred. In the solution management information 38, for example, information indicating a problem expected to occur and information indicating a solution to the problem are associated with each other.

The device function management information 40 is information for managing the functions of devices other than the robot apparatus 10, and indicates, for example, device identification information (device identification information) for identifying the devices and the functions of the devices. It is information indicating the correspondence between the function information and. The device identification information is, for example, a device ID, a device name, information indicating a device type, a device model number, information indicating a position where the device is installed (device position information), an appearance image showing an appearance of the device, Etc. The function information is, for example, a function ID or a function name. For example, when the image forming apparatus as a device has a scan function, a copy function, and a scan transfer function, the device identification information of the image forming apparatus includes function information indicating the scan function, function information indicating the copy function, and Functional information indicating the scan transfer function is associated. By referring to the device function management information 40, the function of each device is specified (identified).

The device managed by the device function management information 40 is, for example, a device (for example, the device 12) included in the device system. Of course, devices not included in the device system may be managed by the device function management information 40. For example, the robot apparatus 10 may acquire information (information including device identification information and function information) about a new device that is not included in the device system, and newly register it in the device function management information 40. The information about the device may be acquired by using the Internet or the like, or may be input by an administrator or the like. Further, the robot apparatus 10 may update the device function management information 40 at an arbitrary timing, periodically, or at a timing designated by an administrator or the like. As a result, in some cases, the device function management information 40 may be registered with the function information indicating the function that the device does not have before the update but that the device has after the update. Similarly, when the function information indicating the function which the device has before the update but which the device does not have after the update is deleted from the device function management information 40 or registered as unusable information. There is. The update information may be acquired by using the Internet or the like, or may be input by an administrator or the like.

The cooperation function management information 42 is information for managing a cooperation function executed by coordinating a plurality of functions. By linking a plurality of functions, one or a plurality of linked functions are executed. The cooperation function may be executed, for example, by coordinating a plurality of functions of one device or by coordinating a plurality of functions of a plurality of devices. Further, a device issuing an operation instruction (for example, the terminal device 14) may be included in the devices to be identified, and the function of the terminal device may be used as a part of the cooperation function, or the function of the robot device 10 may be combined. It may be used as a part of the function.

The cooperation function may be a function executed without using a device as hardware. For example, the cooperation function may be a function executed by coordinating a plurality of software. Of course, the cooperation function may be a function executed by coordinating a function of a device as hardware and a function realized by software.

The cooperation function management information 42 is, for example, information indicating a correspondence between a combination of function information indicating each function used for the cooperation function and the cooperation function information indicating the cooperation function. The cooperation function information is, for example, a cooperation function ID or a cooperation function name. When a single function is updated, the linked function management information 42 is also updated along with the update. This makes it possible to use the linked functions of multiple functions that could not be linked to each other before the update after the update, or conversely, the linked functions that were available before the update are not available after the update. It may be possible. The cooperation function information indicating the cooperation function which is available after the update is registered in the cooperation function management information 42, and the cooperation function information indicating the cooperation function which is not available after the update is deleted from the cooperation function management information 42. Or, it is registered as unusable information.

When linking a plurality of devices, the link function management information 42 is information for managing a link function that uses a plurality of functions of a plurality of devices, and identifies each device used for the link function. It is information indicating a correspondence between a combination of device identification information and cooperation function information. As described above, when the device function management information 40 is updated, the linked function management information 42 is also updated along with the update. As a result, the cooperation function of multiple devices that could not be linked to each other before the update can be used after the update, or conversely, the usable link function before the update cannot be used after the update. It may be possible.

The cooperation function may be a function executed by coordinating a plurality of different functions or may be a function executed by coordinating the same function. The cooperation function may be a function that cannot be used before the cooperation. The functions that could not be used before the cooperation may be the functions that can be used by using the same function among the functions of the device to be linked, or the functions that can be used by combining different functions. May be. For example, by cooperating a device having a print function (printer) and a device having a scan function (scanner), a copy function as a cooperation function is realized. That is, the copy function is realized by linking the print function and the scan function. In this case, the copy function as the cooperation function and the combination of the print function and the scan function are associated with each other. In the cooperation function management information 42, for example, cooperation function information indicating a copy function as a cooperation function, device identification information for identifying a device having a print function, and device identification information for identifying a device having a scan function. Are associated with each other.

The storage unit 32 may store the available function management information. The available function management information is information for managing the functions available to each user, and includes, for example, user identification information for identifying the user and function information indicating the functions available to the user (cooperative function). Information may be included). The function available to the user is, for example, a function provided to the user free of charge, a function purchased by the user, or the like, and may be a single function or a cooperative function. The user identification information is, for example, user account information such as user ID and name. The functions available to each user are identified (specified) by referring to the available function management information. The available function management information is updated, for example, every time the function is provided to the user (for example, each time the function is provided to the user without charge or for a fee).

At least one of the owned function management information 34, the non-owned function management information 36, the solution management information 38, the device function management information 40, the cooperation function management information 42, and the available function management information is information other than the robot device 10. It may be stored in a device (for example, a server, a device 12, a terminal device 14, etc., not shown). In this case, the at least one piece of information may not be stored in the storage unit 32 of the robot apparatus 10.

The situation collection unit 44 has a function of collecting information (hereinafter, referred to as “situation information”) regarding the situation around the robot apparatus 10 by various sensors. As the sensor, the above-described visual sensor, auditory sensor, tactile sensor, taste sensor, olfactory sensor, or the like is used. An image (for example, a moving image or a still image) around the robot apparatus 10 is acquired by the visual sensor and image recognition is performed. The hearing sensor collects sounds around the robot apparatus 10 (for example, a person's conversation or noise) and performs voice recognition. In addition, the temperature, humidity, odor, etc. around the robot apparatus 10 are detected. Of course, sensors other than these may be used to collect information regarding the situation around the robot apparatus 10. The situation collection unit 44 may acquire situation information from a device other than the robot device 10, a sensor, or the like.

The moving unit 46 has a function of moving the robot apparatus 10 by at least one of a means for moving on land, a means for flying, and a means for moving in water. The moving unit 46 is constituted by, for example, the leg 20 shown in FIG.

The working unit 48 has a function of operating a device other than the robot apparatus 10 and lifting or moving an object. The working unit 48 is composed of, for example, the leg unit 20, the arm unit 22 and the finger unit 24 shown in FIG.

The UI unit 50 is a user interface unit and includes a display unit (for example, the display unit 28 shown in FIG. 1) and an operation unit. The display unit is a display device such as a liquid crystal display. The operation unit is an input device such as a touch panel or a keyboard. Of course, it may be a user interface that doubles as a display unit and an operation unit (for example, a touch-type display, a device that electronically displays a keyboard or the like on the display, and the like). The robot device 10 may not include the UI unit 50, or may include a hardware key (for example, various buttons) without including the display unit. The button as a hardware key is, for example, a button specialized for inputting numbers (for example, a ten-key), a button specialized for indicating a direction (for example, a direction instruction key), and the like.

The speaker 52 has a function of outputting sound. For example, a voice (for example, a message for requesting cooperation from a person) or the like regarding a solution is output from the speaker 52.

The control unit 54 controls the operation of each unit of the robot apparatus 10. Further, the control unit 54 includes a detection unit 56, a solving means identification unit 58, a determination unit 60, a search unit 62, and an identification unit 64.

The detection unit 56 detects the situation around the robot apparatus 10 based on the situation information (for example, the values of various sensors) collected by the situation collection unit 44, and whether a problem has occurred around the robot apparatus 10. It has a function to judge whether or not the problem has occurred. The detection unit 56 detects, for example, information about a person existing around the robot apparatus 10 or information other than a person as a situation. The information about a person is, for example, an image of a person obtained by a visual sensor (for example, an image of a person's face, an image of the whole body, other images showing the movement of a person, etc.), or a person's voice collected by an auditory sensor. Etc. The information other than the person is, for example, temperature information obtained by the temperature sensor or humidity information obtained by the humidity sensor. Specifically, the detection unit 56 obtains the image obtained by the visual sensor, the sound collected by the auditory sensor, the information about the sense of touch obtained by the tactile sensor, the information about the sense of taste obtained by the taste sensor, and the information obtained by the smell sensor. By combining the information about the scent and the like, the surrounding situation is detected, and the generated problem is detected. The detection unit 56 may use information collected by a sensor that the robot device 10 does not have. For example, the detection unit 56 acquires information collected by the sensor from a sensor installed outside (for example, a sensor installed in a room or a sensor included in another device), and uses the information to solve the problem. May be detected.

The detection unit 56 may determine that a problem has occurred, for example, when the value obtained by the sensor (for example, the temperature detected by the temperature sensor) is equal to or higher than the threshold value. The threshold may be changed according to the age or sex of a person detected by various sensors included in the robot device 10. By changing the threshold according to the age and sex of the person, the problem can be detected according to the individuality of the person. That is, since the way the problem is felt may differ depending on age and sex, it is possible to detect a problem according to age and sex by changing the threshold according to age and sex.

The solution means identification unit 58 has a function of referring to the solution means management information 38 and identifying (identifying) a solution means for solving the problem detected by the detection unit 56.

Based on a detection result (corresponding to a first detection result) based on information about a person (for example, a person's image or sound) and a detection result based on information other than a person (corresponding to a second detection result) When the problem of the situation is different, the solving means identifying unit 58 may select a solving means for solving the generated problem based on the priority order of the first detection result or the second detection result which is determined in advance. .. In this case, for example, by selecting the solution means based on the first detection result, the solution means for solving the problem that is supposed to actually occur in the person is selected.

The determination unit 60 has a function of determining whether or not the robot device 10 can solve the problem detected by the detection unit 56. The problem that cannot be solved by the robot apparatus 10 is, for example, a problem that cannot be solved only by the function of the robot apparatus 10, or if the time required for solution by the function of the robot apparatus 10 is a preset time or more, it depends on the function of the robot apparatus 10. The problem is that the quality of the work result is less than or equal to the preset quality. When the solving means identified by the solving means identifying unit 58 is included in the function group of the robot apparatus 10, the determining unit 60 determines that the problem can be solved only by the robot apparatus 10. .. That is, when the robot apparatus 10 has all the functions necessary for executing the solving means, the determining unit 60 determines that the problem can be solved only by the robot apparatus 10. On the other hand, if the solving means identified by the solving means identifying unit 58 is not included in the function group of the robot apparatus 10, the determining unit 60 cannot solve the problem only by the robot apparatus 10. Judge that there is. That is, when the robot apparatus 10 does not have all or some of the functions required to execute the solving means, the determination unit 60 determines that the problem cannot be solved by the robot apparatus 10 alone. To do. The determination unit 60 may identify the function of the robot apparatus 10 (or the function that the robot apparatus 10 does not have) by referring to the owned function management information 34, for example. By referring to the unowned function management information 36, a function that the robot apparatus 10 does not have (or a function that the robot apparatus 10 has) may be specified.

When the determination unit 60 determines that the problem can be solved by the robot apparatus 10, the robot apparatus 10 executes the solution means specified by the solution means specification unit 58 under the control of the control unit 54. .. On the other hand, when the determination unit 60 determines that the robot device 10 cannot solve the problem, the control unit 54 determines a solution means that uses an element (other device or person) other than the robot device 10. Performs control for execution. For example, the control unit 54 may cause a device other than the robot device 10 to execute the solving means, or may request a person to cooperate. Of course, the robot apparatus 10 and another element (for example, a person or another device) may cooperate to execute the solving means.

The search unit 62 has a function of searching for a solution that is a solution for solving the problem detected by the detection unit 56 and is not registered in the solution management information 38. The search unit 62 searches for a solution by using the Internet or the like.

The identification unit 64 has a function of identifying a device other than the robot device 10 and identifying (specifying) a function of the device. The identification unit 64 may identify the device based on, for example, an image (for example, an image showing the appearance of the device) obtained by photographing the device with the visual sensor, or by photographing the device with the visual sensor. The device identification information provided in the device may be acquired to identify the device, or the position information indicating the position where the device is installed may be acquired to identify the device. The identification unit 64 further identifies the function of the device. For example, the identification unit 64 identifies the function information indicating the function associated with the device identification information of the identified device in the device function management information 40 stored in the storage unit 32. As a result, the function of the device is identified (specified).

Further, the identification unit 64 identifies a plurality of devices to be cooperated, and in the cooperation function management information 42 stored in the storage unit 32, a cooperation function associated with a combination of device identification information of the plurality of devices. You may specify the cooperation function information which shows. As a result, the cooperation function executed by coordinating the functions of the devices to be cooperated is identified (specified).

When the function available to the user is managed, the identification unit 64 receives the user identification information for identifying the user, and the available function management information stored in the storage unit 32 includes the user identification information. You may specify the function information which shows each function matched with. As a result, the function group available to the user is identified (specified). For example, the user identification information is transmitted from the terminal device 14 to the robot device 10, and the identification unit 64 identifies the function information indicating each function associated with the user identification information. For example, the identification unit 64 receives the device identification information and the user identification information, identifies the function information indicating the function associated with the device identification information in the device function management information 40, and determines the available function management information in the available function management information. The function information indicating the function associated with the user identification information is specified. As a result, the functions that the device specified by the device identification information has and that can be used by the user specified by the user identification information are specified.

The control unit 54 may execute the function purchase process and manage the purchase history. For example, when the pay function is purchased by the user, the control unit 54 may apply the charging process to the user.

The control unit 54 has artificial intelligence (AI) as an intelligence unit, and the function of each unit of the control unit 54 is realized by the artificial intelligence.

At least one of the detection unit 56, the solution specifying unit 58, the determination unit 60, the search unit 62, and the identification unit 64 is a device other than the robot device 10 (for example, a server, a device 12, a terminal device 14 not shown). Etc.) may be provided. in this case. The at least one unit may not be included in the control unit 54 of the robot device 10.

Hereinafter, the configuration of the terminal device 14 will be described in detail with reference to FIG. FIG. 4 shows the configuration of the terminal device 14.

The communication unit 66 is a communication interface and has a function of transmitting data to another device and a function of receiving data from another device. The communication unit 66 may be a communication interface having a wireless communication function or may be a communication interface having a wired communication function.

The camera 68 as a photographing means generates image data (for example, still image data or moving image data) by photographing a photographing target. In addition to using the camera of the terminal device 14, the communication unit 66 receives image data captured by an external camera connected to a communication path such as a network, and the UI unit 72 displays the image data. Image data may be manipulated. The terminal device 14 may not include the camera 68.

The storage unit 70 is a storage device such as a hard disk or a memory (for example, SSD), and stores various programs, various data, and the like. The storage unit 70 stores the address information of the robot apparatus 10, the address information of each device (for example, the device 12), the information about the identified device, the information about the identified cooperation target device, and the information about the function of the identified device. , Information regarding the cooperation function, and the like may be stored. These pieces of information may be stored in different storage devices, or may be stored in the same storage device.

The UI unit 72 is a user interface unit and includes a display unit and an operation unit. The display unit is a display device such as a liquid crystal display. The operation unit is, for example, an input device such as a touch panel, a keyboard, a mouse, or the like. Needless to say, it may be a user interface that also serves as a display unit and an operation unit (including, for example, a touch-type display and one that electronically displays a keyboard or the like on the display).

The control unit 74 controls the operation of each unit of the terminal device 14. The control unit 74 functions as, for example, a display control unit (control unit), and causes the display unit of the UI unit 72 to display various types of information.

On the display unit of the UI unit 72, for example, an image obtained by photographing by the robot device 10, an image photographed by the camera 68, a device to be used (for example, a device used alone or a device to be linked) is displayed. An image associated with the identified device, an image associated with the function, and the like are displayed. The image associated with the device may be, for example, an image (for example, a still image or a moving image) representing the device captured by the robot apparatus 10, or an image (for example, an icon) schematically representing the device. Alternatively, the image may be an image (for example, a still image or a moving image) representing the device captured by the camera 68. The image data schematically represented may be stored in the robot device 10 and provided to the terminal device 14 from the robot device 10, may be stored in the terminal device 14 in advance, or may be stored in another device. It may be provided to the terminal device 14 from the other device. The image associated with the function is, for example, an image such as an icon representing the function.

Hereinafter, each wireless communication technique will be described with reference to FIGS. FIG. 5 shows characteristics for each frequency as characteristics (merits and demerits) of each wireless communication technology, and FIG. 6 shows characteristics for each communication method.

As shown in FIG. 5, the main standard of the wireless communication technology having a frequency of 900 MHz is, for example, RFID. Its merit is that it is strong against obstacles and there are few interference frequency bands, and its demerits are that it has a large antenna and its communication range is short.

The main standard of the wireless communication technology having a frequency of 2.4 GHz is, for example, ZigBee (registered trademark) or Bluetooth. Its advantages include power saving, high communication speed, and small antenna, and its disadvantages include frequent interference.

Major standards of wireless communication technology with a frequency of 5 GHz are, for example, IEEE802.11a and MuLTEfire. The merit is that the interference frequency band is small and the communication speed is fast, and the demerit is that it is weak against obstacles.

As shown in FIG. 6, the advantages of infrared communication include low power consumption and easy miniaturization, and the disadvantage is that infrared rays are invisible.

Visible light communication has a merit that the communication path can be seen by visually observing visible light, and has a strong directivity as a demerit.

As a merit of the near field communication (NFC), it is easy to perform pairing between a plurality of devices, and a demerit is that it cannot be used unless a short distance is used.

When communicating with the communication partner by the wireless communication technology, the communication unit 30 of the robot apparatus 10 communicates with the communication partner by the wireless communication technology having characteristics suitable for the surrounding environment and the communication partner. More specifically, the communication unit 30 wirelessly determines, for example, the distance to the communication partner, the presence/absence of an obstacle between the robot apparatus 10 and the communication partner, the communication method supported by the communication partner, and the like. Communicate with the other party by changing the communication technology.

Hereinafter, the functions of the robot apparatus 10 will be described in detail with reference to FIGS. FIG. 7 shows an example of the owned function management table as the owned function management information 34, and FIG. 8 shows an example of the unowned function management table as the unowned function management information 36.

In the owned function management table, as an example, a management number, information indicating a function of the robot apparatus 10, and information indicating that the robot apparatus 10 can execute the function (operation, processing, operation, etc.) They are associated with each other. For example, the robot apparatus 10 has a function of lifting an object with the arm portion 22, and by the function, for example, an object having a weight of up to 30 kg can be lifted and moved. Further, the robot apparatus 10 has a function of moving, and by the function, the robot apparatus 10 can move at a variable speed within 10 km/h. By referring to the owned function management table, it is possible to identify (identify) the functions of the robot apparatus 10 and the functions that the robot apparatus 10 can execute. Further, since a function that is not registered in the owned function management table or that can be executed is a function that the robot apparatus 10 does not have or that it cannot be executed, by referring to the owned function management table, It is specified (identified) that the function 10 does not have or that it cannot be executed.

In the non-owned function management table, as an example, a management number, information indicating a function that the robot apparatus 10 does not have, and that the robot apparatus 10 cannot execute the function (operation, Information indicating processing, operation, etc.) is associated with each other. For example, the robot device 10 does not have an external cooling function (for example, a function of cooling the periphery of the robot device 10), and therefore the robot device 10 cannot cool the inside of the room. Further, the robot apparatus 10 does not have a print function (print function), and therefore, the voice heard by the robot apparatus 10 and the viewed document cannot be printed. By referring to the non-owned function management table, the functions that the robot apparatus 10 does not have and the fact that the robot apparatus 10 cannot execute are specified (identified). Further, since a function that is not registered in the non-owned function management table or an unexecutable function may be a function that the robot apparatus 10 has or an executable function, refer to the non-owned function management table. By doing so, it is possible to identify (identify) the functions that the robot apparatus 10 has and the executability.

Data of both the owned function management table and the non-owned function management table may be stored in the storage unit 32, and it may be specified that the robot device 10 can execute or cannot execute based on these data. However, either one of the data may be stored, and it may be specified that the robot device 10 can execute or cannot execute based on the data.

Hereinafter, a specific example of the solution management information 38 will be described with reference to FIG. 9. FIG. 9 shows an example of the solution management table as the solution management information 38. In the solution management table, for example, a management number, information indicating a situation (problem), and information indicating a solution to the situation (problem) (means for solving the problem) are associated with each other. Has been. The situation (problem) is detected by the detection unit 56. The solution means identification unit 58 identifies the solution means corresponding to the problem detected by the detection unit 56 in the solution means management table.

For example, when the situation (1) "the room temperature is equal to or higher than a temperature threshold value (for example, 30°C or higher)" is detected, the means for solving the situation is a means for lowering the room temperature, and specifically, , "(1) Cool with a cooler.", "(2) Open windows." The temperature of the room is detected by, for example, various sensors (for example, a temperature sensor) included in the robot device 10. The temperature threshold may be changed according to the age and sex of the people around the robot apparatus 10. For example, the detection unit 56 of the robot device 10 detects a person in the vicinity based on information (for example, image and audio information) obtained by various sensors (for example, a visual sensor and an auditory sensor), and the detected person. Estimate age, sex, etc. For example, different temperature thresholds are used when an old person (a person whose age is above the age threshold) is detected and when a young person (a person whose age is below the age threshold) is detected. You may be asked. For example, a smaller temperature threshold may be used when an older person is detected than when a younger person is detected. As a result, a solution suitable for the age is executed. Of course, the threshold may be changed for each gender. As a result, a solution suitable for gender is executed.

As another example, when the situation (3) "a person in a room is performing an action that seems to be cold" is detected, a means for solving the situation is a means for raising the temperature of the room. Specifically, "(1) Put on the stove" and "(2) Close the door." The detection unit 56 of the robot apparatus 10 detects the situation (3) by detecting the movements of people around it with various sensors. Specifically, the detection unit 56 detects a person's facial expression, the presence/absence of sweat, movements of arms and legs, or detects a person's voice based on an image representing a person's face or body motion. Therefore, the situation (3) is detected.

When a plurality of situations that contradict each other are detected, the detection unit 56 selects one situation from the plurality of situations as a priority situation, and the solution means identification unit 58 solves the priority situation. You may specify the solution for. For example, if the situation (1) "the room temperature is 30°C or higher" is detected, and the situation (3) "The person in the room is performing an action that seems to be cold" is detected, the situation (3) Is selected in preference to the situation (1), and a solution for solving the situation (3) is selected. This priority order is preset. For example, a situation detected based on information about a person is selected with priority over a situation detected based on information other than a person. The situation (3) is a situation detected based on information about a person (for example, a situation detected by analyzing a person's image or sound), and the situation (1) is based on information other than the person. This is the detected situation. By selecting the situation (3) with priority over the situation (1), the solution for the problem that is supposed to occur to the people around will be selected preferentially, which is more reliable. Can solve the problem.

Hereinafter, the device function management information 40 will be described in detail with reference to FIG. FIG. 10 shows an example of the device function management table as the device function management information 40. In the device function management table, for example, a device ID, information indicating a device name (for example, device type), information indicating a function of the device (function information), and an image ID are associated with each other. There is. The device ID and the device name correspond to an example of device identification information. The image ID is an example of image identification information for identifying an image that represents a device (for example, an image that represents the appearance of the device, an image that schematically represents the device (for example, an icon), or the like). Note that the device function management table may not include the image ID. For example, the device having the device ID “B” is a multifunction peripheral (an image forming apparatus having a plurality of image forming functions), and has functions such as a print function and a scan function. An image ID for identifying an image representing the device is associated with the device. The image data representing the device is stored in, for example, the storage unit 32 of the robot device 10 or another device.

For example, a device ID for identifying a device existing around the robot apparatus 10 is acquired by various sensors included in the robot apparatus 10, and the identification unit 64 of the robot apparatus 10 refers to the device function management table, The device name, function, and image ID associated with the device ID are specified. As a result, the devices existing around the robot apparatus 10 are identified. For example, the information indicating the device name and the data of the image indicating the device may be transmitted from the robot device 10 to the terminal device 14 and displayed on the terminal device 14. The image representing the device is displayed as an image associated with the device. The image associated with the device may be an image generated by photographing with a camera or the like, or may be an image (for example, an icon or the like) schematically showing the device. Further, in the terminal device 14, when an image associated with a device is designated by the user, information about a function of the device (for example, function information or function description information) is transmitted from the robot device 10 to the terminal device 14. It may be displayed and displayed on the terminal device 14.

Hereinafter, the cooperation function management information 42 will be described in detail with reference to FIG. FIG. 11 shows an example of the cooperation function management table as the cooperation function management information 42. In the cooperation function management table, as an example, a combination of device IDs, information indicating a device name of a cooperation target (for example, the type of each device), and information indicating a cooperation function (cooperation function information) are associated with each other. ing. For example, the device with the device ID "A" is a PC (personal computer), and the device with the device ID "B" is a multi-function peripheral. By linking the PC (A) and the multifunction peripheral (B), for example, a "scan transfer function" or a "print function" is realized as a linking function. The “scan transfer function” is a function of transferring the image data generated by the scan by the multifunction peripheral (B) to the PC (A). The “print function” is a function of transmitting data (for example, image data or document data) stored in the PC (A) to the multi function peripheral (B) and printing the data in the multi function peripheral (B).

Hereinafter, the outline of the situation confirmation processing will be described with reference to FIG. FIG. 12 is a flowchart showing the processing.

First, the situation collection unit 44 collects situation information (values of various sensors) regarding the situation around the robot apparatus 10 by using various sensors, and the detection unit 56 detects the situation around the robot apparatus 10 based on the situation information. The situation is detected, it is determined whether or not a problem has occurred around the robot apparatus 10, and the problem that has occurred is detected (S01).

The determination unit 60 determines whether the robot device 10 can solve the problem detected by the detection unit 56 (S02). As described above, the problem that cannot be solved by the robot apparatus 10 is, for example, the problem that cannot be solved only by the function of the robot apparatus 10, the problem that the time required for solution by the function of the robot apparatus 10 is a predetermined time or more, the robot For example, there is a problem that the quality of the work result by the function of the apparatus 10 becomes lower than a predetermined quality.

When the problem detected by the detection unit 56 does not correspond to the problem that cannot be solved by the robot apparatus 10 (S02, No), the robot apparatus 10 executes the solution means for solving the problem (S03). The solution is identified by the solution identifying unit 58. In this case, the robot apparatus 10 executes the solving means by its own function without using any equipment other than the robot apparatus 10 and without receiving assistance from a person. For example, when the robot device 10 causes the UI unit 50 of the robot device 10 or the UI unit 72 of the terminal device 14 to display information indicating a problem or information indicating a solving means, and an instruction to execute the solving means is given by the user, The solution may be implemented.

On the other hand, when the problem detected by the detection unit 56 corresponds to a problem that cannot be solved by the robot apparatus 10 (S02, Yes), a search process for a solution means is executed. When the solution is searched for only by the robot device 10 (S04, Yes), the robot device 10 executes the autonomous determination mode (S05). On the other hand, when the solution is not searched for only by the robot device 10 (S04, No), the robot device 10 executes the user determination mode (S06). Whether to execute the autonomous determination mode or the user determination mode is set in advance. That is, when the problem detected by the detection unit 56 corresponds to a problem that cannot be solved by the robot device 10, the robot device 10 executes a preset mode in the autonomous determination mode and the user determination mode. The mode to be executed may be designated by the user.

Hereinafter, the situation confirmation process will be described in detail with reference to FIG. FIG. 13 is a flowchart showing the processing.

First, the situation collection unit 44 collects situation information (values of various sensors) relating to the surrounding situation of the robot apparatus 10 by using various sensors (S10), and the detection unit 56 generates by combining the values of various sensors. Assume a problem (S11). The detection unit 56 assumes a problem that occurs, for example, by combining information obtained by analyzing an image obtained by photographing the surroundings and a value of a sensor other than the image. Of course, the voice of a person may be collected and the analysis result of the voice may be combined to detect the problem. For example, when the detected temperature is equal to or higher than a temperature threshold value (for example, 30° C. or higher) and the image showing the surroundings shows a person who is sweating, the detection unit 56 uses a cooler, a fan, or the like. This detects that the ambient temperature needs to be lowered. When a situation is judged using only one sensor, the judgment may be erroneous, but by combining it with an image representing the surroundings, a more complicated situation can be detected with higher accuracy. For example, by combining the image showing the surroundings and the value of the sensor, not only the uniform judgment but also the individual judgment becomes possible. Of course, by using the sensor, there is an advantage that it is possible to detect even a situation that cannot be detected by human eyes, smell, and the like.

Next, the solution means identification unit 58 assumes solution means for returning the values of various sensors to normal values (S12). For example, the solving means identifying unit 58 identifies the solving means corresponding to the problem detected by the detecting unit 56 in the solving means management table shown in FIG. The information about the solving means specified here is presented at the time of a user inquiry described later, or presented as an instruction (control) request when solving a problem using a device or person other than the robot apparatus 10. Good.

Next, the determination unit 60 determines whether or not the robot apparatus 10 can solve the problem detected by the detection unit 56. Therefore, the determination unit 60 compares the function of the robot apparatus 10 with the solution means specified by the solution means specification unit 58 (S13).

When the solution means identified by the solution means identification unit 58 is included in the functions of the robot apparatus 10, that is, when the robot apparatus 10 has a function for executing the solution means (S14). , Yes), and the process proceeds to step S03 shown in FIG. In this case, the robot apparatus 10 executes the solving means by its own function without using any equipment other than the robot apparatus 10 and without receiving assistance from a person. The robot apparatus 10 may execute the solving means when the execution instruction is given by the user.

If the solution means identified by the solution means identification unit 58 is not included in the functions of the robot apparatus 10, that is, if the robot apparatus 10 does not have the function for executing the solution means (S14). , No), the robot apparatus 10 executes the autonomous determination mode or the user determination mode (S15).

Hereinafter, the processing in the user determination mode will be described in detail with reference to FIG. FIG. 14 is a flowchart showing the processing.

First, under the control of the control unit 54, the communication unit 30 of the robot device 10 transmits the status information (values of various sensors) collected by the status collection unit 44 to the terminal device 14 (S20). At this time, the communication unit 30 may switch the communication method according to the presence or absence of an obstacle, the distance between the robot device 10 and the terminal device 14, and the like. The communication unit 30 may, for example, send the status information to the terminal device 14 registered in advance as the destination, send the status information to the terminal device 14 identified by the robot device 10, or the robot. The status information may be transmitted to the terminal device 14 that has requested the device 10 to communicate. For example, the identification unit 64 of the robot device 10 acquires the device identification information of the terminal device 14 from the image obtained by photographing the terminal device 14 with the visual sensor, and identifies the terminal device 14 based on the device identification information. You may. As another example, the communication unit 30 includes the terminal device 14 existing within a preset range based on the position of the robot device 10 (for example, the terminal device 14 existing within a range in which short-range wireless communication is possible). ), situation information may be transmitted. As still another example, the identification unit 64 of the robot device 10 may identify a user who is assumed to have a problem and transmit the status information to the terminal device 14 that the user possesses.

The robot apparatus 10 may further collect the situation information and transmit it to the terminal apparatus 14 in response to a request from the user of the terminal apparatus 14 (S21). For example, the robot device 10 additionally captures an image of the surroundings by a visual sensor such as a camera while moving, and transmits the image (moving image or still image) obtained by the image capture to the terminal device 14 or the user instructs. The data of the performed items (for example, temperature) is collected again and retransmitted to the terminal device 14.

Next, the robot apparatus 10 presents to the user one or a plurality of solving means (one or a plurality of solving means specified by the solving means specifying unit 58) for solving the problem detected by the detecting unit 56 ( S22). For example, the communication unit 30 of the robot device 10 transmits the information indicating the solving means to the terminal device 14 under the control of the control unit 54. In this case, the information indicating the solving means is displayed on the UI unit 72 of the terminal device 14. As another example, the control unit 54 of the robot apparatus 10 may display information indicating the solving means on the UI unit 50 of the robot apparatus 10 itself.

When the user selects a solution means to be executed from the one or more solution means presented to the user, the control unit 54 of the robot apparatus 10 causes the solution means selected by the user to be the robot apparatus 10. It is determined whether or not the solution means uses a device other than the above (for example, the device 12) (S23). For example, when the user selects a solving means using the terminal device 14, information indicating the solving means is transmitted from the terminal device 14 to the robot device 10, and the control unit 54 of the robot device 10 is based on the information. , It is determined whether or not the solution means selected by the user is a solution means that uses a device other than the robot apparatus 10. Information indicating the solving means is displayed on the UI unit 50 of the robot apparatus 10, and when the user selects the solving means using the UI unit 50, the control unit 54 determines according to the selection.

When the solution means selected by the user corresponds to the solution means that uses a device other than the robot apparatus 10 (S23, Yes), the communication unit 30 of the robot apparatus 10 is controlled by the control unit 54. The information about the device capable of realizing is transmitted to the terminal device 14 (S24). For example, the control unit 54 identifies one or a plurality of devices having a function capable of realizing the solution means selected by the user by referring to the device function management table and the cooperation function management table. For example, when “print” is selected as the solving means, the multi-function peripheral having the printing function is specified as the device that realizes the solving means. The information about the device may include, for example, an appearance image showing the appearance of the device, address information of the device for connecting to the device, information indicating the specifications of the device, and the like. As another example, the control unit 54 of the robot apparatus 10 may display information regarding the device on the UI unit 50 of the robot apparatus 10. In this case, the information about the device may not be transmitted to the terminal device 14.

Next, the solution means is executed by the operation of the user (S25). That is, the function of the device for realizing the solving means is executed. As a function for realizing the solving means, a function of a single device may be executed, or a cooperation function using a function of a plurality of devices may be executed. Of course, the function of the robot apparatus 10 may be used as the function for realizing the solving means. The function execution instruction for the device may be given from the robot device 10 or from the terminal device 14. Operations and the like for executing the function will be described in detail later.

On the other hand, when the solution selected by the user does not correspond to the solution using the device other than the robot apparatus 10 (S23, No), the robot apparatus 10 cooperates with the people around it to execute the solution. Is requested (S24). At this time, the robot apparatus 10 informs the user of the procedure for executing the solving means (procedure of work to be performed by the user) (S25). For example, the control unit 54 of the robot apparatus 10 causes the speaker to generate the procedure as voice, displays the procedure on the UI unit 50 of the robot apparatus 10, or moves to a place where a person is present and touches the person. If the procedure is difficult to convey to people, the voice may be increased. The communication unit 30 of the robot device 10 may transmit information indicating the procedure to the terminal device 14. In this case, the UI section 72 of the terminal device 14 displays information indicating the procedure.

It should be noted that when the user selects the solution means realized by both the device and the person, the above steps S24 to S27 are executed. Also in this case, the function of the robot device 10 may be used as a part of the solution means.

Hereinafter, the processing in the autonomous determination mode will be described in detail with reference to FIG. FIG. 15 is a flowchart showing the processing.

First, the control unit 54 of the robot apparatus 10 specifies, as necessary, an additional function or a support request (a request for support from the user) necessary to realize the solution means specified by the solution means specification unit 58 ( S30).

The control unit 54 of the robot apparatus 10 determines whether or not the solution means identified by the solution means identification unit 58 is a solution means that uses a device other than the robot device 10 (for example, the device 12) (S31).

When the solving means identified by the solving means identifying unit 58 corresponds to the solving means using a device other than the robot apparatus 10 (S31, Yes), the control unit 54 of the robot apparatus 10 determines the solving means in the surroundings. An executable device is searched for (S32). The control unit 54 searches for a device, for example, based on the image obtained by the visual sensor, the position information of each device, the wireless communication status, and the like. For example, the control unit 54 identifies one or a plurality of devices having a function capable of realizing a solving means by referring to the device function management table and the cooperation function management table. For example, when the solution is “print”, the control unit 54 of the robot apparatus 10 searches for a multifunction machine having a print function.

When there is a device capable of executing the solving means in the surroundings (S33, Yes), the communication unit 30 of the robot apparatus 10 under the control of the control unit 54 gives an instruction to execute the solving means (a function for realizing the solving means. Information indicating the execution instruction) is transmitted to the device (S34). The device that has received the information indicating the execution instruction executes the solution according to the instruction.

The control unit 54 of the robot apparatus 10 may check whether or not the control right of the device capable of executing the solving means can be acquired. That is, the control unit 54 confirms whether address information of the device or a controllable driver is stored in the robot apparatus 10. When the driver or the like can be acquired using the network or the like, the control unit 54 executes the download of the driver or the like. The robot apparatus 10 may give an execution instruction to the device by directly operating the operation panel of the device, or may give an execution instruction to the device by operating a remote controller of the device.

Whether or not the solution means has been successfully completed is determined by whether or not the problem has been resolved after the device executes the function. When the detection unit 56 does not detect the problem, the control unit 54 determines that the problem has been solved. When the problem has not been solved, the control unit 54 of the robot apparatus 10 gives an instruction to execute the solution to the device again or searches for another solution.

When there is no device capable of executing the solving means in the vicinity (S33, No), the control unit 54 of the robot apparatus 10 searches for another solving means (S35). At this time, the control unit 54 may switch the mode to the user determination mode.

When the solving means specified by the solving means specifying unit 58 does not correspond to the solving means using the device other than the robot apparatus 10 (S31, No), the robot apparatus 10 executes the solving means, and therefore the surrounding persons need to know. Request cooperation (S36). At this time, the robot apparatus 10 informs the user of the details of the cooperation request (for example, the procedure for executing the solving means) (S37). For example, the control unit 54 of the robot apparatus 10 causes the speaker to indicate the content of the cooperation request as voice, displays the content of the cooperation request on the UI unit 50 of the robot apparatus 10, moves to the place where the person is, and touches the person. Or The communication unit 30 of the robot device 10 may transmit information indicating the cooperation request content to the terminal device 14. In this case, the information indicating the cooperation request content is displayed on the UI unit 72 of the terminal device 14.

When the cooperation is received (S38, Yes), the process ends. In this case, the user executes the solution. When the cooperation is not received (S38, No), the control unit 54 of the robot apparatus 10 searches for another solution (S35). The control unit 54 may switch the mode to the user determination mode. The control unit 54 may determine whether or not the cooperation has been received, for example, by recognizing the user's reply by voice recognition. For example, when the reply to the cooperation is recognized by voice, the control unit 54 determines that the cooperation is accepted. As another example, when an operation supposed to be performed by the user to implement the solution is executed within a preset time and the operation is detected by various sensors (for example, a visual sensor), The control unit 54 may determine that the cooperation was received.

It should be noted that, as time passes, the surrounding situation also changes, so the problems that occur may also change. In this case, the robot apparatus 10 detects a situation (problem) that changes every moment, and identifies a solution means according to the detection result.

Further, the problem that can be solved by the robot device 10 is changed according to the update of the function of the robot device 10. The update of the function is performed, for example, by changing at least one of the hardware and the software included in the robot apparatus 10.

The solving means identifying unit 58 preferentially identifies a solving means that is not executed by a person and is executed by using a device (a solving means that does not require human cooperation), rather than a solving means executed by a person. You may. Solutions performed by people are not always performed by people. Therefore, by preferentially identifying a solution that does not require human cooperation, a solution that can be executed more reliably is identified. In addition, when the person is not detected, the solution means identification unit 58 may identify the solution means that does not require the person's cooperation, without searching for the solution means that requires the person's cooperation. This reduces the load required to search for a solution.

Hereinafter, the execution control of the solving means will be described in detail with reference to FIG. FIG. 16 is a flowchart showing the control. In the following, it is assumed that the robot device 10 cannot solve the problem.

First, the control unit 54 of the robot apparatus 10 includes the control of the device in the solution means selected by the user in the user determination mode described above or the solution means specified by the robot apparatus 10 in the autonomous determination mode described above. It is determined whether or not (S40). That is, the control unit 54 determines whether or not the above-mentioned solving means corresponds to the solving means using the device. In other words, the control unit 54 determines whether the above-mentioned solving means is a solving means realized only by a person without using a device.

When the solution means does not include control of the device (S40, No), that is, when the solution means corresponds to the solution means realized only by a person without using the device, the robot apparatus 10 solves the problem. In order to execute the means, it asks the surrounding people for cooperation (S41). For example, the communication unit 30 of the robot device 10 transmits, to the terminal device 14, information indicating the content of the cooperation request (for example, the procedure for executing the solving means). In this case, the information indicating the cooperation request content is displayed on the UI unit 72 of the terminal device 14. At this time, the communication unit 30 of the robot device 10 transmits information indicating the contents of the cooperation request to the terminal device 14 by using the communication method according to the surrounding environment (S42). The surrounding environment is, for example, the distance between the robot device 10 and the terminal device 14, the presence or absence of an obstacle between the robot device 10 and the terminal device 14, and the like. The control unit 54 of the robot apparatus 10 causes the speaker to indicate the cooperation request content as voice, displays the cooperation request content on the UI unit 50 of the robot apparatus 10, moves to a place where a person is, and touches the person. May be. The control unit 54 of the robot apparatus 10 observes whether or not the operation supposed to be performed by the user to implement the solving means is executed within a preset time (S43). The control unit 54 determines whether the problem has been resolved based on the observation (S44). When the supposed operation is executed within a preset time and the operation is detected by various sensors (for example, a visual sensor), the control unit 54 of the robot apparatus 10 determines that the problem has been resolved ( S44, Yes). In this case, the process ends. On the other hand, when the expected operation is not executed within the preset time and the operation is not detected by the various sensors, the control unit 54 of the robot apparatus 10 determines that the problem has not been resolved (S44, No). ). In this case, the robot apparatus 10 may search for another solution or notify the user of the same cooperation request content (S45).

When the solution means includes control of the equipment (S40, Yes), that is, when the solution means corresponds to the solution means that uses the equipment, the control unit 54 of the robot apparatus 10 is used as the solution means. It is determined whether the device to be used corresponds to a device that can be used free of charge (S46). Specifically, the control unit 54 of the robot apparatus 10 searches (identifies) a device capable of executing the solving means in the surroundings, and determines whether the device corresponds to a device that can be used free of charge. The information indicating free or paid is managed for each device. For example, the device function management table may manage that each device can be used for free or for a fee. As another example, the control unit 54 may acquire, from the identified device, information indicating whether the crisis is a device that can be used for free or for a fee.

When the device used for the solution does not correspond to the device that can be used free of charge (S46, No), that is, when the device corresponds to the device that can be used for a fee, the control unit 54 of the robot apparatus 10 concerned. It is determined whether the robot device 10 has a payment means corresponding to the device (S47). The payment means is, for example, electronic money (electronic currency), virtual currency, cash, credit card, or the like. Of course, payment may be made by means other than these. When the robot apparatus 10 does not have the payment means (S47, No), the control unit 54 of the robot apparatus 10 searches for another solution means (S48). When the robot device 10 has a payment means (S47, Yes), the process proceeds to step S49. In this case, the control unit 54 of the robot apparatus 10 controls the payment operation for using a device that can be used for a fee. That is, the robot device 10 pays by the payment means when using the device. If the robot device 10 does not have a payment means, the robot device 10 receives payment assistance from at least one of the devices other than the robot device 10 and the person (for example, borrow money from the person or the device). The payment operation may be performed (by receiving or handing over).

Next, the control unit 54 of the robot apparatus 10 determines whether or not the device used as the solving means corresponds to the device controllable via the communication function (S49). The information indicating whether control is possible via the communication function is managed for each device. For example, the device function management table may manage whether or not each device can be controlled via the communication function.

When the device used as the solving means corresponds to the device that can be controlled via the communication function (S49, Yes), the control unit 54 of the robot apparatus 10 communicates with the device (the device). Communication system) is selected (S50). The communication method supported by each device is managed for each device. For example, in the device function management table, the communication method supported by each device may be managed. If a communication error occurs even when the device communicates with the device according to the communication method supported by the device, the processes in and after step S57 may be executed. If the device supports a plurality of communication methods, the robot device 10 may attempt communication with the device using each of the plurality of communication methods. In this case, the robot device 10 may communicate with the device using an optimal communication method (for example, a communication method with the highest communication speed, a communication method with the least noise, etc.) among the plurality of communication methods.

In addition, the control unit 54 of the robot apparatus 10 acquires address information, an access password, etc. of the device used as the solving means (S51). The control unit 54 of the robot device 10 may acquire the address information or the like from a device such as a server that stores the address information or the like, or may acquire the address information or the like via the Internet or the like. As another example, the address information and the like may be stored in the robot device 10. Further, when the driver of the device is required to control the device used for the solving means, the control unit 54 of the robot device 10 acquires the driver and installs the driver in the robot device 10 (S52). The control unit 54 of the robot device 10 may acquire the driver from a device such as a server that stores the driver, or may acquire the driver via the Internet or the like.

Next, the communication unit 30 of the robot apparatus 10 transmits information indicating an execution instruction of the solving means (an execution instruction of a function for executing the solving means) to a device used by the solving means (S53). The device that has received the information indicating the execution instruction executes the solution according to the instruction.

The control unit 54 of the robot apparatus 10 observes whether the device solves the problem (S54), and based on the observation, determines whether the problem has been solved (S55). For example, when an operation that is supposed to be performed by the device to implement the solution is performed by the device within a preset time and the operation is detected by various sensors (for example, a visual sensor), The control unit 54 of the robot apparatus 10 determines that the device solves the problem (S55, Yes). In this case, the process ends. On the other hand, when the expected operation is not executed within the preset time and the operation is not detected by the various sensors, the control unit 54 of the robot apparatus 10 determines that the problem has not been resolved (S55, No). ). In this case, the robot apparatus 10 may search for another solution, or may send the same execution instruction to the device (S56). The control unit 54 of the robot apparatus 10 may determine whether another new solving means can be executed. For example, when the temperature of the room is high and a fan is newly installed in the room as a device other than the cooler, the control unit 54 may search for a new solution using the fan. Further, the search unit 62 may search for a new solution by using the Internet or the like.

When the device used as the solving means does not correspond to the device controllable via the communication function (S49, No), the robot apparatus 10 searches for a remote controller for operating the device (S57). The robot apparatus 10 identifies the remote controller by analyzing an image obtained by a visual sensor, for example.

When the remote controller is not found (S58, No), the robot apparatus 10 searches for another solution (S59).

When the remote controller is found (S58, Yes), the robot apparatus 10 operates the remote controller to input an instruction to execute the solving means (S59). The device that has received the instruction executes the solution according to the instruction.

The control unit 54 of the robot apparatus 10 observes whether or not the device solves the problem, and based on the observation, determines whether or not the problem has been resolved (S61). For example, when an operation that is supposed to be performed by the device to implement the solution is executed by the device within a preset time and the operation is detected by various sensors (for example, a visual sensor), The control unit 54 of the robot apparatus 10 determines that the device solves the problem (S61, Yes). In this case, the process ends. On the other hand, when the expected operation is not executed within the preset time and the operation is not detected by the various sensors, the control unit 54 of the robot apparatus 10 determines that the problem has not been resolved (S61, No). ). In this case, the robot apparatus 10 may search for another solution, or may transmit the same execution instruction to the device using the remote controller (S62).

Hereinafter, the application scene of the robot apparatus 10 according to the present embodiment will be described in detail.

(Application scene 1)
Application scene 1 will be described with reference to FIG. FIG. 17 shows a person, the robot device 10, and the like. For example, it is assumed that the person 76 (three people) is having a meeting and the robot apparatus 10 is present at the meeting.

The situation collection unit 44 of the robot apparatus 10 collects surrounding situation information by various sensors. The situation collection unit 44 uses, for example, a voice indicating the conversation of the surrounding person 76, an image indicating the person 76 (for example, an image indicating the face of the person 76 or an image indicating the whole body), temperature, humidity, and the like as the situation information. collect. The detection unit 56 of the robot apparatus 10 detects the surrounding situation based on the situation information (for example, the state (conversation, facial expression, attitude, etc.) of the person 76, temperature, etc.). For example, when the person 76 says “I want you to write down a memo and write on paper”, the situation collection unit 44 of the robot apparatus 10 collects the voice information of the conversation as situation information, and the detection unit of the robot apparatus 10. Based on the conversation, 56 determines whether a problem has occurred. In the above statement, the problem of "printing the content of the statement on paper" is detected.

The solving means identifying unit 58 refers to the solving means management information 38 and identifies a solving means for solving the problem. This problem can be solved by, for example, combining a "function of collecting a statement as voice information and converting the content of the statement into a character string" and a "print function". That is, the solution to this problem is configured by a combination of the "function of collecting utterances as voice information and converting the utterance contents into a character string" and the "print function".

Here, it is assumed that the robot device 10 has a “function of collecting utterances as voice information and converting the utterance contents into a character string”, and does not have a “print function”. In this case, the above problem cannot be solved only by the robot device 10. To deal with this, the robot apparatus 10 searches for a device having a print function. For example, the robot device 10 photographs a peripheral device by a visual sensor (for example, a camera), and the identifying unit 64 of the robot device 10 identifies the photographed device by analyzing an image obtained by the photographing. .. For example, assume that the multi-function peripheral 78 is installed around the robot apparatus 10 and has a print function. The identification unit 64 of the robot apparatus 10 identifies the multifunction device 78 and identifies the function (for example, the printing function) of the multifunction device 78. In this case, the solving means is executed by the robot device 10 and the multifunction device 78. This solving means is a solving means realized by a cooperation function using the robot apparatus 10 and the multi-function peripheral 78, that is, a solving means realized by cooperation between the robot apparatus 10 and the multi-function peripheral 78. For example, the robot apparatus 10 may automatically execute the solving means, or may execute the solving means in response to an execution instruction from the user. When executing the solving means, the robot apparatus 10 communicates with the multi-function peripheral 78, sends information indicating the content of the statement of the person 76 to the multi-function peripheral 78, and gives the multi-function peripheral 78 an instruction to print the information. As a result, the content of the statement made by the person 76 is printed on the paper.

The robot apparatus 10 may move to the multi-function peripheral 78, acquire the sheet on which the content of the statement is printed, and give it to the user. Further, the robot device 10 may notify the user that the solution means has been executed, by voice or the like.

In the example shown in FIG. 17, the robot apparatus 10 is used as the solving means, but the solving means may use the equipment other than the robot apparatus 10 without using the robot apparatus 10. In this case, the robot apparatus 10 gives an instruction to execute the solving means to a device other than the robot apparatus 10.

The robot apparatus 10 may directly operate the multifunction device 78. For example, when the detection unit 56 of the robot apparatus 10 detects a problem such as "I want you to copy a sheet", the solution means identification unit 58 of the robot apparatus 10 identifies the "copy function" as the solution means. In this case, the robot apparatus 10 searches for a device having a copy function (for example, the multifunction device 78) and causes the multifunction device 78 to copy the paper. For example, the robot apparatus 10 receives a sheet from a user, sets the sheet in the multifunction device 78, and directly operates the operation panel of the multifunction device 78 to give a copying instruction to the multifunction device 78. The robot device 10 identifies an operation panel or the like by analyzing an image acquired by a visual sensor, for example, and gives a copy instruction.

As described above, even when a problem that cannot be solved by only the robot apparatus 10 is detected, it is possible to solve the problem by using another device. As a result, it is possible to widen the range of problems that the robot apparatus 10 can solve in cooperation depending on the situation.

(Application scene 2)
Application scene 2 will be described with reference to FIG. FIG. 18 shows the user, the robot device 10, and the like. For example, assume that the person 80 is lying down. The situation collecting unit 44 of the robot apparatus 10 collects an image of the person 80 who is lying down as situation information, and the detecting unit 56 of the robot apparatus 10 determines whether or not a problem occurs based on the image. .. In the example shown in FIG. 18, the problem of “a person is falling” is detected.

The solving means identifying unit 58 refers to the solving means management information 38 and identifies a solving means for solving the problem. This problem can be solved by, for example, "seeking a person to help the fallen person". That is, the solution to this problem includes "to ask a person for help." In this case, the robot apparatus 10 asks a person 82 other than the person 80 who is lying down for help. Therefore, the robot apparatus 10 may generate a voice, or may move to the person 82 and contact the person 82. At this time, the control unit 54 of the robot device 10 may display the information indicating the procedure for rescue on the UI unit 50 of the robot device 10 or on the terminal device 14 carried by the person 82. Good. For example, the robot device 10 may identify the terminal device 14 carried by the person 82 by various sensors, and may transmit information indicating a procedure for rescue to the identified terminal device 14.

In addition, in the case where the solving means specifying unit 58 specifies the solving means of “carrying a fallen person to a safe place”, if the robot apparatus 10 has a function of carrying an object, the robot apparatus 10 is The person 80 may be carried only by the robot apparatus 10, the person 80 may be carried in collaboration with another person, the person 80 may be carried in collaboration with another device, or the person 80 may be carried together with another device. You may carry the person 80 together. The other device is a device identified by the robot apparatus 10. If the robotic device 10 does not have the ability to carry objects, the robotic device 10 may instruct the person 82 to carry the fallen person 80 to a safe location. The information indicating the instruction may be displayed on the UI unit 50 of the robot device 10 or may be transmitted to the terminal device of the person 82 and displayed on the terminal device.

As described above, when a problem that cannot be solved by only the robot apparatus 10 is detected, it is possible to request the cooperation of a person and solve the problem.

(Application scene 3)
Application scene 3 will be described with reference to FIG. FIG. 19 shows a user, the robot device 10 and the like. For example, when the person 84 says "I'm thirsty", the situation collecting unit 44 of the robot apparatus 10 collects the voice information of the statement as the situation information, and the detecting unit 56 of the robot apparatus 10 makes the statement. Based on this, it is determined whether a problem has occurred. In the statement above, the problem of being "thirsty" is detected.

The solving means identifying unit 58 refers to the solving means management information 38 and identifies a solving means for solving the problem. This problem can be solved by, for example, "buying a drink". That is, the solution to this problem is constituted by the "function of providing drinks".

Here, it is assumed that the robot apparatus 10 does not have the “function of providing drinks”. In this case, the above problem cannot be solved only by the robot device 10. To deal with this, the robot apparatus 10 searches for a device that provides a drink. For example, the robot apparatus 10 photographs a peripheral device with a visual sensor, and the identification unit 64 of the robot device 10 identifies the photographed device by analyzing the image obtained by the photographing. For example, it is assumed that a drinking water vending machine 86 (an example of a device) is installed around the robot apparatus 10. The identification unit 64 of the robot device 10 identifies the vending machine 86 and identifies the function of the vending machine 86 (for example, the function of providing drinks for a fee). The identification unit 64 of the robot device 10 may identify the payment method (for example, electronic money or currency) supported by the vending machine 86 based on the image obtained by the visual sensor, or the vending machine 86. May be communicated with to identify the payment method supported by the vending machine 86. The control unit 54 of the robot apparatus 10 prepares a payment method supported by the vending machine 86.

When the robot apparatus 10 has a payment means (for example, electronic money, currency, credit card, etc.) corresponding to the vending machine 86, the control unit 54 of the robot apparatus 10 controls the payment operation. As a result, the robot device 10 purchases the drink by paying by the payment means in the vending machine 86. For example, the robot device 10 moves to the vending machine 86 and directly operates the vending machine 86 to purchase a drink. The robot device 10 identifies the purchase button or the like of the vending machine 86 and analyzes the image acquired by the visual sensor to purchase the drink. The robotic device 10 may deliver the drink to the person 84. The robot apparatus 10 may purchase the drink according to the user's instruction, or may purchase the drink without receiving the user's instruction. When the purchase process is performed according to a user's instruction, the control unit 54 of the robot apparatus 10 causes the UI unit 50 of the robot apparatus 10 to display information indicating a solution (for example, information indicating that a drink is to be purchased). When the user gives a purchase instruction using the UI unit 50, the robot apparatus 10 pays by the payment means and purchases the drink. The information indicating the solving means may be transmitted to the terminal device 14 of the user and displayed on the terminal device 14. When the user gives a purchase instruction using the terminal device 14, information indicating the purchase instruction is transmitted from the terminal device 14 to the robot device 10, and the robot device 10 purchases a drink according to the purchase instruction.

If the robotic device 10 does not have a payment instrument corresponding to the vending machine 86, the payment operation may be performed with payment assistance from at least one of the person 84 and other equipment. .. For example, the robot apparatus 10 may receive money for payment from a person who may be involved in an action for solving a problem, purchase a drink by the money, or pay the payment means from another device having a payment means. You may borrow and perform a payment operation. A person who may be involved in the action for solving the problem is, for example, a person 84 who says “thirsty”, and this person 84 is identified by various sensors of the robot apparatus 10.

(User judgment mode 1)
Hereinafter, an example of the user determination mode (user determination mode 1) will be described in detail. FIG. 20 shows a person, a robot device 10, and the like. For example, a person 87 (three people) is having a conversation, and a plurality of devices (for example, a multifunction device 78, a projector 88, a camera 90, a display 92, and an aroma device 94) are installed around the robot apparatus 10. There is.

The situation collection unit 44 of the robot apparatus 10 collects surrounding situation information by various sensors, and the detection unit 56 detects the surrounding situation based on the situation information. Further, the identification unit 64 identifies the devices existing in the vicinity. For example, it is assumed that the detection unit 56 has detected a situation (problem) that “three people are in a meeting and somehow rubs each other”. Further, in the example shown in FIG. 20, the identification unit 64 identifies the multifunction device 78, the projector 88, the camera 90, the display 92, and the aroma device 94. The identification unit 64 may identify the terminal device 14 carried by a person (for example, a person 87 having a problem).

When the determination unit 60 of the robot apparatus 10 determines that the problem detected by the detection unit 56 cannot be solved by the robot apparatus 10, the communication unit 30 of the robot apparatus 10 controls the robot apparatus 10 under the control of the control unit 54. The status information is transmitted to the terminal device 14 registered in advance or the terminal device 14 identified by the identification unit 64 (for example, the terminal device 14 possessed by the person 87 having the problem). The status information includes information indicating the status (problem) detected by the detection unit 56, information indicating the device identified by the identification unit 64, and the like.

A screen explaining the situation is displayed on the UI unit 72 of the terminal device 14. FIG. 21 shows an example of the screen. A situation explanation screen 96 is displayed on the UI unit 72 of the terminal device 14. The situation explanation screen 96 displays information indicating the situation (problem) detected by the detection unit 56, information indicating the device identified by the identification unit 64, and the like. In the example shown in FIG. 21, a character string indicating “three people are having a meeting and rubbing something” is displayed as the situation (problem), and “around the periphery, a multifunction peripheral, a projector, a camera, a display, and an aroma machine”. Is displayed.” is displayed.

When the user uses the terminal device 14 to request additional information, the information indicating the request is transmitted from the terminal device 14 to the robot device 10. The communication unit 30 of the robot device 10 transmits the additional information to the terminal device 14 in response to the request. For example, when the user requests an image representing the surrounding situation as the additional information, the communication unit 30 of the robot apparatus 10 transmits the image data representing the surrounding situation to the terminal device 14.

The communication unit 30 of the robot apparatus 10 represents the surrounding situation, for example, image data associated with the person 87 who is assumed to have a problem, image data associated with the device identified by the identification unit 64, and the like. The image data is transmitted to the terminal device 14. The image data associated with the person 87 may be image data generated by photographing with a visual sensor or may be image data that schematically represents the person. The image data associated with the device may be image data obtained by shooting with a visual sensor (camera) when the identification unit 64 identifies the device, or image data (image data schematically showing the identified device. For example, it may be an icon). The image data schematically represented may be stored in the robot device 10 in advance, or may be stored in a device such as a server in advance and transmitted to the robot device 10.

The communication unit 30 of the robot device 10 transmits the image data as additional information (for example, the image data associated with the person 87 and the image data associated with the device) to the terminal device 14. The image is displayed on the UI unit 72 of the terminal device 14. FIG. 22 shows an example of the image. A situation explanation screen 98 is displayed on the UI unit 72 of the terminal device 14. A group of images as additional information is displayed on the situation explanation screen 96. For example, an image 100 associated with the person 87, a device image 102 associated with the multifunction device 78, a device image 104 associated with the projector 88, a device image 106 associated with the camera 90, a device image 108 associated with the display 92, and an aroma. A device image 110 associated with the device 94 is displayed.

When the user specifies a device image on the situation explanation screen 98 and gives an execution instruction of a solving means using a device associated with the device image, information indicating the execution instruction is transmitted to the device. The device executes a solution for solving the problem detected by the detection unit 56 according to the execution instruction. The information indicating the execution instruction may be transmitted from the terminal device 14 to the device, or may be transmitted from the robot device 10 to the device.

For example, it is assumed that the device images 108 and 110 are designated by the user. In this case, the solving means identifying unit 58 refers to the solving means management information 38 to solve the problem "three people have a meeting and rub something" detected by the detecting unit 56. That is, a solution means that uses the display 92 associated with the device image 108 and the aroma device 94 associated with the device image 110 is specified. In other words, the solution means identification unit 58 refers to the solution management information 38 to find a solution means for solving the problem "three people have a meeting and rub something" detected by the detection unit 56. By identifying and referring to the device function management information 40 and the cooperation function management information 42, a device group used for each identified solution means (a device group having a function used to execute each solution means) is identified. Identify. When the display 92 and the aroma device 94 are designated by the user, the solution determining unit 58 selects the solution using them.

When the user designates the display 92 and the aroma device 94, an execution instruction screen 112 is displayed on the UI unit 72 of the terminal device 14, as shown in FIG. 23, for example. On the execution instruction screen 112, a device image (for example, device image 108, 110) designated by the user is displayed, and information indicating a function (solving means) that can be executed by using the device designated by the user is displayed. To be done. For example, a solution that can be executed by using the display 92 and the aroma device 94 is to “display a healing image on the display 92 and cause the aroma device 94 to utter a healing scent”. This solving means is a cooperation function that can be executed by coordinating the display 92 and the aroma device 94, and this cooperation function is registered in the cooperation function management information 42. When the user gives an instruction to execute the solving means using the terminal device 14, information indicating the execution instruction is transmitted from the terminal device 14 to the display 92 and the aroma unit 94. Of course, the information indicating the execution instruction may be transmitted to the display 92 and the aroma machine 94 via the robot apparatus 10. The display 92 that has received the information indicating the execution instruction displays a healing image, and the aroma device 94 that has received the information indicating the execution instruction utters a healing scent. The healing image data may be stored in advance in the robot apparatus 10 and may be transmitted from the robot apparatus 10 to the display 92, or may be stored in an apparatus such as a server and transmitted from the apparatus such as a server to the display 92. May be done.

The screens shown in FIGS. 21 to 23 may be displayed on the UI unit 50 of the robot apparatus 10. In this case, each screen may not be displayed on the UI unit 72 of the terminal device 14. Further, the information displayed on the screen may be output as audio information.

The operation of selecting the device that executes the solving means will be described in detail later.

(User judgment mode 2)
Hereinafter, another example of the user determination mode (user determination mode 2) will be described in detail.

When the determination unit 60 of the robot apparatus 10 determines that the problem detected by the detection unit 56 cannot be solved by the robot apparatus 10, the communication unit 30 of the robot apparatus 10 is controlled by the terminal device 14 under the control of the control unit 54. Send status information.

For example, as shown in FIG. 24, a notification screen 114 is displayed on the UI unit 72 of the terminal device 14. The notification screen 114 displays a message indicating that a problem (situation) that cannot be solved by the robot apparatus 10 has occurred.

When the user presses the “Yes” button on the notification screen 114, the screen transitions to the next screen. For example, as illustrated in FIG. 25, the situation explanation screen 116 is displayed on the UI unit 72 of the terminal device 14. On the situation explanation screen 116, an explanation sentence of the situation (problem) detected by the robot apparatus 10 is displayed.

On the situation explanation screen 116, a message asking the user whether or not he/she understands the situation is displayed. When the user presses the “No” button, it becomes possible to make an inquiry to the robot apparatus 10, that is, request additional information. In this case, as shown in FIG. 26, the inquiry screen 118 is displayed on the UI unit 72 of the terminal device 14. On the inquiry screen 118, the user can use the terminal device 14 to input the inquiry content (for example, the user additionally wants to know). In the example shown in FIG. 26, the user inputs inquiry contents such as "I want data of XX." and "I want to see a video of ∆." The information indicating the inquiry content is transmitted from the terminal device 14 to the robot device 10. The situation collecting unit 44 of the robot apparatus 10 collects information (for example, image and voice data) according to the inquiry, and the communication unit 30 of the robot apparatus 10 collects the information collected by the situation collecting unit 44 from the terminal device 14 Send to. The additionally acquired information is displayed on the situation explanation screen 116 of the UI unit 72 of the terminal device 14.

When the user presses the “Yes” button on the situation explanation screen 116, the solution means display screen 120 is displayed on the UI unit 72 of the terminal device 14 as shown in FIG. 27. On the solving means display screen 120, information indicating the solving means identified by the solving means identifying unit 58 of the robot apparatus 10 (explanatory text, name, etc. of the solving means) is displayed. For example, as described above, when the problem of “three people are having a meeting and rubbing something” is detected, the solution means identification unit 58 solves the problem by referring to the solution means management information 38. Identify a solution for doing so. The information indicating the identified solving means is transmitted from the robot device 10 to the terminal device 14 and displayed on the UI unit 72 of the terminal device 14. In the example shown in FIG. 27, four solution means proposals (1) to (4) are specified as recommended solution means for solving the above problems, and information indicating them is displayed. The solution plans may be displayed in a random order, may be displayed in an order expected to have a higher effect in solving the problem, or may be displayed in an order determined by ease of execution. May be displayed in. For example, a solution means that uses a device located closer to the terminal device 14 or the robot device 10 has higher easiness of execution, and the solution means is displayed higher. The position information of the robot device 10, the terminal device 14, and each device can be obtained by using, for example, GPS (Global Positioning System). The control unit 54 of the robot device 10 calculates the distance between the terminal device 14 or the robot device 10 and each device by using the position information obtained by using the GPS. The above ranking is determined based on the calculation result.

When the solution display screen 120 is displayed on the UI unit 72 of the terminal device 14 and the user gives an instruction for screen transition or when a preset time elapses, as shown in FIG. The screen 122 is displayed on the UI unit 72 of the terminal device 14. A message such as “Please select a solution” is displayed on the screen 122 (for example, a message prompting the user to select a solution). Further, when the proposed solution means (see FIG. 27) does not include an appropriate solution means (for example, a solution means that the user desires to execute), the user can give another instruction. It is like this.

For example, if the proposed solution (see FIG. 27) includes an appropriate solution and the user uses the terminal device 14 to select the appropriate solution, the solution shown in FIG. 29 is displayed. As described above, the confirmation screen 124 is displayed on the UI unit 72 of the terminal device 14. On the confirmation screen 124, information indicating the solving means selected by the user is displayed. In the example shown in FIG. 29, the solution means (2) “display a healing image on the display” and (3) “provide a citrus scent with an aroma device” are selected by the user. When the user presses the “Yes” button on the confirmation screen 124, the information indicating the execution instruction of the solving means (2) and (3) is displayed from the terminal device 14 or the robot device 10 of the solving means (2) and (3). It is transmitted to the device used for execution (for example, the display 92 and the aroma device 94). As a result, the display 92 executes the solution (2), and the aroma unit 94 executes the solution (3).

If the user presses the "No" button on the confirmation screen 124, the screen returns to the previous screen 122 (see FIG. 28).

If the proposed solution means (see FIG. 27) does not include an appropriate solution, the user may specify a solution other than the proposed solution solution. When the user uses the terminal device 14 to give an instruction to display an input screen for designating a solution, the user input screen 126 is displayed on the UI unit 72 of the terminal device 14 as shown in FIG. 30, for example. Is displayed. On the user input screen 126, the user instructs a device used as a solution and a solution using the device. In the example shown in FIG. 30, a multi-function peripheral is designated as the device used as the solving means, and "XX" processing is designated as the solving means using the multifunction peripheral. The user may specify the device used for the solving means by inputting the device name in characters, or specify the device used for the solving means by specifying the device image associated with the device. You may. For example, the UI unit 72 of the terminal device 14 displays one or more device images associated with the one or more devices identified by the robot device 10, and the user selects one or more device images from among the one or more device images. A device image associated with the device used as the solving means is selected. By thus specifying the device by the device image, it is possible to specify the device used for the solving means without knowing the detailed name of the device. Also, when the device used for the solving means is designated by the user, a list of functions of the device is displayed on the UI unit 72 of the terminal device 14. The function of the device is specified by, for example, referring to the device function management information 40. The user selects a function used for the solving means from the function list. When the user uses the terminal device 14 to give an instruction to execute the solving means, information indicating the execution instruction is transmitted to the device designated by the user, and the device executes the solving means.

The screens shown in FIGS. 24 to 30 may be displayed on the UI unit 50 of the robot apparatus 10. In this case, each screen may not be displayed on the UI unit 72 of the terminal device 14. Further, the information displayed on the screen may be output as audio information.

In the user determination modes 1 and 2 described above, the information displayed on the UI unit 72 of the terminal device 14 is transmitted from the robot device 10 to the terminal device 14, for example. Of course, the information may be transmitted from the device such as the server to the terminal device 14 under the control of the robot device 10.

(Device identification process)
The device identification process will be described below. As an example, by applying AR (Augmented Reality) technology (augmented reality technology), device identification information is acquired and a device is identified. For example, by applying the AR technology, the device identification information of the device that is used alone is acquired and the device is identified, and the device identification information of the device that is the cooperation target is acquired and the device that is the cooperation target is identified. To be done. AR technology known as AR technology is used. For example, a marker type AR technique using a marker such as a two-dimensional barcode, a markerless type AR technique using an image recognition technique, a position information AR technique using position information, etc. are used. Of course, the device identification information may be acquired and the device may be identified without applying the AR technology. For example, if the device is connected to the network, the device may be identified based on the IP address, or the device ID may be read to identify the device. Further, in the case of a device or a terminal device having various wireless communication functions such as infrared communication, visible light communication, Wi-Fi, and Bluetooth, the device is obtained by acquiring the device ID of the device to be linked using these wireless communication functions. May be identified and the cooperation function may be executed.

Hereinafter, the acquisition processing of the device identification information will be described in detail with reference to FIG. As an example, a case will be described in which the multifunction device 78 is installed around the robot device 10 and the robot device 10 acquires the device identification information of the multifunction device 78. FIG. 31 schematically shows the external appearance of the multi-function peripheral 78. Here, a process for acquiring the device identification information by applying the marker type AR technology will be described. A marker 128 such as a two-dimensional barcode is provided on the housing of the multi-function peripheral 78. The marker 128 is information in which the device identification information of the multifunction device 78 is coded. The robot apparatus 10 captures an image of the marker 128 with a visual sensor. As a result, image data representing the marker 128 is generated. In the robot apparatus 10, the control unit 54 extracts the device identification information by applying the decoding process to the marker image represented by the image data. As a result, the multifunction device 78 is identified. The identification unit 64 of the robot apparatus 10 identifies, in the device function management information 40, function information indicating a function associated with the extracted device identification information. As a result, the function of the multifunction device 78 is specified (identified).

The marker 128 may include coded function information indicating the function of the multi-function peripheral 78. In this case, by applying the decoding process to the image data representing the marker 128, the device identification information of the multifunction peripheral 78 is extracted and the functional information indicating the function of the multifunction peripheral 78 is also extracted. As a result, the multifunction device 78 is specified (identified) and the function of the multifunction device 78 is specified (identified).

When the device identification information is acquired by applying the markerless AR technology, for example, the robot apparatus 10 captures the whole or a part of the external appearance of the device (for example, the multi-function device 78) using a visual sensor. Of course, it is useful to obtain information for identifying the device, such as the device name (for example, product name) and the model number, from the external appearance. Appearance image data representing the whole or part of the appearance of the device is generated by photographing. In the robot device 10, the control unit 54 identifies the device to be used based on the appearance image data. For example, the storage unit 32 of the robot apparatus 10 stores, for each device, appearance image correspondence information indicating correspondence between appearance image data representing all or part of the appearance of the device and device identification information of the device. Remembered The control unit 54 compares, for example, the appearance image data obtained by photographing with each appearance image data included in the appearance image association information, and identifies the device identification information of the device to be used based on the comparison result. To do. For example, the control unit 54 extracts the appearance feature of the device from the appearance image data obtained by shooting, and in the appearance image data group included in the appearance image association information, determines the same or similar feature as the appearance feature. The appearance image data to be represented is specified, and the device identification information associated with the appearance image data is specified. As a result, the device (for example, the multifunction device 78) is identified. As another example, when a device name (for example, product name) or model number is photographed and appearance image data representing the name or model number is generated, based on the name or model number represented in the appearance image data. May identify the device. In the device function management information 40, the identification unit 64 of the robot apparatus 10 identifies the function information indicating each function associated with the identified device identification information. As a result, the function of the device (for example, the multifunction device 78) is specified.

When the device identification information is acquired by applying the position information AR technology, for example, by using the GPS function, the position information indicating the position where the device (for example, the multifunction device 78) is installed is acquired. For example, each device has a GPS function and acquires device position information indicating the position of the device itself. The robot apparatus 10 outputs information indicating a device position information acquisition request to the device, and receives device position information of the device from the device as a response to the acquisition request. In the robot device 10, the control unit 54 identifies the device based on the device position information. For example, the storage unit 32 of the robot apparatus 10 stores, for each device, position correspondence information indicating correspondence between device position information indicating a position where the device is installed and device identification information of the device. ing. The control unit 54 identifies the device identification information associated with the device location information in the location association information. As a result, the device is specified (identified). In the device function management information 40, the identification unit 64 of the robot apparatus 10 identifies the function information indicating each function associated with the identified device identification information. As a result, the function of the device (for example, the multifunction device 78) is specified (identified).

For example, when the multifunction device 78 is identified by the robot device 10, in the user determination mode 1 described above, the device image associated with the multifunction device 78 is displayed on the terminal device 14 as status information. For example, as shown in FIG. 32, the device image 102 associated with the multi-function peripheral 78 is displayed on the UI unit 72 of the terminal device 14. The device image 102 may be, for example, an image generated by photographing with a visual sensor of the robot device 10, or may be an image that schematically represents the multifunction machine 78.

Further, when the device is identified, information indicating the name of the device may be transmitted from the robot device 10 to the terminal device 14, and the device name may be displayed on the UI unit 72 of the terminal device 14. In the example shown in FIG. 32, the name “multifunction machine (B)” is displayed.

For example, when the user specifies the device image 102 using the terminal device 14, for example, as shown in FIG. 33, the UI unit 72 of the terminal device 14 uses a multifunction device 78 linked to the device image 102 in the UI unit 72. Is displayed (for example, a button image for instructing the execution of the solution). The multifunction peripheral (B) has, for example, a print function, a scan function, a copy function, and a facsimile function. When these functions are used as a solution, button images for executing these functions are displayed. It is displayed on the UI unit 72 of the terminal device 14. For example, when the user uses the terminal device 14 to specify a button image representing a print function and gives an instruction to execute the print function, execution instruction information indicating an instruction to execute the print function is output from the terminal device 14 or the robot device 10. Machine 78. The execution instruction information includes control data for executing the print function, data such as image data to which the print function is applied, and the like. Upon receiving the execution instruction information, the multi-function peripheral 78 executes printing according to the execution instruction information.

When a plurality of devices (for example, the multifunction device 78 and the projector 88) are identified by the robot device 10, in the user determination mode 1 described above, the device image associated with the multifunction device 78 and the device image associated with the projector 88 are The information is displayed on the terminal device 14 as information. For example, as shown in FIG. 34, a device image 102 associated with the multifunction device 78 and a device image 104 associated with the projector 88 are displayed on the UI unit 72 of the terminal device 14. The device images 102 and 104 may be, for example, images generated by photographing with a visual sensor of the robot device 10, or may be images that schematically represent the multifunction machine 78 and the projector 88.

When the device is identified, the information indicating the name of the device may be transmitted from the robot device 10 to the terminal device 14 and displayed on the UI unit 72 of the terminal device 14. In the example shown in FIG. 34, the name “multifunction device (B)” of the multifunction device 78 and the name “projector (C)” of the projector 88 are displayed.

For example, when the user uses the terminal device 14 to specify the device images 102 and 104, for example, as shown in FIG. 35, the UI unit 72 of the terminal device 14 includes the multifunction device 78 and the device image associated with the device image 102. Information (for example, a button image for instructing the execution of the solving means) indicating the solving means using the projector 88 associated with 104 is displayed. The solving means is a means realized by a cooperation function using the multifunction device 78 and the projector 88. By making the multifunction device 78 and the projector 88 cooperate with each other, for example, a cooperation function of projecting an image generated by scanning by the multifunction device 78 by the projector 88, or a cooperation function of printing an image projected by the projector 88 by the multifunction device 78. The function becomes executable. Button images for instructing execution of these cooperation functions are displayed on the UI unit 72 of the terminal device 14. For example, when the user uses the terminal device 14 to specify a button image and gives an instruction to execute the solution means (cooperation function), the execution instruction information indicating the execution instruction of the solution means is sent from the terminal device 14 or the robot device 10. It is transmitted to the multi-function device 78 and the projector 88. Upon receiving the execution instruction information, the multi-function device 78 and the projector 88 execute the cooperation function designated by the user.

The user touches the device image 102 and operates the device image 104 up to the device image 104 with a finger or the like (for example, traces with a finger or the like), whereby the device images 102 and 104 are designated, and the multifunction device 78 and the projector 88 are targeted for cooperation. It may be designated as a device. The order in which the device images 102 and 104 are touched and the tracing direction may be opposite to the above example. Of course, screen contact means other than a finger such as a pen for tracing the screen may be used. The user may connect the device image 102 and the device image 104 to specify the device images 102 and 104, and specify the multifunction device 78 and the projector 88 as the cooperation target devices. The device images 102 and 104 may be designated by the user overlapping the device image 102 and the device image 104, and the multifunction device 78 and the projector 88 may be designated as devices to be linked. A device to be linked may be specified by a drawing operation such as a circle, or a device to be linked may be specified by specifying a device image associated with the device to be linked within a preset time. Good. When canceling the cooperation, the user may specify the device to be canceled on the screen, or may press the cooperation cancel button. The device to be released may be designated by a preset operation such as adding a cross mark.

As another example, the cooperation target device may be set in advance as a basic cooperation device. For example, it is assumed that the multi-function device 78 is preset as a basic cooperation device. The device identification information of the basic cooperation device may be stored in advance in a device such as the robot device 10 or a server. The user may specify the basic cooperation device by using the terminal device 14. When the basic cooperation device is set, the user specifies the device image associated with the device other than the basic cooperation device, thereby specifying the device that cooperates with the basic cooperation device.

In the above example, the standalone function and the cooperative function are functions that use the device as hardware, but the standalone function and the cooperative function may be functions realized by software (application). .. For example, instead of the device image, a function image associated with a function realized by software (for example, an image such as an icon) is displayed on the UI unit 72 of the terminal device 14, and one or more function images are designated by the user. By doing so, a function associated with the functional image or a cooperative function using a plurality of functions associated with a plurality of functional images may be designated. Of course, the device image associated with the device as hardware and the function image associated with the function realized by the software are displayed on the UI unit 72 of the terminal device 14, and the device image and the function image are designated by the user. In this case, a cooperation function that uses the device associated with the device image and the function associated with the function image may be designated.

Hereinafter, a process for executing a function of the device will be described. As an example, a process when executing the cooperation function will be described. In this case, a connection request is transmitted from the terminal device 14 to the cooperation target device, and the terminal device 14 and the cooperation target device are connected. Of course, a connection request may be transmitted from the robot apparatus 10 to the cooperation target device, and the robot apparatus 10 and the cooperation target device may be connected. Hereinafter, this connection processing will be described with reference to FIG. FIG. 36 is a sequence diagram showing the processing.

When the user specifies the cooperation function to be executed in the terminal device 14 (S70), the terminal device 14 sends a connection request to the cooperation target device (for example, the multifunction device 78 and the projector 88) that executes the cooperation function. Information is transmitted (S71). For example, when the address information indicating the address of the cooperation target device is stored in the robot device 10, the terminal device 14 acquires the address information of the cooperation target device from the robot device 10. As another example, the terminal device 14 may store address information of devices to be cooperated. Of course, the terminal device 14 may acquire the address information of the cooperation target device by another method. The terminal device 14 uses the address information of the cooperation target device (for example, the multifunction device 78 and the projector 88) to transmit information indicating a connection request to the cooperation target device (for example, the multifunction device 78 and the projector 88).

Upon receiving the information indicating the connection request, the multifunction device 78 and the projector 88 allow or disallow the connection with the terminal device 14 (S72). For example, if the multifunction device 78 and the projector 88 correspond to devices that are not permitted to connect, or if the number of devices requesting the connection exceeds the upper limit, the connection is not permitted. When the connection from the terminal device 14 is permitted, the changing operation may be prohibited so that the unique setting information of the multifunction device 78 and the projector 88 is not changed from the terminal device 14. For example, changes in the color parameters of the multi-function peripheral 78, the set time when shifting to power saving, and the like may be prohibited. As a result, security for the device to be linked is improved. As another example, when the devices are made to cooperate with each other, the change of the setting information may be restricted as compared with the case where the devices are used alone without being made to cooperate. For example, as compared with the case where the device is used alone, fewer setting items may be permitted to be changed. Further, browsing of personal information of other users such as operation history may be prohibited. This improves the security of the personal information of the user.

Result information indicating permission or non-permission of connection is transmitted from the multifunction device 78 and the projector 88 to the terminal device 14 (S73). When the connection to the multifunction device 78 and the projector 88 is permitted, the communication is established between the terminal device 14 and the multifunction device 78 and the projector 88.

Next, the user uses the terminal device 14 to instruct execution of the cooperation function (S74). In response to this instruction, execution instruction information indicating an instruction to execute the cooperation function is transmitted from the terminal device 14 to the multifunction device 78 and the projector 88 (S75). The execution instruction information transmitted to the multi-function peripheral 78 includes information (for example, job information) indicating the process executed by the multi-function peripheral 78, and the execution instruction information transmitted to the projector 88 includes the projector 88. Information (e.g., job information) indicating the processing to be executed is included.

Upon receiving the execution instruction information, the multi-function peripheral 78 and the projector 88 execute the function according to the execution instruction information (S76). For example, like the function of transferring scan data from the multi-function peripheral 78 (multi-function peripheral (B)) to the projector 88 (PC (A)) and projecting the data by the projector 88, the multi-function peripheral 78 and the projector 88 are included in the cooperation function. When the process of transmitting and receiving data to and from is included, communication is established between the multi-function device 78 and the projector 88. In this case, for example, the execution instruction information transmitted to the multifunction device 78 includes the address information of the projector 88, and the execution instruction information transmitted to the projector 88 includes the address information of the multifunction device 78. Communication is established between the multifunction device 78 and the projector 88 using the information.

When the execution of the cooperation function is completed, the information indicating the completion of the execution of the cooperation function is transmitted from the multifunction device 78 and the projector 88 to the terminal device 14 (S77). Information indicating that the execution of the cooperation function is completed is displayed on the UI unit 72 of the terminal device 14 (S78). If the information indicating the completion of execution is not displayed even after the preset time has elapsed from the time when the execution instruction is given, the terminal device 14 causes the UI unit 72 to display the information indicating the error, and the execution instruction is executed again. The information or the information indicating the connection request may be transmitted to the multifunction device 78 and the projector 88.

Next, the user confirms whether or not to cancel the linked state of the multi-function device 78 and the projector 88 (S79), and the process according to the presence/absence of cancellation is executed (S80). When canceling the linked state, the user gives a cancellation instruction using the terminal device 14. As a result, the communication between the terminal device 14 and the multifunction device 78 and the projector 88 is canceled. Similarly, the communication between the multi-function peripheral 78 and the projector 88 is canceled. When the linked state is not released, the execution instruction may be continuously given.

When the single function is executed, the information indicating the instruction to execute the single function is transmitted from the terminal device 14 to the device that executes the single function. The device executes a single function according to the execution instruction.

The execution instruction information may be transmitted from the robot apparatus 10 to each device.

Hereinafter, a process related to a designation operation of a device used as a solving means will be described.

(Switching process for displaying information related to linkage function)
In the present embodiment, the display of the information regarding the cooperation function may be switched according to the order in which the device images associated with the devices are connected. Hereinafter, this processing will be described in detail with reference to FIGS. 37 to 39.

FIG. 37 shows a cooperation function management table as another example of the cooperation function management information 42. In the cooperation function management table, as an example, information indicating a combination of device IDs, information indicating a device name of a cooperation target (for example, the type of each device), information indicating a cooperation function (cooperation function information), and connection The information indicating the order and the information indicating the priority order are associated with each other. The connection order corresponds to the order in which the device images associated with the devices are connected. The priority order is the display priority order of information related to the cooperation function. For example, the device with the device ID "A" is a PC (personal computer), and the device with the device ID "B" is a multi-function peripheral. By linking the PC (A) and the multi-function peripheral (B), for example, a "scan transfer function" and a "print function" are realized as the linking functions. The “scan transfer function” is a function of transferring the image data generated by the scan by the multifunction peripheral (B) to the PC (A). The “print function” is a function of transmitting data (for example, image data or document data) stored in the PC (A) to the multi function peripheral (B) and printing the data in the multi function peripheral (B). For example, when a device is connected from the multi-function peripheral (B) to the PC (A), that is, when a device image is connected from the device image associated with the multi-function peripheral (B) to the device image associated with the PC (A). , The priority of the “scan transfer function” is “first”, and the priority of the “print function” is “second”. In this case, the information regarding the “scan transfer function” is displayed with priority over the information regarding the “print function”. On the contrary, when a device is connected from the PC (A) to the multifunction device (B), that is, the image is linked from the device image associated with the PC (A) to the device image associated with the multifunction device (B). If so, the priority of the “print function” is “first” and the priority of the “scan transfer function” is “second”. In this case, the information regarding the “print function” is displayed with priority over the information regarding the “scan transfer function”.

38 and 39 show examples of screens displayed on the UI unit 72 of the terminal device 14. For example, it is assumed that the multifunction peripheral (B) and the PC (A) are identified. In the user determination mode 1 described above, as shown in FIG. 38A, the UI unit 72 of the terminal device 14 has a device image 102 associated with the multi-function peripheral (B) and a device image associated with the PC (A). 130 is displayed as status information. In this state, the user connects the device images representing the devices to be linked with each other using an indicator (for example, the user's finger, pen, stylus, etc.). The control unit 74 of the terminal device 14 detects the contact of the indicator on the screen and the movement of the indicator on the screen. For example, as indicated by an arrow 132, the user touches the device image 102 on the screen with the operator to operate up to the device image 130 (for example, tracing the operator on the screen), and the device image 102 is displayed. Connect the device images 130. As a result, the multifunction device (B) associated with the device image 102 and the PC (A) associated with the device image 130 are designated as devices to be linked and the order of connection is designated. The order of connecting device images corresponds to the order of connection. The multifunction device (B) corresponds to the first device, and the PC (A) corresponds to the second device. In the example shown in FIG. 38A, since the images are connected from the device image 102 to the device image 130, the device is connected from the multifunction peripheral (B) to the PC (A). The information indicating the connection order of the devices is transmitted from the terminal device 14 to the robot device 10. The control unit 74 of the terminal device 14 may display an image representing the trajectory traced by the user on the screen, or after the devices are connected to each other, replace the trajectory with a preset straight line or the like. It may be displayed on the screen.

As described above, when the devices to be linked (for example, the multifunction device (B) and the PC (A)) are designated, the identification unit 64 of the robot device 10 displays the linked function management table shown in FIG. The cooperation function associated with the combination of PC (A) and multifunction peripheral (B) is specified. As a result, the cooperation function executed by coordinating the PC (A) and the multifunction machine (B) is specified. Further, when the user specifies the connection order of the devices, the identification unit 64 specifies the priority order associated with the connection order in the cooperation function management table. Explaining with a specific example with reference to FIG. 37, since the PC (A) and the multi-function peripheral (B) are designated as devices to be linked, the link function executed by them is the “scan transfer function”. It is a "print function". Further, since the device is connected from the multi-function peripheral (B) to the PC (A) (B→A), the priority order of the “scan transfer function” is “first place” and the priority order of the “print function” is "2nd place".

The information regarding the cooperation function identified as described above and the information indicating the priority order are transmitted from the robot device 10 to the terminal device 14. The control unit 74 of the terminal device 14 causes the UI unit 72 to display information regarding the cooperation function in accordance with the priority order.

For example, as shown in FIG. 38(b), the control unit 74 of the terminal device 14 causes the UI unit 72 to display information regarding the cooperation function candidate. Since the priority of the "scan transfer function" is "1st" and the priority of the "print function" is "2nd", the information about the "scan transfer function" has priority over the information about the "print function". (For example, at the top). For example, as the information regarding the “scan transfer function”, a description of “scan transfer function” of “transfer data scanned by the multifunction device (B) to the PC (A)” is displayed. Further, as the information regarding the “print function”, a description of “print function” of “print data in PC(A).” is displayed.

When the cooperation function is specified by the user and an execution instruction is given, the specified cooperation function is executed. For example, when the “YES” button is pressed by the user, the link function associated with the “YES” button is executed.

It should be noted that the terminal device 14 may perform the cooperation function specifying process and the priority order specifying process.

As an operation other than tracing the device images with the operator, the device to be linked may be specified by a drawing operation such as adding a circle, and the connection order may be specified. For example, the order of drawing operations corresponds to the connection order. As another example, the device to be linked and the connection order may be designated according to the voice instruction from the user.

FIG. 39 shows an example of another operation. For example, as shown in FIG. 39(a), the user touches the device image 130 using an operator and operates the device image 102 in the direction indicated by the arrow 134, thereby connecting the device image 130 and the device image 102. .. As a result, the PC (A) associated with the device image 130 and the multifunction device (B) associated with the device image 102 are specified as devices to be linked, and the order of connection is specified. In this example, since the image is connected from the device image 130 to the device image 102, the device is connected from the PC (A) to the multifunction device (B). Referring to the cooperation function management table shown in FIG. 37, the priority of the “print function” is “first” and the priority of the “scan transfer function” is “second”. In this case, as shown in FIG. 39( b ), the UI unit 72 of the terminal device 14 displays the information about the “print function” preferentially (for example, higher) than the information about the “scan transfer function”. ..

As described above, by linking the device images associated with the device, the cooperation function that uses the function of the device is specified. Further, the display order of the information regarding the cooperation function is changed according to the order of connecting the images, that is, the order of connecting the devices. The device connection order is also the order of the functions used by each device and the order of moving data that moves between linked devices. The operation of connecting the devices (that is, the operation of connecting images) is the order of the functions. It also serves as an operation to specify the data transfer order. Therefore, by changing the display order of the information on the cooperation function according to the connection order, the information on the cooperation function predicted to be used by the user is preferentially displayed. That is, the information regarding the cooperation function that is likely to be used by the user is preferentially displayed. For example, when an image is connected from the multi-function peripheral (B) to the PC (A), "the function of the multi-function peripheral (B) is used before the PC (A), and the multi-function peripheral (B) to the PC (A) is used. It is predicted that the user will use the cooperation function of "transferring data to". Further, when an image is connected from the PC (A) to the multifunction device (B), "the function of the PC (A) is used before the multifunction device (B), and the PC (A) to the multifunction device (B) is used. It is expected that the user will use the cooperation function of "transferring data to". Therefore, by changing the display order of the information regarding the cooperation function according to the order in which the images are connected, the information regarding the cooperation function that is likely to be used by the user is preferentially displayed. Further, the order of the functions to be used and the order of moving the data are designated without performing an operation other than the operation of connecting the device images, and the information on the cooperation functions predicted to be used by the user is displayed.

The above display switching process may be applied when a functional image associated with a function is used. For example, the display of the information regarding the cooperation function is switched according to the order of designation of the device image associated with the first function and the device image associated with the second function.

(Collaboration process using partial images)
Depending on the position in the device image associated with the device, the device assigned to the cooperation function may have a different function. When a specific position is designated by the user in the device image, information regarding the cooperation function that uses the function corresponding to the specific position is preferentially displayed. Hereinafter, this process will be described in detail.

FIG. 40 shows an example of the device function management table. The data of the device function management table is stored in the robot apparatus 10 as the device function management information 40. In the device function management table, as an example, a device ID, information indicating a device name (for example, device type), information indicating a position in the device image, and information indicating a function corresponding to the position in the device image ( (Function information) and the image ID are associated with each other. The position in the device image is a specific position (specific part) in the device image that is associated with the device. For example, a specific position in the device image that schematically represents the device or in the device image captured by the camera. Is a specific position. Different functions are associated with specific positions in the device image.

FIG. 41 shows an example of a screen displayed on the UI unit 72 of the terminal device 14. For example, it is assumed that the multifunction peripheral (B) and the PC (A) are identified. In the user determination mode 1 described above, as shown in FIG. 41A, the device images 102 and 130 are displayed as status information on the UI unit 72 of the terminal device 14. For example, a “print function” is assigned to a specific position (partial image 102a) corresponding to the main body of the multifunction peripheral (B) in the device image 102. In the device image 102, a “scan function” is assigned to a specific position (partial image 102b) corresponding to the document cover, the document glass, or the automatic document feeder of the multifunction peripheral (B). A “stapler stopping function” is assigned to a specific position (partial image 102c) corresponding to the post-processing apparatus in the device image 102. The "stapler stop function" is a function of stopping the output paper by a stapler. Further, in the device image 130, a “data saving function” is assigned to a specific position (partial image 130a) corresponding to the main body of the PC (A). A “screen display function” is assigned to a specific position (partial image 130b) corresponding to the display unit of the PC (A) in the device image 130. The “data saving function” is a function of saving data sent from another device on the PC(A). The "screen display function" is a function of displaying data sent from another device on the PC (A).

The control unit 74 of the terminal device 14 may cause the UI unit 72 to display the name of the function (for example, printing or scanning) assigned to a specific position in the device image. This provides the user with information that makes it easy to understand what function corresponds to a specific position. Of course, the name of the function may not be displayed.

When the user designates the position to which the function is assigned in the device image, the function assigned to the designated position is designated as the function to be linked. The user connects the specific position (partial image) to which the function is assigned in the device image representing the device to be linked, using the indicator. For example, as shown by an arrow 136, the user touches the partial image 102b and operates up to the partial image 130b using the manipulator to connect the partial image 102b and the partial image 130b. As a result, the multifunction machine (B) associated with the device image 102 including the partial image 102b and the PC (A) associated with the device image 130 including the partial image 130b are designated as the cooperation target devices, and the partial image 102b is also specified. The "scan function" assigned to the image and the "screen display function" assigned to the partial image 130b are designated. Further, the order of connection may be designated by this connecting operation. In this case, the order of connecting the partial images corresponds to the order of connection. In the example shown in FIG. 41(a), since the images are connected from the partial image 102b to the partial image 130b, the device is connected from the multifunction peripheral (B) to the PC (A). Also, a "scan function" and a "screen display function" are specified as the functions to be used for the cooperation function. The information indicating the connection order of the devices and the information indicating the specific position designated by the user in the device image are transmitted from the terminal device 14 to the robot device 10.

When the devices (for example, the PC (A) and the multifunction device (B)) to be linked are identified, the identification unit 64 of the robot device 10 displays the PC (A) in the linked function management table shown in FIG. 11, for example. And a coordinating function realized by coordinating the multi-function peripheral (B) with each other. Further, the identification unit 64 identifies the function assigned to the specific position designated by the user in the device image by referring to the device function management table shown in FIG. 40. Then, the identification unit 64 uses a function assigned to the position designated by the user in the function group realized by linking the PC (A) and the multifunction peripheral (B). , And lower the priority of linked functions that do not use that function.

The information regarding the cooperation function identified as described above and the information indicating the priority order are transmitted from the robot device 10 to the terminal device 14. The control unit 74 of the terminal device 14 causes the UI unit 72 to display the information regarding the cooperation function as the information regarding the cooperation function candidate in accordance with the priority order.

For example, as shown in FIG. 41( b ), the control unit 74 of the terminal device 14 causes the UI unit 72 to display information regarding the cooperation function candidate. Information about the link function "scan transfer display function" that is executed by linking "scan function" and "image display function" because "scan function" and "screen display function" are specified by the user in that order However, it is displayed (for example, in a higher order) with priority over information about other cooperation functions. For example, the information about the “scan transfer display function” is preferentially displayed over the information about the cooperative function “scan transfer save function” executed by linking the “scan function” and the “data save function”. The scan transfer display function is a function of transferring data generated by scanning by the multifunction peripheral (B) to the PC (A) and displaying it on the screen of the PC (A). The scan transfer saving function is a function of transferring data generated by scanning by the multifunction peripheral (B) to the PC (A) and saving the data in the PC (A). In the example shown in FIG. 41(b), a description of each cooperation function is displayed as information about each cooperation function.

According to the cooperation process using the partial image, when the device to be cooperated has a plurality of functions, the functions are individually specified, and the information about the cooperation function that uses the specified function is given priority. Is displayed. Thereby, the cooperation function predicted to be used by the user is preferentially displayed.

Note that the cooperation function may be a function that uses a combination of device parts, a function that uses a combination of the entire device and device parts, or a combination of the entire devices. It may be a function of utilizing.

The cooperation process using the partial image may be applied when the functional image associated with the function is used. For example, different functions are assigned according to the position in the functional image, and the cooperation function that uses the function assigned to the position designated by the user is specified.

(Another example of cooperative processing using partial images)
Hereinafter, another example of the cooperation process using the partial images will be described with reference to FIGS. 42 and 43.

FIG. 42 shows an example of the device function management table. The data of the device function management table is stored in the robot apparatus 10 as the device function management information 40. In the device function management table, as an example, a device ID, information indicating a device name (for example, device type), information indicating a device part name (for example, device type), and a part for identifying the part. A partial ID as partial identification information, information indicating a function assigned to the part (a function of the part), and a partial image ID for identifying a partial image associated with the part are associated with each other. Has been. The partial image is an image representing the appearance of the portion of the device obtained by photographing with the camera. Of course, a partial image that schematically represents a device part may be associated with the part. For example, different functions are assigned to different parts of the device.

Explaining with a specific example, the screen display function is assigned to the display unit of the PC (A), and the information indicating the screen display function is associated with the partial image ID of the partial image associated with the display unit. Has been. The screen display function is a function of displaying information on the PC (A). A data storage function is assigned to the main body of the PC (A), and information indicating the data storage function is associated with the partial image ID of the partial image associated with the main body. The data saving function is a function of saving data in the PC (A).

A print function is assigned to the main body of the multifunction peripheral (B), and the partial image ID of the partial image associated with the main body is associated with information indicating the print function. A scanning function is assigned to the reading unit of the multifunction device (B) (for example, a document cover of the multifunction device (B), a portion corresponding to a document glass, or an automatic document feeder), and a portion associated with the reading unit. Information indicating the scan function is associated with the partial image ID of the image. A stapling function is assigned to the post-processing device of the multi-function peripheral (B), and information indicating the stapling function is associated with the partial image ID of the partial image that is subsequently associated with the processing device. The stapling function is a function of binding output sheets by stapling.

The function assigned to the device part is specified (identified) by using, for example, the markerless AR technology. For example, when the device part is photographed by a camera (for example, the visual sensor of the robot device 10), the identification unit 64 of the robot device 10 identifies the function associated with the appearance image data in the device function management table. (Identify) As a result, the function assigned to the captured portion is specified (identified). For example, when the main body of the multifunction peripheral (B) is photographed by the visual sensor, the identification unit 64 of the robot apparatus 10 specifies the print function associated with the appearance image data in the device function management table. As a result, it is specified that the function assigned to the main body of the multifunction peripheral (B) is the print function.

Of course, the function assigned to the device part may be identified (identified) by using the marker type AR technology. For example, each part of the device is provided with a marker such as a two-dimensional barcode in which partial identification information (for example, partial ID) for identifying the part is coded. When the marker provided on the portion is photographed by the visual sensor and the marker type AR technique is applied, the partial identification information (for example, the partial ID) of the portion is acquired. When the partial identification information is acquired in this way, the identification unit 64 of the robot apparatus 10 identifies (identifies) the function associated with the partial identification information (for example, partial ID) in the device function management table.

FIG. 43 shows an example of the cooperation function management table. The data of this cooperation function management table is stored in the robot apparatus 10 as cooperation function management information 42. This cooperation function management table is information indicating cooperation functions that use functions of a plurality of parts. In the cooperation function management table, as an example, information indicating a combination of device parts and a combination of part IDs are combined. The information shown and the information showing the cooperation function that uses the functions of the plurality of parts included in the combination are associated with each other. Of course, in the cooperation function management table, there are information indicating a combination of a device part and the entire device, and information indicating a cooperation function using the function of the device part and the function of the entire device. It may be associated.

Explaining with a specific example, the print function as the cooperation function is assigned to the combination of the display unit of the PC (A) and the main body unit of the multi-function peripheral (B). The information indicating the combination of the partial ID and the partial ID of the main body of the multifunction peripheral (B) is associated with the information indicating the print function as the cooperation function. The print function as the cooperation function is, for example, a function of transmitting data stored in the PC (A) to the multi function peripheral (B) and printing the data by the multi function peripheral (B).

Further, a print function as a cooperation function is assigned to a combination of the main body of the multifunction peripheral (B) and the main body of the projector (C), and the partial ID of the main body of the multifunction peripheral (B) and the projector ( The information indicating the combination with the partial ID of the main body portion in C) is associated with the information indicating the print function as the cooperation function. The print function as the cooperation function is, for example, a function of transmitting the data projected by the projector (C) to the multifunction peripheral (B) and printing the data by the multifunction peripheral (B).

Further, the scan projection function as the cooperation function is assigned to the combination of the reading unit of the multifunction peripheral (B) and the main body of the projector (C), and the partial ID of the reading unit of the multifunction peripheral (B) and the projector The information indicating the combination with the partial ID of the main body portion in (C) is associated with the information indicating the scan projection function as the cooperation function. The scan projection function as the cooperation function is, for example, a function of transmitting data generated by scanning by the multifunction peripheral (B) to the projector (C) and projecting the data by the projector (C).

The cooperation function may be a function that uses a function of a plurality of parts included in the same device, or may be a function that uses a function of a part of a plurality of different devices. Further, the cooperation function may be a function that uses a function of three or more parts.

For example, when a plurality of parts of a device (for example, a plurality of parts of different devices or a plurality of parts of the same device) are identified (identified) by using the marker type AR technology or the markerless type AR technology. The identification unit 64 of the robot apparatus 10 identifies (identifies) the cooperation function associated with the combination of the identified plurality of parts in the cooperation function management table. As a result, the cooperation function that uses the function of the plurality of identified (for example, photographed) portions is specified (identified). For example, when the main body of the multifunction peripheral (B) and the main body of the projector (C) are identified, the robot apparatus 10 uses the main body of the multifunction peripheral (B) and the main body of the projector (C) in the cooperation function management table. A print function or the like as a cooperation function associated with the combination of sets is specified.

(Specify the device to be linked by superimposing device images)
A device group to be linked may be designated by superimposing a plurality of device images. This process will be described below with reference to FIGS. 44 and 45. 44 and 45 show an example of a screen displayed on the UI unit 72 of the terminal device 14.

For example, it is assumed that the multifunction peripheral (B) and the PC (A) are identified. In the user determination mode 1 described above, as shown in FIG. 44A, the device images 102 and 130 associated with the identified device are displayed on the UI unit 72 of the terminal device 14 as status information. In this state, the user uses the indicator (for example, the user's finger, pen, stylus, etc.) to superimpose the device image associated with the first device on the device image associated with the device (second device) of the cooperation destination. For example, as shown in FIG. 44( b ), the user specifies the device image 102 using the operator and superimposes the device image 102 on the device image 130 as indicated by an arrow 138. For example, the device images are superimposed on each other by a drag and drop operation. That is, the user drags the device image 102 and drops the device image 102 at a position overlapping the device image 130. The drag and drop operation itself is a known technique, for example. Note that the device image to be superimposed may be designated according to the voice instruction from the user. For example, the device images 102 and 130 may be designated as the device images to be overlaid according to the voice instruction of the user, and they may be overlaid.

By superimposing the device images 102 and 130 on each other, the multifunction machine (B) associated with the device image 102 and the PC (A) associated with the device image 130 are designated as cooperation target devices.

The control unit 74 of the terminal device 14 may display the device image during the drag operation on the UI unit 72 in a display mode that allows the device image to be identified. For example, the device image during the drag operation may be displayed semitransparently or may be displayed in a specific color.

When the device image 102 is overlaid on the device image 130 and the PC (A) can execute the cooperation function together with the multifunction device (B), as shown in FIG. The confirmation screen 140 is displayed on the UI unit 72. The confirmation screen 140 is a screen for confirming whether or not to cooperate with the device group designated as the cooperation target device. On the confirmation screen 140, when a cooperation instruction is given by the user (for example, the “YES” button is pressed by the user), information regarding the cooperation function is displayed on the UI unit 72 of the terminal device 14.

For example, as shown in FIG. 45(a), the control unit 74 of the terminal device 14 causes the UI unit 72 to display information regarding the cooperation function candidate. By linking the PC (A) and the multi-function peripheral (B), for example, the "scan transfer function" and the "print function" are realized. Therefore, in the UI unit 72, the information about the "scan transfer function" and the "print function" are displayed. Information about "features" is displayed.

When the cooperation function is designated by the user and an execution instruction is given, the terminal device 14 makes a connection request to the cooperation target device. As shown in FIG. 45(b), a standby screen is displayed on the UI unit 72 of the terminal device 14 during the connection request. When the connection between the terminal device 14 and the device to be cooperated with is successful, the specified cooperation function is executed.

As described above, by overlapping the device images associated with the device, the cooperation function that uses the function of the device is specified. Therefore, the functions can be linked without performing an operation other than the image operation, and the functions can be linked by a simple operation.

The cooperation function may be specified by superimposing the partial image on the device image or the partial image. This processing will be described with reference to FIG. FIG. 46 shows an example of a screen displayed on the UI unit 72 of the terminal device 14.

Similar to the above-described cooperation process using the partial image, the function possessed by the device differs depending on the position in the device image associated with the device. By superimposing the partial images in the device image on the partial images in the same or different device images, the cooperation function that uses the function associated with both partial images is specified. Hereinafter, this process will be described in detail.

For example, it is assumed that the multifunction peripheral (B) and the PC (A) are identified. In the user determination mode 1 described above, as shown in FIG. 46A, the device images 102 and 130 are displayed as status information on the UI unit 72 of the terminal device 14. For example, the partial images 102a, 102b, 102c, 130a, and 130b are displayed as images that can be individually moved separately from the other partial images.

When a partial image is designated by the user and the partial image is overlaid on another partial image, a cooperation function that uses the function associated with both partial images is specified, and information regarding the cooperation function is displayed on the UI of the terminal device 14. It is displayed on the section 72.

For example, as shown by an arrow 142 in FIG. 46(b), when the user performs a drag operation on the partial image 102b using the manipulator to perform a drop operation on the partial image 130b, the partial image 102b is included. The multifunction device (B) associated with the device image 102 and the PC (A) associated with the device image 130 including the partial image 130b are designated as devices to be linked, and the "scan function" is assigned to the partial image 102b. ", and the "screen display function" assigned to the partial image 130b is designated as the function to be linked.

In the robot device 10, the functions assigned to the partial images are managed. For example, identification information for identifying a partial image, function information indicating a function associated with the partial image, and cooperative function information indicating a cooperative function executed by linking the functions are associated with each other. Are stored in the robot device 10. When the partial images are selected and overlaid on the other partial images, the identification information indicating the partial images is transmitted from the terminal device 14 to the robot device 10. In the example illustrated in FIG. 46B, the identification information indicating the partial images 102b and 130b is transmitted from the terminal device 14 to the robot device 10. The identification unit 64 of the robot apparatus 10 identifies the function assigned to each of the overlapped partial images 102b and 130b based on the identification information, and identifies the cooperation function that uses the function. Information regarding the cooperation function is transmitted from the robot device 10 to the terminal device 14 and displayed.

According to the above processing, when the device to be linked has a plurality of functions, the function is individually designated, and information regarding the linked function that uses the designated function is preferentially displayed. Thereby, the cooperation function predicted to be used by the user is preferentially displayed.

Further, the display priority of the cooperation function may be changed according to the order in which the partial images are superimposed. In this case, information regarding the cooperation function that uses the function associated with the overlapped partial images is preferentially displayed.

Each of the robot device 10 and the terminal device 14 described above is realized, for example, by cooperation of hardware and software. Specifically, each of the robot device 10 and the terminal device 14 includes one or a plurality of processors such as a CPU (not shown). The functions of the respective units of the robot device 10 and the terminal device 14 are realized by the one or more processors reading and executing the program stored in the storage device (not shown). The program is stored in the storage device via a recording medium such as a CD or a DVD or via a communication path such as a network. As another example, the respective units of the robot device 10 and the terminal device 14 may be realized by hardware resources such as a processor, an electronic circuit, and an ASIC (Application Specific Integrated Circuit). A device such as a memory may be used in the realization thereof. As still another example, the respective units of the robot device 10 and the terminal device 14 may be realized by a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or the like.

10 robot device, 12 devices, 14 terminal device, 34 owned function management information, 36 non-owned function management information, 38 solution means management information, 40 device function management information, 42 cooperative function management information, 44 status collection unit, 46 moving unit , 48 working unit, 54 control unit, 56 detecting unit, 58 solving means specifying unit, 60 determining unit, 62 searching unit, 64 identifying unit.

Claims (6)

Detection means to detect the surrounding situation,
To send information about the situation to a terminal device identified as the user 's terminal device assumed to have the situation problem, when the function of the own device cannot solve the situation problem. Control means for controlling
Have
The control means controls the display of a screen for the user to input additional information when receiving information indicating that the user cannot understand the situation based on the information on the situation,
Information processing device. The additional information is information regarding a user request,
The information processing apparatus according to claim 1, wherein: The control means collects information according to the request and controls display of the information,
The information processing apparatus according to claim 2, wherein: The control means further comprises
Send an image of the device around the device to the terminal device,
The image of the device is displayed on the terminal device,
The information processing apparatus according to any one of claims 1 to 3, characterized in that . The control means further comprises
When an image of the device is designated by the user and an instruction to execute a solution for solving the problem in the situation is given by the user, the instruction to execute the device is transmitted to the device,
The information processing apparatus according to claim 4, wherein: Computer,
Detection means to detect the surrounding situation,
When the function of the information processing device alone cannot solve the situation problem, the information on the situation is transmitted to the terminal device identified as the terminal device of the user assumed to have the situation problem. Control means for performing control for
Function as
The control means controls the display of a screen for the user to input additional information when receiving information indicating that the user cannot understand the situation based on the information on the situation,
program.