WO2018084170A1

WO2018084170A1 - Autonomous robot that identifies persons

Info

Publication number: WO2018084170A1
Application number: PCT/JP2017/039505
Authority: WO
Inventors: 林要
Original assignee: ＧｒｏｏｖｅＸ株式会社
Priority date: 2016-11-07
Filing date: 2017-11-01
Publication date: 2018-05-11
Also published as: JPWO2018084170A1; JP6671577B2

Abstract

A robot 100 takes an image of a user with a built-in camera, and extracts a feature vector that quantitatively represents physical features of the user from the taken image. The robot 100 extracts a feature vector (master vector) from an image taken when the robot 100 is held by the user (master image). A robot system identifies the user by comparing the master vector with a feature vector extracted from an unknown user. The robot 100 may extract a master vector from a master image taken when the robot is touched by the user instead of the master image taken when the robot is held by the user.

Description

Autonomous Behavioral Robot to Identify People

The present invention relates to a robot that autonomously selects an action according to an internal state or an external environment.

Humans acquire various information from the external environment through sense organs and select actions. Sometimes they choose to act consciously, and sometimes they choose to act unconsciously. Repeated actions eventually become unconscious actions, and new actions remain in the conscious domain.

Humans believe that they have the will to freely choose their own actions, that is, the free will. Humans have feelings of affection and hate towards others because we believe that others also have free will. A person with a free will, at least an entity that can be assumed to have a free will, also serves to heal a person's loneliness.

The reason for humans to keep pets is that they provide healing rather than their usefulness. Pets can be good companions to humans because they are more or less free-willing beings.

On the other hand, many people give up their pets for various reasons such as lack of time to take care of their pets, lack of living environment for pets, allergies, and bereavement difficulties. If there is a robot that plays the role of a pet, it may be possible to give healing that a pet gives to a person who can not keep the pet (see Patent Document 1).

JP 2000-323219 A

In recent years, although robot technology has been rapidly advancing, it has not been able to realize its presence as a pet-like companion. It is because I do not think that the robot has free will. Human beings feel the presence of free will in pets, sympathize with pets, and be healed by pets by observing such behavior that only the pet's free will seems to be present.
Therefore, it is considered that a robot capable of expressing human and biological behavior, in particular, a robot that changes its behavior according to the other party, can greatly enhance empathy for the robot.

In order to realize the above-mentioned behavior characteristics, the robot must have the ability to identify humans. In the face recognition technology, the captured image (hereinafter referred to as “master image”) to be the reference of known person A and the captured image (hereinafter referred to as “test image”) of unconfirmed person X are compared It is determined whether the person A and the person X are the same person. When acquiring a master image, the system often instructs a person to be an object regarding the posture and expression at the time of imaging.

A high quality master image is required to improve the identification accuracy of a person, but it is not preferable to place an excessive burden on the user to obtain a master image. In particular, forcing a user to act on a robot that should realize biological behavior characteristics may cause the user to feel the abiotic nature of the robot.

The present invention is an invention completed based on the above recognition, and its main object is to provide a technology for enhancing the identification capability of a robot while suppressing the burden on the user.

An autonomous action type robot according to an aspect of the present invention includes an imaging control unit that controls a camera, a recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object, and a determination result. The robot comprises: an operation selection unit that selects a motion of the robot; a drive mechanism that executes the motion selected by the operation selection unit; and an operation detection unit that detects lifting of the robot by the moving object.
The recognition unit sets a captured image obtained when the robot is held up by the moving object as a master image, and sets a determination reference of the moving object based on the feature vector extracted from the master image.

An autonomous-action robot according to another aspect of the present invention includes an imaging control unit that controls a camera, a recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object, and a determination result. And a drive mechanism for executing the motion selected by the motion selection unit; and a motion detection unit for detecting a touch by a moving object.
The recognition unit sets a captured image when a touch is detected as a master image, and sets a determination reference of a moving object based on a feature vector extracted from the master image.

An autonomous-action robot according to another aspect of the present invention includes an imaging control unit that controls a camera, a recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object, and a determination result. And an operation selection unit that selects a motion of the robot, and a drive mechanism that executes the motion selected by the operation selection unit.
The recognition unit sets, as a master image, an image captured when the moving object is located at a predetermined relative position with respect to the robot as a master image, and sets a determination criterion of the moving object based on the feature vector extracted from the master image. Do.

The behavior control program in an aspect of the present invention is a computer program for object recognition by a robot.
This program has a function of setting as a master image a captured image of a moving object when the robot is held up by the moving object, and a function of setting a determination reference of the moving object based on a feature vector extracted from the master image And causing the robot to exhibit a function of determining a moving object based on a feature vector extracted from a captured image of the moving object.

The behavior control program in another aspect of the present invention is a computer program for object recognition by a robot.
A function of setting as a master image a captured image of the moving object when the robot is touched to the moving object, a function of setting a determination reference of the moving object based on a feature vector extracted from the master image, and imaging of the moving object And causing the robot to exhibit a function of determining a moving object based on a feature vector extracted from an image.

According to the present invention, the identification ability of the robot can be easily enhanced while suppressing the burden on the user.

It is a front external view of a robot. It is a side external view of a robot. FIG. 2 is a cross-sectional view schematically illustrating the structure of a robot. It is a block diagram of a robot system. It is a conceptual diagram of an emotion map. It is a hardware block diagram of a robot. It is a functional block diagram of a robot system. It is an image figure when holding a robot. It is a data structure figure of master information. It is a 1st schematic diagram for demonstrating a user identification method. It is a 2nd schematic diagram for demonstrating a user identification method. It is a flowchart which shows the extraction process process of a master vector. It is a schematic diagram which shows a user's image tracking method. It is a schematic diagram for demonstrating the method to extract a master vector from the distance.

FIG. 1A is a front external view of the robot 100. FIG. FIG. 1 (b) is a side external view of the robot 100.
The robot 100 in the present embodiment is an autonomous action robot that determines an action or gesture (gesture) based on an external environment and an internal state. The external environment is recognized by various sensors such as a camera and a thermo sensor. The internal state is quantified as various parameters representing the emotion of the robot 100. These will be described later.

In principle, the robot 100 takes an indoor range of an owner's home as an action range. Hereinafter, a human being related to the robot 100 is referred to as a "user", and a user who is a member of a home to which the robot 100 belongs is referred to as an "owner". The “moving objects” to be identified by the robot 100 include both human and pet, but in the present embodiment, description will be made for human (user).

The body 104 of the robot 100 has an overall rounded shape, and includes an outer shell formed of a soft and elastic material such as urethane, rubber, resin, or fiber. The robot 100 may be dressed. By making the body 104 round and soft and have a good touch, the robot 100 provides the user with a sense of security and a pleasant touch.

The robot 100 has a total weight of 15 kilograms or less, preferably 10 kilograms or less, and more preferably 5 kilograms or less. By 13 months of age, the majority of babies will start walking alone. The average weight of a 13-month-old baby is just over 9 kilograms for boys and less than 9 kilograms for girls. Therefore, if the total weight of the robot 100 is 10 kilograms or less, the user can hold the robot 100 with almost the same effort as holding a baby that can not walk alone. The average weight of babies less than 2 months old is less than 5 kilograms for both men and women. Therefore, if the total weight of the robot 100 is 5 kg or less, the user can hold the robot 100 with the same effort as holding an infant.

The various attributes such as appropriate weight, roundness, softness, and good touch realize an effect that the user can easily hold the robot 100 and can not hold it. For the same reason, it is desirable that the height of the robot 100 is 1.2 meters or less, preferably 0.7 meters or less. For the robot 100 in the present embodiment, being able to hold it is an important concept.

The robot 100 includes three wheels for traveling three wheels. As shown, a pair of front wheels 102 (left wheel 102a, right wheel 102b) and one rear wheel 103 are included. The front wheel 102 is a driving wheel, and the rear wheel 103 is a driven wheel. The front wheel 102 does not have a steering mechanism, but its rotational speed and rotational direction can be individually controlled. The rear wheel 103 is a so-called omni wheel, and is rotatable in order to move the robot 100 back and forth and right and left. By making the rotation speed of the right wheel 102b larger than that of the left wheel 102a, the robot 100 can turn left or rotate counterclockwise. By making the rotational speed of the left wheel 102a larger than that of the right wheel 102b, the robot 100 can turn right or rotate clockwise.

The front wheel 102 and the rear wheel 103 can be completely housed in the body 104 by a drive mechanism (a rotation mechanism, a link mechanism). Even when traveling, most of the wheels are hidden by the body 104, but when the wheels are completely housed in the body 104, the robot 100 can not move. That is, the body 104 descends and is seated on the floor surface F along with the storing operation of the wheels. In this sitting state, the flat seating surface 108 (grounding bottom surface) formed on the bottom of the body 104 abuts on the floor surface F.

The robot 100 has two hands 106. The hand 106 does not have the function of gripping an object. The hand 106 can perform simple operations such as raising, shaking and vibrating. The two hands 106 are also individually controllable.

The eye 110 can display an image with a liquid crystal element or an organic EL element. The robot 100 mounts various sensors such as a microphone array capable of specifying a sound source direction, an ultrasonic sensor, an odor sensor, a distance measuring sensor, and an acceleration sensor. In addition, the robot 100 can incorporate a speaker and can emit a simple voice. A capacitive touch sensor is installed on the body 104 of the robot 100. The touch sensor allows the robot 100 to detect a touch of the user.

A horn 112 is attached to the head of the robot 100. As described above, since the robot 100 is lightweight, the user can lift the robot 100 by grasping the tongue 112. An omnidirectional camera is attached to the horn 112 so that the entire upper portion of the robot 100 can be imaged at one time.

FIG. 2 is a cross-sectional view schematically showing the structure of the robot 100. As shown in FIG.
As shown in FIG. 2, the body 104 of the robot 100 includes a base frame 308, a body frame 310, a pair of resin wheel covers 312 and a shell 314. The base frame 308 is made of metal and constitutes an axial center of the body 104 and supports an internal mechanism. The base frame 308 is configured by connecting an upper plate 332 and a lower plate 334 by a plurality of side plates 336 up and down. The plurality of side plates 336 is sufficiently spaced to allow air flow. Inside the base frame 308, a battery 118, a control circuit 342 and various actuators are accommodated.

The body frame 310 is made of a resin material and includes a head frame 316 and a body frame 318. The head frame 316 has a hollow hemispherical shape and forms a head skeleton of the robot 100. The body frame 318 has a stepped cylindrical shape and forms the body frame of the robot 100. The body frame 318 is integrally fixed to the base frame 308. The head frame 316 is assembled to the upper end of the body frame 318 so as to be relatively displaceable.

The head frame 316 is provided with three axes of a yaw axis 320, a pitch axis 322 and a roll axis 324, and an actuator 326 for rotationally driving each axis. The actuator 326 includes a plurality of servomotors for individually driving each axis. The yaw shaft 320 is driven for swinging motion, the pitch shaft 322 is driven for loosening motion, and the roll shaft 324 is driven for tilting motion.

A plate 325 supporting the yaw axis 320 is fixed to the top of the head frame 316. The plate 325 is formed with a plurality of vents 327 for ensuring ventilation between the top and bottom.

A metallic base plate 328 is provided to support the head frame 316 and its internal features from below. The base plate 328 is connected to the plate 325 via the cross link mechanism 329 (pantograph mechanism), and is connected to the upper plate 332 (base frame 308) via the joint 330.

Torso frame 318 houses base frame 308 and wheel drive mechanism 370. The wheel drive mechanism 370 includes a pivot shaft 378 and an actuator 379. The lower half of the body frame 318 is narrow to form a storage space S of the front wheel 102 with the wheel cover 312.

The outer cover 314 is made of urethane rubber and covers the body frame 310 and the wheel cover 312 from the outside. The hand 106 is integrally molded with the skin 314. At the upper end of the shell 314, an opening 390 for introducing external air is provided.

FIG. 3 is a block diagram of the robot system 300. As shown in FIG.
The robot system 300 includes a robot 100, a server 200 and a plurality of external sensors 114. A plurality of external sensors 114 (

external sensors

114a, 114b, ..., 114n) are installed in advance in the house. The external sensor 114 may be fixed to the wall of the house or may be mounted on the floor. In the server 200, position coordinates of the external sensor 114 are registered. The position coordinates are defined as x, y coordinates in a house assumed as the action range of the robot 100.

The server 200 is installed in a house. The server 200 and the robot 100 in the present embodiment usually correspond one to one. The server 200 determines the basic behavior of the robot 100 based on the information obtained from the sensors contained in the robot 100 and the plurality of external sensors 114.
The external sensor 114 is for reinforcing the senses of the robot 100, and the server 200 is for reinforcing the brain of the robot 100.

The external sensor 114 periodically transmits a wireless signal (hereinafter referred to as a “robot search signal”) including the ID of the external sensor 114 (hereinafter referred to as “beacon ID”). When the robot 100 receives the robot search signal, the robot 100 sends back a radio signal (hereinafter referred to as a “robot reply signal”) including a beacon ID. The server 200 measures the time from when the external sensor 114 transmits the robot search signal to when the robot reply signal is received, and measures the distance from the external sensor 114 to the robot 100. By measuring the distances between the plurality of external sensors 114 and the robot 100, the position coordinates of the robot 100 are specified.
Of course, the robot 100 may periodically transmit its position coordinates to the server 200.

FIG. 4 is a conceptual view of the emotion map 116. As shown in FIG.
The emotion map 116 is a data table stored in the server 200. The robot 100 selects an action according to the emotion map 116. An emotion map 116 shown in FIG. 4 indicates the size of a bad feeling for the location of the robot 100. The x-axis and y-axis of emotion map 116 indicate two-dimensional space coordinates. The z-axis indicates the size of the bad feeling. When the z value is positive, the preference for the location is high, and when the z value is negative, it indicates that the location is disliked.

In the emotion map 116 of FIG. 4, the coordinate P1 is a point (hereinafter, referred to as a “favory point”) in the indoor space managed by the server 200 as the action range of the robot 100, in which the favorable feeling is high. The favor point may be a "safe place" such as a shade of a sofa or under a table, a place where people easily gather like a living, or a lively place. In addition, it may be a place which has been gently boiled or touched in the past.
Although the definition of what kind of place the robot 100 prefers is arbitrary, generally, it is desirable to set a place favored by small children such as small children and dogs and cats.

A coordinate P2 is a point at which a bad feeling is high (hereinafter, referred to as a “disgust point”). Aversion points are places with loud noise such as near a television, places that are easy to get wet like baths and washrooms, closed spaces or dark places, places that lead to unpleasant memories that have been roughly treated by users, etc. It may be.
Although the definition of what place the robot 100 hates is also arbitrary, it is generally desirable to set a place where small animals such as small children, dogs and cats are scared as a hatred point.

The coordinate Q indicates the current position of the robot 100. The server 200 specifies position coordinates of the robot 100 based on a robot search signal periodically transmitted by the plurality of external sensors 114 and a robot reply signal corresponding thereto. For example, when the external sensor 114 with beacon ID = 1 and the external sensor 114 with beacon ID = 2 respectively detect the robot 100, the distance between the robot 100 is determined from the two external sensors 114, and the position coordinate of the robot 100 is determined therefrom. .

Alternatively, the external sensor 114 with beacon ID = 1 transmits a robot search signal in a plurality of directions, and when the robot 100 receives the robot search signal, it returns a robot reply signal. Thus, the server 200 may grasp how far the robot 100 is from which external sensor 114 and in which direction. In another embodiment, the movement distance of the robot 100 may be calculated from the number of revolutions of the front wheel 102 or the rear wheel 103 to specify the current position, or the current position may be determined based on an image obtained from a camera. It may be specified.
When the emotion map 116 shown in FIG. 4 is given, the robot 100 moves in the direction in which it is drawn to the favor point (coordinate P1) and in the direction away from the aversion point (coordinate P2).

The emotion map 116 changes dynamically. When the robot 100 reaches the coordinate P1, the z-value (favorable feeling) at the coordinate P1 decreases with time. As a result, the robot 100 can reach the favor point (coordinate P1), and emulate the biological behavior of "feeling of emotion" being satisfied and eventually "being bored" at the place. Similarly, bad feelings at coordinate P2 are also alleviated with time. As time passes, new favor points and aversion points are created, whereby the robot 100 makes a new action selection. The robot 100 has an "interest" at a new favor point and continuously selects an action.

The emotion map 116 expresses the ups and downs of emotion as the internal state of the robot 100. The robot 100 aims at the favor point, avoids the disgust point, stays at the favor point for a while, and then takes the next action again. Such control can make the behavior selection of the robot 100 human and biological.

The map that affects the behavior of the robot 100 (hereinafter collectively referred to as “action map”) is not limited to the emotion map 116 of the type shown in FIG. 4. For example, it is possible to define various action maps such as curiosity, fear of fear, feeling of relief, feeling of calmness and dimness, feeling of physical comfort such as coolness and warmth, and so on. Then, the destination point of the robot 100 may be determined by weighted averaging the z values of each of the plurality of action maps.

The robot 100 has parameters indicating the magnitudes of various emotions and senses separately from the action map. For example, when the value of the emotion parameter of loneliness is increasing, the weighting coefficient of the behavior map for evaluating a safe place is set large, and the value of the emotion parameter is lowered by reaching the target point. Similarly, when the value of the parameter indicating a feeling of being boring is increasing, the weighting coefficient of the behavior map for evaluating a place satisfying the curiosity may be set large.

FIG. 5 is a hardware configuration diagram of the robot 100. As shown in FIG.
The robot 100 includes an internal sensor 128, a communicator 126, a storage device 124, a processor 122, a drive mechanism 120 and a battery 118. The drive mechanism 120 includes the wheel drive mechanism 370 described above. Processor 122 and storage 124 are included in control circuit 342. The units are connected to each other by a power supply line 130 and a signal line 132. The battery 118 supplies power to each unit via the power supply line 130. Each unit transmits and receives control signals through a signal line 132. The battery 118 is a lithium ion secondary battery and is a power source of the robot 100.

The internal sensor 128 is an assembly of various sensors incorporated in the robot 100. Specifically, it is a camera (all-sky camera), a microphone array, a distance measurement sensor (infrared sensor), a thermo sensor, a touch sensor, an acceleration sensor, an odor sensor, a touch sensor, and the like. The touch sensor is disposed between the outer skin 314 and the body frame 310 to detect a touch of the user. The odor sensor is a known sensor to which the principle that the electric resistance is changed by the adsorption of the molecule that is the source of the odor is applied. The odor sensor classifies various odors into multiple categories.

The communication device 126 is a communication module that performs wireless communication for various external devices such as the server 200, the external sensor 114, and a portable device owned by a user. The storage device 124 is configured by a non-volatile memory and a volatile memory, and stores a computer program and various setting information. The processor 122 is an execution means of a computer program. The drive mechanism 120 is an actuator that controls an internal mechanism. In addition to this, indicators and speakers will also be installed.

The processor 122 performs action selection of the robot 100 while communicating with the server 200 and the external sensor 114 via the communication device 126. Various external information obtained by the internal sensor 128 also affects behavior selection. The drive mechanism 120 mainly controls the wheel (front wheel 102) and the head (head frame 316). The drive mechanism 120 changes the rotational direction and the rotational direction of the two front wheels 102 to change the moving direction and the moving speed of the robot 100. The drive mechanism 120 can also raise and lower the wheels (the front wheel 102 and the rear wheel 103). When the wheel ascends, the wheel is completely housed in the body 104, and the robot 100 abuts on the floor surface F at the seating surface 108 to be in the seating state.

FIG. 6 is a functional block diagram of the robot system 300. As shown in FIG.
As described above, robot system 300 includes robot 100, server 200, and a plurality of external sensors 114. Each component of the robot 100 and the server 200 includes computing devices such as a CPU (Central Processing Unit) and various co-processors, storage devices such as memory and storage, hardware including wired or wireless communication lines connecting them, and storage It is stored in the device and implemented by software that supplies processing instructions to the computing unit. The computer program may be configured by a device driver, an operating system, various application programs located in the upper layer of them, and a library that provides common functions to these programs. Each block described below indicates not a hardware unit configuration but a function unit block.
Some of the functions of the robot 100 may be realized by the server 200, and some or all of the functions of the server 200 may be realized by the robot 100.

(Server 200)
The server 200 includes a communication unit 204, a data processing unit 202, and a data storage unit 206.
The communication unit 204 takes charge of communication processing with the external sensor 114 and the robot 100. The data storage unit 206 stores various data. The data processing unit 202 executes various processes based on the data acquired by the communication unit 204 and the data stored in the data storage unit 206. The data processing unit 202 also functions as an interface of the communication unit 204 and the data storage unit 206.

In the present embodiment, the communication unit 204 of the server 200 is connected to the communication unit 142 of the robot 100 by two types of communication lines, a first communication line and a second communication line. The first communication line is a 920 MHz ISM frequency (Industrial, Scientific and Medical Band) communication line. The second communication line is a 2.4 GHz communication line. Since the frequency of the first communication line is lower than that of the second communication line, radio waves are likely to get around, but the communication speed is slow.

The data storage unit 206 includes a motion storage unit 232, a map storage unit 216, and a personal data storage unit 218.
The robot 100 has a plurality of motion patterns (motions). Various motions are defined, such as shaking the hand 106, approaching the owner while meandering, staring at the owner with a sharp neck, and the like.

The motion storage unit 232 stores a "motion file" that defines control content of motion. Each motion is identified by a motion ID. The motion file is also downloaded to the motion storage unit 160 of the robot 100. Which motion is to be performed may be determined by the server 200 or the robot 100.

Many of the motions of the robot 100 are configured as complex motions including a plurality of unit motions. For example, when the robot 100 approaches the owner, it may be expressed as a combination of a unit motion that turns toward the owner, a unit motion that approaches while raising the hand, a unit motion that approaches while shaking the body, and a unit motion that sits while raising both hands. . The combination of such four motions realizes a motion of “close to the owner, raise your hand halfway, and finally sit down with your body shaking”. In the motion file, the rotation angle and angular velocity of an actuator provided in the robot 100 are defined in association with the time axis. Various motions are represented by controlling each actuator with the passage of time according to a motion file (actuator control information).

The transition time when changing from the previous unit motion to the next unit motion is called "interval". The interval may be defined according to the time required for unit motion change and the contents of the motion. The length of the interval is adjustable.
Hereinafter, settings relating to behavior control of the robot 100, such as when to select which motion, output adjustment of each actuator for realizing the motion, and the like are collectively referred to as “behavior characteristics”. The behavior characteristics of the robot 100 are defined by a motion selection algorithm, a motion selection probability, a motion file, and the like.

The motion storage unit 232 stores, in addition to the motion file, a motion selection table that defines motion to be executed when various events occur. In the motion selection table, one or more motions and their selection probabilities are associated with an event.

The map storage unit 216 stores, in addition to a plurality of action maps, a map indicating the arrangement of obstacles such as chairs and tables. The personal data storage unit 218 stores information of the user, in particular, the owner. Specifically, master information indicating the closeness to the user and the physical and behavioral characteristics of the user is stored. Other attribute information such as age and gender may be stored. Details of the master information will be described later with reference to FIG.

The robot system 300 (the robot 100 and the server 200) identifies the user based on the physical or behavioral characteristics of the user. The robot 100 captures an image of the periphery with the omnidirectional camera. Then, physical features and behavioral features of the person shown in the image are extracted. Physical characteristics include eye-to-eye size, eye-to-mouth and nose balance, height, clothes you like to wear, glasses, skin color, hair color, ear size, etc. It may be a visual feature associated with the body, or it may include other features such as average temperature, smell, voice quality, and the like. Specifically, the behavioral feature is a feature that accompanies the action, such as the place the user likes, the activity activity, and the presence or absence of smoking. For example, an owner who is identified as a father often does not stay at home and often does not move on a couch when at home, but a mother often extracts behavior characteristics such as being in the kitchen and having a wide range of behavior.
The robot system 300 according to the present embodiment extracts a plurality of parameters indicating physical features from a master image described later, and identifies the user based on the master image. Hereinafter, the process of identifying the user based on the master image is referred to as “user identification process”. Details of the user identification process will be described later.

The robot 100 has an internal parameter called familiarity for each user. When the robot 100 recognizes an action indicating favor with itself, such as raising itself or giving a voice, familiarity with the user is increased. The closeness to the user who is not involved in the robot 100, the user who is violent, and the user who is infrequently encountered is low.

The data processing unit 202 includes a position management unit 208, a map management unit 210, a recognition unit 212, an operation control unit 222, an intimacy management unit 220, and an emotion management unit 244.
The position management unit 208 specifies the position coordinates of the robot 100 by the method described with reference to FIG. The position management unit 208 may also track the user's position coordinates in real time.

The emotion management unit 244 manages various emotion parameters that indicate the emotion (the loneliness, the fun, the fear, etc.) of the robot 100. These emotional parameters are constantly fluctuating. The importance of the plurality of action maps changes according to the emotion parameter, the movement target point of the robot 100 changes according to the action map, and the emotion parameter changes according to the movement of the robot 100 or the passage of time.

For example, when the emotion parameter indicating loneliness is high, the emotion management unit 244 sets the weighting coefficient of the behavior map for evaluating a safe place large. When the robot 100 reaches a point at which loneliness can be eliminated in the action map, the emotion management unit 244 reduces the emotion parameter indicating the loneliness. In addition, various emotional parameters are also changed by the response action described later. For example, the emotion parameter indicating loneliness declines when being "held" from the owner, and the emotion parameter indicating loneliness gradually increases when the owner is not viewed for a long time.

The map management unit 210 changes the parameter of each coordinate in the method described with reference to FIG. 4 for a plurality of action maps. The map management unit 210 may select one of the plurality of behavior maps, or may perform weighted averaging of z values of the plurality of behavior maps. For example, it is assumed that z values at coordinates R1 and R2 are 4 and 3 in action map A, and z values at coordinates R1 and R2 in action map B are -1 and 3, respectively. In the case of simple average, the total z value of the coordinate R1 is 4-1 = 3, and the total z value of the coordinate R2 is 3 + 3 = 6, so the robot 100 moves in the direction of the coordinate R2 instead of the coordinate R1.
When emphasizing the action map A five times the action map B, the total z value of the coordinate R1 is 4 × 5-1 = 19, and the total z value of the coordinate R2 is 3 × 5 + 3 = 18. Head in the direction of

The recognition unit 212 recognizes the external environment. The recognition of the external environment includes various recognitions such as recognition of weather and season based on temperature and humidity, recognition of an object shade (safety area) based on light quantity and temperature. The recognition unit 156 of the robot 100 acquires various types of environment information by the internal sensor 128, performs primary processing on the environment information, and transfers the information to the recognition unit 212 of the server 200. The recognition unit 156 of the robot 100 extracts an image area corresponding to a moving object, in particular, a person or an animal from the image, and extracts a “feature vector” indicating the physical feature or behavioral feature of the moving object from the extracted image area Do. The robot 100 transmits the feature vector to the server 200.

The recognition unit 212 of the server 200 further includes a person recognition unit 214 and a response recognition unit 228.
The person recognition unit 214 compares the feature vector extracted from the image captured by the built-in camera of the robot 100 with the feature vector of the user registered in advance in the personal data storage unit 218 to determine which person the captured user is It determines whether it corresponds to (user identification processing). The person recognition unit 214 includes an expression recognition unit 230. The facial expression recognition unit 230 estimates the user's emotion by performing image recognition on the user's facial expression.
The person recognition unit 214 also performs user identification processing on moving objects other than a person, for example, cats and dogs that are pets.

As described above, in the present embodiment, the recognition unit 156 of the robot 100 extracts an image area corresponding to a moving object (a person and an animal) from a captured image, and extracts a feature vector from the extracted captured image. In the personal data storage unit 218 of the server 200, feature vectors of a plurality of users (hereinafter, referred to as "master vectors") are registered. The master vector is a feature vector extracted based on a user's master image. The person recognition unit 214 of the server 200 identifies the user by comparing the feature vector sent from the robot 100 with the master vector.

Hereinafter, a user whose master vector is registered in the personal data storage unit 218 is referred to as a “registered user”, and an unconfirmed user who is a target of user identification processing recognized by a camera is referred to as an “unknown user”. If the master vector of registered user A and the feature vector of unknown user X (hereinafter also referred to as “test vector”) match or are similar, it is determined that unknown user X is the same person as registered user A.

The response recognition unit 228 recognizes various response actions made to the robot 100, and classifies them as pleasant and unpleasant actions. The response recognition unit 228 also classifies into a positive / negative response by recognizing the owner's response to the behavior of the robot 100.
The pleasant and unpleasant behavior is determined depending on whether the user's response behavior is comfortable or unpleasant as a living thing. For example, holding is a pleasant act for the robot 100, and kicking is an unpleasant act for the robot 100. The positive / negative response is determined depending on whether the user's response indicates a user's pleasant emotion or an unpleasant emotion. For example, being held is a positive response indicating the user's pleasant feeling, and kicking is a negative response indicating the user's unpleasant feeling.

The motion control unit 222 of the server 200 cooperates with the motion control unit 150 of the robot 100 to determine the motion of the robot 100. The motion control unit 222 of the server 200 creates a movement target point of the robot 100 and a movement route for the movement based on the action map selection by the map management unit 210. The operation control unit 222 may create a plurality of movement routes, and then select one of the movement routes.

The motion control unit 222 selects the motion of the robot 100 from the plurality of motions of the motion storage unit 232. Each motion is associated with a selection probability for each situation. For example, a selection method is defined such that motion A is executed with a probability of 20% when a pleasant action is made by the owner, and motion B is executed with a probability of 5% when the temperature reaches 30 degrees or more. .
A movement target point and a movement route are determined in the action map, and a motion is selected by various events described later.

The closeness management unit 220 manages closeness for each user. As described above, the intimacy degree is registered in the personal data storage unit 218 as part of the personal data. When a pleasant act is detected, the closeness management unit 220 increases the closeness to the owner. The intimacy is down when an offensive act is detected. In addition, the closeness of the owner who has not viewed for a long time gradually decreases.

(Robot 100)
The robot 100 includes a communication unit 142, a data processing unit 136, a data storage unit 148, an internal sensor 128, and a drive mechanism 120.
The communication unit 142 corresponds to the communication device 126 (see FIG. 5), and takes charge of communication processing with the external sensor 114, the server 200, and the other robot 100. The data storage unit 148 stores various data. The data storage unit 148 corresponds to the storage device 124 (see FIG. 5). The data processing unit 136 executes various processes based on the data acquired by the communication unit 142 and the data stored in the data storage unit 148. The data processing unit 136 corresponds to a processor 122 and a computer program executed by the processor 122. The data processing unit 136 also functions as an interface of the communication unit 142, the internal sensor 128, the drive mechanism 120, and the data storage unit 148.

The data storage unit 148 includes a motion storage unit 160 that defines various motions of the robot 100.
Various motion files are downloaded from the motion storage unit 232 of the server 200 to the motion storage unit 160 of the robot 100. Motion is identified by motion ID. A state in which the front wheel 102 is accommodated, which causes the robot 100 to rotate by having only the front wheel 102 housed and seated, lifting the hand 106, rotating the two front wheels 102 in reverse, or rotating only one front wheel 102 In order to express various motions such as shaking by rotating the front wheel 102 at a time, stopping and turning back once when leaving the user, operation timing, operation time, operation direction, etc. of various actuators (drive mechanism 120) Temporarily defined in motion file.

Various data may also be downloaded to the data storage unit 148 from the map storage unit 216 and the personal data storage unit 218.

Internal sensor 128 includes a camera 134. The camera 134 in the present embodiment is an omnidirectional camera attached to the horn 112.

The data processing unit 136 includes a recognition unit 156, an operation control unit 150, an operation detection unit 152, an imaging control unit 154, and a distance measurement unit 158.
The motion control unit 150 of the robot 100 determines the motion of the robot 100 in cooperation with the motion control unit 222 of the server 200. Some motions may be determined by the server 200, and other motions may be determined by the robot 100. Also, although the robot 100 determines the motion, the server 200 may determine the motion when the processing load of the robot 100 is high. The base motion may be determined at server 200 and additional motion may be determined at robot 100. How to share the motion determination process in the server 200 and the robot 100 may be designed according to the specification of the robot system 300.

The motion control unit 150 of the robot 100 determines the moving direction of the robot 100 together with the motion control unit 222 of the server 200. The movement based on the action map may be determined by the server 200, and the immediate movement such as turning off the obstacle may be determined by the movement control unit 150 of the robot 100. The drive mechanism 120 drives the front wheel 102 in accordance with an instruction from the operation control unit 150 to direct the robot 100 to the movement target point.

The operation control unit 150 of the robot 100 instructs the drive mechanism 120 to execute the selected motion. The drive mechanism 120 controls each actuator according to the motion file.

The motion control unit 150 can also execute a motion to lift both hands 106 as a gesture that encourages "hug" when a user with high intimacy is nearby, and when the "hug" gets tired, the left and right front wheels 102 By alternately repeating reverse rotation and stop while being accommodated, it is also possible to express a motion that annoys you. The drive mechanism 120 causes the robot 100 to express various motions by driving the front wheel 102, the hand 106, and the neck (head frame 316) according to the instruction of the operation control unit 150.

The motion detection unit 152 detects “hold up” and “hold down” of the robot 100 in addition to the touch by the user. “Holding up” is typically an action where the user lifts the robot 100 by putting both hands on the body 104 of the robot 100. “Holding down” is an action in which the user typically puts the robot 100 on the floor surface F with his hands attached to the body 104 of the robot 100. The motion detection unit 152 detects a touch of the user by a touch sensor installed under the outer skin 314 of the robot 100. On the condition that the acceleration sensor has detected a rise in the touched state, the operation detection unit 152 determines that the “holding up” has been performed. Similarly, when the descent is detected by the acceleration sensor in the touched state, or when a load on the seating surface 108 or the front wheel 102 is detected, the operation detection unit 152 determines that the “holding down” is performed. The moving image of the outside world may be captured by the camera 134, and “lifting” and “holding down” may be determined by recognizing the ascent and descent of the robot 100 from changes in the image.

The imaging control unit 154 controls the camera 134. The imaging control unit 154 images a subject when holding up or holding down, when a touch is detected, or at various timings described later.

The distance measuring unit 158 detects a distance to a moving object (person and pet) to be a subject by using a distance measuring sensor (infrared sensor) included in the internal sensor 128. The recognition unit 156 also detects the relative angle between the robot 100 and the subject by performing image recognition on the subject. The captured image when the robot 100 is positioned at a predetermined relative position with respect to the subject may be used as a master image candidate (hereinafter, referred to as “master candidate image”). The method of acquiring the master candidate image based on the distance measurement will be described later with reference to FIG.

The recognition unit 156 of the robot 100 interprets external information obtained from the internal sensor 128. The recognition unit 156 is capable of visual recognition (visual unit), odor recognition (olfactory unit), sound recognition (hearing unit), and tactile recognition (tactile unit).
The recognition unit 156 periodically images the outside world with the built-in omnidirectional camera, and detects a moving object such as a person or a pet. The feature vector extracted from the captured image of the moving object by the recognition unit 156 is transmitted to the server 200, and the person recognition unit 214 of the server 200 identifies the user. The recognition unit 156 of the robot 100 also detects the smell of the user and the voice of the user. Smells and sounds (voices) are classified into multiple types by known methods.

When a strong impact is given to the robot 100, the recognition unit 156 recognizes this by the built-in acceleration sensor, and the response recognition unit 228 of the server 200 recognizes that the "abuse act" is performed by the user in the vicinity. Even when the user holds the tongue 112 and lifts the robot 100, it may be recognized as a violent act. When the user directly facing the robot 100 utters in a specific sound volume region and a specific frequency band, the response recognition unit 228 of the server 200 may recognize that the “voice call action” has been performed on itself. In addition, when a temperature at a temperature close to the body temperature is detected, it may be recognized that the user has made a "contact act".
In summary, the robot 100 acquires the action of the user as physical information by the internal sensor 128, and the action detection unit 152 determines the action such as "hold up" or "hold down", and the response recognition unit 228 of the server 200 The discomfort is determined, and the recognition unit 212 of the server 200 executes a user identification process based on the feature vector.

The response recognition unit 228 of the server 200 recognizes various responses of the user to the robot 100. Of the various types of response actions, some typical response actions correspond to pleasure or discomfort, affirmation or denial. In general, most pleasurable actions are positive responses, and most offensive actions are negative. Pleasure and discomfort are related to intimacy, and affirmative and negative responses affect the action selection of the robot 100.

Among the series of recognition processes including detection, analysis, and determination, the recognition unit 156 of the robot 100 selects and extracts information necessary for recognition, and interpretation processes such as determination are executed by the recognition unit 212 of the server 200. . The recognition processing may be performed only by the recognition unit 212 of the server 200, or may be performed only by the recognition unit 156 of the robot 100, or both perform the above-mentioned recognition processing while sharing roles. It is also good.

In accordance with the response action recognized by the recognition unit 156, the closeness management unit 220 of the server 200 changes the closeness to the user. In principle, the intimacy with the user who has performed pleasure is increased, and the intimacy with the user who has performed offensive activity decreases.

The recognition unit 212 of the server 200 determines the comfort / discomfort according to the response, and the map management unit 210 changes the z value of the point where the comfort / discommitment was performed in the action map expressing “attachment to a place”. May be For example, when a pleasant act is performed in the living, the map management unit 210 may set a favor point in the living with a high probability. In this case, a positive feedback effect is realized in that the robot 100 prefers a living and enjoys an activity in the living, and thus prefers a living more and more.

The closeness to the user changes depending on what action is taken from the moving object (user).

The robot 100 sets a high degree of intimacy for people who frequently meet, people who frequently touch, and people who frequently speak. On the other hand, the intimacy with the people who rarely see, those who do not touch very much, the violent people, the people who speak loudly becomes low. The robot 100 changes the intimacy degree of each user based on various external information detected by sensors (vision, touch, hearing).

The actual robot 100 autonomously performs complex action selection in accordance with the action map. The robot 100 acts while being influenced by a plurality of action maps based on various parameters such as loneliness, boredom and curiosity. The robot 100 tries to approach people with high intimacy and leaves people with low intimacy, in principle, when the influence of the action map is excluded or in an internal state where the influence of the behavior map is small. I assume.

The behavior of the robot 100 is categorized as follows according to closeness.
(1) The user robot 100 with a very high degree of intimacy approaches the user (hereinafter referred to as “proximity action”), and performs the affection of love by predefining a gesture of love for people. Express strongly.
(2) The user robot 100 with relatively high intimacy performs only the proximity action.
(3) The user robot 100 with relatively low intimacy does not perform any particular action.
(4) The user robot 100 with a particularly low intimacy performs a leaving action.

According to the above control method, when the robot 100 finds a user with high intimacy, it approaches that user, and conversely, when finding a user with low intimacy, it leaves the user. By such a control method, it is possible to express so-called "human sight" behavior. In addition, when a visitor (user A with low intimacy) appears, the robot 100 may move away from the visitor and head toward the family (user B with high intimacy). In this case, the user B can feel that the robot 100 is aware of strangers and feels uneasy, and relies on himself. Such a behavioral expression evokes the user B the joy of being selected and relied upon, and the accompanying attachment.

On the other hand, when the user A who is a visitor frequently visits, calls and makes a touch, the intimacy with the user A of the robot 100 gradually increases, and the robot 100 does not act as an acquaintance with the user A (disengagement behavior) . The user A can also have an attachment to the robot 100 by feeling that the robot 100 has become familiar with himself.

Note that the above action selection is not always performed. For example, when the internal parameter indicating the curiosity of the robot 100 is high, the robot 100 may not select the behavior influenced by the intimacy because the action map for finding a place satisfying the curiosity is emphasized. . In addition, when the external sensor 114 installed at the entrance detects that the user has returned home, the user may be asked to give priority to the user's meeting action.

FIG. 7 is an image view when a robot is held.
The robot 100 has a round, soft, well-touched body 104 and an appropriate weight, and recognizes a touch as a pleasure, so it is easy for the user to feel that he / she wants to hold the robot 100. The robot 100 applies this involuntary feeling to the user identification process.

In order for the robot 100 to identify the user, information to be a clue is needed. For example, eyebrow thickness, eye size, eye shape, skin color, skin brightness, eyebrow shape, hair brightness, bangs length, eye or nose size in the entire face The clues are physical characteristics such as proportions and eye-to-eye spacing. In the present embodiment, first, the robot 100 acquires a master image. The recognition unit 156 of the robot 100 extracts a feature vector (master vector) from the master image. The feature vector has a plurality of vector components. The feature vector component is a numerical value that quantifies the various physical features described above. For example, the width of the eye is quantified in the range of 0 to 1, and these form feature vector components. The method of extracting feature vectors from a captured image of a person is an application of known face recognition technology. The master vector of the user A is stored as master information 224 of the personal data storage unit 218.
Hereinafter, the process of extracting a feature vector from a captured image is referred to as “vector extraction process”.

When the robot 100 captures an unknown user X, the recognition unit 156 extracts a feature vector (inspection vector) from the captured image (inspection image) of the unknown user X. If the inspection vector of unknown user X and the master vector of registered user A are similar, person recognition unit 214 of server 200 determines that unknown user X and registered user A are the same person.

In order to improve identification accuracy, it is necessary to image the user at a high quality master image where the master vector can be easily extracted, more specifically, at a short distance. The recognition unit 156 in the present embodiment sets a captured image when the motion detection unit 152 detects holding of the robot 100 as a master image. When the robot 100 is held, it can be imaged with high accuracy by the built-in camera 134. This is because when the robot 100 is lifted, the distance between the user's face and the camera 134 incorporated in the robot 100 falls within a certain range. Instead of giving the user a "action instruction" for capturing a master image, it is possible to obtain a good-quality master image without putting a burden on the user, at a timing when the user holds the robot 100 by his own intention.

FIG. 8 is a data structure diagram of master information.
The master information 224 is stored in the agent recognition unit 228. In FIG. 8, three master vectors are associated with a user with user ID = 01 (hereinafter referred to as “user (01)”). The master image is acquired not only from the front of the user (01) but also from profile faces such as the right side and left side. For this reason, a plurality of master vectors are associated with one registered user by imaging the user from a plurality of angles and a plurality of distances. The master vector is identified by a master ID. The master vector (01) is extracted from the master image when the face of the user (01) is captured from the front, and the master vector (02) is extracted from the master image when the face of the user (01) is captured from the right .

In order to simplify the description, the master vector shown in FIG. 8 is described as a five-dimensional vector having five vector components. The five vector components a to e correspond to arbitrary feature amounts such as the eye-to-eye distance and the skin color. The master vector (01) includes feature quantities a1, b1 and c1 corresponding to three vector components a to c. On the other hand, no feature amount is set for the vector components d and e. For example, when the vector component d is a feature indicating the ear size, there is a possibility that the component d can not be extracted from the front master image.

The master vector (02) includes feature quantities a2, c2 and e2 corresponding to three vector components a, c and e, but does not include feature quantities corresponding to vector components b and d. The master vector (03) includes feature quantities a3, b3, d3 and e3 corresponding to the four vector components a, b, d and e, but does not include feature quantities corresponding to the vector component c. If a plurality of master images are acquired by imaging the user (01) from a plurality of directions, physical features of the user (01) can be three-dimensionally grasped.

The person recognition unit 214 calculates the center of gravity vector MB by arithmetically averaging the three master vectors of the user (01). The vector component a of the centroid vector MB is an average value of the a components (a1, a2, a3) of the three master vectors. Since only the master vector (03) has the vector component d, the vector component d of the gravity center vector MB becomes the feature amount d3 of the master vector (03). The person recognition unit 214 executes user identification processing based on the master vector or the gravity center vector MB (described later).

Assume that there is no registered user.
The motion detection unit 152 acquires a master image when being held by the unknown user A. The person recognition unit 214 associates the user ID = 01 with the master vector (01) extracted from the master image of the unknown user A, and records it in the master information 224. At this time, the person recognition unit 214 also records the acquisition date and time of the master vector (01). By the above processing, the unknown user A is registered in the master information 224 as a registered user (01). In FIG. 8, the master vector (01) is acquired on June 7, 2016.

Even when a new unknown user X holds the robot 100 after registration of the user (01), the motion detection unit 152 acquires a master image. The person recognition unit 214 compares the master vector MX extracted from the master image of the unknown user X with the master vector (01) of the registered user (01).

(1) When Not Registered If the vector distance between the master vector MX and the master vector (01) is equal to or greater than a predetermined distance, the person recognizing unit 214 determines that the unknown user X is different from the user (01). The distance of the feature vector may be calculated as Euclidean distance, or may be calculated based on another definition such as Chebyshev distance. The person recognition unit 214 registers the unknown user X in the master information 224 as a new registered user (02), assigns the user ID = 02 to the user X, and assigns the master ID = 04 to the master vector MX. As a result of the above processing, two users, user (01) and user (02), are registered in the master information 224.

(2) In the Case of Already Registered If the distance between the master vector MX and the master vector (01) is less than a predetermined distance, the person recognizing unit 214 determines that the unknown user X and the registered user (01) are the same person. The person recognition unit 214 sets master ID = 02 in the master vector MX, and associates it with the user (01). The number of master vectors of the user (01) is two, and the information for identifying the user (01) is enriched.

Since high-quality master vectors are obtained from the master image, it is possible to easily determine whether the user holding the robot 100 is a registered user and a person by comparing the master vectors.

When there are multiple registered users, the master vectors of each registered user are to be compared. When one registered user has two or more master vectors, the gravity center vector of the registered user and the master vector of the unknown user are to be compared.

FIG. 9 is a first schematic diagram for explaining the user identification method.
In FIG. 9 and FIG. 10, in order to illustrate the principle of the user identification process, among the vector components contained in the feature vector, two vector components a and b will be described as targets. The processing method is the same when having three or more vector components.
In FIG. 9, one master vector MA and one master vector MB are extracted for each of registered user A and registered user B. Master vector MA = (a1, b1) and master vector MB = (a2, b2). In such a situation, it is assumed that the captured image (inspection image) of the unknown user X who walks from the front of the robot 100 is acquired. The recognition unit 156 determines which one of the registered users A and B the unknown user X shown in the inspection image is. As not being held, the feature vector (inspection vector) obtained from the inspection image of the unknown user X usually does not have the accuracy as the master vector.

The recognition unit 156 first extracts the inspection vector DX = (ax, bx) from the inspection image of the unknown user X. The communication unit 142 of the robot 100 transmits the inspection vector DX to the communication unit 204 of the server 200. The person recognition unit 214 of the server 200 calculates the distance ra between the test vector DX and the master vector MA, and the distance rb between the test vector DX and the master vector MB.

When an arbitrary threshold value rm is set, if rb <ra and rb <rm, the person recognizing unit 214 determines that the unknown user X is the registered user B. On the other hand, if ra <rb and ra <rm, the person recognizing unit 214 determines that the unknown user X is the registered user A. On the other hand, when ra> rm and rb> rm, the unknown user X does not correspond to any of the registered users A and B. When it is determined that the unknown user X is a registered user A with high intimacy, the operation control unit 150 may select an intimacy action such as running to the unknown user X. On the other hand, when it is determined that the unknown user X is a registered user B with low intimacy, the operation control unit 150 may select a evasion action such as fleeing from the unknown user X.

When the unknown user X can not be identified, the person recognition unit 214 may notify the robot 100 that the unknown user X has not been confirmed, and the operation control unit 150 of the robot 100 may select a motion to hold the unknown user X. Specifically, motions such as approaching the unknown user X, raising the hand 106, and sitting in front of the unknown user X can be considered.

When the unknown user X holds the robot 100 and the motion detection unit 152 detects “hold”, the imaging control unit 154 controls the camera 134 to pick up an image of the unknown user X at a short distance. The recognition unit 156 extracts a master vector MX from the master image of the unknown user obtained at the time of holding. The person recognition unit 214 of the server 200 may execute the user identification process again by comparing the master vector MX of the unknown user X with the existing master vectors MA and MB. Since the comparison is between master vectors, more accurate identification is possible. If the unknown user X is determined to be another person from the registered users A and B also by comparison of the master vectors, the person recognition unit 214 sets the unknown user X as the third registered user in the master information 224 together with the master vector MX. sign up.

When it is determined that the unknown user X is the registered user A, the person recognition unit 214 may register the test vector obtained from the test image of the unknown user X as a new master vector of the registered user A.

FIG. 10 is a second schematic diagram for explaining the user identification method.
In FIG. 10, a plurality of master vectors are extracted for each of registered user A and registered user B. The person recognition unit 214 calculates the gravity center vector MB (A) of the registered user A and the gravity center vector MB (B) of the registered user B. In such a situation, it is assumed that the robot 100 acquires a captured image (examination image) of the unknown user X who walks from the front. The recognition unit 156 determines which one of the registered users A and B the unknown user X shown in the inspection image is.

The recognition unit 156 first extracts the inspection vector DX = (ax, bx) from the inspection image of the unknown user X. The communication unit 142 of the robot 100 transmits the inspection vector DX to the communication unit 204 of the server 200. The person recognition unit 214 of the server 200 calculates the distance ra between the test vector DX and the barycentric vector MB (A) and the distance rb between the test vector DX and the barycenter vector MB (B).

When an arbitrary threshold value rm is set, if rb <ra and rb <rm, the person recognizing unit 214 determines that the unknown user X is the registered user B. On the other hand, if ra <rb and ra <rm, the person recognizing unit 214 determines that the unknown user X is the registered user A. On the other hand, when ra> rm and rb> rm, the unknown user X does not correspond to any of the registered users A and B.

FIG. 11 is a flowchart showing a process of extracting a master vector.
When the motion detection unit 152 of the robot 100 detects lifting of the robot 100, the vector extraction process of FIG. 11 is performed. When holding up is detected, the operation control unit 150 executes a predetermined guidance motion (S10). Guided motion is a motion defined in advance to bring the user's attention. Specifically, non-verbal motions are assumed such as shaking the hand 106, shaking the body 104, pointing the head frame 316 to the user, shaking the head frame 316 up and down or left and right. Induction motion is not limited to mechanical motion. The operation control unit 150 causes the organic EL element to display an image of “pupil” on the eye 110. The operation control unit 150 may instruct image control such as opening the pupil, shaking the pupil, or causing a wink by enlarging the pupil image.

The user's face is directed to the robot 100 by pulling the user's attention with the induced motion. Further, by preparing various induction motions, it is possible to extract various master vectors corresponding to various expressions by extracting various expressions of the user. For example, features specific to smiles, such as smiles and bumps can be included as vector components of the master vector.

After executing the induction motion, the imaging control unit 154 controls the camera 134 to image the user (S12). The captured image at this time is the “master candidate image”. By imaging the user at the timing when the user gazes at the robot 100 by the guidance motion, it is possible to acquire a high-quality master candidate image that easily recognizes the user's face.

The recognition unit 156 determines the quality of the master candidate image (S14). Hereinafter, the quality determination of the master candidate image is referred to as “quality inspection”. A master candidate image that has passed the quality inspection is set as a master image. If the quality inspection fails (N in S14), the process returns to S10 to reacquire the master candidate image. At this time, another type of induction motion may be performed. For the quality inspection, a plurality of evaluation items are set in advance with respect to the user's face size, light intensity, expression, and the like. For example, when the user has a closed eye, when the master candidate image is too dark or too bright, or when the master candidate image is not in focus, the quality check fails. It is optional what kind of evaluation item is set for quality inspection.

The recognition unit 156 adopts the master candidate image that has passed the quality inspection as a formal master image (Y in S14). The recognition unit 156 extracts a master vector from the master image (S16). The communication unit 142 transmits the master vector to the server 200 (S18).

The person recognition unit 214 compares the newly obtained master vector with the master vector already registered in the master information 224 (S20). When the distance of the master vector in which the newly obtained master vector is already registered is close (Y in S20), the master vector is additionally registered (S22). For example, when a master vector similar to the master vector (01) of the user (01) is obtained, the new master vector is also mapped to the user (01). If none of the registered master vectors is close (N in S20), a new user ID and a master ID are added to newly register a master vector (S24).

In S20, the registered master vector may be compared with the master vector of the new extraction, or, as described with reference to FIG. 10, the registered centroid vector may be compared with the master vector of the new extraction.

The extraction process of the master vector is not limited to holding, and may be executed when the user touches the robot 100 as a trigger. When the user touches the robot 100, the user may be able to obtain a good-quality master image because the user is near the robot 100.

Also when the motion detection unit 152 detects that the robot 100 is being lowered, the recognition unit 156 executes a master vector extraction process. The motion detection unit 152 continuously captures an image of the user when the hold down is detected. The recognition unit 156 sequentially performs quality inspection on the plurality of master candidate images obtained at this time, and extracts a plurality of master vectors. At the time of holding and lowering, it is possible to image physical features such as the jaw, the waist and the legs at a short distance.

The master vector obtained when the user lifts the robot 100 is referred to as “first master vector”, and the master vector obtained when the user lowers the robot 100 or after being lowered is referred to as “second master vector”. . After obtaining the first master vector of the user (01), the recognition unit 156 also extracts one or more second master vectors from the master image at the time of holding and lowering. As described above, when the first master vector with high accuracy is obtained, the master vector of the user (01) can be enriched by acquiring the second master vector even when holding down. Here, the “second master vector” also includes master vectors obtained from various distances and angles, including the back view of the user even after the robot 100 is lowered. The first and second master vectors are associated with each other for one user as shown in master information 224.

FIG. 12 is a schematic view showing the image tracking method of the user.
Even after the robot 100 is lowered to the floor F, the imaging control unit 154 further tracks the user with the camera 134 (all-sky camera). A celestial imaging range 418 shown in FIG. 12 is an imaging range by the omnidirectional camera. The omnidirectional camera is capable of imaging the entire upper hemisphere of the robot 100 at one time. After extracting the first master vector, the recognition unit 156 of the robot 100 tracks the user in the celestial imaging range 418 for a predetermined period, for example, about 10 seconds. The imaging control unit 154 captures a master image of the user from various angles and different distances during tracking. For example, the length of hair, the thinness of waist, etc. are information that can not be obtained without leaving the user. The recognition unit 156 enriches the master vector by extracting various second master vectors from the master image obtained during tracking. These second master vectors are managed in association with the first master vector. In addition to tracking the user on the image in the celestial imaging range 418, the operation control unit 150 may execute tracking actions such as following the user, moving around the user, and the like. Then, the imaging control unit 154 may enrich the master vector by imaging the user even during the tracking action. The operation control unit 150 may instruct the tracking behavior, or the operation control unit 222 of the server 200 may instruct the operation control unit 150.

FIG. 13 is a schematic diagram for explaining a method of remotely extracting a master vector.
The imaging control unit 154 picks up not only a hug or a touch but also a master candidate image when the user is positioned at a predetermined relative point with respect to the robot 100. Here, the relative point includes both the distance between the user and the robot 100 and the relative angle. The distance measuring unit 158 periodically measures the distance to one or more users recognized in the celestial imaging range 418. In FIG. 13, the robot 100 is located at a horizontal angle a to the front direction of the user, an elevation angle b to the position of the user's face, and a distance r from the user. The recognition unit 156 determines the orientation of the user's body by image recognition. The imaging control unit 154 captures a master candidate image when the distance, the horizontal angle, and the elevation angle are in a predetermined range (hereinafter, referred to as a “master shot range”). The recognition unit 156 performs quality inspection on the master candidate image, and if it passes, extracts a master vector.

The distance measuring unit 158 has a plurality of master shot ranges set in advance. The distance measuring unit 158 notifies the imaging control unit 154 each time the user enters the master shot range, and the imaging control unit 154 acquires a master candidate image. For example, when the master shot range R1 to R3 is defined, when a new user C enters the master shot range R1, a master candidate image corresponding to the master shot range R1 is acquired. In this manner, master vectors corresponding to each of the master shot ranges R1 to R3 are extracted. The user C can be multilaterally imaged from a plurality of master shot ranges, in other words, from a plurality of relative points, and master vectors from multiple directions can be acquired to three-dimensionally grasp the physical characteristics of the user.

At the time of contact such as holding or touching, the user can be photographed from a close distance, so it is easy to obtain good information on the user's face. On the other hand, even when holding or touching is not performed, when the distance measuring unit 158 detects a user at a close distance, the imaging control unit 154 may obtain a master candidate image. For example, even if a small child resists holding or touching, it can extract a master vector when approaching with interest. Also, in order to obtain information on the length and shape of the user's hair, the robot 100 has to be away from the user to some extent. By setting various master shot ranges, various master vectors including not only the face of the user but also the figure can be obtained.

The first master vector is a feature vector useful for identifying a user because it is based on a master image taken at a close distance to the user. On the other hand, in the second master vector, the physical features of the user often do not appear as clearly as the first master vector. Therefore, the imaging control unit 154 enters the tracking mode in response to extraction of the first master vector (A) from the master image A when being held or touched. The tracking mode may continue for a predetermined time. The imaging control unit 154 acquires, for example, the master image B1 when it is detected that the holding and lowering is performed. The second master vector (B1) is extracted from the master image B1 and is associated with the first master vector (A) extracted earlier. The tracking mode continues after holding down, and further acquires the master image B2 when the user enters the master shot range. The second master vector (B2) obtained from the master image B2 is also associated with the first master vector (A) that triggered the tracking mode. In this way, various second master vectors obtained thereafter are associated with the first master vector (A) which can easily identify the user. Even if it is the second master vector where features do not easily appear like "back view", it is possible to enrich the master vector group corresponding to one user by associating it with the first master vector that triggered the acquisition. .

The robot 100 and the robot system 300 including the robot 100 have been described above based on the embodiment.
In face recognition technology, a user is often given a language instruction such as “please face forward” or “please look at the camera”, and then a master image is often acquired. Such language instructions such as voice and text tend to be burdensome to the user. In addition, the language instruction for acquiring the master image is not desirable in that it makes the user aware of the inanimate nature of the robot 100. The robot 100 in the present embodiment can acquire a master image casually at the timing when the user holds the robot 100. The robot 100 has a small, soft, light, round shape that human beings like to touch. Instead of forcing the user to take some action, it is possible to capture high-quality master images by capturing the timing at which the user naturally “held”. The master image can be acquired casually by utilizing the characteristic of the robot 100 that stimulates the feeling of wanting to hold and touch.

The robot 100 further draws the user's attention by non-verbal induced motion such as flipping the hand 106. The user is less likely to have a sense of being forced because it is a method of making the user pay attention to it by non-verbal communication.

The robot 100 captures an image of the user when being held and acquires a first master vector. Furthermore, the robot 100 can also acquire a plurality of second master vectors by capturing a master image even when being held down or after being held down. By accumulating the second master vector also at the timing at which the first master vector is extracted, it becomes easier to grasp the physical characteristics of the user in multiple ways.

According to the present embodiment, it becomes easy to establish a fine discrimination criterion for user identification processing based on high quality and a large number of master images. The user identification process is a premise of the recognition of the response action and the closeness calculation. By identifying the user with high accuracy based on the master vector, the robot 100 can change the behavior characteristic according to the user.

The present invention is not limited to the above-described embodiment and modification, and the components can be modified and embodied without departing from the scope of the invention. Various inventions may be formed by appropriately combining a plurality of components disclosed in the above-described embodiment and modifications. Moreover, some components may be deleted from all the components shown in the above-mentioned embodiment and modification.

Although the robot system 300 is described as being configured of one robot 100, one server 200, and a plurality of external sensors 114, part of the functions of the robot 100 may be realized by the server 200, or the functions of the server 200 A part or all of may be assigned to the robot 100. One server 200 may control a plurality of robots 100, or a plurality of servers 200 may cooperate to control one or more robots 100.

A third device other than the robot 100 or the server 200 may have a part of the function. An aggregate of the functions of the robot 100 and the functions of the server 200 described with reference to FIG. 6 can also be generally understood as one “robot”. How to allocate a plurality of functions necessary to realize the present invention to one or more hardwares will be considered in view of the processing capability of each hardware, the specifications required of the robot system 300, etc. It should be decided.

As described above, the “robot in a narrow sense” refers to the robot 100 not including the server 200, while the “robot in a broad sense” refers to the robot system 300. Many of the functions of the server 200 may be integrated into the robot 100 in the future.

The master vector may include feature quantities other than the feature quantities extracted from the master image as vector components. For example, the smell detected by the odor sensor, the voice quality detected by the microphone, and the body temperature detected by the temperature sensor may be included as vector components. In particular, it is easy to detect the user's smell, body temperature, etc. with high accuracy when carrying. The master image may not be a still image, but may be a moving image (hereinafter referred to as "master moving image"). The recognition unit 156 may extract wrinkles, such as how the user walks or a poor baby, from the master moving image, and may include these feature information in the master vector component.

The camera 134 in the present embodiment is an omnidirectional camera, but the camera 134 may be a normal camera. The camera 134 may be built into the horn 112 or may be built into the eye 110. In addition, both an omnidirectional camera and a normal camera may be incorporated.

Although in FIG. 8 the centroid vector is formed by arithmetic averaging of a plurality of master vectors, as a modification, the median of a plurality of master vectors may be used as a component of the centroid vector. For example, if a1 <a2 <a3 in FIG. 8, the a component of the gravity center vector may be a2.

A plurality of evaluation items may be weighted at the time of quality inspection of the master candidate image. As an evaluation item, (E1) is facing the front, (E2) is appropriate, (E3) is the eye open, or the like. The master candidate image may be scored for each evaluation item, and the quality of the master candidate image may be determined by weighted averaging those item points. For example, if the coefficients of p1, p2 and p3 are set to E1 to E3 (p1 + p2 + p3 = 1), and the item values of E1 to E3 are s1, s2 and s3, the combined point is p1 · s1 + p2 · s2 + p3 · s3 . If the combined point is equal to or greater than a predetermined threshold, the master image is adopted, and the master vector is registered in the master information 224.

The induced motion may be performed other than when being held. The motion control unit 150 may execute the induction motion when the distance between the robot 100 and the user is within a predetermined range. For example, when the user is in the master shot range shown in FIG. 13, a master candidate image may be captured after executing the induction motion. In summary, the robot 100 extracts a master vector without missing a “shutter chance” that is effective in grasping the physical and behavioral characteristics of the user while engaging in various ways with the user. A variety of high quality master vectors can be collected without making the user aware. In addition, it is possible to create a "shutter opportunity" positively by non-verbal approach by induced motion.

The induction motion in the present embodiment is a type of non-verbal communication. The non-verbal motion referred to here may include speech that does not make sense as a verb like an animal call. Alternatively, the robot 100 may ask the user in a simple language.

The robot 100 may acquire three face images of a front face, a right face, and a left face as a master image when being held by the user. The recognition unit 156 may determine which direction the user is looking at by recognizing the user's ears and nose. As one of the internal sensors 128, the robot 100 may mount a gyroscope. The recognition unit 156 may detect the tilt direction of the robot 100 when being held by the user using a gyroscope and thereby determine from which direction the user is viewed.

In addition to the methods shown in FIG. 9 and FIG. 10 (hereinafter referred to as “distance determination method”), user identification may be performed by the Mahalanobis distance. In FIG. 10, when a plurality of master vectors are obtained, the person recognition unit 214 determines the Mahalanobis distance (Mahalanobis' Distance) between the test vector DX and the master vector group of the user A in consideration of the variance value. . Similarly, the person recognition unit 214 obtains the Mahalanobis distance between the test vector DX and the master vector group of the user B. Then, based on the Mahalanobis distance for each group, it may be determined whether the unknown user X is the user A or the user B by a known discriminant analysis method (hereinafter referred to as "the Mahalanobis determination method". Say).

The person recognition unit 214 may form a neural network using the master vector group of each user as teacher data, and perform user identification based on the matching between the test vector of the unknown user X and the master vector. (Hereafter, it is called "neural network judgment method").

The person recognition unit 214 may identify a user by combining a plurality of distance determination methods, Mahalanobis determination methods, and neural network determination methods. Further, not only comparison of the inspection vector and the master vector, but also when comparing the master vector of the registered user and the master vector of the unknown user, similarity determination may be performed by the various methods described above.

In the present embodiment, the imaging control unit 154 has been described as acquiring the master image at timing such as holding or touching. As a modification, the imaging control unit 154 may periodically image the user, and the recognition unit 156 may select a large number of captured images as master candidate images. For example, the user is imaged at a timing of once every 10 seconds, and the recognition unit 156 performs quality inspection as a master candidate image. The recognition unit 156 extracts a master vector from the passed master image. According to such a method, it is possible to extract a master vector also from a high-quality captured image obtained by chance.

The person recognition unit 214 may delete the old master vector from the personal data storage unit 218 when the number of master vectors is equal to or more than a predetermined number. Alternatively, old master vectors, for example, master vectors obtained three or more years ago may be deleted. According to such a control method, not only can the amount of data stored in the personal data storage unit 218 be reduced, but it is also possible to cope with changes in physical characteristics as the user ages and grows.

In the present embodiment, it is described that the user is identified by extracting feature vectors in the robot 100 and comparing the feature vectors in the server 200. As a modification, the robot 100 may send a captured image to the server 200, and the person recognition unit 214 of the server 200 may perform both extraction of feature vectors and user identification. Alternatively, the robot 100 may execute the user identification process in the recognition unit 156 without relying on the processing capability of the server 200. In this case, the robot 100 may manage the master vector of each user in the data storage unit 148 of the robot 100.

Not only the camera and various sensors incorporated in the robot 100 but also the sensors incorporated in the external sensor 114 may extract physical and behavioral features of the user. The external sensor 114 captures an image of the user when the user is nearby, and transmits the captured image to the robot 100. The recognition unit 156 of the robot 100 may execute quality inspection and component extraction of this captured image.

In the present embodiment, although the personal data storage unit 218 is described as storing the master vector instead of the master image, both the master image and the master vector may be stored.

The robot system 300 does not have to be provided with a user identification function by a master vector from the time of factory shipment. For example, the robot system 300 may perform user identification by a clustering technology applying deep learning. The functional enhancement of the robot system 300 may be realized by downloading the behavior control program for realizing the user identification function by the master vector via the communication network after the shipment of the robot system 300.

As described above, the recognition unit 156 selects a captured image when the robot 100 is held up as a master candidate image. The recognition unit 156 may detect the position and orientation of the user's face by a temperature sensor such as a thermal camera, or may detect the distance between the user and the robot 100 by a distance measurement sensor. The recognition unit 156 may select, as a master candidate image, a captured image when a predetermined specific condition is satisfied for both or one of temperature information by a thermal camera and distance information by a distance measurement sensor. For example, the recognition unit 156 can confirm that the user is facing the robot 100 by the thermal camera, and the captured image when the distance between the user and the robot 100 is within the predetermined range by using the distance measurement sensor as the master candidate image You may choose. According to such a control method, it becomes easy to select an appropriate master candidate image based on a plurality of types of sensors.

The camera mounted on the robot 100 may be an omnidirectional camera. When the robot 100 is held by the user from the back side, in other words, even when the user does not face the robot 100, the robot 100 can capture the rear user with the omnidirectional camera. Therefore, even when the robot 100 is held from the back side, the recognition unit 156 can acquire an appropriate master candidate image, and thus the acquisition opportunity of the master candidate image can be expanded. Alternatively, the recognition unit 156 may specify the master candidate image on condition that the robot 100 and the user are facing each other.

When a plurality of users appear in the captured image when being held, the recognition unit 156 may not select this captured image as a master candidate image.

When the registered user P1 is detected in the captured image including a plurality of users, the recognition unit 156 extracts the feature vector of the registered user P1 from this captured image, and additionally registers it as a new master vector of the registered user P1. It may be When a registered user is not detected in a captured image including a plurality of users, in other words, when a captured image including only a plurality of unknown users is obtained, the recognition unit 156 faces the front, for example. A feature vector may be extracted for an unknown user P2 that satisfies the condition, and this may be newly registered as a master vector of the unknown user P2.

The recognition unit 156 may also acquire the user's voice (voice information) with a microphone at the time of shooting. The master vector may include not only image information but also feature vectors based on audio information. Similarly, the recognition unit 156 may acquire the user's odor (olfactory information) by the odor sensor when photographing. As described above, various pieces of sensor information such as voice information and olfactory information may be included as information for identifying a registered user, in addition to image information.

The robot 100 may include a plurality of microphones. At the time of voice registration, voice may be detected only from a microphone corresponding to the direction in which the user is present, for example, a microphone attached to the front of the robot 100. The recognition unit 156 may invalidate other microphones. According to such a control method, environmental sounds other than the user are less likely to be taken into the master vector. It is desirable that the microphones, in particular the microphones mounted on the front, have directivity.

The recognition unit 156 may acquire the user's voice information as part of the master vector, on the condition that the voice information is detected when a motion is detected on the user's lips in the captured image. When an unknown user is detected, the recognition unit 156 may execute a motion that approaches the unknown user and asks for holding.

Claims

An imaging control unit that controls the camera;
A recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object;
An operation selection unit that selects a motion of the robot according to the determination result;
A drive mechanism for executing the motion selected by the operation selection unit;
A motion detection unit for detecting lifting of the robot by a moving object;
The recognition unit sets a captured image when the robot is held up by the moving object as a master image, and sets a determination reference of the moving object based on a feature vector extracted from the master image. Autonomous robot.
The autonomous action type robot according to claim 1, wherein the recognition unit sets a plurality of captured images obtained by capturing the moving object from a plurality of angles as a master image.
The operation selection unit causes the drive mechanism to execute a predetermined induction motion.
The autonomous action type robot according to claim 1 or 2, wherein the imaging control unit captures a master image of the moving object in response to execution of the guidance motion.
The autonomous motion robot according to claim 3, wherein the guidance motion is a non-verbal motion.
The autonomous action type robot according to claim 1, wherein the recognition unit further sets, as a master image, a captured image when the robot is held down by the moving object.
The motion selection unit causes a motion object to be selected to track the moving object after the robot is held down by the moving object.
The autonomous action type robot according to claim 1, wherein the imaging control unit captures a master image of the moving object at the time of tracking.
The imaging control unit tracks the moving object even after the robot is held down by the moving object, and captures a second master image of the moving object at a predetermined timing.
The recognition unit extracts a plurality of feature vectors related to the moving object by associating a first master image acquired when the robot is held up with the moving object with the second master image. The autonomous behavior robot according to claim 1, characterized in that
An imaging control unit that controls the camera;
A recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object;
An operation selection unit that selects a motion of the robot according to the determination result;
A drive mechanism for executing the motion selected by the operation selection unit;
A motion detection unit for detecting a touch by a moving object;
The recognition unit sets a captured image when the touch is detected as a master image, and sets a discrimination reference of a moving object based on a feature vector extracted from the master image. robot.
An imaging control unit that controls the camera;
A recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object;
An operation selection unit that selects a motion of the robot according to the determination result;
And a drive mechanism for executing the motion selected by the operation selection unit,
The recognition unit sets, as a master image, an image captured when the moving object is positioned at a predetermined relative point with respect to the robot as a master image, and determines a moving object based on a feature vector extracted from the master image. An autonomous behavior robot characterized by setting.
A computer program for object recognition by a robot, comprising
A function of setting, as a master image, a captured image of the moving object when the robot is held up by the moving object;
A function of setting a discrimination reference of a moving object based on a feature vector extracted from the master image;
And a function of causing a robot to exhibit a function of determining a moving object based on a feature vector extracted from a captured image of the moving object.
A computer program for object recognition by a robot, comprising
Setting a captured image of a moving object when the robot is touched to the moving object as a master image;
A function of setting a discrimination reference of a moving object based on a feature vector extracted from the master image;
And a function of causing a robot to exhibit a function of determining a moving object based on a feature vector extracted from a captured image of the moving object.