JP6705738B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

Image-based object recognition has become a popular technique in recent years. In the image-based object recognition, for example, the feature amount of the image captured by the imaging device is extracted, and the feature amount is matched with the feature amount registered in advance as dictionary data to recognize the object in the image. To be done. Here, since the feature amount changes when the angle of the object shown in the image changes, it is necessary to prepare dictionary data for each angle of the object in order to improve the availability of object recognition.

Patent Documents 1 and 2 are examples of techniques for preparing dictionary data for each angle of an object in image-based object recognition. Patent Document 1 describes a technique for recognizing an object based on eight images obtained by rotating the object at intervals of 45 degrees. Patent Document 2 describes a technique for learning a model of an object by recognizing a common part from a large number of images obtained by rotating the object in the horizontal angle direction and the zenith angle direction in units of 5 degrees.

JP, 2003-346152, A US Patent Application Publication No. 2013/0202212

Techniques such as these are techniques for recognizing objects in the image, i.e. estimating what the object in the image is, and it is not possible to extract any additional information from the image. Not intended. However, in view of the diversification of fields in which the technique of object recognition is used in recent years, it is considered to be advantageous to provide additional information regarding an object based on an image.

Therefore, the present invention is capable of estimating the angle of an object based on an image and autonomously updating data for estimation, which is a new and improved information processing apparatus, information processing method, and program. The purpose is to provide.

According to an aspect of the present invention, a dictionary data acquisition function for acquiring dictionary data regarding an object, an image acquisition function for acquiring a first image of an object, and a first image based on the first image and the dictionary data. A first angle estimation function for estimating the angle of the object in the image, a second angle estimation function for re-estimating the angle of the object in the first image after a physical operation on the object, and a first angle estimation function. An information processing apparatus including a processor that realizes a dictionary data update function that updates dictionary data according to a result of estimation and re-estimation by the second angle estimation function is provided.

According to another aspect of the present invention, a step of acquiring dictionary data regarding the object, a step of acquiring a first image of the object, and an object in the first image based on the first image and the dictionary data. Angle of the object, re-estimating the angle of the object in the first image after physical manipulation of the object, and updating the dictionary data by the processor according to the result of the estimation and the re-estimation. An information processing method including is provided.

According to still another aspect of the present invention, based on a dictionary data acquisition function for acquiring dictionary data regarding an object, an image acquisition function for acquiring a first image of an object, and a first image and dictionary data. A first angle estimation function for estimating the angle of the object in the first image, a second angle estimation function for re-estimating the angle of the object in the first image after a physical operation on the object, A program for causing a processor to realize a dictionary data updating function for updating dictionary data according to the result of the estimation by the angle estimating function and the re-estimation by the second angle estimating function is provided.

It is a figure which shows schematically the system which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the function structure of the terminal in the system shown in FIG. FIG. 2 is a schematic perspective view showing the configurations of a pan head device and a jig used in the system shown in FIG. 1. It is the II sectional view taken on the line of FIG. It is a figure for notionally explaining the dictionary data produced|generated in the 1st Embodiment of this invention. It is a figure for demonstrating the schematic structure of the robot in the system shown in FIG. FIG. 2 is a block diagram showing a functional configuration of a robot in the system shown in FIG. 1. It is a flow chart which shows an example of dictionary data generation processing in a 1st embodiment of the present invention. It is a flow chart which shows an example of recognition processing in a 1st embodiment of the present invention. It is a flow chart which shows an example of pruning processing in a 1st embodiment of the present invention. It is a figure for demonstrating notionally the pruning process shown in FIG. It is a flow chart which shows an example of dictionary data update processing in a 1st embodiment of the present invention. It is a block diagram which shows the function structure of the robot which concerns on the 2nd Embodiment of this invention. It is a figure for explaining roughly the 3rd Embodiment of the present invention. It is a block diagram which shows the function structure of the robot which concerns on the 3rd Embodiment of this invention. It is a block diagram showing an example of hardware constitutions of an information processor in an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, and duplicate description will be omitted.

(First embodiment)
FIG. 1 is a diagram schematically showing a system 10 according to a first embodiment of the present invention. Referring to FIG. 1, the system 10 includes a terminal 100, a database 200, and a robot 300, which are connected to each other by a network NW. A camera 150 and a pan head device 160 are connected to the terminal 100. The robot 300 has a camera 310 and a manipulator 320.

In the system 10 as described above, the camera 150 captures an image of the object obj attached to the pan head device 160 via a jig 170 described later. The terminal 100 acquires an image from the camera 150 and also acquires angle information indicating the angle of the object obj from the camera platform device 160. In the following description, the angle of the object obj is an angle in a three-dimensional space, for example, an angle represented by the amount of rotation around three axes in a rectangular coordinate system, unless otherwise specified. The terminal 100 generates dictionary data based on the acquired image and angle information (and the identification information of the object obj). The generated dictionary data is stored in the database 200.

On the other hand, the robot 300 captures an image of the object obj using the camera 310 while the manipulator 320 holds the object obj. The robot 300 recognizes the object obj in the image based on the captured image and the dictionary data acquired from the database 200, and further estimates the angle of the object obj in the image.

Thereby, the robot 300 can recognize the object obj and further estimate the angle of the object obj held by the manipulator 320. This angle indicates, for example, how much the object obj is rotated with respect to the reference posture. The robot 300 can rotate the object obj by controlling the manipulator 320 based on the estimation result of the angle, thereby making the object obj take a desired posture.

The system 10 as described above is useful, for example, when automating the work of arranging or organizing articles using the robot 300. Also, how should the system 10 rotate the object obj, for example, in order to read information (printed code, RFID (radio frequency identifier), etc.) arranged at a predetermined portion of the object obj. It is also useful for identifying The application of the system 10 is not limited to the above example, and various other applications may be possible.

(Structure for generating dictionary data)
FIG. 2 is a block diagram showing a functional configuration of the terminal 100 in the system shown in FIG. Referring to FIG. 2, the terminal 100 includes an image acquisition unit 110, an angle information acquisition unit 120, and a dictionary data generation unit 130. The terminal 100 is, for example, a personal computer, a tablet, a smartphone, or the like, and the function of each unit is realized by the hardware configuration of the information processing device described below. Specifically, for example, the functions of the image acquisition unit 110, the angle information acquisition unit 120, and the dictionary data generation unit 130 are realized by the processor included in the information processing device. The dictionary data 210 generated by the dictionary data generation unit 130 is stored in the database 200 connected to the terminal 100 via the network. The function of the database 200 is realized by the storage of one or more information processing devices connected to the network. When the terminal 100 includes a plurality of processors, the plurality of processors may cooperate to realize the functions of the above units. Further, as will be described later, some or all of the functions realized by the processor of the terminal 100 can be realized by the server. The function of each unit will be described below.

The image acquisition unit 110 acquires an image of the object obj captured by the camera 150. Here, the camera 150 is an example of an imaging device that captures an image of an object. Specifically, the camera 150 is, for example, a digital camera having an image sensor, and the image acquisition unit 110 receives the image data generated by the camera 150. In the illustrated example, the camera 150 is connected to the terminal 100 via a wired communication interface such as a USB (Universal Serial Bus), but in another example, the camera 150 is a wireless communication such as Bluetooth (registered trademark). It may be connected to the terminal 100 via an interface. Alternatively, the camera 150 may be built in the terminal 100 and transmit image data to the image acquisition unit 110 via a bus.

The angle information acquisition unit 120 acquires angle information indicating the angle of the object obj from the platform device 160. Here, in the present embodiment, the angle information acquired by the angle information acquisition unit 120 of the terminal 100 indicates the angle of the object obj based on the coordinate system of the platform device 160. Even when the angle information acquisition unit 120 itself generates the angle information of the object obj and transmits the angle information to the camera platform device 160 and provides it to the dictionary data generation unit 130, “the angle information acquisition unit 120 acquires the angle information”. Included in the case of "getting". In this case, the platform device 160 sets the angle for holding the object obj according to the angle information received from the angle information acquisition unit 120. In this embodiment, the platform device 160 is an example of a holding unit that holds the object obj. Like the camera 150, the platform device 160 may be connected to the terminal 100 via a wired communication interface or may be connected to the terminal 100 via a wireless communication interface.

As described above, the angle of the object obj is the angle in the three-dimensional space, for example, the angle represented by the rotation amount about the three axes in the orthogonal coordinate system. Therefore, the angle information acquisition unit 120 represents, for example, the angle information by the rotation amount corresponding to the difference between the current posture of the object obj and the reference posture. Here, the reference posture is, for example, the posture of the object obj when the camera platform device 160 is reset. Alternatively, the reference attitude may be the attitude of the object obj when the image acquisition unit 110 first acquires the image of the object obj to generate the dictionary data 210.

The dictionary data generation unit 130 generates the dictionary data 210 based on the image acquired by the image acquisition unit 110, the identification information of the object obj, and the angle information acquired by the angle information acquisition unit 120. Here, the identification information of the object obj may be specified by any means. For example, the identification information of the object obj may be specified based on the information that the user inputs to the terminal 100. Further, the identification information of the object obj may be specified by matching the image acquired by the image acquisition unit 110 with dictionary data separately provided for image-based object recognition. Alternatively, the dictionary data generation unit 130 may assign the identification information to the object obj included in common in the plurality of images acquired by the image acquisition unit 110.

Note that among the information used for generating the dictionary data 210 in the present embodiment, with respect to the combination of the image and the identification information of the object obj, the already known technology in image-based object recognition can be appropriately used. .. For example, the dictionary data generating unit 130 may extract the feature amount from the image by an appropriate method used in the image-based object recognition, and associate the extracted feature amount with the identification information and the angle information of the object obj. .. Further, for example, the dictionary data generating unit 130 may use the identification information of the object obj that is classified and labeled by an appropriate method used in image-based object recognition.

Further, in the present embodiment, the dictionary data 210 is described as being generated based on the identification information of the object obj, but the dictionary data 210 does not necessarily have to be generated based on the identification information of the object obj. For example, if the system 10 is provided for a single type of object obj, the dictionary data 210 may not include the identification information of the object obj. On the other hand, when the dictionary data 210 includes the identification information of the object obj as in the present embodiment, it is possible to further estimate the angle of the object obj after recognizing the object obj for a plurality of types of objects obj.

(Structure of platform and jig)
Hereinafter, the configurations of the camera platform device 160 used together with the terminal 100 in the system 10 according to the present embodiment, and the jig 170 for attaching the object obj to the camera platform device 160 will be further described.

FIG. 3 is a schematic perspective view showing the configurations of the camera platform device 160 and the jig 170 used in the system shown in FIG. FIG. 4 is a cross-sectional view taken along the line II of FIG. Referring to FIGS. 3 and 4, the platform device 160 includes a base 161, a pair of columns 162, a pair of arms 163, a pair of pins 164, a holder 165, a beam 166, and a control unit. And section 167. The jig 170 includes a mounting member 171, a connecting member 172, an object holder 173, and a background plate 174. The background plate 174 is omitted in FIG. 3. Hereinafter, each part will be described.

In the platform device 160, the base 161 is, for example, a rotary table, and is rotated about the axis A ₁ by driving a motor (not shown) controlled by the control unit 167. Here, the axis A ₁ is orthogonal to the optical axis of the camera 150 (shown as axis A _{3 in} FIG. 4). The pair of support columns 162 are fixed on the base 161 at positions symmetrical with respect to the axis A ₁ . Therefore, the midpoint of the pair of struts 162 substantially coincides with the axis A ₁ . The pair of arms 163 is connected to each of the pair of columns 162 using a pin 164 on the side opposite to the base 161. The pin 164 is located on the axis A ₂ which is orthogonal to the axis A ₁ . The pair of arms 163 are each pivotable about an axis A ₂ . Specifically, the pair of support columns 162 and the pin 164, or the pin 164 and the pair of arms 163 are connected via a gear, and a motor controlled by the control unit 167 is connected to the gear (see FIG. The connection of (not shown) pivots the pair of arms 163 about axis A ₂ .

The holder 165 is fixed via a beam 166 between the ends of the pair of arms 163 on the side opposite to the pair of columns 162. The holder 165 is a member to which the camera is attached, for example, when the camera platform device 160 is used as an automatic camera platform, but in the present embodiment, the attachment member 171 of the jig 170 is attached to the holder 165 as described later. It is attached. When the pair of arms 163 pivots about the axis A ₂ as described above, the holder 165 pivots about the axis A ₂ . Here, due to the configuration of the pair of arms 163 as described above, the mounting surface 165s of the holder 165 is maintained in the state of being oriented to the axis A ₂ while the holder 165 is pivoting around the axis A2.

The control unit 167 is, for example, a microcontroller incorporated in the platform device 160, and controls the rotation of the base 161 and the pivotal movement of the pair of arms 163 by controlling the motor as described above. The control unit 167 controls the motor according to, for example, a predetermined procedure or an instruction from the terminal 100. In this way, the control unit 167 sets the angle at which the base 161 rotates about the axis A ₁ and the angle at which the pair of arms 163 pivots about the axis A ₂ . The angle information acquisition unit 120 of the terminal 100 acquires, for example, the information indicating the set value of the angle by the control unit 167 as the angle information.

The pan head device 160 as described above is originally distributed as a device that automates pan (rotation around the axis A ₁ ) and tilt (swing around the axis A ₂ ) of a camera attached to the holder 165. .. In this embodiment, it is intended to efficiently generate the dictionary data 210 covering various angles by automatically setting the angle of the object obj using the platform device 160. However, when the object obj is directly attached to the holder 165 of the platform device 160, when the pair of arms 163 is pivoted, the holder 165 swings around the axis A ₂ , and as a result, the position of the object obj is changed by the light of the camera 150. It will deviate significantly from the axis (shown as axis A _{3 in} FIG. 4). Therefore, in this embodiment, the object obj is attached to the pan head device 160 via a jig 170 as described below.

In the jig 170, the attachment member 171 is a member that can be attached to the holder 165 of the platform device 160. For example, the mounting member 171 is provided with a mounting structure corresponding to the structure for fixing the camera provided in the holder 165. Specifically, when the holder 165 is provided with a screw for fixing the camera, the mounting member 171 is provided with a screw hole. Alternatively, the mounting member 171 may be provided with a mounting structure that can be used regardless of the structure of the holder 165. Specifically, the attachment member 171 may be provided with a clip that holds the holder 165, a belt wound around the holder 165, or the like.

The object holder 173 is a member to which the object obj can be attached. For example, the object holder 173 is provided with a mounting structure capable of fixing the object obj while reducing the contact area with the object obj as much as possible. This is because the contact area between the attachment structure and the object obj can be an occlusion area in the image of the object obj captured by the camera 150. Specifically, the object holder 173 may be provided with a clip that holds the object obj, a hook that holds the object obj, an adhesive surface to which the object obj is attached, and the like. A magnet may be provided in the object holder 173 for the object obj which is a magnetic body.

The connecting member 172 connects the mounting member 171 and the object holder 173. Further, the connecting member 172 is located near the intersection of the axis A ₁ and the axis A ₂ with the object obj attached to the object holder 173 when the attachment member 171 is attached to the holder 165 of the camera platform device 160. Thus, the positional relationship between the mounting member 171 and the object holder 173 is defined. For example, the connecting member 172 is connected to the mounting member 171 so as to extend along the pair of arms 163 when the mounting member 171 is mounted on the holder 165. At this time, the length of the connecting member 172 in the direction along the pair of arms 163 is determined by the distance between the holder 165 and the axis A ₂ from the thickness of the attachment member 171 and the object holder 173 and the thickness of the object obj. Is almost equal to the half of. The connection member 172 may have a structure capable of adjusting the length in the direction along the arm 163. This makes it possible to adjust the length of the connecting member 172 according to the size of the object obj and bring the center of the object obj close to the intersection of the axis A1 and the axis A2.

The object obj attached to the platform device 160 via the jig 170 as described above is located near the intersection of the axis A ₁ and the axis A ₂ . Therefore, even when the base 161 of the platform device 160 is rotated around the axis A _{1 and} the pair of arms 163 is pivoted around the axis A ₂ , the position of the object obj is substantially unchanged. In other words, the optical axis of the camera 150 (shown as the axis A _{3 in} FIG. 4) is not largely deviated. Therefore, in this embodiment, the control unit 167 of the platform device 160 sets the angle at which the base 161 rotates about the axis A ₁ and the angle at which the pair of arms 163 pivots about the axis A _2. Then, these angles can be regarded as the rotation amounts of the object obj about the axis A ₁ and the axis A ₂ .

When using the platform device 160 and the jig 170 as described above, the object obj cannot be rotated around the axis A ₃ orthogonal to the axis A ₁ and the axis A ₂ , that is, the optical axis of the camera 150. However, the rotation around the axis A ₃ can be accurately complemented by rotating the image captured by the camera 150 in a plane. Further, in the above description, the object obj is on the optical axis of the camera 150 for simplification, but the object obj is not necessarily on the optical axis of the camera 150.

The background plate 174 is attached to the connecting member 172 or the object holder 173 and provides a background of the object obj. For example, the background plate 174 may be provided with a mounting structure for selectively mounting the screen. The screen may include, for example, a plurality of screens made of different materials. The material can include, for example, paper, cloth, or film. The screen may also include a plurality of screens having different colors or different reflection characteristics. By exchanging the screen, it is possible to exchangeably provide a plurality of backgrounds of the object obj having different materials, colors, reflection characteristics, or the like. Further, for example, the background plate 174 may be detachably attached to the connecting member 172 or the object holder 173. In this case, by selectively attaching the plurality of background plates 174, it is possible to exchangeably provide a plurality of backgrounds of the object obj having different materials, colors, reflection characteristics, or the like. Specifically, for example, the background plate 174 may include a plurality of background plates 174 whose surfaces facing the object obj are made of different materials. The material can include, for example, paper, cloth, or film. In addition, the background plate 174 may include a plurality of background plates 174 in which surfaces facing the object obj have different colors or different reflection characteristics.

(Conceptual explanation of dictionary data)
FIG. 5 is a diagram for conceptually explaining the dictionary data generated in the first embodiment of the present invention. FIG. 5 illustrates dictionary data 210 associated with an object obj (a connector in the illustrated example) specified by certain identification information. In the illustrated example, the angle of the object obj is a vector amount represented by the amount of rotation around the three axes (X axis, Y axis, Z axis) of the orthogonal coordinate system in the three-dimensional space. Regarding the angle of the object obj, the dictionary data 210 divides the entire circumference into N _X pieces about the rotation amount around the X axis (rot_X), and makes the entire circumference into N _Y pieces around the rotation amount around the Y axis (rot_Y). It includes at least N _X ×N _Y ×N _Z elements defined by dividing and dividing the entire circumference into N _Z pieces about the rotation amount (rot_Z) about the Z axis. Each element is associated with at least one piece of information corresponding to the image of the object obj. Here, the information corresponding to the image of the object obj is, for example, a feature amount extracted from the image captured by the camera 150 when the object obj has an angle represented by a rotation amount (rot_X, rot_Y, rot_Z). sell.

In the above example, the division widths of the rotation amounts (rot_X, rot_Y, rot_Z) about the respective axes may be different (that is, at least one of N _X , N _Y , and N _Z is different from the others). May be different). Further, the rotation amount does not necessarily have to be evenly divided. For example, in the angle estimation of the object obj described later, if there is an angle that is difficult to estimate with high reliability, the division width of the rotation amount is set to be smaller than other portions in the vicinity of the rotation amount corresponding to the angle. May be done.

For example, when the camera 310 of the robot 300 captures an image of an object obj whose angle is unknown, the feature amount extracted from the captured image and the feature amount associated with the elements of the dictionary data 210 as described above. By matching and, the angle of the object obj can be estimated.

Here, the dictionary data 210 may include a plurality of elements generated based on the angle information of the same object obj and a plurality of different images. In this case, the dictionary data 210 has more elements than N _X ×N _Y ×N _Z. For example, the environmental conditions at the time of capturing may be different between the plurality of images associated with the same angle information. The environmental conditions can be, for example, the background or the placement of light. By generating the dictionary data 210 under a plurality of different environmental conditions, it is possible to provide the dictionary data 210 capable of estimating the angle of the object obj under various environmental conditions.

In the above case, the image acquisition unit 110 of the terminal 100 acquires a plurality of different images of the object obj. For example, the image acquisition unit 110 respectively displays images of the object obj when the control unit 167 of the pan head device 160 sets the same angle before and after the background of the object obj is exchanged using the background plate 174 of the jig 170. You may get it. In this case, the dictionary data generation unit 130 is based on a plurality of images with different backgrounds, identification information of the object obj common to these images, and angle information indicating the angle of the object obj common to these images. Then, a plurality of elements of the dictionary data 210 are generated.

(Robot configuration)
FIG. 6 is a diagram for explaining a schematic configuration of the robot 300 in the system shown in FIG. Referring to FIG. 6, the robot 300 includes a camera 310, a manipulator 320, a controller 330, a sensor 340, and a motor 350. Under the control of the control unit 330, the robot 300 can grasp the object obj using the manipulator 320 and capture an image of the object obj using the camera 310, for example. In the present embodiment, the manipulator 320 is also an example of a holding unit that holds the object obj, similarly to the pan head device 160 described above. The control unit 330 is realized, for example, by a hardware configuration of an information processing device described later.

The sensor 340 includes a sensor that is used by the robot 300 or is transmitted from the robot 300 to another device to obtain various measurement values. Specifically, the sensor 340 may include an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, and/or a GNSS (Global Navigation Satellite System) receiver. Further, the sensor 340 may include a depth sensor or a laser range scanner such as a LIDAR (Laser Imaging Detection and Ranging).

The motor 350 operates each unit of the robot 300 under the control of the control unit 330. The motor 350 may include, for example, a motor (actuator) that operates a joint structure (not shown) to change the posture of the robot 300 or move the robot 300. The motor 350 may include a motor for rotating the wheels to move the robot 300. The configuration of each part of the robot 300 including the motor 350 can be appropriately configured based on a known robot design method. Here, the robot 300 does not necessarily have to change its posture and may not move. Similarly, the robot 300 need not necessarily include joint structures (other than the manipulator 320) and need not include wheels.

(Structure for estimating the angle of the object)
FIG. 7 is a block diagram showing a functional configuration of the robot 300 in the system shown in FIG. Referring to FIG. 7, in addition to the camera 310 and the manipulator 320, the robot 300 includes an image acquisition unit 331, a dictionary data acquisition unit 332, an object recognition/angle estimation unit 333, a result output unit 334, and a dictionary data update. A unit 335, a manipulator control unit 336, and an angle information acquisition/angle estimation unit 337 are included. Each unit other than the camera 310 and the manipulator 320 is realized by, for example, a processor of an information processing device that realizes the control unit 330 of the robot 300 described above. When the control unit 330 includes a plurality of processors, the plurality of processors may cooperate to realize the functions of the above units. Further, as will be described later, it is possible to realize some or all of the functions realized by the processor of the control unit 330 in the server. The function of each unit will be described below. The function relating to the update of the dictionary data will be described in detail later with reference to the flowchart, and therefore the description will be limited to a simple description here.

The image acquisition unit 331 acquires the image of the object obj captured by the camera 310. Here, in the present embodiment, the camera 310 is also an example of an imaging device that captures an image of an object, like the camera 150 described above. The image captured by the camera 150 and the image captured by the camera 310 are different images including the same type of object obj. Specifically, the camera 310 is a digital camera having an image sensor, for example, and the image acquisition unit 331 receives the image data generated by the camera 310. For example, the robot 300 holds the object obj using the manipulator 320. In this case, the image acquired by the image acquisition unit 331 includes the object obj held by the manipulator 320. Alternatively, the image acquisition unit 331 may include an object obj placed on the table, floor, or the like without being held by the manipulator 320. In the illustrated example, the camera 310 is built in the robot 300 and transmits image data to the image acquisition unit 331 via a bus, but the camera 310 is externally connected to the robot 300 via a wired communication interface or a wireless communication interface. May be done.

The dictionary data acquisition unit 332 acquires the dictionary data 210 from the database 200 connected to the robot 300 via the network. As described above, the dictionary data 210 is generated based on the image of the object obj and the angle information (and the identification information of the object obj). The robot 300 uses the dictionary data 210 to estimate the angle of the object obj held by the manipulator 320. The dictionary data acquisition unit 332 does not necessarily have to acquire the entire dictionary data 210. For example, when the dictionary data 210 is generated for a plurality of types of objects and the object obj included in the image acquired by the image acquisition unit 331 has already been identified, the dictionary data acquisition unit 332 determines that the object in the dictionary data 210 is the object. The element associated with the obj identification information is selectively acquired.

The object recognition/angle estimation unit 333 estimates the angle of the object obj in the image based on the image of the object obj acquired by the image acquisition unit 331 and the dictionary data 210 acquired by the dictionary data acquisition unit 332. When the dictionary data 210 is generated for a plurality of types of objects and the object obj included in the image acquired by the image acquisition unit 331 is not identified, the object recognition/angle estimation unit 333 determines that the object obj by the image-based object recognition. Specify the identification information of. As for the image-based object recognition, a known technique can be applied, and thus a detailed description thereof will be omitted. For example, when the dictionary data 210 is generated for a single type of object, or when the object obj included in the image acquired by the image acquisition unit 331 is already identified, the object recognition/angle estimation unit 333 performs object recognition. Do not run.

On the other hand, the object recognition/angle estimation unit 333 estimates the angle of the object obj, for example, by matching the image acquired by the image acquisition unit 331 with the element of the dictionary data 210. In this case, the angle associated with the element of the dictionary data 210 having the highest matching score is estimated as the angle of the object obj in the image. As will be described later, the dictionary data 210 for estimating the angle of the object obj can include many elements. Therefore, the object recognition/angle estimation unit 333 may prun the dictionary data 210 based on the image acquired by the image acquisition unit 331 and execute matching between the pruned dictionary data 210 and the image. Here, in the present embodiment, the pruning is a process of determining the dictionary data 210 that is not a target of the matching by a process having a lighter processing load than the matching for estimating the angle of the object obj.

The result output unit 334 outputs the result of recognition by the object recognition/angle estimation unit 333. As described above, the robot 300 may use the estimation result of the angle of the object obj for the operation of the robot 300 itself, for example, the control of the manipulator 320, but if necessary, the estimation result may be output in some form. May be. More specifically, for example, the estimation result may be displayed as an image on the display of the robot 300 or may be output as sound from the speaker. The estimation result may be transmitted from the communication device of the robot 300 to another device via the network. The result output unit 334 controls the output of the estimation result as described above. The result output unit 334 is not provided when it is not necessary to output the estimation result.

The dictionary data updating unit 335 stores the dictionary data 210 according to the result of the estimation of the angle of the object obj by the object recognition/angle estimation unit 333 and the result of the angle information acquisition/angle estimation by the angle estimation unit 337 described later. Update. More specifically, when the reliability of the angle estimated by the object recognition/angle estimation unit 333 does not exceed the threshold, the dictionary data update unit 335 performs the angle information acquisition/angle estimation by the angle estimation unit 337. The dictionary data 210 is updated based on the result. In the following description, the angle estimation function of the object recognition/angle estimation unit 333 is the “first angle estimation function”, and the angle re-estimation function of the angle information acquisition/angle estimation unit 337 is the “second angle estimation function”. Also called "function." These angle estimation functions are not necessarily performed independently of each other. For example, as will be described later, the angle information acquisition/angle estimation unit 337 uses the estimation result of the angle by the object recognition/angle estimation unit 333 when re-estimating the angle. That is, the "first angle estimating function" may be executed independently, and the "second angle estimating function" may call the "first angle estimating function".

The manipulator control unit 336 controls the manipulator 320 of the robot 300 holding the object obj. When the dictionary data updating unit 335 updates the dictionary data 210, the manipulator control unit 336 controls the manipulator 320 to rotate the object obj. The rotation referred to here means a change in the angle of the object obj. The rotation of the object obj is an example of a physical operation relating to the object obj, which is executed in re-estimating the angle of the object obj.

The angle information acquisition/angle estimation unit 337 acquires angle information indicating the angle of the object obj from the manipulator control unit 336. Here, in the present embodiment, the angle information acquired by the angle information acquisition/angle estimation unit 337 of the robot 300 indicates the angle of the object obj based on the coordinate system of the robot 300 or the manipulator 320. Therefore, in this embodiment, the angle information acquired from the manipulator control unit 336 is not necessarily directly associated with the angle information of the dictionary data 210. Therefore, in the present embodiment, the angle information acquisition/angle estimation unit 337 calculates the rotation amount Δθ of the object obj from the angle information before and after the manipulator control unit 336 controls the manipulator 320 to rotate the object obj, and this rotation The quantity Δθ is used for re-estimation of the angle described later.

Further, the angle information acquisition/angle estimation unit 337 calculates the angle θ _{2 of the} object obj estimated by the object recognition/angle estimation unit 333 based on the rotated image (second image) of the object obj and the dictionary data 210, The angle θ ₁ of the object obj in the image before rotation of the object obj (first image) is re-estimated based on the rotation amount Δθ (briefly expressed, θ ₁ =θ ₂ −Δθ). Here, the rotation amount Δθ is an example of the amount of physical operation on the object obj. It should be noted that each of the angle θ ₁ , the angle θ ₂ , and the rotation amount Δθ is a vector amount including an element of rotation around each axis of the coordinate system (rot_X, rot_Y, rot_Z in the example of FIG. 5). sell.

As described above, the dictionary data update unit 335 determines the reliability of the angle θ ₂ of the object obj estimated by the object recognition/angle estimation unit 333 based on the rotated image (second image) of the object obj and the dictionary data 210. Is greater than the threshold value, the dictionary information is obtained based on the angle information indicating the angle θ ₁ re-estimated by the angle information acquisition/angle estimation unit 337 and the image before rotation of the object obj (first image). Update 210.

On the other hand, when the reliability of the angle θ ₂ of the object obj estimated based on the rotated image (second image) of the object obj and the dictionary data 210 by the object recognition/angle estimation unit 333 does not exceed the threshold value, the manipulator control is performed. By controlling the manipulator 320 by the unit 336, the object obj is further rotated by the rotation amount Δθ′, and the object recognition/angle estimation unit 333 causes the object obj to rotate based on the rotated image (third image) and the dictionary data 210. Estimate the angle θ _{3 of the} object obj. When the reliability of the angle θ ₃ exceeds the threshold, the angle information acquisition/angle estimation unit 337 re-estimates the angle θ ₁ based on the angle θ ₃ and the total amount of rotation (Δθ+Δθ′), and based on this result. The dictionary data updating unit 335 updates the dictionary data 210.

In this way, the dictionary data updating unit 335, based on the angle θ ₁ and the image before rotation of the object obj (first image), when the angle θ ₁ is reestimated with sufficient reliability. The data 210 is updated. Specifically, the dictionary data updating unit 335 adds or replaces an element of the dictionary data 210. Thereby, when the camera 310 subsequently captures an image of the object obj at the angle θ ₁ under the same environmental condition, the possibility that the angle θ ₁ can be estimated with high reliability is increased without re-estimation.

(Example of processing flow)
Hereinafter, an example of the processing flow in the system 10 according to the present embodiment will be described with reference to FIGS. 8 to 12.

FIG. 8 is a flowchart showing an example of dictionary data generation processing according to the first embodiment of the present invention. Referring to FIG. 8, in the registration process, first, the image acquisition unit 110 of the terminal 100 acquires an image (step S101), and the angle information acquisition unit 120 acquires angle information (step S103). Either of steps S101 and S103 may be executed first, or may be executed in parallel. For example, the angle information acquisition unit 120 may acquire the angle information from the camera platform device 160, when the image acquisition unit 110 acquires the image captured by the camera 150 in real time. Further, the image acquisition unit 110 may acquire the image captured by the camera 150 in real time, triggered by the angle information acquisition unit 120 transmitting the angle information to the camera platform device 160. Alternatively, the image acquisition unit 110 continuously acquires the images captured by the camera 150 in time series, and the angle information acquisition unit 120 continuously acquires the angle information set in the camera platform device 160 in time series. Good.

Next, the dictionary data generation unit 130 of the terminal 100 associates the image in step S101 with the angle information acquired in step S103. For example, when both the image and the angle information are acquired in real time, the dictionary data generation unit 130 associates the image and the angle information acquired at substantially the same time. On the other hand, when the image and the angle information are acquired with a time difference or afterwards, the dictionary data generating unit 130 associates the image having the common key with the angle information. The key in this case may be, for example, a time stamp or a sequence number allocated separately from the time stamp.

Next, the dictionary data generation unit 130 generates the dictionary data 210 based on the images and the angle information associated with each other in step S105 (step S107). Here, as described above, the dictionary data generation unit 130 may generate the dictionary data 210 by applying a known image-based object recognition technique. Further, for example, when the substantially same set of image and angle information is continuously acquired, the dictionary data generation unit 130 determines that the information is redundant and generates the dictionary data 210. May be omitted.

FIG. 9 is a flowchart showing an example of recognition processing according to the first embodiment of the present invention. Referring to FIG. 9, in the recognition process, first, the image acquisition unit 331 of the robot 300 acquires an image (step S301). As described above, the image acquisition unit 331 acquires the image captured by the camera 310, and the image includes the object obj held by the manipulator 320, for example. Next, the dictionary data acquisition unit 332 acquires the dictionary data 210 from the database 200 (step S303).

Next, the object recognition/angle estimation unit 333 recognizes the object obj based on the image acquired in step S301 and the dictionary data 210 acquired in step S302 (step S305). Note that the image-based object recognition can be applied to a known technique, and thus detailed description thereof will be omitted. Further, as described above, for example, when the dictionary data 210 is generated for a single type of object, or when the object obj included in the image is already identified, the object recognition in step S305 is omitted.

Next, the object recognition/angle estimation unit 333 executes pruning of the dictionary data 210 (step S307). For example, when the dictionary data 210 as shown in FIG. 5 is generated by dividing the entire circumference into 52 pieces about the rotation amount (rot_X, rot_Y, rot_Z) around each axis (that is, N _X =N _Y =N _Z =52), dictionary data 210 having at least 52 ³ =140,608 elements will be generated. As described above, when the dictionary data 210 is generated by associating a plurality of different images with the same angle, the number of elements is further increased. Since the processing load for executing matching with all the elements of the dictionary data 210 is enormous, the benefit of pruning the dictionary data 210 is great.

FIG. 10 is a flowchart showing an example of the pruning process in the first embodiment of the present invention. FIG. 11 is a diagram for conceptually explaining the pruning process shown in FIG. 10. Referring to FIG. 10, the object recognition/angle estimation unit 333 first determines a pruning procedure corresponding to the object obj (step S331). The pruning procedure corresponding to the object obj is, for example, determined in advance and stored in the database 200 together with the dictionary data 210. When step S305 shown in FIG. 9 is executed, the object recognition/angle estimation unit 333 determines the pruning procedure according to the object recognition result in step S305.

Subsequent steps S333 and S335 are an example of processing executed according to the pruning procedure corresponding to the object obj in the example shown in FIG. The processing executed here may differ depending on the type of object. In the above example, the object recognition/angle estimation unit 333 masks the image (step S333) and further reduces the color of the image (step S335). Next, the object recognition/angle estimation unit 333 executes pruning (step S337). In the illustrated example, for example, a plurality of characteristic portions are extracted from the image subjected to masking and color reduction as described above, and the positional relationship of the plurality of similarly extracted characteristic portions of the dictionary data 210 is not common to the image. The element is excluded from matching.

In the example shown in FIG. 11, the object obj is a connector. In the illustrated example, a pruning procedure focusing on the colors of the cables (cable1 to cable3) is set. In step S333 shown in FIG. 10, the portion of the image other than the cable is masked (mask). This eliminates the influence of the shadow of the terminal cover existing in the masked portion. Further, although not shown in the figure, in step S335, the image is reduced in color under the condition that the difference in color between the cables (cable1 and cable3) at both ends is expressed. This facilitates extraction of the cables (cable1 and cable3) at both ends as two characteristic parts in each element of the image and the dictionary data 210.

Further, in step S337 shown in FIG. 10, pruning of the dictionary data 210 is executed based on the mask and the color-reduced image as described above. Specifically, for example, in the image, cable1 is located at the upper right when viewed from cable3. On the other hand, in the element group 210b of the dictionary data 210 (where the connector is rotated around the viewpoint axis), the cable1 is located at the upper left as viewed from the cable3. In addition, in the element group 210c (the connector is turned inside out), the cable1 is located at the lower left when viewed from the cable3. Therefore, in step S337, the element groups 210b and 210c are excluded from the matching targets. As a result, the matching is executed only for the element group 210a (similar to the image, the cable1 is located on the upper right side when viewed from the cable1).

Returning to FIG. 9, after pruning the dictionary data 210 in step S307, the object recognition/angle estimation unit 333 executes matching between the image and the dictionary data 210 (step S309). The matching can be, for example, template matching. It should be noted that since a known technique can be applied to image matching, detailed description thereof will be omitted. Note that in the already known image-based object recognition, a score in object units is calculated as a result of matching, but in step S307, a score in angle units of objects is calculated.

Next, the object recognition/angle estimation unit 333 estimates the angle of the object obj based on the result of the matching in step S309 (step S311). The estimation result in step S311 can be, for example, the angle indicated by the angle information associated with the element of the dictionary data 210 for which the highest score was calculated in the matching in step S309.

Next, the object recognition/angle estimation unit 333 determines whether or not the score calculated by the matching in step S309 exceeds the threshold value (step S313). The score compared with the threshold here is, for example, the highest matching score. Alternatively, it may be determined whether or not some percentage (for example, 10%) in the top of the matching scores exceeds the threshold value. When the matching score does not exceed the threshold value in the determination of step S313 (NO), the dictionary data updating unit 335 updates the dictionary data 210 (S315). On the other hand, if the matching score exceeds the threshold in the determination of step S313 (YES), the process of updating the dictionary data 210 may not be executed. The result of estimation in step S311 is output by the result output unit 334 as needed.

FIG. 12 is a flowchart showing an example of dictionary data update processing according to the first embodiment of the present invention. Referring to FIG. 12, in the update process, first, the angle information acquisition/angle estimation unit 337 of the robot 300 stores the angle information of the object obj provided from the manipulator control unit 336 (step S351). Here, the angle stored in step S351 indicates, for example, the angle of the object obj in the coordinate system based on the manipulator 320. Next, the manipulator control unit 336 rotates the object obj by controlling the manipulator 320 (step S353).

After the object obj is rotated, the angle of the object obj is estimated (step S355). The processing of step S355 corresponds to the processing of steps S301 to S311 shown in FIG. 9, for example. Specifically, the image acquisition unit 331 acquires the rotated image (second image) of the object obj, and the object recognition/angle estimation unit 333 determines the angle of the object obj in the rotated image (second image). To estimate. The dictionary data 210 may be the one acquired in step S303 executed previously, or the object obj may be treated as already recognized in step S305 executed previously.

Next, the dictionary data updating unit 335 determines whether the matching score in the estimation in step S355 exceeds the threshold value (step S357). This determination can be performed, for example, in the same manner as step S309 shown in FIG. If the matching score does not exceed the threshold value in the determination of step S357 (NO), the processes of step S353 and step S355 are re-executed. That is, the manipulator control unit 336 controls the manipulator 320 to further rotate the object obj (step S353), and the object recognition/angle estimation unit 333 estimates the angle of the object obj in the rotated image (third image). Yes (step S355).

On the other hand, when the matching score exceeds the threshold value in the determination of step S357 (YES), the angle information acquisition/angle estimation unit 337 determines the initial angle θ from the angle θ ₂ estimated in step S355 and the rotation amount Δθ of the object obj. ₁ is reestimated (step S359). Here, the initial angle θ ₁ is an angle before the rotation of the object obj, and is an angle that the object recognition/angle estimation unit 333 could not estimate with sufficient reliability. On the other hand, the angle θ ₂ is the angle of the object obj estimated by the object recognition/angle estimation unit 333 based on the rotated image (second image) of the object obj and the dictionary data 210, and is sufficiently determined by the determination in step S357. It has been proved that it is estimated with high reliability. The rotation amount Δθ is calculated based on the angle information of the object obj stored in step S351 and the angle information of the object obj provided from the manipulator control unit 336 at the time of step S357.

As a result of the determination in step S357, when the processes of step S353 and step S355 are repeated N times, the angle information acquisition/angle estimation unit 337 causes the angle θ _N+1 estimated in step S355 executed last, and The initial angle θ ₁ is re-estimated from the total amount of rotation Δθ _TTL of the object obj in step S353 executed N times. The total amount of rotation Δθ _TTL is calculated based on the angle information of the object obj stored in step S351 and the angle information of the object obj provided from the manipulator control unit 336 at the time of step S357.

Next, the dictionary data update unit 335, the angle information corresponding to the initial angle θ ₁ re-estimated in step S359, and the pre-rotation image (first image) of the object obj acquired in step S301 illustrated in FIG. ) Are associated with each other (step S361). Further, the dictionary data updating unit 335 updates the dictionary data 210 based on the image and the angle information associated with each other in step S361 (step S363). Here, updating the dictionary data 210 includes adding an element of the dictionary data 210 and/or replacing an element of the dictionary data 210.

In the above step S363, the dictionary data updating unit 335 adds an element of the dictionary data 210 based on the image and the angle information. Thereby, when the camera 310 of the robot 300 subsequently captures the image of the object obj at the angle θ ₁ under the same environmental condition, the possibility that the angle θ ₁ can be estimated with high reliability increases. Note that, for example, when the dictionary data 210 is dedicated to the robot 300 and it is expected that the environmental condition when the camera 310 captures the image of the object obj does not change, the dictionary data updating unit 335 determines that the image and the angle information. The elements of the dictionary data 210 may be replaced based on

By updating the dictionary data 210 as described above, the additional dictionary data 210 is accumulated for the angle or the environmental condition of the object obj that is difficult to estimate with high reliability in the first generated dictionary data 210. be able to. In this way, the robot 300 that estimates the angle of the object obj using the dictionary data 210 autonomously enhances the dictionary data 210, and thus the robustness of estimation can be improved.

(Example of verification process before update)
Here, the dictionary data update process described above with reference to FIG. 12 may include a verification process before updating the dictionary data 210 as an additional process. As a first example, before step S351 shown in FIG. 12, a process of verifying whether or not to execute the dictionary data update process may be executed (illustrated as step S371 “verification process 1”). ). In the verification process according to the first example, the image acquisition unit 331 reacquires the image of the object obj before the object obj is rotated in step S353. The object recognition/angle estimation unit 333 estimates the angle of the object obj in the reacquired image. If the matching score in this estimation exceeds the threshold (unlike the estimation in step S311 shown in FIG. 9), the dictionary data update processing is stopped, and at least the dictionary data update in step S363 is not executed.

In the first example described above, for example, in the image acquired by the image acquisition unit 331 in step S301 illustrated in FIG. 9 described above, the focus of the camera 310 is delayed or instantaneous (for example, due to lightning or flash light). Accidental factors, such as changes in lighting conditions, can cause unexpected changes in the image, which can reduce the reliability of the estimate. The verification process as in the first example described above is effective in preventing the dictionary data 210 from being updated based on information with low reproducibility due to an accidental factor.

As a second example, after step S361 shown in FIG. 12, a process of verifying whether or not to update the dictionary data based on the prepared angle information and image may be executed (step). This is shown as S373 "Verification process 2"). In the verification process according to the second example, the dictionary data update unit 335 generates temporary dictionary data based on the angle information and the image associated in step S361. Next, the manipulator control unit 336 controls the manipulator 320 to rotate the object obj, contrary to step S353. As a result, the object obj is returned to the original angle θ ₁ . Further, the image acquisition unit 331 newly acquires the image of the object obj returned to the original angle θ ₁ , and the object recognition/angle estimation unit 333 detects the object obj of the image newly acquired by the image acquisition unit 331. The angle is estimated based on the temporary dictionary data generated by the dictionary data updating unit 335. Here, the original angle θ ₁ can be estimated, and when the matching score exceeds the threshold value, the dictionary data updating unit 335 executes the update of the dictionary data 210 in step S363. If not, the dictionary data 210 is not updated in step S363.

The second example described above is effective, for example, to prevent updating of the dictionary data 210 that does not contribute to improving the reliability of angle estimation. Depending on the environmental conditions when the camera 310 captures the image of the object obj, even if the dictionary data 210 is updated based on the image acquired by the image acquisition unit 331, the estimation of the angle in the similar image acquired thereafter may be performed. The reliability may not always improve. The verification process as in the second example described above is effective in preventing the capacity of the dictionary data 210 from increasing due to unnecessary elements that do not necessarily contribute to improving the reliability of angle estimation.

(Other modifications)
Note that, in the above example, the angle information acquisition/angle estimation unit 337 re-estimates the angle after the object obj is rotated, but in another example, the angle information acquisition/angle estimation unit 337 indicates that the robot 300 has the motor 350. The angle may be re-estimated after being moved with the object obj by. Due to the movement of the robot 300, the environmental condition when the camera 310 captures an image changes, and there is a possibility that the angle can be estimated with high reliability without rotating the object obj. The configuration for moving the robot 300 is described in more detail in the third embodiment described later.

Further, the movement of the robot 300 as described above may be combined with the rotation of the object obj. For example, if the angle information acquisition/angle estimation unit 337 does not obtain sufficient reliability even by re-estimating the angle after rotating the object obj, the angle information is re-adjusted after the robot 300 is moved together with the object obj. It may be estimated. For example, when the environmental condition when the camera 310 captures the image of the object obj is significantly different from the environmental condition of the camera 150 when the dictionary data 210 is generated, the re-estimation process as described above is effective. sell.

The functions of the system 10 according to the present embodiment are realized by being distributed to the terminal 100, the database 200, and the robot 300 in the examples shown in FIGS. In other examples, most of the functionality of system 10 may be implemented in a server. That is, the functions described as being realized by the processors of the terminal 100 and the robot 300 in the above example may be realized by the processor of the server including the database 200. In this case, the terminal 100 transmits the image of the object obj captured by the camera 150 and the angle information of the object obj acquired from the camera platform device 160 to the server, and the server associates these with each other to generate the dictionary data 210. On the other hand, the robot 300 transmits the image of the object obj captured by the camera 310 to the server, and the server estimates the angle of the object obj based on this image. The robot 300 receives the angle estimation result from the server. If the reliability of the estimated angle does not exceed the threshold value, the server requests the robot 300 to rotate the object obj for re-estimating the angle and acquire an image of the rotated object obj. Good. Note that the number of servers that realize these functions does not have to be one, and the above functions may be realized by a plurality of servers distributed on the network. The server that realizes the above function may be a device other than the storage that includes the database 200.

(Second embodiment)
Next, a second embodiment of the present invention will be described. It should be noted that parts that are configured in the same manner as in the above-described first embodiment will be denoted by common reference numerals, and redundant description may be omitted.

FIG. 13 is a block diagram showing a functional configuration of a robot 300a according to the second embodiment of the present invention. Referring to FIG. 13, in the present embodiment, the entire functions of generating the dictionary data 210 and estimating the angle of the object obj using the dictionary data 210 are realized in the robot 300a. Specifically, the processor of the control unit 330 of the robot 300a includes image acquisition units 110 and 331, angle information acquisition/angle estimation units 120 and 337, dictionary data generation/update units 130 and 335, and dictionary data acquisition units. 332, an object recognition/angle estimation unit 333, a result output unit 334, and a manipulator control unit 336 are realized. When the control unit 330 includes a plurality of processors, the plurality of processors may cooperate to realize the functions of the above units. Further, as will be described later, some or all of the functions realized by the processor of the control unit 330 can be realized by the server. Further, the database 200 is stored in the storage of the control unit 330 of the robot 300a. Hereinafter, each part will be further described.

The image acquisition units 110 and 331 have both functions of the image acquisition unit 110 described above with reference to FIG. 2 and the image acquisition unit 331 described with reference to FIG. 7. That is, the image acquisition units 110 and 331 use the dictionary data 210 to determine the angle of the object obj for the dictionary data generation/update units 130 and 335 when generating the dictionary data 210 for the image of the object obj captured by the camera 310. When estimating, they are provided to the object recognition/angle estimation unit 333, respectively.

The angle information acquisition/angle estimation units 120 and 337 have functions of both the angle information acquisition unit 120 described above with reference to FIG. 2 and the angle information acquisition/angle estimation unit 337 described with reference to FIG. 7. Have. That is, when generating the dictionary data 210, the angle information acquisition/angle estimation units 120 and 337 provide the angle information acquired from the manipulator control unit 336 to the dictionary data generation/update units 130 and 335. Further, when updating the dictionary data 210, the angle information acquisition/angle estimation units 120 and 337 calculate the rotation amount Δθ of the object obj based on the angle information acquired from the manipulator control unit 336, and further, the rotation amount Δθ and the object. The initial angle θ ₁ is estimated based on the angle θ ₂ estimated by the recognition/angle estimation unit 333.

In the present embodiment, the angle information acquired by the angle information acquisition/angle estimation unit 337 of the robot 300 may indicate the angle of the object obj based on the coordinate system of the manipulator 320. In this case, the angle of the object obj indicated by the angle information acquired by the angle information acquisition/angle estimation unit 337 is connected to the manipulator 320 such as an arm as well as the rotation amount of the manipulator 320 set by the manipulator control unit 336. It may also change depending on the amount of movement of other components of the robot 300. Further, which surface of the object obj the manipulator 320 grips may vary depending on the time. Therefore, even if the same manipulator 320 that grips the object obj as when the dictionary data 210 was generated, it may be useful to use the dictionary data 210 to estimate the angle of the object obj in the image.

The dictionary data generation/update units 130 and 335 have both functions of the dictionary data generation unit 130 described above with reference to FIG. 2 and the dictionary data update unit 335 described with reference to FIG. 7. That is, the dictionary data generation/update units 130 and 335 generate the dictionary data 210 based on the images acquired by the image acquisition units 110 and 331 and the angle information acquired by the angle information acquisition/angle estimation units 120 and 337. The dictionary data 210 is generated. Further, when estimating the angle of the object obj using the dictionary data 210, the dictionary data generating/updating units 130 and 335 perform the result of the estimation of the angle of the object obj by the object recognition/angle estimating unit 333, and the angle information acquisition/ The dictionary data 210 is updated according to the result of angle re-estimation by the angle estimators 120 and 337.

As shown by the second embodiment above, the functions of the system 10 according to the first embodiment can be realized in a single device, for example the robot 300a. In this case, it can be said that the system 10 is realized by a single device. Similarly, the configuration of system 10 can be implemented with various device configurations. For example, the system 10 may include a plurality of robots 300, and each robot 300 may generate the dictionary data 210 and estimate the angle of the object using the dictionary data 210. In this case, the dictionary data 210 stored in the database 200 is shared by the plurality of robots 300.

Further, for example, most of the functions of the system 10 may be realized by the server. In this case, the server including the database 200 realizes the function as realized by the control unit 330 of the robot 300a in the second embodiment described above. In this case, the terminal 100 transmits the image of the object obj captured by the camera 150 and the angle information of the object obj acquired from the camera platform device 160 to the server, and the server associates these with each other to generate the dictionary data 210. On the other hand, the robot 300 transmits the image of the object obj captured by the camera 310 to the server, and the server estimates the angle of the object obj based on this image. The robot 300 receives the angle estimation result from the server. If the reliability of the estimated angle does not exceed the threshold value, the server requests the robot 300 to rotate the object obj for re-estimating the angle and acquire an image of the rotated object obj. Good.

(Third Embodiment)
Next, a third embodiment of the present invention will be described. It should be noted that the same components as those in the second embodiment described above will be denoted by the same reference numerals and overlapping description will be omitted.

FIG. 14 is a diagram schematically illustrating the third embodiment of the present invention. Referring to FIG. 14, in the present embodiment, the robot 300b moves relative to the object obj instead of holding the object using the manipulator. In the illustrated example, the movement of the robot 300b includes a revolving movement around the object. At this time, in the image captured by the camera 310, the object obj rotates around the axis A ₁ . Further, the movement of the robot 300b includes a tilt of the camera 310 with respect to the object obj. At this time, in the image captured by the camera 310, the object obj rotates around the axis A ₂ .

FIG. 15 is a block diagram showing the functional configuration of a robot 300b according to the third embodiment of the present invention. The robot 300b according to the present embodiment is different from the robot 300a illustrated in FIG. 13 in that a motor control unit 339 that controls the motor 350 is included instead of the manipulator control unit 336 that controls the manipulator 320.

The motor control unit 339 controls the motor 350 of the robot 300. As described above with reference to FIG. 6, the motor 350 moves the robot 300b or rotates the robot 300b by operating the joint structure of the robot 300 or rotating the wheels of the robot 300b. Includes a motor for changing. As described above with reference to FIG. 14, the motor control unit 339 causes the robot 300b to perform a turning movement about the object obj and/or a tilt of the camera 300 of the robot 300b with respect to the object obj. , And control the motor 350.

The angle information acquisition/angle estimation units 120 and 337b acquire angle information indicating the angle of the object obj. Here, the angle information is acquired, for example, by executing image-based SLAM (Simultaneous Localization and Mapping) using a plurality of time-series images acquired by the image acquisition unit 331 while the robot 300 and the camera 310 are moving. To be done. The SLAM may be executed using the measurement result of another sensor 340 included in the robot 300a, such as a depth sensor or a laser range scanner. In this case, the angle information acquisition/angle estimation units 120 and 337b specify the amount of movement of the camera 310 by SLAM, and then, based on the positional relationship between the camera 310 and the object obj separately specified, the angle information of the object obj. To get. Alternatively, the angle information acquisition/angle estimation units 120 and 337b may specify the movement amount of the camera 310 based on the control value of the motor 350 by the motor control unit 339.

In this embodiment, the dictionary data 210 can be generated using the angle information acquired as described above. In addition, when the object recognition/angle estimation unit 333 cannot estimate the angle with sufficient reliability based on the dictionary data 210, the motor control unit 339 controls the motor 350 so that the object obj in the image is controlled. Can be rotated to re-estimate the angle and update the dictionary data 210. In the present embodiment, the relative movement of the camera 310 with respect to the object obj is an example of a physical operation relating to the object obj, which is executed in re-estimating the angle of the object obj.

According to the configuration of the third embodiment of the present invention as described above, the dictionary data 210 for estimating the angle of the object obj even when the object obj is large or cannot be moved even if it is small. Can be generated. Here, the robot 300b also includes the manipulator 320 and the manipulator control unit 336 as described with reference to FIG. 7, and when the object obj can be grasped, the robot 300b and the first and second embodiments described above are used. Similarly, the manipulator 320 may be used to rotate the object obj.

Note that, in the example of the third embodiment described above, as in the second embodiment, the entire functions relating to the generation of the dictionary data 210 and the estimation of the angle of the object obj using the dictionary data 210 are robots. Although implemented in 300b, other examples are possible. For example, in the system 10 according to the first embodiment, the robot 300 may include a motor control unit 339 instead of the manipulator control unit 336 or together with the manipulator control unit 336.

For example, when the pan head device 160 (or the robot 300) used when generating the dictionary data 210 and the robot 300 that estimates the angle of the object obj using the dictionary data 210 have different sizes, the dictionary data 210 is generated. While it is possible to rotate the object obj using the pan head device 160 or the manipulator 320, it may be difficult to rotate the object obj when updating the dictionary data 210, or vice versa. It can happen.

Further, for example, when the robot 300 includes the manipulator control unit 336 and the motor control unit 339 as described above, the motor control unit 339 may control the motor 350 so that the camera 310 moves together with the object obj. In this case, the manipulator control unit 336 controls the manipulator 320 so that the angle of the object obj in the image does not change. Specifically, the manipulator control unit 336 determines the positional relationship between the manipulator 320 and the camera 310 and the angle at which the manipulator 320 grips the object obj while the motor control unit 339 controls the motor 350 and the robot 300 moves. Hold.

In this way, by moving the camera 310 together with the object obj, it is possible to change the environmental conditions when the camera 310 captures an image, for example, without changing the angle of the object obj in the image. Thereby, for example, when it is difficult to estimate the angle of the object obj based on the dictionary data 210 with high reliability under certain environmental conditions, it is possible to perform highly reliable estimation by changing the environmental conditions. There is a nature. In addition, when the dictionary data 210 is generated, the dictionary data 210 includes a plurality of elements in which a plurality of images acquired under different environmental conditions are associated with common angle information, thereby improving the robustness of the angle estimation. Can be made

In the above example, in the updating process of the dictionary data 210, first, the motor control unit 339 controls the motor 350 to move the camera 310 together with the object obj. After the camera 310 and the object obj are moved, the image acquisition unit 331 acquires the moved image (second image) of the object obj, and the object recognition/angle estimation unit 333 acquires the moved image (second image). Re-estimate the angle of the object obj in the (image). When the matching score in this estimation exceeds the threshold, the dictionary data update unit 335 determines the angle information corresponding to the re-estimated angle of the object obj and the image acquired by the image acquisition unit 331 before the movement of the object obj (first Image) and the dictionary data is updated. In this example, the movement of the camera 310 together with the object obj is a physical operation for the object obj executed in re-estimating the angle of the object obj. In addition, in this example, the object recognition/angle estimation unit 333 performs both the above-mentioned “first angle estimation function” and “second angle estimation function”.

(Example of hardware configuration of information processing device)
Next, with reference to FIG. 16, a hardware configuration example of the information processing apparatus according to the embodiment of the present invention will be described. FIG. 16 is a block diagram showing a hardware configuration example of the information processing apparatus according to the embodiment of the present invention.

The information processing device 900 includes a processor 901, a memory 903, an input device 905, an output device 907, and a bus 909. The information processing device 900 may further include a storage 911, a drive 913, a connection port 915, and a communication device 917.

The processor 901 is configured by a processing circuit such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), and/or an FPGA (Field-Programmable Gate Array). The processor 901 functions as an arithmetic processing device and a control device, and controls the operation of the information processing device 900 according to a program recorded in the memory 903, the storage 911, or the removable recording medium 919.

The memory 903 includes, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM stores, for example, a program for the processor 901 and operation parameters. The RAM temporarily stores, for example, a program expanded when the processor 901 is executed, parameters when the program is executed, and the like.

The input device 905 is a device operated by a user, such as a mouse, a keyboard, a touch panel, a button, or various switches. The input device 905 does not necessarily have to be integrated with the information processing device 900, and may be, for example, a remote controller that transmits a control signal by wireless communication. The input device 905 includes an input control circuit that generates an input signal based on information input by a user and outputs the input signal to the processor 901.

The output device 907 is configured by a device capable of outputting information to the user by using senses such as sight, hearing, and touch. The output device 907 may include, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, an audio output device such as a speaker or headphones, or a vibrator. The output device 907 outputs the result obtained by the processing of the information processing device 900 as a video such as a text or an image, a voice such as a voice or a sound, or a vibration.

The storage 911 includes, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage 911 stores, for example, a program for the processor 901, various kinds of data read out when the program is executed, or generated by executing the program, and various kinds of data acquired from the outside.

The drive 913 is a reader/writer for a removable recording medium 919 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. The drive 913 reads out the information recorded in the mounted removable recording medium 919 and outputs it to the memory 903. Further, the drive 913 writes various kinds of data in the mounted removable recording medium 919.

The connection port 915 is a port for connecting the external connection device 921 to the information processing apparatus 900. The connection port 915 can include, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, and the like. Further, the connection port 915 may include an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, and the like. By connecting the external connection device 921 to the connection port 915, various data can be exchanged between the information processing apparatus 900 and the external connection device 921.

The communication device 917 is connected to the network 923. The network 923 may be an open communication network to which an unspecified number of devices such as the Internet are connected. For example, a limited device such as Bluetooth (registered trademark), such as two devices, may be used. It may be a closed communication network connected. The communication device 917 may include, for example, a LAN (Local Area Network), Bluetooth (registered trademark), Wi-Fi, or WUSB (Wireless USB) communication card. The communication device 917 transmits/receives signals or data to/from other information processing devices using a predetermined protocol according to the network 923.

Heretofore, an example of the hardware configuration of the information processing apparatus 900 has been shown. Each component described above may be configured by using a general-purpose member, or may be configured by hardware specialized for the function of each component. Further, those skilled in the art can appropriately change the configuration of the information processing apparatus 900 as described above according to the technical level at the time of implementation.

In the embodiment of the present invention, for example, the system, the jig, the information processing device, the information processing method executed by the information processing device, the program for operating the information processing device, and the program described above are recorded. May include non-transitory tangible media.

Although some embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in claims. It is understood that these also belong to the technical scope of the present invention.

10... System, 100... Terminal, 110... Image acquisition part, 120... Angle information acquisition part, 130... Dictionary data generation part, 150... Camera, 160... Platform device, 161,... Base, 162... Strut, 163... Arm 164... Pin, 165... Holder, 167... Control part, 170... Jig, 171... Mounting member, 172... Connecting member, 173... Object holder, 174... Background plate, 200... Database, 210... Dictionary data, 300, 300a, 300b... Robot, 310... Camera, 320... Manipulator, 330... Control part, 331... Image acquisition part, 332... Dictionary data acquisition part, 333... Object recognition/angle estimation part, 334... Result output part, 335... Dictionary Data update unit, 336... Manipulator control unit, 337... Angle information acquisition/angle estimation unit, 339... Motor control unit, 340... Sensor, 350... Motor.

Claims (12)

A dictionary data acquisition function that acquires dictionary data related to objects,
An image acquisition function for acquiring a first image of the object,
A first angle estimating function for estimating an angle of the object in the first image based on the first image and the dictionary data;
A second angle estimation function for re-estimating the angle of the object in the first image after a physical operation on the object;
An information processing apparatus including a processor that realizes a dictionary data updating function that updates the dictionary data according to a result of estimation by the first angle estimating function and re-estimation by the second angle estimating function. The said dictionary data update function updates the said dictionary data based on the said 1st image and the angle information which shows the angle of the said object reestimated by the said 2nd angle estimation function. Information processing equipment. The dictionary data update function, based on the result of re-estimation by the second angle estimation function, when the reliability of the angle of the object estimated by the first angle estimation function does not exceed a threshold value, the dictionary. The information processing apparatus according to claim 1, which updates data. The image acquisition function acquires a second image of the object after physical manipulation of the object,
The first angle estimation function estimates an angle of the object in the second image based on the second image and the dictionary data,
The second angle estimation function is based on an angle of the object estimated by the first angle estimation function based on the second image and the dictionary data, and an amount of physical operation on the object, The information processing apparatus according to claim 1, wherein the angle of the object in the first image is re-estimated. When the reliability of the angle of the object estimated by the first angle estimation function based on the second image and the dictionary data does not exceed a threshold value, the image acquisition function performs a physical operation of the object. After the rerun, take a third image of the object,
The first angle estimation function estimates an angle of the object in the third image based on the third image and the dictionary data,
If the reliability of the angle of the object estimated by the first angle estimation function based on the third image and the dictionary data exceeds the threshold value, the second angle estimation function may be used. The estimating function re-estimates the angle of the object in the first image based on the angle of the object estimated based on the third image and the dictionary data, and the total amount of physical operations on the object. The information processing device according to claim 4. The physical operation regarding the object includes rotation of the object by a holding unit that holds the object,
The second angle estimation function re-estimates the angle of the object in the first image based on the rotation amount of the object provided from the holding unit. The information processing device described. The physical operation relating to the object includes a relative movement with respect to the object of an imaging device that provides an image to the image acquisition function,
The information processing apparatus according to claim 1, wherein the second angle estimation function re-estimates the angle of the object in the first image based on the amount of movement of the imaging device. The information processing apparatus according to claim 7, wherein the movement of the imaging device includes a turning movement around the object. The image acquisition function acquires a plurality of images in time series during movement of the imaging device,
The information processing apparatus according to claim 7, wherein the second angle estimation function specifies a movement amount of the imaging device based on the plurality of images. The information processing apparatus according to claim 1, wherein the physical operation related to the object includes that an imaging device that provides an image to the image acquisition function moves together with the object. Obtaining dictionary data about the object,
Obtaining a first image of the object,
Estimating an angle of the object in the first image based on the first image and the dictionary data;
Re-estimating the angle of the object in the first image after physical manipulation of the object;
A processor updating the dictionary data according to the results of the estimation and the re-estimation. A dictionary data acquisition function that acquires dictionary data related to objects,
An image acquisition function for acquiring a first image of the object,
A first angle estimating function for estimating an angle of the object in the first image based on the first image and the dictionary data;
A second angle estimation function for re-estimating the angle of the object in the first image after a physical operation on the object;
A program for causing a processor to realize a dictionary data updating function for updating the dictionary data according to the result of the estimation by the first angle estimating function and the re-estimation by the second angle estimating function.

Images