Virtual Training for a Real Application: Accurate Object-Robot Relative Localization Without Calibration

Localizing an object accurately with respect to a robot is a key step for autonomous robotic manipulation. In this work, we propose to tackle this task knowing only 3D models of the robot and object in the particular case where the scene is viewed from uncalibrated cameras—a situation which would be typical in an uncontrolled environment, e.g., on a construction site. We demonstrate that this localization can be performed very accurately, with millimetric errors, without using a single real image for training, a strong advantage since acquiring representative training data is a long and expensive process. Our approach relies on a classification Convolutional Neural Network trained using hundreds of thousands of synthetically rendered scenes with randomized parameters. To evaluate our approach quantitatively and make it comparable to alternative approaches, we build a new rich dataset of real robot images with accurately localized blocks.

1. The project page with this UnLoc dataset (Uncalibrated Relative Localization) is imagine.enpc.fr/\(\sim \)loingvi/unloc.

Authors and Affiliations

1. Navier (UMR 8205), Ecole des Ponts ParisTech, UPE, Champs-sur-Marne, France

   Vianney Loing

2. LIGM (UMR 8049), Ecole des Ponts ParisTech, UPE, Champs-sur-Marne, France

   Renaud Marlet & Mathieu Aubry

Corresponding author

Correspondence to Vianney Loing.

Communicated by Adrien Gaidon, Florent Perronnin and Antonio Lopez.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

A Training Dataset Based on Synthetic Images

As introduced in Sect. 2.2, we generated three synthetic datasets for training a network for each subtask:

1. 1.

   robot and block in random pose, for coarse estimation,

2. 2.

   robot with clamp in random vertical pose pointing downwards, for 2D tool detection,

3. 3.

   close-up of vertical clamp and random block nearby, for fine estimation.

We created a simple room model where a robot is placed on the floor and a cuboid block is laid nearby. The robot we experimented with is an IRB120 from ABB company, for which we have a 3D model. We also have a 3D model of the clamp. However, we did not model the cables on the robot base nor on the clamp (compare, e.g., Fig. 1a–c). We considered configurations that are similar to what can be found in the evaluation dataset, although with slightly greater variations for robustness. The actual randomization for the generation of images is as follows:

Representation of some parameters defining a synthetic configuration of the relative pose between the robot and a camera

Full size image

• The size of the room is 20 m \(\times \) 20 m so that the walls are visible on some images.

• The robot base (20 cm \(\times \) 30 cm) orientation and position are sampled randomly. (The robot height is around 70 cm and the arm length around 50 cm.)

• The orientations (angles) of the robot joints are sampled randomly among all possible values, except for the clamp 2D detection task and the fine estimation task, where the arm extremity is placed vertically on top of the floor.

• Each dimension of the cuboid block is sampled randomly between 2.5 and 10 cm.

• The block is laid flat on the floor with an orientation and a position sampled randomly in the attainable robot positions for the coarse estimation task or in a 12 cm square below the clamp for the fine estimation task.

• All the textures, for the floor, robot and block are sampled among 69 widely different texture images.

• The camera center is sampled randomly at a height between 70 and 130 cm from the floor in a cylindrical sleeve of minimum radius 1 m and maximum radius 2.8 m, centered on the robot, as illustrated in Fig. 9.

• For the coarse estimation setting (wide views), the camera target is sampled in a cylinder of 30 cm radius and 50 cm height around the robot base.

• For the fine estimation setting (close-ups), the target is the center of the clamp, with a small random shift.

• The camera is rotated around its main axis (line between camera center and camera target) with an angle sampled randomly between -8 and +8 degrees.

• The camera focal length is randomly sampled between 45 mm and 75 mm for an equivalent sensor frame size of 24 mm \(\times \) 24 mm.

• Synthetic images are 256 \(\times \) 256 pixels.

The pictures were generated with the Unreal Engine 4 game engine. The dataset we created for coarse estimation consists of approximately 420k images (examples can be seen on Fig. 4a), the one for 2D clamp detection consists approximately of 2800k images (examples can be seen on Fig. 4b) and the dataset for fine estimation consists approximately of 600k images (examples can be seen on Fig. 4c). We used 90% of the images for training and the remaining 10% for validation.

B Evaluation Dataset Based on Real Images

As explained in Sect. 5.1, our evaluation dataset actually divides in three parts, corresponding to different settings, illustrated on Fig. 8:

1. 1.

   in the ’lab’ dataset, the robot and the block are on a table with no particular distractor or texture,

2. 2.

   in the ’field’ dataset, the table is covered with dirt, sand and gravels, making the flat surface uneven,

3. 3.

   in the ’adv’ dataset, the table is covered with pieces of paper that can be confused with cuboid blocks.

We use the robot itself to accurately move the block to random poses, which provides an reliable measure of its relative position and orientation, for each configuration. In practice, the block can slowly drift from its expect position as the robot repeatedly picks it, moves it and puts it down. To ensure there is no drift, the block position is checked and realigned every ten position. Because of the limited stroke of the clamp, we considered only a single block sizes: 5 cm \(\times \) 8 cm \(\times \) 4 cm. Note however that our method does not exploit this information; we believe a robust method should be able to process a wide range of block shapes. As we want to model situations where the robot can pick a block, we have to restrict the reach of the arm to a range for which the tool can be set vertically above the block, i.e., 0.505 m.

We collected images from 3 cameras at various viewpoints, looking at the scene slightly from above. To make sure the block is visible on most pictures, we actually considered successively different regions of the experiment table for sampling block poses, moving the cameras to ensure a good visibility for each region. The cameras are moved manually without any particular care. The distance between a camera and the block is typically between 1 and 2.5 m. The maximum angle between the left-most and the right-most camera is typically on the order of 120 degrees. This setting is illustrated on Fig. 6.

For each block position with respect to the robot base, we consider two main articulations of the robot arm: a random arm configuration for the coarse location subtask, and an articulation where the clamp is vertical, pointing downward and positioned next to the block for the fine location subtask. In the latter case, we positioned the clamp 150 mm above the table surface, at a random horizontal position in a 120 mm square around the block.

We actually recorded two clamp orientations along the vertical axis: a first random orientation, and then a second orientation where the clamp is rotated by 90 degrees (see Fig. 5). As the clamp orientation, with respect to which the fine estimation is defined, can be hard to estimate for some configurations, using two orientations allows more accurate predictions.

In total, we considered approximately 1300 poses (positions and orientations) of the robot and the block together. This lead to a total of approximately 12,000 images, of size 1080 \(\times \) 1080 for wide views and 432 \(\times \) 432 for close-ups with corresponding ground-truth relative position and orientation. The cameras used are eLight full HD 1080p webcams from Trust. Camera intrinsics were not available nor estimated. Nevertheless, the focal length was roughly determined to be about 50 mm and, in the synthetic images, the camera focal length was randomly set between 45 mm and 75 mm (see “Appendix A”).

Datasets and 3D models are available from imagine.enpc.fr/\(\sim \)loingvi/unloc.

C Network Architecture Details

We define here what are the bins for the three different networks, addressing the three different subtasks.

The number of bins for the last layer of the coarse estimation network depends on the bin size and on the maximum range of the robotic arm with the tool maintained vertically, i.e., 0.505 m (see “Appendix B”). In practice, we defined bins of 5 mm for the coarse estimation and 2 mm for the fine estimation, both visualized by the fine red grid in Figs. 1a and 3. These pictures also give a sense of how accurate the localization is compared to the sizes of the block and of the robot. For the angular estimation, we used bins of 5 and 2 degrees respectively. For the clamp detection network, we defined bins whose size is 2% of the picture width. Since as stated above, we predict each dimension separately, this leads to 202 bins for x and y and 36 bins for \(\theta \) for the coarse localization network, 60 bins for x and y and 90 bins for \(\theta \) for the fine localization network and 50 bins for x and y for the clamp localization network.

Reprints and permissions