WO2018134866A1

WO2018134866A1 - Camera calibration device

Info

Publication number: WO2018134866A1
Application number: PCT/JP2017/001337
Authority: WO
Inventors: 秀行粂; 三好　雅則; 媛李
Original assignee: 株式会社日立製作所
Priority date: 2017-01-17
Filing date: 2017-01-17
Publication date: 2018-07-26

Abstract

When a calibration jig is moved comprehensively in order to improve calibration precision, the amount of time required for measurement becomes longer, and when the calibration jig is moved in a limited fashion in order to reduce measurement time, calibration precision decreases. To fix this, a camera calibration device according to the present invention comprises: a mobile marker calibration unit in which a fixed camera and a sensor-equipped mobile marker are connected and the position and attitude of the mobile marker are estimated on the basis of measurement values from the sensor of the sensor-equipped mobile marker; a fixed camera calibration unit in which the position and attitude of the fixed camera are estimated on the basis of a photographed image of the fixed camera and the estimated position and attitude of the mobile marker; a movement planning unit which creates a movement plan including a target movement position of the mobile marker in accordance with the estimated position and attitude of the fixed camera; a movement instruction unit which gives instructions for movement on the basis of the movement plan; and an output unit which outputs the estimated position and attitude of the fixed camera.

Description

Camera calibration device

The present invention relates to a camera calibration device, and more particularly to a camera calibration device that automatically creates a movement plan for a calibration jig during calibration work.

In recent years, in the surveillance field, there is an increasing need for image recognition technology that detects the position and size of an object from video captured by an imaging device. In order to realize such an image recognition technique, it is necessary to have a camera installation position / posture in a coordinate system (hereinafter referred to as “world coordinate system”) installed in a real space.

As a conventional technique for calibrating the installation position and orientation of a camera, for example, in paragraph 0022 of Patent Document 1, “second imaging device for imaging a real space in which a second index is arranged and the second imaging” When a calibration jig configured with a first index arranged on the apparatus is moved within the imaging range of the first imaging apparatus, a plurality of images captured by the first imaging apparatus at a plurality of times First acquisition means for acquiring a first image, and first extraction for extracting image coordinates of the first index from each of the first images and acquiring image coordinates of the first index at the plurality of times Means, second acquisition means for acquiring a plurality of second images taken by the second imaging device at the plurality of times, and extracting the image coordinates of the second index from each of the second images, Image coordinates of the second index at the plurality of times Utilizing simultaneously the image coordinates of the first index and the second index at the plurality of times extracted by the second extracting means to be acquired, the first extracting means, and the second extracting means, There is a description of an information processing apparatus that includes a calculation unit that obtains a camera parameter of the first photographing apparatus as an unknown parameter.

Further, this information processing apparatus can obtain the effect of “the mutual orientation and absolute orientation processes in camera calibration can be integrated” described in paragraph 0031 of the same document.

Japanese Patent No. 4886560

However, in the calibration work of Patent Document 1, there is a trade-off between measurement time and calibration accuracy depending on how the calibration jig is moved. That is, if the calibration jig is moved comprehensively in order to improve the calibration accuracy, the time required for measurement will increase, and if the calibration jig is moved in a limited way to shorten the measurement time, the calibration accuracy will decrease. There is a problem of doing.

Further, since the calibration work of Patent Document 1 is executed by the method of moving the calibration jig planned by the calibration operator, it takes time and knowledge to create the plan, and the skill of the movement plan is the skill of the calibration operator. There is also a problem that depends heavily on the degree.

In order to solve the above-described problems, a camera calibration apparatus according to the present invention is a movement in which a fixed camera and a movement marker with a sensor are connected, and the position / posture of the movement marker is estimated from the measured value of the sensor of the movement marker with sensor. A marker calibration unit, a fixed camera calibration unit that estimates the position / orientation of the fixed camera from the captured image of the fixed camera and the estimated position / orientation of the moving marker, and an estimated position / position of the fixed camera A movement plan unit that creates a movement plan including a target movement position of the movement marker according to the posture, a movement instruction unit that instructs movement based on the movement plan, and outputs the estimated position and posture of the fixed camera And an output unit.

According to the camera calibration apparatus of the present invention, the positions and postures of a plurality of fixed cameras can be calibrated with high accuracy in a short time regardless of the skill level of the calibration operator.

The figure which shows the block structure of the camera calibration apparatus 100 The figure which shows an example of the fixed camera 200 and the movement marker 300 with a sensor. The flowchart which shows the process performed in the fixed camera calibration part 102 The figure which shows an example of the feature point 311 detected from the image 210 and the image 210 The flowchart which shows the process performed in the movement plan part 103 The figure which shows an example of the three-dimensional histogram 220 The figure which shows an example of the bin 221 of the three-dimensional histogram 220 in a three-dimensional space. The figure which shows an example of the two-dimensional histogram 230 The figure which shows an example of the bin 231 of the two-dimensional histogram 230 The flowchart which shows the process performed in step S660. The figure which shows an example of the three-dimensional map which the movement instruction | indication part 104 displays The figure which shows an example of the two-dimensional map which the output part 105 outputs The figure which shows the block structure of the camera calibration apparatus 400 The flowchart which shows the process performed in the stop determination part 401

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Hereinafter, the camera calibration apparatus 100 according to the first embodiment will be described with reference to FIGS.
(Block configuration)
FIG. 1 is a diagram illustrating a block configuration of a camera calibration apparatus 100 according to the present embodiment. The camera calibration apparatus 100 is connected to a plurality of fixed cameras 200 and a movement marker 300 with a sensor, and calibrates the position and orientation of the fixed camera 200. In the following description, the identifier of each of the fixed camera 200 and i, the total number of fixed cameras 200 N _i. That is, the camera calibration apparatus 100 shall be fixed camera 200 of N _i stand is connected.

Here, the fixed camera 200 is a camera whose position and posture are fixed, such as a surveillance camera installed on the ceiling. The movement marker 300 with a sensor includes, for example, a sensor such as a camera that can measure its own movement and a marker such as a checkerboard pattern that can be easily detected from a captured image of the fixed camera 200. Accordingly, the camera moves in the environment where the fixed camera 200 is installed.

As shown in FIG. 1, the camera calibration apparatus 100 includes a movement marker calibration unit 101, a fixed camera calibration unit 102, a movement planning unit 103, a movement instruction unit 104, and an output unit 105. Note that these can be realized, for example, by operating an arithmetic device such as a CPU in accordance with a program stored in a storage device such as a semiconductor memory in the camera calibration device 100, and each is necessarily provided as hardware. There is no need.

The moving marker calibration unit 101 estimates the position / posture of the sensor-equipped moving marker 300 in the world coordinate system from the measurement values measured by the sensor of the sensor-equipped moving marker 300. On the other hand, the fixed camera calibration unit 102 uses the image including the moving marker 300 with the sensor imaged by the fixed camera 200 and the position / posture in the world coordinate system of the moving marker 300 with the sensor estimated by the moving marker calibration unit 101. Estimate the position and orientation in 200 world coordinate systems.

The movement planning unit 103 plans the movement of the sensor-equipped moving marker 300 according to the position / orientation of each fixed camera 200 estimated by the fixed camera calibration unit 102. The movement instruction unit 104 issues an instruction for causing the movement marker with sensor 300 to execute the movement plan planned by the movement planning unit 103. The output unit 105 outputs the position / posture of the fixed camera 200 estimated by the fixed camera calibration unit 102.

Although not shown in FIG. 1, the fixed camera 200, the sensor-equipped moving marker 300, and the camera calibration apparatus 100 may be connected by a wired connection such as USB or Ethernet (registered trademark) or wirelessly via a wireless network. Connection is also acceptable. Further, data recorded in a recording medium existing in the fixed camera 200 and the sensor-equipped moving marker 300 may be input to the camera calibration apparatus 100 via a storage medium such as an SD card.

FIG. 2 is a diagram illustrating an example of the fixed camera 200 and the movement marker 300 with a sensor. The fixed camera 200 fixed to the ceiling or the like captures images at a predetermined cycle and outputs captured images to the camera calibration device 100. The sensor-equipped moving marker 300 can move in the environment where the fixed camera 200 is installed. Here, the marker 310 for facilitating detection from the captured image of the fixed camera 200 and the marker 310 are fixed on the marker 310. In this example, the moving marker with sensor 300 composed of the moving camera 320 is mounted on the moving robot 350 that moves in the environment and moved, but the calibration operator 300 moves the moving marker with sensor by hand. Alternatively, the calibration operator may move the cart or tripod on which the sensor-equipped movement marker 300 is placed. It is assumed that the fixed position / orientation of moving camera 320 in the coordinate system (hereinafter referred to as “marker coordinate system”) installed on marker 310 is known to camera calibration apparatus 100.

In FIG. 2, a checkerboard pattern is used as the marker 310, but another pattern such as a circular pattern that can be easily detected from an image may be used. In addition, the marker 310 is not limited to a planar pattern, and may be a three-dimensional object such as a cube or a sphere, or a pattern that exists in advance in the mobile robot 350 may be used as the marker 310. Further, in FIG. 2, the moving camera 320 is used as a sensor, but other sensors that can measure its own motion, such as an IMU (Inertial Measurement Unit), a wheel encoder, a steering angle meter, GPS, and a laser range finder, are used. Also good.

The sensor typified by the moving camera 320 of the moving marker 300 with sensor measures at a predetermined cycle and outputs a captured image as a measurement result to the camera calibration device 100. In this embodiment, the fixed camera 200 and the moving camera 320 are time-synchronized, and photographing and measurement are performed at the same time. The camera calibration device 100 executes the process of the fixed camera calibration unit 102 every time an image is input from the fixed camera 200 or a certain number of images are input. Further, the process of the movement marker calibration unit 101 is performed every time a measurement result is input from the movement marker 300 with sensor or a certain number of measurement results are input.

The moving marker calibration unit 101 estimates the position / orientation in the world coordinate system of the moving marker 300 with sensor at each measurement time of the moving camera 320. Since the position / posture of the mobile camera 320 in the marker coordinate system is fixed and known, the position / posture of the marker 310 in the world coordinate system is estimated by estimating the position / posture of the mobile camera 320 in the world coordinate system. Can do. As shown in FIG. 2, when the moving camera 320 is used as the sensor of the sensor-equipped moving marker 300, Structure from Motion that estimates the position / orientation of the moving camera 320 at the time of capturing each image from a plurality of captured images. Method or Visual Simultaneous Localization and Mapping (vSLAM) method can be used. For example, G. Klein and D. Murray, Parallel Tracking and Mapping for Small AR Workspaces, Proc. IEEE and ACM Int. Symp. On Mixed and Augmented Reality, pp.225-234, 2007. Can do.

When an IMU is used as a sensor, the position / orientation can be estimated by integrating the acceleration and angular velocity measured by the IMU. When a wheel encoder and a steering angle meter are used as sensors, the position / posture can be estimated by dead reckoning. Further, a plurality of sensors such as a camera, an IMU, and a wheel encoder may be used as the sensors, and the position / posture may be estimated by using the measured values of the sensors together.

The fixed camera calibration unit 102 uses the image including the moving marker 300 with sensor captured by the fixed camera 200 and the position / posture of the moving marker 300 with sensor estimated by the moving marker calibration unit 101 corresponding to the shooting time. 200 positions and postures are estimated. Details of the processing will be described later. Note that, as described above, since the fixed camera 200 and the moving camera 320 perform time-synchronized shooting, in the estimation process in the moving marker calibration unit 101 and the estimation process in the fixed camera calibration unit 102, captured images at the same time are used. Use.

The movement planning unit 103 creates a movement plan for the sensor-equipped movement marker 300 according to the position / orientation of the fixed camera 200 estimated by the fixed camera calibration unit 102. Details of the processing will be described later.

The movement instruction unit 104 issues an instruction for causing the movement marker 300 with a sensor to execute the movement plan created by the movement planning unit 103. Details of the processing will be described later.

The output unit 105 outputs the position / posture of the fixed camera 200 estimated by the fixed camera calibration unit 102. Details of the processing will be described later. (Operation of Fixed Camera Calibration Unit 102)
Next, details of the processing of the fixed camera calibration unit 102 will be described with reference to FIGS. 3 and 4. FIG. 3 is a flowchart showing a process that the fixed camera calibration unit 102 repeats during the operation of the camera calibration apparatus 100. FIG. 4 is an example of an image 210 taken by the fixed camera 200.

First, in step S500, the marker 310 and the feature point 311 on the marker 310 are detected from the image 210 taken by the fixed camera 200. Here, the feature point 311 is a point where it is easy to detect the position from the image, such as the corners of each square included in the checkerboard pattern included in the image 210 illustrated in FIG. In FIG. 4, only some feature points 311 are illustrated for convenience of illustration. Since detection of the checkerboard pattern and the feature points on the pattern from the image is a known technique, a detailed description thereof is omitted.

In the following description, it is assumed that the identifier when the images 210 are arranged in time series is j, and the total number of images taken by the i-th fixed camera 200 when the fixed camera calibration unit 102 is processed is N _ij . The identifier of the feature point 311 is k, and the total number of feature points 311 detected in the j-th image 210 photographed by the i-th fixed camera 200 is N _ijk . The two-dimensional position in the image coordinate system of the image 210 of the k-th feature point 311 detected in the j-th image 210 photographed by the i-th fixed camera 200 is defined as (u ′ _ijk , v ′ _ijk ) ^T.

In step S510, the three-dimensional position in the world coordinate system of the feature point 311 on the marker 310 detected in step S500 is calculated using the position / orientation of the marker 310 estimated by the moving marker calibration unit 101, and the process proceeds to step S520. move on. The three-dimensional position p _ijk ^M of the feature point 311 in the marker coordinate system is known from the specification of the checkerboard pattern. From the position / posture of the marker 310 at the shooting time of the j-th image 210 captured by the i-th fixed camera 200 estimated by the moving marker calibration unit 101, the three-dimensional position p _ijk ^W of the feature point in the world coordinate system is Calculated by (Equation 1).

Here, R _ij ^MW and t _ij ^MW are a rotation matrix and a translation vector from the marker coordinate system to the world coordinate system.

In step S520, from the two-dimensional position in the image coordinate system of the feature point 311 on the marker 310 detected in step S500 and the three-dimensional position in the world coordinate system of the feature point 311 on the marker 310 calculated in step S510, The position / orientation of the fixed camera 200 in the world coordinate system is estimated. Specifically, the position and orientation of the i-th fixed camera 200 in the world coordinate system is calculated by solving (Equation 2) using a known nonlinear least square method such as the Levenberg-Marquardt method or the Gauss-Newton method. To do.

Here, R _i ^WCi and t _i ^WCi are a rotation matrix and a translation vector from the world coordinate system to the fixed camera coordinate system of the i-th fixed camera 200, and R ′ _i ^WCi and t ′ _i ^WCi are estimated. A rotation matrix and a translation vector from the world coordinate system to the fixed camera coordinate system of the i-th fixed camera 200. E _i is the total reprojection error. The reprojection error refers to a projection position obtained by projecting the three-dimensional position of the feature point 311 onto the image 210 using camera internal parameters such as the position / posture of the fixed camera 200, focal length, and lens distortion, and the feature point in the image 210. This is the distance between 311 detection positions. The total re-projection error E _i is calculated by (Equation 3).

Here, (x _ijk , y _ijk ) ^T is a position in the normalized image coordinate system of the k-th feature point 311 in the image 210 photographed j-th by the i-th fixed camera 200. (x ′ _ijk , y ′ _ijk ) ^T projects the position in the world coordinate system of the k-th feature point 311 in the image 210 photographed j-th by the i-th fixed camera 200 onto the i-th fixed camera 200. Coordinates.

The position (x _ijk , y _ijk ) ^T in the normalized image coordinate system of the k-th feature point 311 in the image 210 photographed j-th by the i-th fixed camera 200 uses, for example, a perspective projection model as a camera model. If there is, it is calculated by (Equation 4). However, the camera model is not limited to the perspective projection model, and other camera models such as a camera model for an omnidirectional camera may be used.

Here, (c _ix , c _iy ) ^T is the position of the optical center of the i-th fixed camera 200, and (f _ix , f _iy ) ^T is the focal length of the i-th fixed camera 200. (u _ijk , v _ijk ) ^T is the position of the k-th feature point 311 in the image 210 photographed j-th by the i-th fixed camera 200 in the image coordinates from which lens distortion has been removed. When distortion in the radial direction of the lens is used, it is calculated according to (Equation 5). However, the lens distortion model is not limited to distortion in the radial direction of the lens, and other lens models such as distortion in the tangential direction orthogonal to the radial direction of the lens may be used.

Here, κ ₁ and κ ₂ are lens distortion parameters.

Coordinates (x ′ _ijk , y ′ _ijk ) ^T where the position of the k-th feature point 311 in the image 210 photographed j-th by the i-th fixed camera 200 is projected to the i-th fixed camera 200. Is calculated by (Expression 6.1) and (Expression 6.2).

(Operation of the movement planning unit 103)
Next, details of the processing of the movement planning unit 103 will be described with reference to FIGS. FIG. 5 is a flowchart showing a process executed by the movement planning unit 103 during the operation of the camera calibration apparatus 100. A fixed camera 200 having a high calibration priority is selected and a target movement position is calculated.

In step S600, the movable area of the moving marker 300 with sensor is calculated. The reason why the movable area is obtained in advance is that if there is an area where the movement marker 300 with the sensor cannot be arranged due to the influence of an obstacle or the like, it is necessary to create a movement plan excluding the area. is there.

First, based on the position / orientation of the marker 310 at each time estimated by the moving marker calibration unit 101, the three-dimensional in the world coordinate system of the feature point 311 on the marker 310 at each time is calculated by the same calculation as step S510 in FIG. Calculate the position and fit a plane to the calculated 3D position. Since plane fitting for a three-dimensional position is a known technique, the details are omitted, but a least square method or principal component analysis can be used.

Next, the signed distance between each three-dimensional position and the fitted plane is calculated. For each code, the distances are rearranged in ascending order, and the q quantiles are calculated as w ₁ and w ₂ , respectively. Here, q is a preset value of 0 ≦ q ≦ 1. By using the q quantile, it is possible to calculate the moving range in the three-dimensional space while eliminating the influence of outliers.

Finally, an area sandwiched between two planes obtained by moving the fitted plane by w ₁ and w ₂ in the normal direction of the plane is defined as a movable area of the sensor-equipped movement marker 310. Here, when it is known in advance that the movement of the movement marker 310 with sensor is limited to a plane, such as when the movement marker 310 with sensor is moved on a robot or carriage that moves on a plane, w ₁ , W ₂ should be set to 0.

In the loop processing of steps S610 to S640 following step S600, the priority of calibration is calculated for each fixed camera 200.

In step S610, the distance d _i between the i-th fixed camera 200 and the sensor-equipped moving marker 300 is calculated. Here, d _i is estimated by the moving marker calibration unit 101, and is estimated by the rotation matrix from the latest marker coordinate system to the world coordinate system, translation vectors R _ij ^MW and t _ij ^MW, and the fixed camera calibration unit 102. The rotation matrix from the world coordinate system to the fixed camera coordinate system of the i-th fixed camera 200 and the translation vectors R ′ _i ^WCi and t ′ _i ^WCi are used ( ^Equation 7.1) and ( ^Equation 7.2). ).

In step S620, the three-dimensional distribution a _i ^3D in the world coordinate system of the feature point 311 on the marker 310 photographed by the i-th fixed camera 200 is calculated.

For this calculation, first, a three-dimensional histogram 220 of the number of feature points 311 is created using the position of the feature point 311 on the marker 310 in the image 210 and the depth from the i-th fixed camera 200. FIG. 6 is a diagram illustrating an example of the three-dimensional histogram 220. The three-dimensional histogram 220 is a histogram of the number of feature points 311 in a space composed of the position of the feature point 311 on the marker 310 in the image 210 and the depth from the i-th fixed camera 200. The bin 221 includes an image x coordinate minimum value x _min , a maximum value x _max , an image y coordinate minimum value y _min , a maximum value y _max , a depth minimum value z _min , and a region defined by the maximum value z _max . This bin constitutes the three-dimensional histogram 220 that holds the number of feature points 311. FIG. 7 is a diagram illustrating an example of the bin 221 of the three-dimensional histogram 220 in the three-dimensional space.

In the process of creating the three-dimensional histogram 220, first, as illustrated in FIG. 7, an area of 160x120 pixels is generated by dividing the image 210 of 640x480 pixels into 4x4 areas, and each area is further divided in the depth direction. Thus, the bin 221 is defined. For example, 10 bins 221 having a depth of 1 m can be defined by dividing the depth from 0 m to 10 m into 10 equal parts. Note that the size of the bin 221 shown here is merely an example, and the bin 221 having an arbitrary size may be set according to the environment. Then, by counting the number of feature points 311 present in each bin, the three-dimensional histogram 220 illustrated in FIG. 6 is created. Here, in the creation of the three-dimensional histogram 220, the number of feature points 311 detected in all the images 210 taken by the i-th fixed camera 200 is counted. That is, N _ij images are used. Also, the depth Z ′ _ijk ^Ci of the feature point 311 is calculated by (Equation 8).

Next, for each bin 221 of the three-dimensional histogram 220, it is determined whether or not the movement marker 300 with sensor can be reached. Specifically, the bin 221 including the movable region of the moving marker 300 with sensor calculated in step S600 is a bin that the moving marker 300 with sensor can reach, and the bin that does not include the bin that the moving marker 300 with sensor cannot reach. And

Here, when an external sensor capable of measuring surrounding three-dimensional information, such as a moving camera 320 or a laser range finder, is used as the sensor of the moving marker 300 with sensor, the measured three-dimensional information is used as the moving marker 300 with sensor. This is used to determine whether or not the destination is reachable. Specifically, when there are three-dimensional measurement points in the bin 221 that are equal to or greater than a preset threshold, it is considered that there is an obstacle and the bin is determined as an unreachable bin. In addition, using the measured three-dimensional information, it is determined whether the fixed camera 200 can shoot each bin 221 of the three-dimensional histogram 220. Specifically, for each area on the image 210, the bin 221 having a depth larger than the bin 221 determined as an unreachable bin by the obstacle is determined as a bin 221 that cannot be captured from the fixed camera 200 due to the obstacle. To do. Since the bin 221 that cannot be photographed from the fixed camera 200 cannot be used for calibration of the fixed camera 200, the moving marker with sensor 300 is processed as a bin that cannot be reached.

Finally, the number N _{i3eb of} bins 221 that can be reached by the sensor-equipped moving marker 300 and the bin 221 that can be reached by the sensor-equipped moving marker 300 and the number of feature points 311 in the bin 221 is greater than a preset threshold th ^3D. from number N _I3b, the three-dimensional distribution of a _i ^3D in the world coordinate system of the feature point 311 on the marker 310 calculates the _{^{_{a i 3D = N i3b / N}}} i3eb.

In step S630 following step S620, the two-dimensional distribution a _i ^2D in the image coordinate system of the feature point 311 on the marker 310 photographed by the i-th fixed camera 200 is calculated.

For this calculation, first, a two-dimensional histogram 230 of the number of feature points 311 is created using the positions of the feature points 311 on the marker 310 in the image 210. FIG. 8 is a diagram illustrating an example of the two-dimensional histogram 230. The two-dimensional histogram 230 is a histogram of the number of feature points 311 in the space formed by the positions of the feature points 311 on the marker 310 in the image 210. The bin 231 holds the number of feature points 311 in the region defined by the minimum value x _min , the maximum value x _{max of} the image x coordinate, the minimum value y _min of the image y coordinate, and the maximum value y _max . This is a bin constituting the histogram 230. FIG. 9 is a diagram illustrating an example of the bin 231 of the two-dimensional histogram 230. In this step, the two-dimensional histogram 230 is created by using the three-dimensional histogram 220 created in step S620 and adding the number of feature points 311 included in all depth bins 221 for each region on the image.

Next, for each bin 231 of the two-dimensional histogram 230, it is determined whether or not the movement marker 300 with sensor can be reached. Using the three-dimensional histogram 220 created in step S620, for each bin 231 of the two-dimensional histogram 230, if there is at least one bin 221 of the three-dimensional histogram 220 that can be reached with respect to the same image area, it can be reached. Judge that the bin is correct. If no reachable 3D histogram bin 221 exists, it is determined that the bin is not reachable.

Finally, the number N _{i2eb of} bins 231 that can be reached by the sensor-equipped moving marker 300 and the bin 231 that can be reached by the sensor-equipped moving marker 300 and the number of feature points 311 in the bin 231 is greater than a preset threshold th ^2D. from number N _i2b, the two-dimensional distribution of a _i ^2D in the image coordinate system of the feature point 311 on the marker 310 calculates the _{^{_{a i 2D = N i2b / N}}} i2eb.

In step S640 following step S630, each fixed camera is calibrated based on the distance d _i obtained in steps S610 to S630, the three-dimensional distribution a _i ^3D , the two-dimensional distribution a _i ^2D , and (Equation 9). Priority a _i is calculated.

Here, λ _d , λ _2D , and λ _3D are preset weights for the distance d _i from the sensor-equipped moving marker 300, the three-dimensional distribution a _i ^3D , and the two-dimensional distribution a _i ^2D .

After the priorities a _i of all the fixed cameras 200 are calculated, the fixed camera 200 having the highest priority a _i is selected from all the fixed cameras 200 in step S650.

In step S660 following step S650, the target movement position of the sensor-equipped movement marker 300, which is required to calibrate the fixed camera 200 having the highest priority selected in step S650, is calculated. Details of this calculation will be described with reference to the flowchart of FIG.

Steps S661 to S664 are processed for each bin 221 of the three-dimensional histogram 220 created in Step S620 for the fixed camera 200 selected in Step S650.

In step S661, the distance d _b between the center position of the bin 221 of the three-dimensional histogram 220 and the moving marker 300 with sensor is calculated. Fixed camera coordinate system of i-th fixed camera 200

The center position p _b ^Ci of the bin 221 at is calculated by (Equation 10).

The distance d _b between the center position p _b ^Ci of the bin 221 and the moving marker 300 with sensor is calculated by (Expression 11.1) and (Expression 11.2).

In step S662, the three-dimensional sufficiency b _b ^3D of the bin 221 of the three-dimensional histogram 220 is calculated. Dimensional fullness b _b ^3D bin 221, if the number of feature points 311 included in the bottle 221 is equal to or greater than the threshold th ^3D 1, if it is less than the threshold th ^3D to 0.

In step S663, the two-dimensional sufficiency b _b ^2D of the bin 221 of the three-dimensional histogram 220 is calculated. The two-dimensional sufficiency b _b ^2D of the bin 221 has the same threshold value th ^{2D as} the number of feature points 311 included in the bin 231 of the two-dimensional histogram 230 created in step S630 having the same image area as the bin 221 of the three-dimensional histogram 220. It is set to 1 in the above case, and set to 0 if less than the threshold th ^2D .

In step S664, the priority b _b of the bin 221 of the three-dimensional histogram 220 is calculated from the distance d _b from the sensor-equipped moving marker 300, the three-dimensional sufficiency b _b ^3D , and the two-dimensional sufficiency b _b ^2D . The priority b _b of the bin 221 is calculated by (Equation 12).

Here, λ ′ _d , λ ′ _3D , and λ ′ _2D are preset weights for the distance d _b from the sensor-equipped moving marker 300, the three-dimensional sufficiency b _b ^3D , and the two-dimensional sufficiency b _b ^2D . is there.

In step S665, the world coordinate system of the bin 221 having the highest priority b _b from among the reachable bins 221 using the reachability of the sensor-attached movement marker 300 with respect to the bin 221 of the three-dimensional histogram 220 calculated in step S620. The center position p _b ^W at is output as the target movement position.

Here, the case where the movement marker 300 with a sensor cannot move to the target movement position output by the movement plan part 103 is assumed. In such a case, the movement marker 300 with the sensor cannot move through the interface of the movement instruction unit 104 that instructs the movement marker 300 with the sensor the target movement position output by the movement planning unit 103. Send a signal to convey. When receiving the signal, the movement planning unit 103 excludes the bin 221 of the three-dimensional histogram 220 having the highest priority from the selection candidates in step S665 and moves the center position of the bin 221 having the next highest priority to the target movement. Output as position. If a signal indicating that the movement cannot be continuously performed for the same fixed camera 200 more than a preset number of times is received, in step S650, the fixed camera 200 with the highest priority is selected from the selection candidates. The process of step S660 is executed on the fixed camera 200 with the next highest priority.
(Operation of the movement instruction unit 104)
Next, the operation of the movement instruction unit 104 will be described with reference to FIG. The movement instruction unit 104 issues an instruction for moving the sensor-equipped movement marker 300 to the target movement position output by the movement planning unit 103.

As shown in FIG. 2, when the mobile robot 350 moves the sensor-equipped movement marker 300, the movement instructing unit 104 outputs a control signal for moving the mobile robot 350 to the target movement position. 350 is moved.

On the other hand, when the calibration operator moves the movement marker 300 with the sensor, the movement instruction unit 104 gives an instruction to the calibration operator by displaying the target movement position 250 on a three-dimensional or two-dimensional map. FIG. 11 is a diagram illustrating an example of a three-dimensional map created by the movement instruction unit 104. In this example, the current position / posture of the moving marker with sensor 300 estimated by the moving marker calibration unit 101 and the locus 360 are displayed in computer graphics. Further, the position / orientation of the fixed camera 200 estimated by the fixed camera calibration unit 102 is displayed by computer graphics, and the right fixed camera 200 to be calibrated selected in step S650 is colored and emphasized. Yes.

Here, when an external sensor capable of measuring surrounding three-dimensional information, such as a moving camera 320 or a laser range finder, is used as the sensor of the movement marker 300 with a sensor, the measured three-dimensional information is three-dimensional. Alternatively, it may be displayed on a two-dimensional map with computer graphics.

Further, the movement instruction unit 104 may instruct movement based on the amount of movement from the current position / posture of the movement marker 300 with sensor. The coordinate p _b ^M of the target movement position in the marker coordinate system of the movement marker 300 with sensor is calculated by (Equation 13).

p _b ^M corresponds to the amount of movement from the current position / posture of the moving marker 300 with sensor, for example, 10 m forward, 5 m right, and 0.5 m upward. When the mobile robot 350 moves the sensor-equipped movement marker 300, the mobile robot 350 is moved by outputting a control signal for moving the movement amount. When the calibration operator moves the sensor-equipped movement marker 300, the movement amount is instructed by, for example, sound output from a speaker or the like, or screen output from a display.
(Operation of output unit 105)
Next, the operation of the output unit 105 will be described with reference to FIG. The output unit 105 outputs the position / posture of the fixed camera 200 estimated by the fixed camera calibration unit 102 to a RAM in the fixed camera 200, a management server of the fixed camera 200, and the like. The output unit 105 may display the position / orientation of the fixed camera 200 estimated by the fixed camera calibration unit 102 using a three-dimensional or two-dimensional map.

FIG. 12 is a diagram illustrating an example of a two-dimensional map output by the output unit 105. FIG. 12 is drawn from the normal direction of the plane representing the movable region of the movement marker 300 with sensor, which is applied in step S600 of the movement planning unit 103. In FIG. 12, the position / orientation of the fixed camera 200 estimated by the fixed camera calibration unit 102 is displayed together with the imageable range 240 by computer graphics. A trajectory 360 indicates the position of the moving marker 300 with the sensor estimated by the moving marker calibration unit 101. The shootable range 240 is a shootable range of the fixed camera 200, and can be calculated from the focal length and the angle of view of the fixed camera 200. A point 370 is a three-dimensional measurement that seems to be an obstacle measured by the external sensor when an external sensor capable of measuring surrounding three-dimensional information, such as a camera or a laser range finder, is used as the sensor of the moving marker 300 with the sensor. Is a point. Here, the display method of the measurement result by the external sensor is not limited to the point. For example, a plurality of planes may be applied to the three-dimensional measurement points measured by the external sensor, and the applied planes may be displayed.
(Effect)
According to Example 1 mentioned above, the following effects are obtained.

(1) In the camera calibration apparatus 100, the movement planning unit 103 plans the target movement position of the sensor-equipped movement marker 300 based on the position and orientation of the fixed camera in the world coordinate system estimated by the fixed camera calibration unit 102, and moves The instruction unit 104 instructs the target movement position to the movement marker 300 with a sensor. Therefore, regardless of the skill level of the calibration operator, the positions and postures of the plurality of fixed cameras 200 can be calibrated with high accuracy in a short time.

(2) The heel movement planning unit 103 includes the position / posture of the sensor-equipped moving marker 300 estimated by the moving marker calibration unit 101 in the world coordinate system and the position / posture of the fixed camera 200 estimated by the fixed camera calibration unit 102 in the world coordinate system. The distance between the sensor-equipped moving marker 300 and the fixed camera 200 is calculated from the posture, and the fixed camera 200 with a short distance is preferentially selected as a calibration target (steps S610 and S650 in FIG. 5). Further, the distance between the movement marker with sensor 300 and the center position of the bin 221 of the selected three-dimensional histogram 220 of the fixed camera 200 is calculated, and the center position of the bin 221 having a short distance is set as the target movement position (FIG. 10 step S661, step S665). Therefore, by selecting a target movement position with a short distance from the sensor-equipped movement marker 300, the time required to move to the target movement position is shortened, and a plurality of fixed cameras 200 can be calibrated in a short time.

(3) The heel movement planning unit 103 determines the tertiary of the feature point 311 on the marker 310 based on the position / posture in the world coordinate system of the moving marker with sensor 300 estimated by the moving marker calibration unit 101 and the result of the fixed camera calibration unit 102. The original distribution degree and the two-dimensional distribution degree are calculated, and the fixed camera 200 having a small distribution degree is preferentially selected as a calibration target (steps S620 to S650 in FIG. 5). Further, for each bin 221 of the selected three-dimensional histogram 220 of the fixed camera 200, the three-dimensional satisfaction degree and the two-dimensional satisfaction degree of the feature point 311 on the marker 310 are calculated, and the three-dimensional histogram 220 having a small satisfaction degree is calculated. The center position of the bin 221 is set as the target movement position (FIG. 10, step S662 to step S665). Therefore, by using information on various positions in the three-dimensional space and the two-dimensional space for calibration of the fixed camera 200, the position and orientation of the fixed camera 200 can be calibrated with high accuracy.

(4) The heel movement planning unit 103 uses the distance between the sensor-equipped moving marker 300 and the fixed camera 200 and the three-dimensional distribution degree and the two-dimensional distribution degree of the feature point 311 on the marker 310 at the same time. The fixed camera 200 having a high priority is selected and set as a calibration target (steps S610 to S650 in FIG. 5). Furthermore, for each bin 221 of the three-dimensional histogram 220 of the selected fixed camera 200, the distance between the sensor-equipped moving marker 300 and the center position of the bin 221 and the three-dimensional satisfaction of the feature point 311 on the marker 310 Further, the priority of the bin 211 is calculated using the two-dimensional sufficiency at the same time, and the center position of the bin 221 of the three-dimensional histogram 220 having a high priority is set as the target movement position (step S661 to step 665 in FIG. 10). Therefore, by selecting a target movement position having a short distance from the movement marker 300 with the sensor and using information on various positions in the three-dimensional space and the two-dimensional space for calibration of the fixed camera 200, a plurality of pieces can be obtained in a short time. The position / posture of the fixed camera 200 can be calibrated with high accuracy.

(5) The heel movement planning unit 103 calculates whether or not the movement marker 300 with sensor reaches the bin 221 of the three-dimensional histogram 220 and the bin 231 of the two-dimensional histogram 230 by calculating the movable region of the movement marker 300 with sensor. Using the reachable bin, the fixed camera 200 is selected and the target movement position is calculated (step S600 in FIG. 5, step S620 to step S650, step S665 in FIG. 10). Therefore, it is prevented that the movement marker 300 with the sensor indicates a target movement position that cannot be reached, and a plurality of fixed cameras 200 can be calibrated in a short time.

(6) When the sensor of the sensor-equipped movement marker 300 is an external sensor capable of measuring three-dimensional information, the kite movement planning unit 103 performs a cubic operation on the bin 221 of the three-dimensional histogram 220 and the bin 231 of the two-dimensional histogram 230. Using the original information, it is determined whether or not the moving marker with sensor 300 can be reached and whether or not the fixed camera 200 can shoot, and using the reachable and shootable bin, the fixed camera 200 is selected and the target moving position is calculated. (FIG. 5 step S600, steps S620 to S650, FIG. 10 step S665). Therefore, it is possible to prevent a target movement position that cannot be reached by the sensor-equipped movement marker 300 and a target movement position that cannot be photographed by the fixed camera 200, and a plurality of fixed cameras 200 can be calibrated in a short time.

(7) The heel movement instruction unit 104 moves the mobile robot 350 by outputting a control signal for the mobile robot 350 to move to the target movement position when the mobile robot 350 moves the sensor-equipped movement marker 300. Let For this reason, a plurality of fixed cameras 200 can be automatically calibrated.

(8) When the calibration operator moves the sensor-equipped movement marker 300, the heel movement instruction unit 104 fixes the position / posture in the world coordinate system of the sensor-equipped movement marker 300 estimated by the movement marker calibration unit 101. By displaying the position / posture of the fixed camera 200 in the world coordinate system estimated by the camera calibration unit 102 and the target movement position output by the movement planning unit 103 on a two-dimensional or three-dimensional map, movement can be achieved. Instruct (FIG. 11). Therefore, since the calibration operator who moves the movement marker 300 with the sensor can easily grasp the target movement position, the time required to move to the target movement position is shortened, and a plurality of fixed cameras 200 can be quickly connected. Can be calibrated.

(9) In the camera calibration apparatus 100, the output unit 105 includes the position / posture in the world coordinate system of the movement marker 300 with sensor estimated by the movement marker calibration unit 101, and the fixed camera 200 estimated by the fixed camera calibration unit 102. The position / posture in the world coordinate system and the shootable range of the fixed camera 200 calculated from the focal length and angle of view of the fixed camera 200 are displayed on a two-dimensional or three-dimensional map (FIG. 12). ). Therefore, the calibration operator can easily confirm the calibration result of the fixed camera 200.
(Modification 1)
In the first embodiment described above, the movement planning unit 103 calculates the priority of each bin 221 of the three-dimensional histogram 220 and sets the center position of the bin 221 having the highest priority as the target movement position (step S665 in FIG. 10). . However, the output of the movement planning unit 103 is not limited to this.

For example, the movement planning unit 103 sets the minimum value x _min of the image x coordinate, the maximum value x _max , the minimum value y _min of the image y coordinate, the maximum value y _max , and the minimum value z _{min of the} bin 221 having the highest priority. The entire area of the predetermined bin 221 having the maximum value z _max may be output as the target movement position. The movement instruction unit 104 then estimates the current position / posture of the moving marker 300 with sensor estimated by the moving marker calibration unit 101, the past position of the moving marker 300 with sensor, and the fixed camera calibration unit 102. The movement is instructed by displaying the position / posture of the fixed camera 200 and the entire area of the predetermined bin output by the movement planning unit 103 on a two-dimensional or three-dimensional map.

According to the first modification, the following operational effects can be obtained. That is, the movement planning unit 103 outputs the entire area of the predetermined bin 221 as the target movement position, and the movement instruction unit 104 displays the area output by the movement planning unit 103 as the target movement position. Therefore, since the target movement position can be easily grasped, the time required to move to the target movement position is shortened, and a plurality of fixed cameras 200 can be calibrated in a short time.
(Modification 2)
In the first embodiment and the first modification described above, the movement planning unit 103 calculates the priority of each bin 221 of the three-dimensional histogram 220, and uses the center position or the entire area of the bin 221 having the highest priority as the target movement position. This is output (step S665 in FIG. 10). However, the output of the movement planning unit 103 is not limited to this.

For example, in addition to the center position or the entire region of the bin 221 having the highest priority, the movement planning unit 103 sets all the bins 221 whose priority is equal to or higher than a preset threshold, such as second priority and third priority. The center position or the entire area may be output as the target movement position. Alternatively, the center position or area of the bin 221 having a preset number of high priorities may be output as the target movement position. The movement instruction unit 104 instructs movement by displaying all target movement positions output by the movement planning unit 103 on a two-dimensional or three-dimensional map.

Furthermore, the movement planning unit 103 may output the center position or the entire area of all the bins 221 together with the bin priority. The movement instructing unit 104 instructs the movement by displaying the center positions or areas of all the bins 221 output from the movement planning unit 103 on the two-dimensional or three-dimensional map by color coding according to the bin priority.

According to the second modification, the following operational effects can be obtained. That is, the movement planning unit 103 outputs a plurality of target movement positions with high priority, and the movement instruction unit 104 displays a plurality of target movement positions with high priority output by the movement planning unit 103. Therefore, when the calibration operator moves the sensor-equipped movement marker 300, it can move so as to efficiently pass through a plurality of high priority target movement positions, and the position / posture of the plurality of fixed cameras 200 can be achieved in a short time. Can be calibrated with high accuracy.
(Modification 3)
In the first embodiment and the first modification described above, the movement planning unit 103 calculates the priority of each bin 221 of the three-dimensional histogram 220 and outputs the center position or region of the bin 221 having the highest priority as the target movement position. (Step S665 in FIG. 10). However, the output of the movement planning unit 103 is not limited to this.

For example, the movement planning unit 103 may calculate a route on which the bin 221 having a high priority can be efficiently moved based on the priority of each bin 221 of the three-dimensional histogram 220, and output it as the target movement position. . Specifically, assuming that the bin 221 where the sensor-equipped movement marker 300 currently exists is a start position, the movement to the adjacent bin 221 with respect to image coordinates and depth is counted as one time, and the bin is moved a preset number of times. For all the movement paths, the sum of the priorities of the bins 221 on the movement path that can be reached by the sensor-attached movement marker is calculated, and the movement path with the highest priority sum is output as the target movement position. However, the moving route calculation method is not limited to this, and other known route planning methods can be used.

When the mobile robot 350 moves the sensor-equipped movement marker 300, the movement instruction unit 104 outputs a control signal so that the mobile robot 350 passes the route output as the target movement position by the movement planning unit 103. The mobile robot 350 is moved. On the other hand, when the calibration operator moves the movement marker with sensor 300, the movement is instructed by displaying the route output as the target movement position by the movement planning unit 103 on a two-dimensional or three-dimensional map.

According to the third modification, the following effects can be obtained. That is, the movement planning unit 103 outputs, as the target movement position, a route for efficiently moving a plurality of movement targets having high priority, and the movement instruction unit 104 outputs the route output by the movement planning unit 103 as the target movement position. Instruct. Therefore, it can move so that it may pass through a plurality of movement targets efficiently, and the position and posture of a plurality of fixed cameras 200 can be calibrated with high accuracy in a short time.

Hereinafter, the camera calibration device 400 according to the second embodiment will be described with reference to FIGS. 13 to 14 for the case where the measurement times of the sensors of the fixed camera 200 and the sensor-equipped moving marker 300 are not synchronized. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and redundant description is omitted. Differences will be mainly described.

In the first embodiment, since the fixed camera 200 and the moving camera 320 perform time-synchronized shooting, a movement plan can be created using a pair of captured images of the fixed camera 200 and the moving camera 320 at the same time. Images taken by the fixed camera 200 and the moving camera 320 at the same time cannot be extracted based on the time information. Therefore, by adding each configuration described later, a set of captured images when the relative positions of the fixed camera 200 and the moving camera 320 are the same condition is extracted, and a moving plan can be created using these.
(Block configuration)
FIG. 13 is a diagram illustrating a block configuration of the camera calibration apparatus 400. The camera calibration device 400 calibrates the position and orientation of the connected fixed camera 200. In addition to the functions provided in the camera calibration device 100 of the first embodiment, the stop determination unit 401, the stop plan unit 402, A stop instruction unit 403. Note that these can be realized, for example, by operating an arithmetic device such as a CPU in accordance with a program stored in a storage device such as a semiconductor memory in the camera calibration device 400, and each is necessarily provided as hardware. There is no need.

The stop determination unit 401 combines the image coordinates of the feature point 311 on the marker 310 and the three-dimensional coordinates in a state where the moving marker 300 with the sensor is stopped, and outputs the combination to the fixed camera calibration unit 102. The stop planning unit 402 plans the stop of the sensor-equipped moving marker 300 according to the results of the stop determination unit 401, the moving marker calibration unit 101, and the movement planning unit 103. The stop instruction unit 403 instructs the stop marker with the sensor to the movement marker 300 with the sensor.

In the second embodiment, the measurement times of the sensors of the fixed camera 200 and the sensor-equipped moving marker 300 are not synchronized. The movement marker 300 with sensor repeats movement and stop according to the instruction of the stop instruction unit 403. In order to realize the stop in the three-dimensional space, it is desirable that the sensor-equipped movement marker 300 is mounted on the mobile robot 350, a carriage, a tripod, or the like.
(Operation of stop judgment unit)
Details of the processing of the stop determination unit 401 will be described with reference to FIG. FIG. 14 is a flowchart illustrating processing executed by the stop determination unit 401 during the operation of the camera calibration apparatus 400.

Steps S800 to S802 are processed for each fixed camera 200.

In step S800, the feature point 311 on the marker 310 of the sensor-equipped moving marker 300 is detected from the image 210 obtained by capturing with the fixed camera 200, and the process proceeds to step S801. The process of step S800 is the same as step S500 of the fixed camera calibration unit 102.

In step S801, if the marker 310 is detected in step S800, the process proceeds to step S802. If the marker 310 is not detected, the process proceeds to the next process of the fixed camera 200.

In step S802, it is determined from the image coordinates of the feature point 311 detected in step S800 whether the sensor-equipped moving marker 300 is stopped. Specifically, for each feature point 311, the distance between the position in the latest image and the position in the previous image is calculated. The average of the distances of all the feature points 311 is calculated, and when the average distance is smaller than a preset threshold, it is determined that the sensor-equipped moving marker 300 is stopped.

Step S803 determines whether or not the sensor-equipped moving marker 300 is stopped based on the position / orientation of the sensor-equipped moving marker 300 estimated by the moving marker calibrating unit 101, and the process proceeds to step S804. The distance between the latest position of the moving marker 300 with sensor and the position of the moving marker 300 with sensor calculated from the measured value of the previous sensor is calculated. If the distance is smaller than a preset threshold value, the sensor It is determined that the attached movement marker 300 is stopped.

In step S804, the image coordinates of the feature point 311 and the three-dimensional coordinates are synchronized. Specifically, after the stop instruction is issued by the stop instructing unit 403, the image coordinates of the feature point 311 that is first determined that the sensor-equipped moving marker 300 is stopped in step S802, and the sensor that is first detected in step S803. The three-dimensional coordinates of the feature point 311 determined that the attached movement marker 300 is stopped are regarded as the coordinates in a state where the movement marker with sensor 300 is stopped at the same position, and the image coordinates of the feature point 311 The combination of the three-dimensional coordinates is output to the fixed camera calibration unit 102.
(Operation of outage planning department)
Next, details of the processing of the stop planning unit 402 will be described.

The stop planning unit 402 includes a detection result of the marker 310 from the image 210 captured by the fixed camera 200 by the stop determination unit 401, the position of the moving marker 300 with sensor estimated by the moving marker calibration unit 101, and a movement planning unit. An instruction is issued to stop or move the sensor-equipped movement marker 300 from the target movement position output by 103.

The stop planning unit 402 instructs the moving marker with sensor to stop via the stop instructing unit 403 when the moving marker with sensor 300 is first detected in each fixed camera 200.

Further, the stop plan unit 402 is configured such that the distance between the position of the sensor-equipped movement marker 300 estimated by the movement marker calibration unit 101 and the target movement position output by the movement planning unit 103 is equal to or less than a preset threshold value. In this case, the stop instruction unit 403 is used to instruct the sensor-equipped movement marker 300 to stop.

When the image coordinate of the feature point 311 and the three-dimensional coordinate are synchronized in step S804 of the stop determination unit 401, the stop planning unit 402 cancels the stop of the sensor-equipped moving marker 300 via the stop instruction unit 403. And instruct them to move.
(Operation of stop instruction section)
Next, details of the processing of the stop instruction unit 403 will be described.

The stop planning unit 403 stops or moves the mobile robot 350 by outputting a control signal to the mobile robot 350 when the mobile robot 350 moves the sensor-equipped movement marker 300. When the calibration operator moves the sensor-equipped movement marker 300, for example, the stop or movement is instructed by sound output from a speaker or screen output from a display. The same interface as the movement instruction unit 104 may be used as an interface such as a speaker or a display.
(Effect)
According to the second embodiment, the following operational effects can be obtained. That is, in the camera calibration apparatus 400, the stop planning unit 402 instructs the sensor-equipped movement marker 300 to stop via the stop instruction unit 403, and the stop determination unit 401 causes the sensor-equipped movement marker 300 to have the same position / posture. The image coordinates of the feature point 311 on the marker 310 and the three-dimensional coordinates in a stopped state are combined and output to the fixed camera calibration unit 102. Therefore, the fixed camera 200 can be calibrated even when the measurement times of the sensors of the fixed camera 200 and the sensor-equipped moving marker 300 are not synchronized.

In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Other embodiments conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment. Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files for realizing each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

DESCRIPTION OF

SYMBOLS

100, 400 ... Camera calibration apparatus, 101 ... Movement marker calibration part, 102 ... Fixed camera calibration part, 103 ... Movement plan part, 104 ... Movement instruction part, 105 ... Output part, 200 ... Fixed camera, 210 ... Image, 300 ... Moving marker with sensor, 310 ... marker, 311 ... feature point, 320 ... moving camera, 350 ... moving robot 401 ... stop determination unit, 402 ... stop planning unit, 403 ... stop instruction unit,

Claims

A camera calibration device in which a fixed camera and a moving marker with a sensor are connected,
A moving marker calibration unit that estimates the position / posture of the moving marker from the measurement value of the sensor of the moving marker with sensor;
A fixed camera calibration unit that estimates the position / posture of the fixed camera from the captured image of the fixed camera and the estimated position / posture of the moving marker;
A movement planning unit that creates a movement plan including a target movement position of the movement marker according to the estimated position / posture of the fixed camera;
A movement instruction unit for instructing movement based on the movement plan;
An output unit for outputting the estimated position and orientation of the fixed camera;
A camera calibration device comprising:
The camera calibration device according to claim 1,
The said movement plan part sets the said target movement position using the distance from the said movement marker with a sensor, The camera calibration apparatus characterized by the above-mentioned.
The camera calibration device according to claim 1,
The movement planning unit uses a distribution of two-dimensional positions of feature points on the movement marker in a captured image of the fixed camera and a distribution of three-dimensional positions of feature points on the movement marker in a three-dimensional space. A camera calibration apparatus that sets the target movement position.
The camera calibration device according to claim 1,
The movement planning unit includes a distance from the movement marker with the sensor, a distribution of two-dimensional positions of feature points on the movement marker in a captured image of the fixed camera, and a feature point on the movement marker in a three-dimensional space. A camera calibration apparatus characterized in that the target movement position is set by simultaneously using a three-dimensional position distribution.
In the camera calibration device according to any one of claims 1 to 4,
The said movement plan part sets the said target movement position based on the arrival possibility of the said movement marker with a sensor determined from the movement possible area | region of the said movement marker with a sensor, The camera calibration apparatus characterized by the above-mentioned.
The camera calibration device according to claim 5,
When the sensor of the moving marker with sensor is an external sensor capable of measuring surrounding three-dimensional information,
The movement planning unit is configured to determine whether or not the movement marker with sensor is determined based on the three-dimensional information measured by the external sensor, and whether or not the fixed camera is used to determine whether or not the fixed marker is determined based on the three-dimensional information measured by the external sensor. And the target movement position is set based on the camera calibration apparatus.
In the camera calibration apparatus according to any one of claims 1 to 6,
The camera calibration apparatus, wherein the movement instruction unit outputs a control signal for moving to the target movement position to a mobile robot equipped with the movement marker with sensor.
In the camera calibration apparatus according to any one of claims 1 to 6,
The movement instruction unit displays the estimated position / posture of the movement marker with sensor, the estimated position / posture of the fixed camera, and the target movement position on a two-dimensional or three-dimensional map. A camera calibration device characterized by
In the camera calibration device according to any one of claims 1 to 8,
The output unit displays the estimated position / orientation of the moving marker with sensor, the estimated position / orientation of the fixed camera, and the photographing range of the fixed camera on a two-dimensional or three-dimensional map. A camera calibration device.
In the camera calibration device according to any one of claims 1 to 9,
A stop determination unit that combines the image coordinates and three-dimensional coordinates of the feature points on the moving marker in a state where the moving marker with sensor is stopped;
A stop planning unit for planning a stop of the moving marker with the sensor;
A stop instruction unit for instructing a stop planned by the stop planning unit;
A camera calibration device further comprising: