Keywords

1 Introduction

The acreage and production of China’s kiwi fruit rank first in the world. However, at present, the kiwi fruit is mainly harvested manually, which is highly labor-intensive. With the progress of urbanization and industrialization, more and more young and middle aged people are attracted to work in cities. As a result, the loss of labor force in agriculture is becoming serious, which in turn raises the cost of agricultural production and lowers market competitiveness of our agricultural products. Therefore, the development of kiwi fruit picking robot is of great significance to the development of China’s kiwi fruit industry.

The key techniques of Kiwi picking robot involve three parts: fruit identification, location and nondestructive picking. The widespread adoption of standardized scaffolding pattern in kiwi fruit production makes robot picking fruit feasible. However, there are still several factors that hinders the development of kiwi fruit picking robot. Firstly, kiwi plants grow in clusters, each of which is usually composed of 3–5 fruits, and the fruits usually grow too close to one another and even overlap. Moreover, foliage sheltering and similar color between fruits and the background make the harvesting robot difficult to perform precise fruit identification and separation as well as feature extraction of fruits. Secondly, kiwi fruit positioning and spatial coordinate acquisition are also problems to be solved for the development of the harvesting robot. The existing fruit and vegetable harvesting robot positioning system is low in precision, time-consuming, complex in structure and high in cost. So it is imperative to develop a new, efficient positioning system.

Among the existing fruit and vegetable harvesting robots at home and abroad, some can harvest fruits whose colors are greatly different from the background colors, such as strawberry picking robot [1], tomato picking robot [2–4], citrus harvesting robots [5, 6], and some can harvest the target fruits whose colors are similar to the background colors, such as cucumber picking robots [7]. Such robots usually adopt near-infrared spectroscopy or laser technology for detection. In terms of detection and identification of kiwi fruits, Zhan et al. [8] used Adaboot algorithm. Ding et al. [9] used RB color component method to separate kiwi fruits. These two methods can only separate the regions with fruits from the ones without fruits, but they failed to identify the fruit individually. Cui et al. [10] used 0.9R-G color features in the fruit image segmentation, but in a complex background environment, this method involves a large amount of calculation and time. In terms of kiwi fruit positioning and coordinate acquisition, some other methods are used, such as monocular vision, binocular vision, multi-purpose vision, hyper-spectrum, laser etc. However, these methods have drawbacks, such as complex computation, low accuracy, high cost, poor reliability and so on. Meanwhile, the conversion among pixel coordinates, spatial coordinates and mechanical arms is still a problem to be solved.

In on-site investigation, it is found that the places below the fruits are spacious with less sheltering and the background is simple, so the writer proposes that the fruits be identified, positioned, and picked from the bottom parts of the plants. The principle is like this: determine the sequence of fruit identification, feature point extraction and fruit picking by using elliptic Hough conversion; acquire the feature point coordinates of the images with Kinect sensors made by Microsoft Company; obtain the image coordinates of feature points by Kinect sensor referring the foreign research results for Microsoft Kinect sensor in robot navigation [11, 12] and feature recognition [13–15]; finally, conduct coordinates conversion between the camera and sensors and construct mathematical model of the coordinates conversion to obtain the 3D coordinates of the feature points.

2 Information Perception

2.1 Feature Extraction and Image Acquisition

The kiwi fruit pictures were taken in October 2014 during harvest time at Kiwi Experimental Station of Northwest Agriculture and Forestry University and the breed is “Hayward”. Camera used is Microsoft Life Camera studio with COMS sensor and auto-focus. Image pixel acquisition is of 640 × 360, jpg format. Each picture containing 2–5 fruits was taken from the bottom with a distance of 20 cm to the fruits. Image acquisition mainly comes from the side and bottom, as shown in Fig. 1.

Fig. 1.
figure 1

Two different shooting directions

Full size image

In Fig. 1, due to the greater scene depth, and complicated background, it can be seen that the images of fruits taken from the side contain not only the leaves of the near-byplant branches, but also distant non-target fruits, and serious mutual occlusion between the target fruits. All of these affect the accuracy of target fruit segmentation and recognition. In contrast, the picture shot from the bottom has less mutual occlusion between fruits and no interference of other distant non-target fruits, which is favorable for extracting target fruits. Due to the mutual occlusion between fruits, fruit identification can only be performed from outside to inside, picking one by one, which results in low harvest efficiency. In contrast, the shadow area between fruits is less in image shot from the bottom, which makes it possible that all the fruits can be identified at a time. As a result, the picking sequence can be determined and efficiency improved.

In order to improve fruit recognition success rate under complex background, Cui et al. [16], researchers of Northwest Agriculture and Forestry University, presented a comprehensive method to identify fruits and extract fruit features according to kiwi fruit characteristics and color features, and elliptical Hough conversion. This method can minimize the impact of different complex background and illumination on the identification and the extraction of fruit features. The specific steps are shown in Fig. 2.

Fig. 2.
figure 2

Flow chart of fruit recognition

Full size image

2.2 Picking Order

Figure 3 is the pixel coordinates of the feature points when each fruit is identified. X-axis is within the range of 0–360, Y axis 0–640.

Fig. 3.
figure 3

Coordinates of target fruit

Full size image

The picking sequence is determined according to the values of the feature point coordinates. In Fig. 3, 1, 2, 3, 4, 5 represent picking sequence, which is determined according to the Y coordinate values of the feature points from small to big, thus the picking arm of the robot reaches minimum stroke, and maximum efficiency in the whole picking process.

3 Coordinate Conversion

The principle of Kiwi fruit extraction is shown in Fig. 4. On the left is a front view and on the right is atop view, where 1 stands for kiwi fruits, 2 for Microsoft camera, and 3 for Kinect sensors. The camera is used to identify fruits, extract features and determine the picking sequence. Kinect sensors are used to obtain the spatial coordinates of the feature points of kiwi fruits. The intersection between the optical center and the outer surface of the infrared camera is used as the origin of the coordinates; the intersection between Microsoft camera lens surface and the optical center is used as the coordinate origin of its coordinate system as is shown in Fig. 4, point ‘O’ and point ‘E’.

Fig. 4.
figure 4

The Schematic of fruit space coordinates acquisition

Full size image

As is shown in Fig. 4, the horizontal distance between the feature point and the Microsoft camera is ‘h’; the distance between the feature point and Kinect sensor is ‘H’, then the distance between the Microsoft camera and Kinect sensor is ‘H-h’. When the distance between the Microsoft camera and Kinect sensor remains unchanged, let the spatial coordinates between the camera and Kinect sensor be (X, Y, Z), then we get Fig. 5.

Fig. 5.
figure 5

Relationship between two coordinate systems

Full size image

The coordinate system diagram of Microsoft camera image is shown in Fig. 5(a) with a pixel area of 640 × 360. The value of the center coordinates is (320,180), which is also the projection position of the camera’s optical center in the image. Letthe pixel coordinates of the feature point A recognized by the camera at the image be (x, y), so the value of the relative pixel coordinates in the image is (\( \Delta {\text{x}},\Delta {\text{y}} \)). Then:

$$ \Delta {\text{x}} = {\text{x}} - 320 $$
(1)
$$ \Delta {\text{y}} = {\text{y}} - 180 $$
(2)

The relationship between \( \Delta {\text{x}} \) and \( \Delta {\text{y}} \) is positive and negative, and the image can be divided into our regions, that is, ‘1’, ‘2’, ‘3’, ‘4’. When \( \Delta {\text{x}} < 0 \) and \( \Delta {\text{y}} < 0 \), it corresponds to region ‘1’, representing that the feature point is on the upper left of the projection point. When \( \Delta {\text{x}} < 0 \) and \( \Delta {\text{y}} > 0 \), it corresponds to region ‘2’, showing that the feature point is on the lower left of the projection point. When \( \Delta {\text{x}} > 0 \) and \( \Delta {\text{y}} > 0 \), it corresponds to region ‘3’, meaning that the feature point is at the right bottom of the projection point; When \( \Delta {\text{x}} > 0 \) and \( \Delta {\text{y}} < 0 \), it corresponds to region ‘4’, indicating that the feature point is on the upper right of the projection point.

When the distance between the camera and the feature point is h, letone of the image pixels be a (mm), then the three 3D coordinates \( {\text{Xw}},{\text{Yw}},{\text{Zw}} \) of the feature points recognized relative to the origin are respectively as follows:

$$ X_{w} = a\Delta x $$
(3)
$$ Y_{w} = a\Delta y $$
(4)
$$ Z_{w} = h $$
(5)

In addition, the positive or negative values of \( X_{w} \) and \( Y_{w} \) determines the regions where the feature points distribute.

The Kinect sensor is installed below the Microsoft camera with a fairly great distance between them, so that images acquired by the Kinect sensor include the shooting area of the Microsoft camera. As is shown in Fig. 5 (b), ‘XOY’ is the Kinect image screen coordinates, and ‘\( {\text{xoy}} \)’ is the Microsoft camera image screen coordinates. When the pixel range of the Kinect sensor in capturing image is 640 × 480, the projection of the optical center of the infrared video camera in the image is the pixel coordinates (320 × 240) of the center point of image projected.

When the spatial position of the feature points recognized by the Microsoft camera remains unchanged, and let the coordinates of feature point A in the screen coordinate system of the Kinect sensor be x′, y′, then its relative pixel coordinates to the center point in the image are:

$$ \Delta x^{'} = x^{'} - 320 $$
(6)
$$ \Delta y^{'} = y^{'} - 240 $$
(7)

Supposing one pixel of the image on plane H of the Kinect sensor represents the actual length b (mm), the 3D coordinates \( {\text{X}}_{k} \), \( Y_{k} \), \( Z_{k} \) of feature point A relative to the origin of the Kinect sensor are respectively:

$$ X_{k} = b\Delta x^{'} $$
(8)
$$ Y_{k} = b\Delta y^{'} $$
(9)
$$ Z_{k} = H $$
(10)

Similarly, the positive and negative of \( {\text{X}}_{k} \), \( {\text{Y}}_{k} \) correspond to the locations of four images ① ② ③ ④ and the actual locations of information. Since the spatial position of the Microsoft camera and Kinect sensors remain unchanged, pixel coordinates of feature point A in the imaged captured by Kinect sensor can be derived from the pixel coordinates of the feature point ‘A’ in the camera image shot by the Microsoft camera. That is

$$ a\left( {{\text{x}} - 320} \right) + {\text{X}} = {\text{b}}({\text{x}}^{ '} - 320) $$
(11)
$$ a\left( {{\text{y}} - 180} \right) + {\text{Y}} = {\text{b}}({\text{y}}^{ '} - 240) $$
(12)

From Eqs. (11) and (12), we can get the following formulas:

$$ {\text{x}}^{ '} = \frac{\text{a}}{\text{b}}\left( {{\text{x}} - 320} \right) + \frac{X}{b} + 320 $$
(13)
$$ {\text{y}}^{ '} = \frac{\text{a}}{\text{b}}\left( {{\text{y}} - 180} \right) + \frac{Y}{b} + 240 $$
(14)

When the values of distance H and h remain unchanged, the values of X, Y, a and b would be constant, so would be the values of \( \frac{\text{a}}{b} \), \( \frac{X}{b} \), \( \frac{Y}{b} \). When Microsoft camera recognizes and extracts pixel coordinates(x, y) of the feature point of kiwi fruits, the corresponding pixel on the Kinect sensor screen is (\( {\text{x}}^{'} \), \( {\text{y}}^{'} \)) in theory. However, in practice, image acquired by the obtained by RGB video camera on the Kinect sensor is just opposite along the left-right direction. That is to say, it is reversed along the direction of axis X. In this case, the value of the coordinates (x, y) of the point corresponding to the coordinates (x″, y″) is (\( 640 - {\text{X}}^{'} \), \( {\text{y}}^{'} \)).

4 Obtaining Spatial Coordinates

As far as the existing fruit and vegetable harvesting robots at home and abroad are concerned, a variety of methods are adopted for target positioning and coordinates extraction, such as monocular vision, binocular vision, multi-purpose visualization, close infrared spectroscopy, laser scanners etc. but each of them has some problems to be solved.

In this study, Microsoft’s Kinect sensor is used to obtain the spatial coordinates of the feature points of the fruits, and the development platform is Kinect for Windows SDK. The sensing device is shown in Fig. 6(a). It mainly consists of three parts. They are, from left to right, an infrared projector, a RGB camera, and an infrared camera. The function of the infrared projector is to project near-infrared spectrum actively. As is known, when the infrared spectrum is projected ontothe objects with rough surfaces or ground glass, there would be distorted spectrum, which in turn would will generate random points of reflected light (also called speckles). The speckles are then read by the infrared camera. The infrared camera is used to analyze the close infrared spectrum and to create depth images of the objects within our vision. RGB camera is used to shoot colored images within our vision. The measurement range is shown in Fig. 6(b). The range centers on the infrared camera with upper angle 43° and lower angle 43°, 400–4000 mm away in front of the video camera. The precision of the depth images captured within this area can reach millimeter.

Fig. 6.
figure 6

Hardware components and field of view of the Kinect sensor

Full size image

The flow chart of Kiwi fruit spatial coordinates acquisition is shown in Fig. 7. First, register color image data and depth image data flow to get respective color image and depth image, and to map RGB image onto the depth image. Then load the mapped image into the Kinect built-in spatial coordinates system, flowed by loading Map Depth Point to Skeleton Point function. Afterwards, judge whether the coordinate values are within the range with higher accuracy. If the values are within the range, then output the spatial coordinates with the infrared camera as its origin; if not, the output distance is beyond range.

Fig. 7.
figure 7

Flow chart of spatial coordinates acquisition

Full size image

5 Test and Analysis

5.1 Test Method

In order to verify the conversion relationship between feature point coordinates acquired by Microsoft camera and the coordinates of the Kinect sensor, as well as the accuracy of the mathematical model, verification test was carried out in the laboratory. Firstly, graph paper was used to calibrate the length represented by one pixel (e.g. numerical values of ‘a’ and ‘b’) in the images captured by Microsoft camera and Kinect sensor in fixed positions. Then the feature point images and 3D coordinates were acquired with the Microsoft camera and the Kinect sensor. The specific steps are as follows:

  1. (1)

    In order to facilitate verification testing, the whole coordinate conversion system was inverted: Kiwi fruit was placed at the bottom, and Microsoft camera was placed at the upper part with the Kinect sensor on top. The testing platform constructed is shown in Fig. 8(a). Kinect sensor and Microsoft camera are fixed onto the bracket on the same level with desktop supporting Kiwi fruit. In this test, in order to facilitate verification, the kiwi fruit surface was randomly marked with a point as the recognition feature point. Figure 8(b) is an image acquired by the sensor and Fig. 8(c) is the image captured by Microsoft camera.

    Fig. 8.
    figure 8

    Test equipment and images acquired

    Full size image
  2. (2)

    In this test, the vertical distance from the Microsoft camera to the feature point is 200 mm, and the vertical distance from the Kinect sensor to the feature point is 928 mm. Graph paper was used to calibrate the actual length between the image plane and the place 200 mm away from the Microsoft camera as well as the actual length of one pixel 928 mm away from the Kinect sensor (i.e. the values of a and b). The way of calibration is shown in Fig. 9, where Microsoft camera is fastened to the height gauge parallel to the coordinate plane. Kinect sensor calibration method is same with Microsoft camera calibration method.

    Fig. 9.
    figure 9

    Calibration correspondence between the pixel value and the actual distance

    Full size image
  3. (3)

    Microsoft camera was used to obtain the images of feature points and the pixel coordinates of the feature point. Kinect sensor was used to obtain the images of feature points, pixel coordinates and spatial coordinates.

  4. (4)

    Equations (13), (14) were verified with the pixel coordinates acquired by Microsoft camera and Kinect sensor and the values of a, b. Errors in coordinate conversion were derived according to the actual pixel coordinates of the feature points while Kinect sensor acquiring images, D-values between the images coordinates derived from equations, D-values between pixel coordinates from Kinect sensor image and calculation, and the actual length represented by each D-value.

5.2 Result and Analysis

Through calibration, it is found that the actual length represented by one pixel 200 mm away from at Microsoft camera is 0.445 mm, that is to say, a = 0.445 m and that the actual length represented by one pixel 928 mm away from the Kinect sensor is 1.32 mm, that is to say, b = 1.778. In this experiment, we got the pixel coordinates of 24 groups of feature points in different positions on the images captured by Microsoft camera and the images captured by Kinect sensor. In addition, with the help of Kinect sensor, we got the spatial coordinates of the feature points at such positions.

By plugging into formulas (13), (14) the pixel coordinates of point 1 and point 24 on the images acquired by Microsoft camera and the symmetrical image of the image acquired by Kinect sensor, we have the following results:

$$ \frac{a}{b} = 0.25 $$
$$ \frac{X}{b} = 32.00 $$
$$ \frac{Y}{b} = 186.374 $$

The value of a/b coincides with the previous calibration. By plugging into formula (13), (14) the values of a/b, X/b, Y/b and the coordinate values of the rest 22 points on the images acquired by the Microsoft camera, we can work out the coordinates of these 22 points on the symmetrical images acquired of the Kinect sensor image.

Figure 10 is a diagram drawn with MATLAB to express the calculated coordinate values and the actual coordinate values.

Fig. 10.
figure 10

The calculated and actual coordinate distribution curves

Full size image

In the diagram, the red curves represent the actual distribution of points, and the blue ones represent the distribution of calculated points. It can be seen that the two curves coincide with each other. Table 1 indicates that the error between the calculated point coordinates and the actual coordinates is less than 3 pixels or 5 mm, thus formulas (13), (14) can accurately reflect the correspondence of a same point on the image acquired by Microsoft camera and on the image acquired by the Kinect sensor.

Table 1. The coordinates of points in different coordinate systems
Full size table

6 Conclusions

  1. (1)

    In light of the kiwi fruit growth characteristics, an automatic identification method is studied, including how to acquire fruit image from the bottom and an integrated application of fruit shape and color features in recognition.

  2. (2)

    Considering the drawbacks of the existing fruit and vegetable harvest robots, a fresh method based on Kinect sensor is proposed to acquire the coordinates of the target kiwi fruits.

  3. (3)

    This paper discusses the coordinate conversion between Microsoft camera and Kinect sensor and the mathematical model constructed can perform accurate coordinates conversion.