A Survey of Computer Vision Technologies in Urban and Controlled-environment Agriculture

The classic problem of image recognition is to classify an image containing a single object to the corresponding object class. The success of deep convolutional networks in this area dates (at least) back to LeNet [173] of 1998, which recognizes hand-written digits. The fundamental building block of such networks is the convolution operation. Using the principles of local connections and weight sharing, convolutional networks benefit from an inductive bias of translational invariance. That is, a convolutional network applies (approximately) the same operation to all pixel locations of the image.

The victory of AlexNet [163] in the 2012 ImageNet Large Scale Visual Recognition Challenge [247] is often considered as a landmark event that introduced deep neural networks into the AI mainstream. Subsequently, many variants of convolutional networks [148, 170, 271, 289] have been proposed. Due to space limits, here, we provide a brief review of a few influential works, which is by no means exhaustive. ResNet [134] introduces residual connections that allow the training of networks of more than 100 layers. ResNeXT [336] and MobileNet [138] employ grouped convolution that reduces interaction between channels and improves the efficiency of the network parameters. ShuffleNet [365] utilizes the shuffling of channels, which complements group convolution. EfficientNet [292] shows simultaneous scaling of the network width, height, and image resolution is key to efficient use of parameters.

Recently, the transformer model has proven to be a highly competitive architecture for image recognition and other computer vision tasks [90]. These models cut the input image into a sequence of small image patches and often apply strong regularization such as RandAugment [75]. Variants such as CaiT [302], CeiT [350], Swin Transformer [195], and others [72, 78, 337, 371] achieve outstanding performance on ImageNet.

Despite the maturity of the technology for image classification, the assumption that an image contains only one object may not be easily satisfied in real-world scenarios. Thus, it is often necessary to adopt a problem formulation as object detection or semantic/instance segmentation.

The object detection task is to identify and locate all objects in the image. It can be understood as the task resulted from relaxing the assumption that the input image contains a single object. This is one natural problem formulation for real-world images and has seen wide adoption in agricultural applications.

In broad strokes, contemporary object detection methods can be categorized into anchor-box-based and point-based/proposal-free approaches. In anchor-box methods [110, 239], the process starts with a number of predefined anchor boxes that are periodically tiled to cover the entire input image. For each anchor box, the network makes two types of predictions. First, it determines if the anchor box contains one of the predefined object classes. Second, if the box contains an object, then the network attempts to move and reshape the box to become closer to the ground-truth location of the object. One-stage anchor-box detectors [77, 101, 187, 193, 236, 367] make these predictions all at once. In comparison, two-stage detectors [110, 132, 186, 239], in the first stage discard anchor boxes that do not contain any object and classify the remaining boxes into finer object categories in the second stage. The location adjustment, known as bounding box regression, can happen in both stages. It is also possible to employ more than two stages [48]. When the objects have diverse shapes and scales, these methods must create a large number of proposal boxes and evaluate them all, which can lead to high computational cost.

While point-based object detectors [91, 158, 171, 300, 372] still need to identify rectangular boxes around the objects, they make predictions at the level of grid locations on the feature maps. The networks predict if a grid location is a corner or the center of an object bounding box. After that, the algorithm assembles the corners and centers into bounding boxes. The point-based approaches can reduce the total number of decisions to be made. A careful comparison and analysis of anchor-box methods and point-based methods can be found in Reference [360].

Segmentation is a pixel-level classification task, aiming to classify every pixel in the image into a type of object or an object instance. The variations of the task differ by their definitions of the classes. In semantic segmentation [73, 94, 113, 167, 196], each type of object, such as cat, cow, grass, or sky, is its own class, but different instances of the same object type (e.g., two cats) share the same class. In instance segmentation [76, 129, 131, 226], different instances of the same object type become unique classes, so two cats are no longer the same class. However, object types such as sky or grass, which are not easily divided into instances, are ignored. In the recently proposed panoptic segmentation [69, 109, 155, 178, 189, 363], objects are first separated into things and stuff. Things are countable and each instance of things is its own class, whereas stuff is uncountable, impossible to separate into instances, appearing as texture or amorphous regions [12], and remains as one class. We note that the distinction between things and stuff is not rigid and can change, depending on the application. For example, grass is typically considered as stuff, but in the leaf instance segmentation task, each leaf of a plant becomes an instance and is a separate class.

The primary requirement of pixel-level classification is to learn pixel-level representations that consider sufficient context and within reasonable computational budget. A typical solution is to introduce a series of downsampling followed by a series of upsampling operations. Since classic works such as the Fully Convolutional Network (FCN) [196] and U-Net [246], this has been the mainstream strategy for various segmentation strategies.

Due to its use in leaf segmentation, a problem in plant phenotyping, instance segmentation may be the most relevant segmentation formulation for urban farming. Despite the apparent similarity to semantic segmentation, instance segmentation poses challenges due to the variable number of instance classes and possible permutation of class indices [80]. This could be handled by combining proposal-based object detection and segmentation [61, 68, 129, 180, 227]. Mask-RCNN [132] exemplifies this approach. Leveraging its object detection capability, the network associates each object with a bounding box. After that, the network predicts a binary mask for the object within the bounding box. However, such methods may not perform well when there is substantial occlusion among objects or when objects are of irregular shapes [80].

Departing from the detect-then-segment paradigm, recurrent methods [238, 245, 251] that output one segmentation mask at one time may be considered as implicitly modeling occlusion. Pixel embedding methods [62, 80, 213, 221, 326, 331, 344] learn vector representations for every pixel and cluster the vectors. These methods are especially suitable for segmenting plant leaves, and we will discuss them in greater detail in Section 3.1. Taking a page from the proposal-free object detector YOLO [235], SOLO [316] and SOLOv2 [317] divide the image into grids. The grid that the center an object falls into is responsible for predicting the segmentation mask of the object.

Real-world applications often require qualification of the amount of uncertainty in the predictions made by machine learning, especially when the predictions carry serious implications. For example, if the system incorrectly determines that fruits are not mature enough, then it may delay harvesting and cause overripe fruits with diminished values. Thus, users of the ML system are justified to ask how certain we are about the decision. In addition, when facing real-world input, it is desirable for the network to answer “I don’t know” when facing an input that it does not recognize [183]. Well-calibrated uncertainty measurements may enable such a capability.

However, research shows that deep neural networks exhibit severe vulnerability to overconfidence, or under-estimation of the uncertainty in its own decisions [117, 203]. That is, the accuracy of the network decision is frequently lower than the probability that the network assigns to the decision. As a result, proper calibration of the networks should be a concern for systems built for real-world applications.

Calibration of deep neural networks may be performed post hoc (after training) using temperature scaling and histogram binning [87, 117, 312]. Also, regularization during training such as label smoothing [289] and mixup [137] have been shown to improve calibration [211, 224, 297]. Researchers propose new loss functions to replace existing ones that are susceptible to overconfidence [210, 343]. Moreover, ensemble methods such as Vertical Voting [335], Batch Ensemble [323], and Multi-input Multi-output [130] can derive uncertainty estimates.

Modern AI systems are known for their inability to provide faithful and human-understandable explanations for its own decisions. The unique characteristics of deep learning, such as network over-parameterization, large amount of training data, and stochastic optimization, while being beneficial to the predictive accuracy (e.g., References [27, 179, 274, 281]), all create obstacles toward understanding how and why a neural network reaches its decisions. The lack of human-understandable explanations leads to difficulties in the verification and trust of network decisions [52, 366].

We categorize model interpretation techniques into a few major classes, including visualization, feature attribution, instance attribution, inherently explainable models, and approximation by simple models. Visualization techniques present holistically what the model has learned from the training data by visualizing the model weights for direct visual inspection [34, 93, 97, 202, 209, 290]. In comparison, feature attribution and instance attribution are often considered as local explanations, as they aim to explain model predictions on individual samples. Feature attribution methods [22, 58, 60, 207, 230, 255, 265, 273, 287, 340] generate a saliency map of an image or video frame, which highlights the pixels that contribute the most to its prediction. Instance attribution methods [32, 46, 67, 156, 229, 265, 341] attribute a network decision to training instances that, through the training process, exert positive or negative influence on the particular decision. Moreover, inherently explainable models [33, 59, 175, 259, 345] incorporate explainable components into the network architecture, which reduces the need to apply post hoc interpretation techniques. In contrast, researchers also try to post hoc approximate complex neural networks with simple models such as rule-based models [84, 102, 115, 153, 222, 318] or linear models [14, 105, 106, 160, 241] that are easily understandable.

The most significant benefit of interpretation in the context of CEA lies in its ability to aid with the auditing and debugging of AI systems and datasets. With feature attribution, users can make sure the system captures the robust features, or semantically meaningful features, that generalize to real-world data. As in the well-known case of husky vs. wolf image classification, due to a spurious correlation, the neural network learns to classify all images with white backgrounds as wolf and those with green backgrounds as husky [206]. Such shortcut learning can be identified by feature attribution and subsequently corrected. Moreover, instance attribution allows researchers to pinpoint outliers or incorrectly labeled training data that may lead to misclassification [67].

Growth monitoring, a critical component of plant phenotyping, aims to understand the life cycle of plants and estimate yield by monitoring various growth indicators such as the plant size, number of leaves, leaf sizes, land area covered by the plant, and so on. Plant growth monitoring facilitates in quantifying the effects of biological/environmental factors on growth and thus is crucial for finding the optimal growing condition and developing high-yield crops [212, 294].

As early as 1903, Wilhelm Pfeffer recognized the potential of image analysis in monitoring plant growth [225, 279]. Traditional machine vision techniques such as gray-level pixel thresholding [220], Bayesian statistics [45], and shallow learning techniques [147, 348] have been applied to segment the objects of interest, such as leaves and stems, from the background to analyze plant growth. Compared to traditional methods, deep-learning techniques provide automatic representation learning and are less sensitive to image quality variations. For this reason, deep learning techniques for growth monitoring have recently gained popularity.

Among various growth indicators, leaf size and number of leaves per plant are the most commonly used [121, 169, 252]. Therefore, in the section below, we first discuss leaf instance segmentation, which can support both indicators at the same time, followed by a discussion of techniques for only leaf counting or for other growth indicators.

Algorithms for fruit and flower detection find the location and spatial distribution of fruits and fruit flowers. This task supports various downstream applications such as fruit count estimation, size estimation, weight estimation, robotic pruning, robotic harvesting, and disease detection [31, 108, 199, 342]. In addition, fruit or flower detection may help devise plantation management strategies [108, 127], because fruit or flower statistics such as positions, facing directions (the directions the flowers face), and spatial scatter can reveal the status of the plant and the suitability of environmental conditions. For example, the knowledge of flower distribution may allow pruning strategies that focus on regions of excessive density and achieve even distribution of fruits, which optimizes the delivery of nutrients to the fruits.

Traditional approaches for fruit detection rely on manual feature engineering and feature fusion. As fruits tend to have unique colors and shapes, one natural thought is to apply thresholding on color [219, 322] and shape information [192, 217]. Additionally, References [55, 184, 208] employ a combination of color, shape, and texture features. However, manual feature extraction suffers from brittleness when the image distribution changes with different camera resolutions, camera angles, illumination, and species [30].

Deep learning methods for fruit detection include object detection and segmentation. Reference [351] applies SSD for cherry tomato detection. Reference [139] leverages Faster R-CNN to detect tomatoes. Inside the generated bounding boxes, color thresholding and fuzzy-rule-based morphological processing methods are applied to remove image background and obtain the contours of individual tomatoes. Reference [249] leverages Faster R-CNN with VGG-16 as the backbone for sweet pepper detection. RGB and near-infrared (NIR) images are used together for detection. Two fusion approaches, early and late fusion, are proposed. Early fusion alters the first pretrained layer to allow four input channels (RGB and NIR), whereas late fusion aggregates the two modalities by training independent proposal models for each modality and then combining the proposed boxes by averaging the predicted class probabilities. Reference [356] trains three multi-task cascaded convolutional networks (MTCNN) [355] for detecting apples, strawberries, and oranges. MTCNN contains a proposal network, a bounding box refinement network, and an output network in a feature pyramid architecture with gradually increased input sizes for each network. The model is trained on synthetic images, which are random combinations of cropped negative patches and fruits patches, in addition to real-world images. Reference [346] proposed R-YOLO with MobileNet-V1 as the backbone to detect ripe strawberries. Different from regular horizontal bounding boxes in object detection, the model generates rotated bounding boxes by adding a rotation-angle parameter to the anchors.

Delicate fruits, such as strawberries and tomatoes, are particularly vulnerable to damage during harvesting. Therefore, much research has been devoted to segmenting such fruits from backgrounds to determine the precise picking point. Precise fruit masks are expected to enable robotic fruit picking while avoiding damages on the neighboring fruits. Reference [185] performs semantic segmentation for guava fruits and determines their poses using FCN with RGB-D images as input. The FCN outputs a binary mask for fruits and another binary mask for branches. With the fruit binary mask, the authors employ Euclidean clustering [248] to cluster single guava fruit. From the clustering result and the branch binary mask, fruit centroids and the closest branch are located. Finally, the system predicts the vertical axis of the fruit as the direction perpendicular to the closest branch to facilitate robotic harvesting. Similarly, Reference [13] leverages Mask R-CNN with ResNet as backbone for semantic segmentation of tomatoes. In addition, the authors filter the false positive detection of tomatoes from the non-targeted rows by setting a depth threshold. Reference [107] utilizes Mask R-CNN with a ResNet101 backbone to perform instance segmentation of ripe strawberries, raw strawberries, straps, and tables. Depth images are aligned with the segmentation mask to project the shape of strawberries into 3D space to facilitate automatic harvesting. Reference [347] also applies Mask R-CNN with a ResNet101 + FPN backbone to perform instance segmentation and ripeness classification on strawberries. Reference [141] leverages a similar network for instance segmentation of tomatoes. With the segmentation mask, the systems determine the cut points of the fruits.

Besides accuracy, the processing speed of neural networks is also important for their deployment on mobile devices or agricultural robots. Reference [262] performs network pruning on YOLOv3-tiny to form a lightweight mango detection network. A YOLOv3-tiny pretrained on the COCO dataset has learned to extract fruit-relevant features, because the COCO dataset contains apple and orange images, but it also has learned irrelevant features. The authors thus use a generalized attribution method [266] to determine the contribution of each layer to fruit features extraction and remove convolution kernels responsible for detecting non-fruit classes. They find that the lower-level features are shared across all classes detection and pruning in the higher layers does not harm fruit detection performance. After pruning, the network achieves significantly lower float-point operations (FLOPs) at the same level of accuracy.

Object detection is also applied for flower detection. Reference [199] proposes a modified YOLOv4-Tiny with cascade fusion (CFNet) to detect citrus buds, citrus flowers, and gray mold, which is a disease commonly found on citrus plants. The authors propose additionally a block module with channel shuffle and depth separable convolution for YOLOv4-Tiny. Reference [284] shrinks the anchor boxes of Faster-RCNN to fit small fruits and applies soft non-maximum suppression to retain boxes that may contain occluded objects. As flowers usually have similar morphological characteristics, flowers from other non-targeted species could possibly be used as training data in a transfer learning scenario. In Reference [285], the authors fine-tune a DeepLab-ResNet model [63] for fruit flower detection. The model is trained on apple flower dataset but achieves high F1 scores on pear and peach flower images (0.777 and 0.854, respectively).

Pre-harvest estimation of yields plays an important role in the planning of harvesting resources and marketing strategies [135, 338]. As fruits are usually sold to consumers as a pack of uniformly sized fruits or individual fruits, the fruit count also provides an effective yield metric [157], besides the distribution of fruit sizes. Traditional yield estimation is obtained through manual counting of samples from a few randomly selected areas [135]. Nonetheless, when the production is large-scale, to counteract the effect of plant variability, accurate estimation would require a large quantity of samples from different areas of the field, resulting in high cost. Thus, researchers resort to CV-based counting methods.

A direct counting method is to regress on the image and output the fruit count. In Reference [234], the authors apply a modified version of Inception-ResNet for direct tomato counting. The authors train the model on simulated images and test on real images, which suggests, once again, the viability of using simulated images to circumvent the cost for formulating a large dataset.

Besides direct regression, object detection [157, 320], semantic segmentation [154], and instance segmentation [215] have also been used for fruit counting. These methods provide an intermediate level of results from which the count can be easily gathered. Reference [157] proposes MangoYOLO based on YOLOv2-tiny and YOLOv3 for mango detection and counting. The authors increase the resolution of the feature map to facilitate detection of small fruits. Reference [124] proposes pre-trained Faster R-CNN network, building upon DeepFruits [249], to estimate the quantity of sweet pepper. The authors design a tracking sub-system for sweet pepper counting. The sub-system identifies new fruits by measuring the IoU between and comparing the boundary of detected and new fruits. Reference [154] performs semantic segmentation for mango counting using a modification of FCN. The coordinates of blob-like regions in the semantic segmentation mask are used to generate bounding boxes corresponding to mango fruits. Finally, Reference [215] applies Mask R-CNN to for instance segmentation of blueberries. The model also classifies the maturity of individual blueberries and counts the number of berries according to the masks.

Occlusion poses a difficult challenge for counting. Due to this issue, automatic count from detection or segmentation results is almost always lower than the actual number of fruits. To solve this, [157] calculates and applies the ratio between the actual hand harvest count and the automatic fruit count; it also uses both front and back views of mango trees to mitigate occlusion from one angle. Taking this idea one step further, Reference [320] uses dual-view videos to detect and track mangoes when the camera moves. Utilizing different views of the same tree in a video, Reference [320] recognizes around 20% more fruits. However, the detected count is still significantly lower than the actual number, underscoring the research challenge of exhaustive and accurate counting.

Maturity-level classification aims to determine the ripeness of fruits or vegetables to aid in proper harvesting and food quality assurance. Premature harvesting results in plants that are unpalatable or incapable of ripening, while delayed harvesting can result in overripe plants or food decay [141].

The optimal maturity level differs for different targeted products and destinations. Fruits and vegetables can be consumed at different growing stages. For example, lettuce can be consumed either as baby lettuce or fully grown lettuce. The same situation happens with baby corn and normal corn. Products are to be transported to different destinations, so we must consider the length of transportation and ripening speed when deciding the correct maturity level at harvest [358].

Manually distinguishing the subtle differences in maturity levels is time-consuming, prone to inconsistency, and costly. The labor cost of harvesting accounts for a large percentage of operation cost in farms, with 42% of variable production expenses in U.S. fruit and vegetable farms being spent on labor for harvesting [142]. Automatic maturity-level classification with computer vision, in contrast, can assist automatic harvesting [20, 107, 358] and reduce cost.

Similar to fruit detection, we can apply thresholding methods on color to detect ripeness. For example, Reference [25] applies color thresholding on HSI and YIQ color spaces. Reference [296] applies linear color models. Reference [176] utilizes the combination of color and texture features. References [96, 165, 256, 257, 329] apply shallow learning methods based on a multitude of features.

More recently, researchers evaluate the performance of deep learning-based computer vision methods on maturity level classification and attain satisfactory results. For example, Reference [357] applies CNN to classify tomato maturity into five levels. However, to further facilitate automatic harvesting, object detection and instance segmentation are more commonly used for getting the exact shape, location and maturity level of fruits, and position of peduncles for robotic end-effectors to cut on.

With object detection, Reference [346] applies the R-YOLO network described in the fruit detection section (Section 3.2) to detect ripe strawberries. Reference [124], as mentioned in the fruit counting section (Section 3.3), proposes pre-trained Faster R-CNN network to estimate both the ripeness and quantity of sweet pepper. Two formulations of the model are tested. One treats ripe/unripe as additional classes on top of foreground/background, and the other performs foreground/background classification first and then performs ripeness classification on foreground regions. The second approach generates better ripeness classification results, as the ripe/unripe classes are more balanced when only the foreground regions are considered.

Using the segmentation methods discussed in Section 3.2, Reference [13] classifies semantic segmentation masks of tomatoes into raw and ripe tomatoes. References [107, 347] perform instance segmentation and classify instance masks into ripe and raw strawberries. Reference [141] performs instance segmentation on tomatoes first. After transforming the mask region into HSV color space, the authors employ a fuzzy system to classify tomatoes into four classes: immature (completely green), breaker (green to tannish), preharvest (light red), and harvest (fully colored).

Plants are susceptible to environmental disorders caused by temperature, humidity, nutritional excess/deficiency, light changes, and biotic disorders due to fungi, bacteria, virus or other pests [103, 272]. Infectious diseases or pest pandemic induce inferior plant quality or plant death, resulting in at least 10% of global food production losses [282].

Although controlled vertical farming restricts the entry of pests and diseases, it cannot eliminate them. Pests and diseases can enter the farm from accidental contamination from employees, seeds, irrigation water and nutrient solution, poorly maintained environment or phytosanitation protocols, unsealed entrance, and ventilation systems [242]. For this reason, pest and disease detection is still worth studying in the context of CEA.

Manual diagnosis of plant is complex due to the large quantity of vertically arranged plants in the field and numerous possible symptoms of diseases on different species. In addition, plants show different patterns along infection cycles, and their symptoms can vary in different parts of the plant [43]. Consequently, autonomous computer vision systems that recognize diseases according to the species and plant organs are gaining traction. From a technological perspective, we sort existing techniques into three parts, single- and multi-label classification, handling unbalanced class distributions, as well as label noise and uncertainty estimates.

The ability to handle realistic data is a critical competence that has not received sufficient research attention (with a few notable exceptions [44, 116, 182, 216, 263]). Unlike well-curated datasets that have accurate and abundant labels and relatively balanced label distributions, real-world data exhibit skewed label distribution as well as substantial noise in the labels. For effective real-world application, it is important that the CV algorithms can maintain good predictive performance under these conditions. In addition, the algorithmic tolerance of data imperfection can lower annotation cost and enable wider applications of CV. There has been substantial research on these topics in the computer vision community, such as long-tail recognition [81, 190, 261, 313, 369, 370], few-shot and zero-shot learning [177, 275, 276, 277, 334], as well as noise-resistant classification [17, 70, 150, 321, 368] and metric learning [145, 188, 310]. We believe that research on smart agriculture could benefit from the existing body of literature.

Real-world applications call for reliable estimation of the quality of automated decisions. An incorrect prediction made by an AI system may have profound implications. For example, if the system incorrectly determines that fruits are not mature enough, then it may delay harvesting and cause overripe fruits with diminished values. However, it is impossible to eliminate incorrect or uncertain predictions, as they originate from factors difficult to control and precisely measure, including model assumptions, test data shift, incomplete training data, and so on [11, 144]. Thus, we argue that uncertainty quantification is another crucial factor for real-world deployment. Such quantification would allow farmers to make informed decisions on whether to follow the machine recommendation or not. For the convenience of readers, we provide a brief review of such deep learning techniques in Section 2.4.

Besides uncertainty quantification, pairing the model with explanation on its decisions could enhance user confidence and assist auditing and debugging of the AI system. Specifically, instance attribution methods, as discussed in Section 2.5, enable detection of the biased or low-quality data points with extreme influence on prediction [67]. For example, if the model is trained with an image of dry leaves with dust that resembles a certain disease of the plant, then in the inference process, the model might misclassify diseased leaves as normal dry leaves or vice versa and induce plant death or unnecessary treatments. With instance attribution interpretation, researchers can identify misleading data points and perform adversarial training to improve model accuracy.

Real-world deployment usually requires the coordination of multiple CV capabilities provided by different networks. When the system is designed well, these networks could facilitate each other and achieve synergistic effects. For example, instance segmentation can be used for fruit and flower localization (Section 3.2), growth monitoring (Section 3.1), and fruit maturity level detection (Section 3.4). However, academic research tends to study these problems in isolation, thereby unable to reap benefits of multi-task learning.

Multi-task learning [29, 51, 191] focuses on leveraging mutually beneficial supervisory signals from multiple correlated tasks. Recently, CV researchers have built large-scale networks [66, 71, 120, 149, 152, 197, 314, 374] that perform a wide range of tasks and achieve state-of-the-art results on most tasks. This demonstrates the benefits of multi-task learning and could inspire similar work dedicated to smart farming in CEAs.

Another motivation for considering multi-task learning and system integration is that errors can propagate in a pipeline architecture. For example, a network could first incorrectly detect a leaf occluding a mature fruit as the fruit and then classify it as an immature fruit. As a result, simply concatenating multiple techniques will result in inferior overall performance than what practitioners may expect. Thus, we encourage system designers to consider end-to-end training or other innovative techniques [119, 332, 339] for aligning and interfacing different components within a system.

Finally, multi-task learning handles multiple tasks simultaneously, which saves computation power, enhances data efficiency, and alleviates the necessity to maintain and iterate multiple models. Such benefits are crucial for popularizing CEAs, as they facilitate the efficient use of energy, computation power, and human resources. Consequently, both the initial setup and ongoing maintenance investments for CEA farms can be reduced, expediting the emergence of economically viable CEAs. Furthermore, mindful selection and combination of targeted tasks have the potential to further improve overall efficiency [280].

Fusion of multi-modal data enhances inference ability of models by incorporating complementary view of data [168]. In the context of CEA, thermal or depth images capture the depth or temperature differences between foreground and background and enable filtering of non-target objects (e.g., fruits or leaves). Abnormal temperature changes during growth cycle can also indicate disease infection before visual symptoms appear [53, 54]. Furthermore, as different materials absorb, reflect, and transmit light in different ways and at different wavelengths, multi-spectral imaging (MSI) and hyper-spectral imaging (HSI), which capture images at multiple wavelengths of light, can be used to perform more specific internal inspection of leaves, fruits, and plants as compared to thermal and depth images. Finally, LiDAR and RGB-D systems allow the generation of high-density 3D point clouds of plants, fruits [107, 185], or environment [309], which facilitate 3D volume measurement or cut-point detection during harvesting.

Existing works have demonstrated the efficacy of MSI and HSI [13, 42, 311]. MSI has been utilized for yield prediction [301] and early disease detection [223, 307]. However, current literature explored majorly the power of MSI with shallow machine learning. We found only one work that leverages deep learning on MSI input [301], which applies a pruned VGG-16 for wheat yield estimation. HSI provides finer-grained resolution and divides the range of wavelength into many more spectral bands than MSI, typically ranging from tens to hundreds of bands, though at a higher cost. Hyper-spectral images have been used as the sole modality in early disease detection with both shallow machine learning methods [18, 19, 288] and deep learning methods [98, 122, 214, 311]. Due to relevancy and space limit, we will only talk about the deep learning methods here. Specifically, with a GAN-based data augmentation method, Reference [311] performs early detection of tomato spotted wilt virus before visible symptoms appear using hyper-spectral images. Reference [214] performs early detection of grapevine vein-clearing virus and shows the discriminative power of HSI in combination with CNN and shallow machine learning algorithms. Reference [98] attains early barley disease detection through generating future prediction of hyper-spectral barley leaf images using GAN. Moreover, HSI has also been utilized for yield prediction through fruit counting. Reference [122] leverages CNN and HSI to segment semantic mango masks and count the number of fruits.

However, systematic exploration of fusion techniques for multimodal inputs remains relatively rare in CEA applications. Many existing approaches adopt pipeline-based multimodal integration techniques that do not exhaust the potential of deep learning due to the lack of end-to-end training. For example, in Reference [13], the authors set a depth threshold to filter false positive tomato detection from the background. Reference [42] first performs broccoli segmentation on the RGB image. Within the segmentation mask, the authors find the mode of the depth value distribution, which is used to calculate the diameter of the broccoli head. Reference [185] conducts semantic segmentation for guava fruits using RGB images and reconstructs their 3D positions from the depth input. Reference [107] utilizes Mask R-CNN to perform instance segmentation of strawberries and align depth image with the segmentation mask to obtain 3D shape of strawberries. These methods use the two modalities separately and do not apply end-to-end training of the pipeline. As exceptions, Reference [249] proposes late fusion of RGB and near-infrared images in sweet pepper detection. Reference [308] incorporates depth information by replacing the blue channel with depth channel and applies masked R-CNN to locate tomatoes.

In computer vision research, numerous techniques for fusing and joint utilization of multimodal information have been proposed over the years, which we believe could contribute to CV applications in CEA. Due to space limits, we list only a few examples here. Reference [260] proposes two different ways to combine multiple modalities in object detection, Concatenation and Element-wise Cross Product. The former combines feature maps from different modalities along the channel dimension and lets the network discover the best way to combine them from data. The latter technique, Element-wise Cross Product, applies element-wise multiplication to every possible pair of feature maps from the two modalities. Reference [50] experiments with a variety of fusion techniques for RGB and optical flow and discovers a high-performing late-fusion strategy in action recognition. In self-supervised learning, Reference [125] identifies similar data points using one modality and treats them as positive pairs in another modality. This technique provides another paradigm to leverage the complementary nature of multimodality.