Open AccessArticle

Solder Joint Defect Detection in the Connectors Using Improved Faster-RCNN Algorithm

Kaihua Zhang

and

Haikuo Shen

^2,*

School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, Ministry of Education, Beijing 100044, China

Key Laboratory of Vehicle Advanced Manufacturing, Measuring and Control Technology, School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, Ministry of Education, Beijing 100044, China

Author to whom correspondence should be addressed.

Appl. Sci. 2021, 11(2), 576; https://doi.org/10.3390/app11020576

Submission received: 8 December 2020 / Revised: 6 January 2021 / Accepted: 6 January 2021 / Published: 8 January 2021

Download

Browse Figures

Figure 1
Examples of five connector solder joints. (a) the qualified solder joint, (b) the multi-tin solder joint, (c) the less-tin solder joint, (d) the connected welding solder joint and (e) the tin tip solder joint. "> Figure 2
Faster-RCNN uses 9 anchor boxes to get suggestion boxes. Rectangular boxes in the same color represent the same scale. "> Figure 3
Operational process of Faster-RCNN. The target classification and positioning module uses full connection layers to calculate the final scores and target position coordinates. "> Figure 4
VGG-16 feature extract network. For the pool layer of VGG-16, Max pool is adopted and the final generated feature map scale is 1/16 of the resized image scale. "> Figure 5
Operational process of RPN. After RPN, there are about 300 boxes, where the scale variables and label variables are preserved. "> Figure 6
Results of data augmentation in this paper. (a) the original image, (b) the result of left-right flipping, (c) the result of up-down flipping, (d) the result of random brightness, (e) the result of diagonal flipping and (f) the result of random contrast. "> Figure 7
Anchor boxes average IOU under different number of clusters. In this paper, 9 clusters are selected to realize the generation of anchor boxes, with an average IOU of 80.25%. "> Figure 8
Two kinds of residual unit. (a) 2-layers residual unit; (b) 3-layers residual unit. "> Figure 9
RPN bounding box regression loss curve. (a) Original anchor boxes scale; (b) K-means clustering anchor boxes scale. The red curve is the data fitting curve, showing a general downward trend. After 30,000 iterations, the loss of original algorithm is <math display="inline"><semantics> <mrow> <mn>2.034</mn> <mo>×</mo> <msup> <mrow> <mn>10</mn> </mrow> <mrow> <mo>−</mo> <mn>4</mn> </mrow> </msup> </mrow> </semantics></math>, and the loss after using k-means clustering generates anchor boxes algorithm is <math display="inline"><semantics> <mrow> <mn>1.293</mn> <mo>×</mo> <msup> <mrow> <mn>10</mn> </mrow> <mrow> <mo>−</mo> <mn>4</mn> </mrow> </msup> </mrow> </semantics></math>. "> Figure 10
Different training model’s precision-recall curve. (a) Faster-RCNN training model without data augmentation; (b) Faster-RCNN training model with data augmentation; (c) Faster-RCNN training model with data augmentation and k-means clustering; (d) Training model with data augmentation and ResNet-101 transfer learning; (e) Training model proposed in this paper. Each color represents one type of solder joint and the mAP of each type is shown as the label. "> Figure 11
Different network’s total loss. (a) Faster-RCNN training model without data augmentation; (b) Faster-RCNN training model with data augmentation; (c) Faster-RCNN training model with data augmentation and k-means clustering; (d) Training model with data augmentation and ResNet-101 transfer learning; (e) Training model proposed in this paper. After removing the bad point data, it can be seen that the algorithm proposed in this paper has a good optimization effect on model parameters. "> Figure 12
Some of typical detect results. In the pictures, hg represents qualified solder joint; dx represents multi-tin solder joint; sx represents less-tin solder joint; lh represents connected welding solder joint; xj represents tin tip solder joint. "> Figure 12 Cont.
Some of typical detect results. In the pictures, hg represents qualified solder joint; dx represents multi-tin solder joint; sx represents less-tin solder joint; lh represents connected welding solder joint; xj represents tin tip solder joint. ">

Versions Notes

Abstract

The miniaturization and high integration of electronic products have higher and higher requirements for welding of internal components of electronic products. A welding quality detection method has always been one of the important research contents in the industry, among which, the research on solder joint defect detection of a connector has gradually attracted people’s attention with the development of image detection algorithm. The traditional solder joint detection method of connector adopts manual detection or automatic detection methods, which is inefficient and not safe enough. With the development of deep learning, the application of a deep convolutional neural network to target detection has become a research hotspot. In this paper, a data set of connector solder joint samples was made and the number of image samples was expanded to more than 3 times of the original by using data augmentation. Clustering generates anchor boxes and transfer learning with ResNet-101 were fused, so an improved faster region-based convolutional neural networks (Faster RCNN) algorithm was proposed. The experiment verified that the improved algorithm proposed in this paper had a great improvement in all aspects compared with the original algorithm. The average detection accuracy of this method can reach 94%, and the detection rate of some defects can even reach 100%, which can completely meet the industrial requirements.

Keywords:

connectors; solder joint defect; transfer learning; target detection

1. Introduction

Recently, the rapid development of various industries has brought earth-shaking changes to the world, especially the electronic products industry. Electronic products are getting smaller and smaller, with more and more parts, and more and more powerful functions. Therefore, it is very important to check the quality of electronic products to ensure their high accuracy, stability, and safety. Currently, most electronic products use all kinds of connectors to complete power supply and information transmission. The connector is mainly connected to the cable by welding. The welding quality directly determines the ability of the connector cable, one of the important test indicators [1].

The earliest method of solder joint quality inspection for connectors was manual inspection. However, in this method, workers are prone to eye fatigue and other uncontrollable situations during the detection process, resulting in low detection efficiency and even false detection and omission. Once the above situation occurs, it will greatly reduce the service life of the cable, and even cause a short circuit of the cable due to misconnection and other phenomena, resulting in serious industrial accidents. In order to eliminate such hidden dangers, experts and scholars have successively developed automatic detection systems, committed to replacing people with instruments for completing the boring and repetitive work of cable welding defect detection, and proposed a series of detection methods, such as resistance measurement method [2], voltage drop measurement method and X-ray detection method [3]. However, these methods have not been widely used in the industrial field because of low detection accuracy and high cost.

With the development of cameras and computers, digital image quality is getting higher and higher, and computers are becoming more and more capable of image processing. Scholars have found that detection of welding defects by image processing can achieve higher accuracy and efficiency, thus an automatic optical inspection (AOI) system [4,5,6,7,8] was proposed. Since all solder joints on the printed circuit board (PCB) are fixed, with quite different solder color, brightness, and other features from the background plate, it is relatively easy to the corresponding algorithm to extract the solder joint features, so as to analyze the solder joint quality. Therefore, AOI has received much attention in the task of PCB solder joint detection. Wu et al. designed an automatic optical inspection algorithm to extract the position, shape and logic characteristics of PCB solder joints, and then successfully identify various welding defects [9]. Aiming at the low efficiency and accuracy of traditional PCB solder joint detection, Wang et al. proposed a PCB solder joint detection method based on automatic threshold segmentation algorithm and image shape feature extraction algorithm, which improved the efficiency and accuracy of PCB solder joint detection [10]. However, there are some differences between connector solder joints and PCB solder joints. First of all, all components on the PCB have the corresponding location of the welding spot, easy to automatic welding. While the connector usually does not need to weld all the pins, so the welding process requires more human intervention, with more uncertain solder joint features. Secondly, the connector uses tin to weld the core wire to a pin cup. The color of tin is similar to the pin color of most connectors, which is easy to be interfered by environmental factors in the image, so it is not conducive to extracting effective solder joint features. Therefore, AOI cannot be used for connector solder joint detection as a PCB.

In 2012, Hinton’s team participated in the ImageNet image recognition competition for the first time, and won the first prize by using convolutional neural network AlexNet [11]. Since then, the convolutional neural network has attracted worldwide attention on the problem of target classification. Compared with the traditional target detection classification algorithm, the convolutional neural network does not need to design feature extraction algorithm manually. At present, mainstream target detection algorithms include Faster-RCNN [12], You Only Look Once (YOLO) [13], Single Shot MultiBox Detector (SSD) [14], etc. In the industrial field, many enterprises have begun to use deep convolutional neural network to replace traditional algorithms and complete a series of tasks such as production, planning and quality detection. As experts and scholars explored continuously, deep convolutional neural network has gradually shown its powerful functions and advantages [15,16,17,18,19,20]. Abdul proposed an automatic coding system including an encoder and a decoder, and designed three full connection layers for feature extraction with the idea of deep learning [21]. The two feature matching strategies are designed for images with different backgrounds (smooth or texture). However, the threshold value of this method needs to be designed according to experience, and the size of the block needs to be changed under different defect targets and overall image proportion, otherwise the score will be affected. Li proposed an improved YOLO-v3 algorithm to complete defect detection of PCB electronic components [22]. The training data was expanded to 20 times by means of data combination and data enhancement. Because YOLO-v3 has a low recognition rate for small electronic components on PCB, the author adds a shallow layer on the basis of YOLO-v3, and uses the features obtained from the shallow layer to identify and detect the widgets. Finally, the mean average precision (mAP) was raised from 77.08 to 93.07%. Urbonas took Faster-RCNN as the main algorithm to solve the detection task of 5 kinds of defects on the wood surface [23]. He cropped the collecting board images and enhanced them, then used the AlexNet, Visual Geometry Group-16 Network (VGG-16), ResNet, GoogleNet to achieve transfer learning. The accuracy of test results under different combinations (batch size, learning rate) of different networks is compared. The results show that the highest detection accuracy of 80.6% can be obtained by using ResNet for transfer learning. Gao used the classical Faster-RCNN algorithm to identify the defect location and defect type of the tunnel wall, and then obtained a series of proposal boxes on the image. Then the marked image was sent into the adaptive border region of interest (ROI) boundary layer, and the minimum external rectangular box was selected for various marked boxes to remove the redundancy of data set and reduce the difficulty of interference identification in the process of data set creation. Finally, the three-layer full connection layer is used to complete the defect recognition task within a single marker frame of the adaptive boundary ROI boundary layer output. This algorithm reduced the error detection rate of defects in tunnels from an average of 0.3 to 0.019 [24].

Although the target detection algorithm based on deep learning has been widely used in the industry, there are few researches on solder joint defect detection of connectors. The reasons are as follows: (1) there are various kinds of solder joint defects and it is difficult to classify them; (2) The connector pin arrangement is complex, easy to block the solder joints, and difficult to obtain images; (3) The detection accuracy cannot meet the industrial requirements. Based on the above reasons and actual industrial requirements, this paper will adopt Faster-RCNN as the basic target detection algorithm to carry out appropriate algorithm improvement and finally improve the detection accuracy of five solder joint types (one qualified solder joint and four defective solder joints). The qualified connector solder joints shall be smooth and uniform between the cup and the core wire, the tin shall be filled with more than 75% of the cup and the length of core wire exposed outside the cup shall not exceed 1.5 times of the outer diameter of the cup. The four defective solder joints include multi-tin, less-tin, connected welding, and tin tip. Detailed descriptions of the four defect solder joints are shown in Table 1. Figure 1 shows examples of the five connector solder joints tested in this article.

2. Materials and Methods

Faster-RCNN [12] is the third generation of RCNN series algorithm, which was proposed by Ren Shaoqing in 2016. In this algorithm, a regional proposal network (RPN) is proposed for the first time to replace the selective search used in the previous two versions to obtain proposals. Then proposals fused with the feature map extracted by convolution- rectified linear unit (ReLU)-pooling network are sent into the full connection network for target classification and location. The proposed method raised the target detection efficiency to 17 fps, and the network accuracy obtained from Visual Object Classes 2012 (VOC 2012) data set to 75.9% at that time. First, we will introduce the key technology of Faster-RCNN, and then we will elaborate on the important improvement points of the algorithm in this paper.

2.1. Key Technology of Faster-RCNN

2.1.1. Anchor Boxes

The regional proposal network uses anchor boxes to get the suggestion boxes of the feature map. Faster-RCNN uses three sets of rectangular boxes with different length-width ratios (2:1, 1:1, 1:2), and each set uses three fixed-size rectangular boxes with different scales (128, 256, 512) to stack at each pixel on the feature map, as shown in Figure 2. Then, each box was compared with ground truth box to calculate the intersection over union (IOU). The box with larger IOU than the large preset threshold is demarcated as foreground, and the box with smaller IOU than the small preset threshold is demarcated as background. Both the boxes with IOU between the two thresholds and the overflow boundary were directly discarded. Finally, we got the suggestion box to train the region proposal network (RPN).

The introduction of anchor boxes have brought many advantages. By setting different scales, all the targets can be covered as far as possible, while reducing the calculation amount and greatly reducing the difficulty of subsequent regression algorithm optimization.

2.1.2. Network Architecture

Faster-RCNN can be divided into four modules, including feature extraction module, regional proposal network (RPN) module, ROI pooling, and target classification and positioning module. The overall structure of the algorithm is shown in Figure 3. Currently, the released code of Faster-RCNN usually takes the convolution-pooling part of VGG-16 as the feature extraction module.

The feature extraction module firstly resizes the input data of any scale to a fixed scale (the long side of the image shall not be greater than 1000, and the short side of the image shall not be greater than 600), then extracts the data features through a set of convolution-ReLU-pooling layers, and generates n-dimensional feature map (512-dimensional feature map is generated by VGG-16 network). VGG-16 feature extraction network is shown in Figure 4.

RPN uses anchor boxes to create rectangular boxes for each pixel on the feature map, assigns labels through the IOU value calculated with the ground truth boxes, and uses non-maximum suppression algorithm to eliminate overlapping boxes. The reserved foreground or background boxes will be assigned six variables, including four scale variables (x, y, w, h) and two label variables (Fg, Bg). The softmax algorithm is then used to calculate the target probability score for each box. The box regression algorithm uses four parameters to complete the regression of each anchor box, so that the proposed box is closer to the actual position. The algorithm process of RPN is shown in Figure 5.

The ROI pooling integrated feature maps and the proposal boxes generated by RPN, and proposal feature maps were calculated and generated. As the size of the proposal box is different, the dimension of input variables should be fixed when the subsequent full connection layer is used for classification. Therefore, Faster-RCNN uses ROI pooling to divide the proposal feature maps into seven equal parts horizontally and vertically, and performs the maximum pool on each square, so that the final size of each proposal feature map is 7 × 7.

The target classification and positioning module uses the full connection layer and softmax algorithm to categorize each proposal feature map, and output the probability vector. At the same time, bounding box regression is used again to obtain the positional offset of each suggested feature map and to generate a more accurate detection box.

2.1.3. Loss Function

The loss function of Faster-RCNN mainly consists of two parts, including the loss function of RPN and the loss function of RCNN, and each loss function includes the classification loss and the regression loss. The classification loss

L_{c l s} (*)

was calculated using cross-entropy, and the regression loss

L_{r e g} (*)

was calculated using smooth-L1. RPN loss function and RCNN loss function are shown in Equations (1) and (2).

L_{R P N} ({p_{i}}, {t_{i}}) = \frac{1}{N_{c l s}} \sum_{i} L_{c l s} (p_{i}, p_{i}^{*}) + λ \frac{1}{N_{r e g}} \sum_{i} p_{i}^{*} L_{r e g} (t_{i}, t_{i}^{*})

(1)

where

N_{c l s}

represents the number of images in each minibatch, and

N_{r e g}

represents the number of anchor boxes in each minibatch, with a difference for nearly 10 times. Therefore, in order to balance the classification loss and regression loss, a coefficient

λ = 10

is added in the regression loss function section.

p_{i}

represents the category score vector of the proposal box,

p_{i}^{*}

represents the label of the proposal box,

t_{i}

represents the position parameter of the proposal box,

t_{i}^{*}

represents the position parameter of the proposal box, and

i

represents each annotation box.

L_{R C N N} (p, u, t^{u}, v) = L_{c l s} (p, u) + L_{r e g} (t^{u}, v)

(2)

where

p

represents the score of the category predicted by the network,

u

represents the score of the ground truth,

t^{u}

represents the coordinate of the ground truth, and

v

represents the coordinate of the predicted box.

2.2. Important Improvements

In order to detect solder joint defects, we welded 625 solder joint samples and completed the images collection. In terms of algorithm, we used the open source TensorFlow version of Faster-RCNN algorithm on the network to train and test the test images. We found that the automatic detection of solder joint defects could not be realized with obvious effect, and a large number of undetected solder joint defects would lead to insufficient product performance. After comprehensive analysis, we believe that the main reasons are as follows: 1. There are too few training samples, so the network does not converge well after training; 2. The depth of the model is insufficient, and the ability to extract solder joint features of the connector is insufficient; 3. The scales of the default anchor do not match the size of the solder joint, and the positioning is not accurate, so the features are not fully learned by the network. Therefore, the following three improvement strategies are proposed, including data augmentation, k-means clustering to generate anchor boxes, and transfer learning with ResNet-101.

2.2.1. Data Augmentation

In this paper, 625 solder joint sample images were collected, including five types of qualified solder joints, multi-tin solder joints, less-tin solder joints, connected welding solder joints, and tin tip solder joints. Among them, 100 images were randomly selected as test images, and the remaining 525 images were used as training data, including 336 images of training set, 84 images of verification set and 105 images of test set. The deep neural network contains a large number of hidden layers and weights, and the data amount is too small to adjust the weights, which leads to the under-fitting of the model after training, thus affecting the accuracy of the model, and the actual detection effect will be poor. However, the limited welding capacity makes it difficult to continue expanding the data volume beyond the existing solder spot samples, and this is leading to the use of data augmentation to realize the expansion of the data set.

Data augmentation is a means of applying a set of basic forms of image processing to generate new images and, at the same time, corresponding label files for deep network training, usually involving rotation, mirror flipping, random shearing, brightness switching, contrast switching, etc. They can increase the data set. At the same time, different transformation eliminates the effect of target’s position, brightness, color and other properties in the image, so that the model can better recognize the target and improve the accuracy of the model. The data augmentation technology employed in this paper includes left-right flipping, up-down flipping, diagonal flipping, random brightness, and random contrast, and the results of this change are shown in Figure 6. In order to improve the robustness of the model, all transformations are random.

After data augmentation, the training data in this paper were expanded from 525 images to 1654 images, including 1059 images in the training set, 265 images in the validation set and 330 images in the test set.

2.2.2. K-Means Clustering Generates Anchor Boxes

The size of anchor boxes in the Faster-RCNN algorithm refers to the target size in VOC 2007 and VOC 2012 data set. The generated boxes can cover most of the targets, and a more accurate suggestion box can be obtained through regression. However, the size of the pin in the connector is small in the image. Although the official anchor box can get a more accurate suggestion box by training the regression coefficient, the adjustment scale is too large, which is not conducive to convergence. Therefore, k-means clustering is adopted in this paper to analyze all training data and obtain a set of anchor boxes with more appropriate scale. K-means clustering pseudo code is shown in Table 2.

Anchor boxes generated by k-means clustering are more in line with the actual size of detection targets compared with the preset anchor boxes based on experience, which is conducive to regression and obtaining a suggestion box with higher accuracy. Generally speaking, the larger a k value is, the more anchor boxes are generated and the higher the accuracy is. However, when k value is increased to a certain extent, the accuracy is basically unchanged, while too many suggestion boxes will greatly reduce the computational efficiency of the network. Therefore, the value range of k is generally defined in [2,10]. The average Intersection-over-Union (IOU) of anchor boxes generated by k-means clustering under different k values are shown in Figure 7.

2.2.3. Transfer Learning with ResNet-101

In the target detection task, the depth of the convolutional neural network is usually up to dozens or even hundreds of layers due to the complexity of the shape, color, and other characteristics of the target in the image. However, the deeper layer is not conducive to the parameter optimization of the network, and gradient explosion or gradient disappearance are likely to occur during the training process [25]. Although batch-norm, random gradient descent and other algorithms can be used to achieve a certain degree of optimization, the effect is not obvious. Therefore, in 2015, He proposed a new network structure unit, which is called the residual unit. Meanwhile, in 2016, He improved the residual unit to make it easier to train and increase its generalization ability [26]. The remaining units can be divided into 2 layers of remaining units and 3 layers of remaining units, as shown in Figure 8. The deep convolutional network consisting of the remaining units is called ResNet.

The residual unit changes the operation mode of the traditional convolutional network, in which, the final value obtained by convolutional layer and ReLU is shown in Equation (3).

H (x) = F (x) + x

(3)

where

F (x)

is the difference between the output

H (x)

and the input

x

, namely the residual. In an ideal situation, when the network reaches a certain depth, if the network state is already optimal,

F (x)

should be set as 0, which is equivalent to the output

H (x)

of the residual unit as

x

, so that the current depth network does not degrade, which ensures the accuracy of the deep network model.

ResNet has a variety of network structures, which can be divided into 18, 34, 50, 101, and 152 layers according to the convolutional layer depth. The residual units contained in different network structures are slightly different, as shown in Table 3. ResNet-18 and ResNet-34 are composed of 2-layers residual units, while ResNet-50, ResNet-101, and ResNet-152 are composed of 3-layers residual units.

At present, ResNet is widely used due to its remarkable parameter optimization ability in training [27,28,29,30]. In this paper, considering the accuracy of defect detection and actual computing ability, the convolution part of ResNet-101 was selected as the feature extraction module, and the original VGG-16 was replaced to improve the accuracy of the final detection results.

3. Results

In this paper, we chose the open-source version of TensorFlow Faster-RCNN on GitHub as the basic algorithm framework. In terms of hardware, Inter Core I7-8750h CPU and Nvidia GTX 1050Ti 8G GPU were used to achieve efficient operation rate. As for the data, 625 solder joint images were collected, including five types, namely qualified, multi-tin, less-tin, connected welding, and tin tip. A total of 100 images were randomly selected as the test images, and the remaining 525 images were expanded through the data augmentation and made into the training set, verification set and test set. In terms of network model parameters, the maximum steps of training were 30000, the batch-size was selected as 256, and the learning rate was selected as 0.001.

In the experiment, we first used the initial training set to train the original version of Faster-RCNN, and proposed three improvement strategies by analyzing the results. Subsequently, we used the enhanced data set for comparison experiments to verify the effectiveness of k-mean clustering, and used ResNet-101 to generate anchor boxes and transfer learning, respectively. Finally, we fused all the improved algorithms and completed the task of connector solder joint defect detection. By comparing and analyzing the experimental results, it is proved that the algorithm proposed in this paper is efficient and feasible for the detection of connector solder joint defects.

In the original version of Faster-RCNN, 9 fixed anchor boxes were selected, which were obtained through the comprehensive analysis of the target scale of Pascal VOC data set. Although the anchor boxes generated can contain most of the areas of the solder joint defect target in this paper, the offset coefficient in the regression algorithm is too large compared to the boxes with closer scales, resulting in the inaccurate position of the calculated proposals. This problem is particularly evident at the beginning of training, as shown in Figure 9a. The proposal loss obtained after using k-means clustering to generate anchor boxes algorithm is shown in Figure 9b. If the loss mutation caused by insufficient batch size (GPU memory limit) is not taken into account, it can be seen that the concussion range of loss in the early training period is reduced to a certain extent after using k-means clustering generates anchor boxes algorithm, and the final loss of network model has no obvious change.

A deep learning target detection model usually used the mAP as an index to measure the detection accuracy. By gradually reducing the threshold value of sample classification, the recall rate and corresponding precision of samples under each threshold value were calculated, and the precision-recall curve was drawn, and the area of the curve and the coordinate axis was mapped. The closer the mAP of a network model approaches 1, the higher the accuracy of the current network model will be. Table 4 and Table 5 list the mAP and detection accuracy of the network model under different conditions, respectively. Figure 10 and Figure 11 show the precision-recall curves of different training models and the total loss curves in the training process respectively. Figure 12 lists some of typical detection results of the method proposed in this paper. According to the experimental results, the model mAP and detection accuracy are improved to some extent after the training set is enhanced. In the case of using enhanced training set, k-means clustering generates anchor boxes and transfer learning with ResNet-101 both improve the accuracy of model mAP and detection accuracy, among which ResNet-101 plays a more obvious role. The final mAP and detection accuracy of the algorithm proposed are 0.941 and 94%, respectively, which significantly improves the effect of the original Faster-RCNN under the same training set.

4. Discussion

In this paper, the algorithm of Faster-RCNN was improved. The five transformation modes of left-right flipping, up-down flipping, diagonal flipping, random brightness, and random contrast were used in the data set to solve the problem of small amount of data in the original data set and avoid the situation of under-fitting during network model training. According to the generation mode of anchor boxes in RPN, this paper proposed to cluster the data sets using k-means clusters, analyzed the average IOU of anchor boxes generated by clustering algorithm in the case of different k values, and then selected appropriate clusters number to generate anchor boxes, which improved the positioning accuracy of proposals. Given the insufficiency of feature extraction network for connector’s solder joint defect feature extraction capability, the convolutional neural network with deeper layers is needed. In addition, considering that the deepening of the network may cause gradient explosion and reduce the accuracy of the model, this paper proposed to replace the VGG-16 with ResNet-101. After network replacement, the algorithm mentioned in this paper has enhanced its own ability to extract solder joint defect characteristics of connectors, and the network model has higher accuracy and stability.

After comparing and analyzing the actual detection capability and model precision of different algorithms, we find that the algorithm proposed in this paper is superior to the original Faster-RCNN in all aspects. The mAP of the algorithm increased from 0.8554 to 0.941, increasing by 8%. The final top-1 accuracy rate increased from 78 to 94%, with an improvement of 16%. Nevertheless, 94% of top-1 accuracy has been able to meet the needs of industrial detection.

We hope to detect the solder joint defects automatically with machine vision, so the algorithm proposed can only detect the solder joint image collected by the optical camera, but cannot detect the defects such as cracks or bubbles inside the solder joint. Currently, there are various types of connectors used in electronic products, and the sample used in this paper is only D-type data interface connector with 9 pins. However, the defect type and appearance of solder joint of connector are basically the same because the shape of welding cup of different connector is similar. Therefore the improved Faster-RCNN algorithm proposed in this paper can still obtain similar high accuracy rate, for the detection of solder joint defects in different connectors.

In the industry, there are many kinds of defects in connector solder joints. This paper only took 4 typical defect types and qualified solder joints as examples to train the model in this paper, it is still a long way to go in terms of ultimate practicality. Meanwhile, due to the lack of computing power, the algorithm of batch size 256 can see the loss jumping in the training process. The reason is that the smaller batch size does not match the images of more data sets, so the batch normalization effect of each batch of images is poor, which affects the model training and reduces the test accuracy to a certain extent. Therefore, in the following research, we will increase the types and images of solder joint defects, and at the same time consider more optimized training methods, so as to increase the algorithm’s ability to detect more kinds of solder joint defects on the premise of ensuring high accuracy and at the same time reduce the detection cost.

5. Conclusions

This paper has captured images of connectors containing the five solder joint types, achieved data expansion and trained, verified, and tested them with data augmentation. After training, the algorithm proposed in this paper has been verified by comparison experiments to be more accurate in the detection of solder joint defects. The main contributions of this paper are as follows: First, there are few studies on solder joint quality inspection at present, and most of them are detected by workers’ eyes during welding. The method proposed in this paper has obtained high detection accuracy, which provides a new way of thinking for solder joint quality detection of connectors. Secondly, a data set containing five solder joint types is produced to train the solder joint quality inspection network. At the same time, the data set will be filled in the following research process to enable the network to detect more connector solder joint defect types; Thirdly, the network model performance and the final detection accuracy have been significantly improved after optimization of the algorithm proposed in this paper. Therefore, the optimization method can also provide a new optimization strategy for solving similar target detection problems.

Author Contributions

Conceptualization, H.S.; methodology, K.Z.; software, K.Z.; validation, K.Z. and H.S.; formal analysis, K.Z.; investigation, K.Z.; data curation, H.S.; writing—original draft preparation, K.Z.; writing—review and editing, K.Z.; project administration, H.S.; All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Conflicts of Interest

We declare that we have no conflict of interest.

References

Lei, X. Research on Qualification Detection of Cable Joint Solder Joint Based on Deep Learning; Lanzhou University of Technology: Lanzhou, China, 2020. [Google Scholar]
Furlanetto, V.; Stocco, D.; Batalha, G.F. Electric resistance spot welding evaluation by A-scan and B-scan ultrasound analyses. In Proceedings of the AIP Conference, American Institute of Physics, College Park, MD, USA, 28–31 July 2011. [Google Scholar]
Zhou, X.; Zhou, J.; Tian, G.; Wang, Y. Research on defects inspection of solder balls based on eddy current pulsed thermography. Sensors 2015, 15, 25882–25897. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Kim, H.Y.; Han, S.S.; Hong, S.B.; Hong, S.J. Statistical process monitoring system for SMT industry using automatic optical inspection system. Mater. Sci. Forum 2008, 733, 561–564. [Google Scholar] [CrossRef]
Chiu, M.C.; Yeh, L.J.; Hsu, C.J. The deficiency recognition in PCBA`s automatic optical inspection system by using back-propagation network method. J. Appl. Sci. 2008, 8, 2814. [Google Scholar] [CrossRef] [Green Version]
Jong, H.J.; Chia, H.C.; Tyng, B.H. Development of an automatic optical inspection system and its application to defect examination. Appl. Mech. Mater. 2014, 2872, 636–640. [Google Scholar]
Ssu-Han, C.; Der-Baau, P. Automatic optical inspection system for IC molding surface. J. Intellig. Manufact. 2016, 27, 915–926. [Google Scholar]
Lu, R.; Wu, A.; Zhang, T.; Wang, Y. Review on automated optical (visual) inspection and its applications in defect detection. Acta Optic. Sin. 2018, 38, 23–58. [Google Scholar]
Wu, F.; Zhang, X.; Kuan, Y.; He, Z. An AOI algorithm for PCB based on feature extraction. In Proceedings of the 7th World Congress on Intelligent Control and Automation, Chongqing, China, 25–27 June 2008. [Google Scholar]
Yuan Wang, S.; Zhao, Y.; Wen, L. PCB welding spot detection with image processing method based on automatic threshold image segmentation algorithm and mathematical morphology. Circuit World 2016, 42, 97–103. [Google Scholar] [CrossRef]
Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. Commun. ACM. 2017, 60, 84–90. [Google Scholar] [CrossRef]
Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intellig. 2015, 39, 1137–1149. [Google Scholar] [CrossRef] [Green Version]
Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016. [Google Scholar]
Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.Y.; Berg, A.C. SSD: Single shot MultiBox detector. In Proceedings of the European Conference on Computer Vision, Santiago, Chile, 7–13 December 2015. [Google Scholar]
Yin, S.; Li, H.; Teng, L. Airport detection based on improved faster RCNN in large scale remote sensing images. Sens. Imag. 2020, 21, 1747–1755. [Google Scholar] [CrossRef]
Mo, N.; Yan, L. Improved faster RCNN based on feature amplification and oversampling data augmentation for oriented vehicle detection in aerial images. Remote. Sens. 2020, 12, 2558. [Google Scholar] [CrossRef]
Li, P.; Zhao, W. Image fire detection algorithms based on convolutional neural networks. Case Stud. Therm. Eng. 2020, 19, 9. [Google Scholar] [CrossRef]
Ju, M.; Luo, H.; Wang, Z.; Hui, B.; Chang, Z. The application of improved YOLO V3 in multi-scale target detection. Appl. Sci. 2019, 9, 3775. [Google Scholar] [CrossRef] [Green Version]
Ding, F.; Zhuang, Z.; Liu, Y.; Jiang, D.; Yan, X.; Wang, Z. Detecting defects on solid wood panels based on an improved SSD algorithm. Sensors 2020, 20, 5315. [Google Scholar] [CrossRef] [PubMed]
Leng, J.; Liu, Y. An enhanced SSD with feature fusion and visual reasoning for object detection. Neural Comput. Appl. 2019, 31, 6549–6558. [Google Scholar] [CrossRef]
Mujeeb, A.; Dai, W.; Erdt, M.; Sourin, A. One class based feature learning approach for defect detection using deep autoencoders. Adv. Eng. Inform. 2019, 42, 100933. [Google Scholar] [CrossRef]
Li, J.; Gu, J.; Huang, Z.; Wen, J. Application research of improved YOLO V3 algorithm in PCB electronic component detection. Appl. Sci. 2019, 9, 3750. [Google Scholar] [CrossRef] [Green Version]
Urbonas, A.; Raudonis, V.; Maskeliūnas, R.; Damaševičius, R. Automated identification of wood veneer surface defects using faster region-based convolutional neural network with data augmentation and transfer learning. Appl. Sci. 2019, 9, 4898. [Google Scholar] [CrossRef] [Green Version]
Gao, X.; Jian, M.; Hu, M.; Tanniru, M.; Li, S. Faster multi-defect detection system in shield tunnel using combination of FCN and faster RCNN. Adv. Struct. Eng. 2019, 22, 2907–2921. [Google Scholar] [CrossRef]
He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision & Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016. [Google Scholar]
He, K.; Zhang, X.; Ren, S.; Sun, J. Identity Mappings in Deep Residual Networks. In Proceedings of the ECCV 2016: Computer Vision—ECCV 2016, Amsterdam, The Netherlands, 8–16 October 2016; pp. 630–645. [Google Scholar]
Liang, J. Image classification based on RESNET. J. Phys. 2020, 1634, 012110. [Google Scholar] [CrossRef]
Li, Z.; Lin, Y.; Elofsson, A.; Yao, Y. Protein contact map prediction based on ResNet and DenseNet. BioMed Res. Int. 2020, 2020, 7584968. [Google Scholar] [CrossRef] [PubMed]
Li, L.; Qin, S.; Lu, Z.; Xu, K.; Hu, Z. One-shot learning gesture recognition based on joint training of 3D ResNet and memory module. Multimed. Tools Appl. 2020, 79, 6727–6757. [Google Scholar] [CrossRef]
Wang, A.; Wang, M.; Wu, H.; Jiang, K.; Iwahori, Y. A novel LiDAR data classification algorithm combined CapsNet with ResNet. Sensors 2020, 20, 1151. [Google Scholar] [CrossRef] [PubMed] [Green Version]

Figure 1. Examples of five connector solder joints. (a) the qualified solder joint, (b) the multi-tin solder joint, (c) the less-tin solder joint, (d) the connected welding solder joint and (e) the tin tip solder joint.

Figure 2. Faster-RCNN uses 9 anchor boxes to get suggestion boxes. Rectangular boxes in the same color represent the same scale.

Figure 3. Operational process of Faster-RCNN. The target classification and positioning module uses full connection layers to calculate the final scores and target position coordinates.

Figure 4. VGG-16 feature extract network. For the pool layer of VGG-16, Max pool is adopted and the final generated feature map scale is 1/16 of the resized image scale.

Figure 5. Operational process of RPN. After RPN, there are about 300 boxes, where the scale variables and label variables are preserved.

Figure 6. Results of data augmentation in this paper. (a) the original image, (b) the result of left-right flipping, (c) the result of up-down flipping, (d) the result of random brightness, (e) the result of diagonal flipping and (f) the result of random contrast.

Figure 7. Anchor boxes average IOU under different number of clusters. In this paper, 9 clusters are selected to realize the generation of anchor boxes, with an average IOU of 80.25%.

Figure 8. Two kinds of residual unit. (a) 2-layers residual unit; (b) 3-layers residual unit.

Figure 9. RPN bounding box regression loss curve. (a) Original anchor boxes scale; (b) K-means clustering anchor boxes scale. The red curve is the data fitting curve, showing a general downward trend. After 30,000 iterations, the loss of original algorithm is

2.034 \times 10^{- 4}

, and the loss after using k-means clustering generates anchor boxes algorithm is

1.293 \times 10^{- 4}

2.034 \times 10^{- 4}

, and the loss after using k-means clustering generates anchor boxes algorithm is

1.293 \times 10^{- 4}

Figure 10. Different training model’s precision-recall curve. (a) Faster-RCNN training model without data augmentation; (b) Faster-RCNN training model with data augmentation; (c) Faster-RCNN training model with data augmentation and k-means clustering; (d) Training model with data augmentation and ResNet-101 transfer learning; (e) Training model proposed in this paper. Each color represents one type of solder joint and the mAP of each type is shown as the label.

Figure 11. Different network’s total loss. (a) Faster-RCNN training model without data augmentation; (b) Faster-RCNN training model with data augmentation; (c) Faster-RCNN training model with data augmentation and k-means clustering; (d) Training model with data augmentation and ResNet-101 transfer learning; (e) Training model proposed in this paper. After removing the bad point data, it can be seen that the algorithm proposed in this paper has a good optimization effect on model parameters.

Figure 12. Some of typical detect results. In the pictures, hg represents qualified solder joint; dx represents multi-tin solder joint; sx represents less-tin solder joint; lh represents connected welding solder joint; xj represents tin tip solder joint.

Table 1. Detailed descriptions of solder joints defects.

Defect Type	Description
Multi-tin	The solder joint is too large, exceeding the distance between adjacent pins by 50%
Less-tin	The solder joint is too small, without covering more than 75% of the pin cup
Connected welding	The solder is connected to the adjacent pin, causing a short circuit
Tin tip	The surface of the solder joint has a sharp thorn, easy to cause short circuit

Table 2. K-means clustering pseudo-code.

Ground Truth Boxes: ${(w_{i}^{}, h_{i}^{})}, i \in [1, N]$
Initialize k clustering centers: ${(w_{j}, h_{j}), j \in [1, k]}$ Iteration: stop if the change of new clustering centers is smaller than threshold. for $i$ in range ( $N$ ): for $j$ in range ( $k$ ): $d_{i} (j) = 1 - I O U ((w_{i}^{}, h_{i}^{}), (w_{j}, h_{j}))$ $c l s (i) = I n d e x (\underset{j \in [1, k]}{m i n} d_{i} (j))$ Calculate new $k$ clustering centers: $w_{j}^{'} = \frac{1}{N_{j}} \sum w_{i}^{} (j), h_{j}^{'} = \frac{1}{N_{j}} \sum h_{i}^{} (j), i \in [1, N], j \in [1, k]$

Ground Truth Boxes:

{(w_{i}^{*}, h_{i}^{*})}, i \in [1, N]

Initialize k clustering centers:

{(w_{j}, h_{j}), j \in [1, k]}

Iteration: stop if the change of new clustering centers is smaller than threshold.
for

i

in range (

N

):
for

j

in range (

k

d_{i} (j) = 1 - I O U ((w_{i}^{*}, h_{i}^{*}), (w_{j}, h_{j}))

c l s (i) = I n d e x (\underset{j \in [1, k]}{m i n} d_{i} (j))

Calculate new

k

clustering centers:

w_{j}^{'} = \frac{1}{N_{j}} \sum w_{i}^{*} (j), h_{j}^{'} = \frac{1}{N_{j}} \sum h_{i}^{*} (j), i \in [1, N], j \in [1, k]

Table 3. ResNet-N network structure and accuracy.

Layer Name	Output Size	18-Layer	34-Layer	50-Layer	101-Layer	152-Layer
conv1	$112 \times 112$	7 × 7, 64, stride 2
conv2_x	$56 \times 56$	3 × 3 max pool, stride 2
conv2_x	$56 \times 56$	$[\begin{matrix} 3 \times 3, 64 \\ 3 \times 3, 64 \end{matrix}] \times 2$	$[\begin{matrix} 3 \times 3, 64 \\ 3 \times 3, 64 \end{matrix}] \times 3$	$[\begin{matrix} 1 \times 1, 64 \\ 3 \times 3, 64 \\ 1 \times 1, 256 \end{matrix}] \times 3$	$[\begin{matrix} 1 \times 1, 64 \\ 3 \times 3, 64 \\ 1 \times 1, 256 \end{matrix}] \times 3$	$[\begin{matrix} 1 \times 1, 64 \\ 3 \times 3, 64 \\ 1 \times 1, 256 \end{matrix}] \times 3$
conv3_x	$28 \times 28$	$[\begin{matrix} 3 \times 3, 128 \\ 3 \times 3, 128 \end{matrix}] \times 2$	$[\begin{matrix} 3 \times 3, 128 \\ 3 \times 3, 128 \end{matrix}] \times 4$	$[\begin{matrix} 1 \times 1, 128 \\ 3 \times 3, 128 \\ 1 \times 1, 512 \end{matrix}] \times 4$	$[\begin{matrix} 1 \times 1, 128 \\ 3 \times 3, 128 \\ 1 \times 1, 512 \end{matrix}] \times 4$	$[\begin{matrix} 1 \times 1, 128 \\ 3 \times 3, 128 \\ 1 \times 1, 512 \end{matrix}] \times 8$
conv4_x	$14 \times 14$	$[\begin{matrix} 3 \times 3, 256 \\ 3 \times 3, 256 \end{matrix}] \times 2$	$[\begin{matrix} 3 \times 3, 256 \\ 3 \times 3, 256 \end{matrix}] \times 6$	$[\begin{matrix} 1 \times 1, 256 \\ 3 \times 3, 256 \\ 1 \times 1, 1024 \end{matrix}] \times 6$	$[\begin{matrix} 1 \times 1, 256 \\ 3 \times 3, 256 \\ 1 \times 1, 1024 \end{matrix}] \times 23$	$[\begin{matrix} 1 \times 1, 256 \\ 3 \times 3, 256 \\ 1 \times 1, 1024 \end{matrix}] \times 36$
conv5_x	$7 \times 7$	$[\begin{matrix} 3 \times 3, 512 \\ 3 \times 3, 512 \end{matrix}] \times 2$	$[\begin{matrix} 3 \times 3, 512 \\ 3 \times 3, 512 \end{matrix}] \times 3$	$[\begin{matrix} 1 \times 1, 512 \\ 3 \times 3, 512 \\ 1 \times 1, 2048 \end{matrix}] \times 3$	$[\begin{matrix} 1 \times 1, 512 \\ 3 \times 3, 512 \\ 1 \times 1, 2048 \end{matrix}] \times 3$	$[\begin{matrix} 1 \times 1, 512 \\ 3 \times 3, 512 \\ 1 \times 1, 2048 \end{matrix}] \times 3$
	$1 \times 1$	average pool, 1000-d fc, softmax
FLOPs		$1.8 \times 10^{9}$	$3.6 \times 10^{9}$	$3.8 \times 10^{9}$	$7.6 \times 10^{9}$	$11.3 \times 10^{9}$
Top-1 error		27.88%	25.03%	22.85%	21.75%	21.43%

Table 4. mAP of original network and different training model.

	mAP
VGG-16	0.8364
VGG-16 + data augmentation ¹	0.8554
VGG-16 + data augmentation + k-means clustering ²	0.8670
ResNet-101 + data augmentation ³	0.9337
This paper	0.941

^1,2,3 Training model using different improved algorithm.

Table 5. Detection accuracy of 5 solder joint types and mean detection accuracy.

	Qualified	Multi-tin	Less-tin	Connected Welding	Tin Tip	Mean Accuracy
Original	60%	95%	65%	90%	75%	77%
Training model ¹	65%	90%	70%	95%	70%	78%
Training model ²	70%	85%	70%	95%	80%	80%
Training model ³	90%	95%	90%	95%	85%	91%
This paper	90%	95%	95%	100%	90%	94%

^1,2,3 Training model using different improved algorithm. Corresponds to the number in Table 4.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Zhang, K.; Shen, H. Solder Joint Defect Detection in the Connectors Using Improved Faster-RCNN Algorithm. Appl. Sci. 2021, 11, 576. https://doi.org/10.3390/app11020576

AMA Style

Zhang K, Shen H. Solder Joint Defect Detection in the Connectors Using Improved Faster-RCNN Algorithm. Applied Sciences. 2021; 11(2):576. https://doi.org/10.3390/app11020576

Chicago/Turabian Style

Zhang, Kaihua, and Haikuo Shen. 2021. "Solder Joint Defect Detection in the Connectors Using Improved Faster-RCNN Algorithm" Applied Sciences 11, no. 2: 576. https://doi.org/10.3390/app11020576

APA Style

Zhang, K., & Shen, H. (2021). Solder Joint Defect Detection in the Connectors Using Improved Faster-RCNN Algorithm. Applied Sciences, 11(2), 576. https://doi.org/10.3390/app11020576

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu

Solder Joint Defect Detection in the Connectors Using Improved Faster-RCNN Algorithm

Abstract

1. Introduction

2. Materials and Methods

2.1. Key Technology of Faster-RCNN

2.1.1. Anchor Boxes

2.1.2. Network Architecture

2.1.3. Loss Function

2.2. Important Improvements

2.2.1. Data Augmentation

2.2.2. K-Means Clustering Generates Anchor Boxes

2.2.3. Transfer Learning with ResNet-101

3. Results

4. Discussion

5. Conclusions

Author Contributions

Funding

Conflicts of Interest

References

Share and Cite

Article Metrics

Article Access Statistics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI