Nothing Special   »   [go: up one dir, main page]
Skip to main content

Advertisement

Springer Nature Link
Log in

A dual-structure attention-based multi-level feature fusion network for automatic surface defect detection

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The detection of surface defects is crucial to industrial manufacturing. In recent years, numerous detection methods based on computer vision have been successfully applied in the industry. However, industrial defect detection is still full of challenges. In one aspect, most of the industrial defects are extremely small. In another aspect, even though the intra-class defects have numerous similar elements, their outward appearances differ significantly. In this paper, we propose a dual-structure attention-based multi-level feature fusion network (DaMFFN) to address these two issues. In the first attention-based multi-level feature extraction structure, we introduce novel attention pooling to preserve more detailed information about the defective features of the tiny defect by giving certain regions varying weights. In the second attention-based multi-level feature fusion structure, we propose channel attention to capture the defect feature with the greatest potential for discrimination rather than all possible defect features, which is employed to prevent the incorrect detection of intra-class defects. The experiments demonstrate that the detection performance of the DaMFFN is better than other methods in five surface defect datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The data used to support the findings of this study contains two parts: AIRSKIN_DET is our own collection of aircraft skin surface defect datasets, which are available from the corresponding author on reasonable request; the other parts are openly available, including DAGM 2007, NEU_DET, PCB, and TianChi_Fabric.

References

  1. Tao, J., Zhu, Y., Jiang, F., Liu, H., Liu, H.: Rolling surface defect inspection for drum-shaped rollers based on deep learning. IEEE Sens. J. 22(9), 8693–8700 (2022). https://doi.org/10.1109/JSEN.2022.3159743

    Article  Google Scholar 

  2. Dong, H., Song, K., He, Y., Xu, J., Yan, Y., Meng, Q.: PGA-Net: pyramid feature fusion and global context attention network for automated surface defect detection. IEEE Trans. Ind. Infor. 16(12), 7448–7458 (2020). https://doi.org/10.1109/TII.2019.2958826

    Article  Google Scholar 

  3. Su, B., Chen, H., Chen, P., Bian, G., Liu, K., Liu, W.: Deep learning-based solar-cell manufacturing defect detection with complementary attention network. IEEE Trans. Ind. Inform. 17(6), 4084–4095 (2021). https://doi.org/10.1109/TII.2020.3008021

    Article  Google Scholar 

  4. Zhu, Z., Zhu, P., Zeng, J., Qian, X.: A surface fatal defect detection method for magnetic tiles based on semantic segmentation and object detection: IEEE ITAIC(ISSN:2693-2865), 2022 IEEE 10th joint international information technology and artificial intelligence conference (ITAIC), pp. 2580–2586 (2022). https://doi.org/10.1109/ITAIC54216.2022.9836478

  5. Li, Y., Liu, P., Xiang, Y., Liu, C., Guo, H.: Detection approach based on an improved YOLOX for Flange Surface Defects, In: 2022 4th International conference on intelligent control, measurement and signal processing (ICMSP), pp. 227-232 (2022). https://doi.org/10.1109/ICMSP55950.2022.9859056

  6. Bin Roslan, M.I., Ibrahim, Z., Aziz, Z.A.: Real-time plastic surface defect detection using deep learning. In: 2022 IEEE 12th symposium on computer applications & industrial electronics (ISCAIE), pp. 111–116 (2022). https://doi.org/10.1109/ISCAIE54458.2022.9794475

  7. Hang, J., Sun, H., Yu, X., Rodríguez-Andina, J.J., Yang, X.: Surface defect detection in sanitary ceramics based on lightweight object detection network. IEEE Open J. Ind. Electr. Soci. 3, 473–483 (2022). https://doi.org/10.1109/OJIES.2022.3193572

    Article  Google Scholar 

  8. Wang, W., et al.: A real-time steel surface defect detection approach with high accuracy. IEEE Trans. Instrument. Measur. 71(1–10), 5005610 (2022). https://doi.org/10.1109/TIM.2021.3127648

    Article  Google Scholar 

  9. Li, Y., Liu, P., Xiang, Y., Liu, C., Guo, H.: Detection approach based on an improved YOLOX for flange surface defects In: 2022 4th international conference on intelligent control, measurement and signal processing (ICMSP), pp. 227–232 (2022). https://doi.org/10.1109/ICMSP55950.2022.9859056

  10. Jiang, W., Liu, M., Peng, Y., Wu, L., Wang, Y.: HDCB-Net: a neural network with the hybrid dilated convolution for pixel-level crack detection on concrete bridges. IEEE Trans. Ind. Inform. 17(8), 5485–5494 (2021). https://doi.org/10.1109/TII.2020.3033170

    Article  Google Scholar 

  11. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. IEEE Conf. Comput. Visi. Patt. Recognit. (CVPR) 2017, 5987–5995 (2017). https://doi.org/10.1109/CVPR.2017.634

    Article  Google Scholar 

  12. He, Y., Song, K., Meng, Q., Yan, Y.: An end-to-end steel surface defect detection approach via fusing multiple hierarchical features. IEEE Trans. Instrum. Meas. 69(4), 14931504 (2020)

    Article  Google Scholar 

  13. Bhatt, P.M., Malhan, R.K., Rajendran, P., Shah, B.C., Thakar, S., Yoon, Y.J., Gupta, S.K.: Image-based surface defect detection using deep learning: a review. J. Comput. Inform. Sci. Eng. (2021). https://doi.org/10.1115/1.4049535

    Article  Google Scholar 

  14. Dong, H., et al.: PGA-Net pyramid feature fusion and global context attention network for automated surface defect detection. IEEE Trans. Ind. Inform 16(12), 7448–7458 (2019)

    Article  Google Scholar 

  15. Cao, J., Yang, G., Yang, X.: A pixel-level segmentation convolutional neural network based on deep feature fusion for surface defect detection. IEEE Trans. Instru. Measur. 70, 1–12 (2020)

    Google Scholar 

  16. Hu, B., et al.: A lightweight spatial and temporal multi-feature fusion network for defect detection. IEEE Trans. Image Process. 30, 472–486 (2020)

    Article  Google Scholar 

  17. Konovalenko, I., et al.: Defectoscopic and geometric features of defects that occur in sheet metal and their description based on statistical analysis. Metals 11(11), 1851 (2021)

    Article  Google Scholar 

  18. Üzen, Hüseyin., et al.: Depth-wise squeeze and excitation block-based efficient-Unet model for surface defect detection. Vis. Comput. 39(5), 1745–1764 (2022)

    Article  Google Scholar 

  19. Konovalenko, I., et al.: Research of U-Net-based CNN architectures for metal surface defect detection. Machines 10(5), 327 (2022)

    Article  Google Scholar 

  20. Üzen, H., et al.: Swin-MFINet: swin transformer based multi-feature integration network for detection of pixel-level surface defects. Expert Sys. Appl. 209, 118269 (2022)

    Article  Google Scholar 

  21. Singh, S.A., Kumar, A.S., Desai, K.A.: Comparative assessment of common pre-trained CNNs for vision-based surface defect detection of machined components. Expert Sys. Appl. 218, 119623 (2023)

    Article  Google Scholar 

  22. Üzen, H., Turkoglu, M., Hanbay, D.: Multi-dimensional feature extraction-based deep encoder-decoder network for automatic surface defect detection. Neural Comput. Appl. 35(4), 3263–3282 (2023)

    Article  Google Scholar 

  23. Lin, T., Dollar, P., Girshick, R., He, K., Hatiharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2017)

  24. Ma, N., Zhang, X., Zheng, H., Sun, J.: Shufflenet v2: Practical guidelines for efficient CNN architecture design. In: Proceedings of the European Conference on computer vision (ECCV), pp. 116-131 (2018)

  25. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Patt. Anal. Mach. Intell. 39(6), 11371149 (2017)

    Article  Google Scholar 

  26. Girshick, R.: Fast R-CNN. In: Proceeding of IEEE International Conference Computer Vision (ICCV), Santiago, Chile, pp. 14401448 (2015)

  27. Long, J., Shelhamer, E., Darrell, T. (2015) Fully convolutional networks for semantic segmentation, In: Proceeding of IEEE Computer Vision Pattern Recognition (CVPR), Columbus, OH, USA, pp. 34313440

  28. Paszke, A., Gross, S., Massa, F., Lerer, A. et al.: Pytorch: an imperative style, high-performance deep learning library. Adv. Neural Inform. Process. Syst. 32 (2019)

  29. Wieler, M., Hahn, T.: Weakly supervised learning for industrial optical inspection. Accessed: Jun. 25, (2017) [Online]. Available: https://hci.iwr.uni-heidelberg.de/node/3616

  30. Wei, P.: PKU-Market-PCB, Open lab on human robot interaction, [Online]. Available: https://robotics.pkusz.edu.cn/resources/dataset/

  31. Tianchi: smart diagnosis of cloth flaw dataset, 2020. [Online]. Available:https://tianchi.aliyun.com/dataset/dataDetail?dataId=79336

  32. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. IEEE Trans. Patt. Anal. Mach. Intell. 42(2), 318–327 (2020). https://doi.org/10.1109/TPAMI.2018.2858826

    Article  Google Scholar 

  33. Redmon, J., Ali F.: Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)

  34. Bochkovskiy, A., Chien-Yao W., Hong-Yuan M. L.: Yolov4: optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)

  35. Chen, J., et al.: Transunet: transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021)

  36. Cao, H., et al.: Swin-unet: Unet-like pure transformer for medical image segmentation. Computer Vision-ECCV, et al.: Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part III, p. 2023. Springer Nature Switzerland, Cham (2022)

  37. Oktay, O., et al.: Attention u-net: learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 (2018)

  38. Baheti, B., et al.: Eff-unet: a novel architecture for semantic segmentation in unstructured environment. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition workshops. (2020)

  39. Wei, Y., et al.: Cross-modal retrieval with CNN visual features: a new baseline. IEEE Trans. Cybernet. 47(2), 449–460 (2016)

    Google Scholar 

  40. Hosang, J., Benenson, R., Dollár, P., Schiele, B.: What makes for effective detection proposals? IEEE Trans. Patt. Anal. Mach. Intell. 38(4), 814–830 (2016). https://doi.org/10.1109/TPAMI.2015.2465908

    Article  Google Scholar 

  41. He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016)

  42. Hu, J., Li S., Gang S.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2018)

  43. Kong, T., Yao, A., Chen, Y., Sun, F.: HyperNet: towards accurate region proposal generation and joint object detection. IEEE Conf. Comput. Vis. Patt. Recognit. (CVPR) 2016, 845–853 (2016). https://doi.org/10.1109/CVPR.2016.98

    Article  Google Scholar 

  44. Li, Z., et al.: Detnet: a backbone network for object detection. arXiv preprint arXiv:1804.06215 (2018)

Download references

Acknowledgements

This research was supported by funding from the National Natural Science Foundation of China (62173331, 52005500), Natural Science Foundation of Tianjin Municipal Science and Technology Commission (2020KJ013), Civil Aviation University of China Research Innovation Project for Postgraduate Students (2022YJS018), and The Basic Science-research Funds of National University (3122023044, 3122023PY06).

Author information

Authors and Affiliations

  1. College of Electronic Information and Automation, Civil Aviation University of China, 2898 Jinbei Road, Dongli District, Tianjin, 300300, China

    Xiaoyu Zhang, Jinping Zhang, Jiusheng Chen & Runxia Guo

  2. College of Aeronautical Engineering, Civil Aviation University of China, 2898 Jinbei Road, Dongli District, Tianjin, 300300, China

    Jun Wu

Authors
  1. Xiaoyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinping Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiusheng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Runxia Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jun Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoyu Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: the photos of Runxia Guo and Jun Wu were exchanged.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Zhang, J., Chen, J. et al. A dual-structure attention-based multi-level feature fusion network for automatic surface defect detection. Vis Comput 40, 2713–2732 (2024). https://doi.org/10.1007/s00371-023-02980-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02980-1

Keywords

Search

Navigation