Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Microinstrument contact force sensing based on cable tension using BLSTM–MLP network

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Minimally invasive surgical robotic systems established the foundation for precise and refined surgery, and the intelligentization of robotic systems is an important direction for future development. Among the methods of intelligentization, microinstrument external force sensing is an open and challenging research area. Force sensing information is used not only to ensure that surgeons apply the appropriate amount of force but also to prevent unintentional tissue damage. Because a microinstrument is a compact and small-sized construction, indirect force sensing method instead of the integration of sensors into the microinstrument is used, yielding better biocompatibility, sterilizability and monetary cost savings. This paper focuses on microinstrument-tissue contact force sensing, and the microinstrument used is a three degrees of freedom cable-driven manipulator. A contact force estimation strategy based on the differences in cable tension is established with consideration of the kinematics, dynamics and friction of the manipulator. A principle prototype of a surgical microinstrument force measurement system is developed, and then zero-drift, hysteresis and force loading experiments are studied. Based on the experimental data of the force loading experiments, the relationship between cable tension and contact forces is established by using a bidirectional long short-term memory plus multilayer perceptron network. The results show that the L2 cost of the network in the training set converges to 0.006 and that the RMSE of the network in the testing set converges to 0.053, and the network can meet the measurement requirements without overfitting. Therefore, the indirect force estimation method is a viable method of measuring forces of cable-driven microinstrument and can be used to integrate force sensing information into surgical robotic systems to improve the operability of surgical robots.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Abbreviations

P :

Motor displacement

T :

Cable tension

F f :

Friction

M f :

Friction torque

τ :

Driving torque

F :

Contact force

q :

Generalized coordinate

r :

Centroid position

m :

Mass

J :

Pseudoinertia matrix

ANN:

Artificial neural network

RMIS:

Robot-assisted minimally invasive surgery

DOF:

Degrees of freedom

F.S.:

Full span

DC:

Direct current

RNN:

Recurrent neural network

LSTM:

Long short-term memory network

BLSTM:

Bidirectional long short-term memory network

MLP:

Multilayer perceptron

RMSE:

Root-mean-squared error

References

  1. Freschi C, Ferrari V, Melfi F, Ferrari M, Mosca F, Cuschieri A (2013) Technical review of the da Vinci surgical telemanipulator. Int J Med Robot Comput Assist Surg 9:396–406. https://doi.org/10.1002/rcs.1468

    Article  Google Scholar 

  2. Ng AT, Tam PC (2014) Current status of robot-assisted surgery. Hong Kong Med J 20:241–250. https://doi.org/10.12809/hkmj134167

    Article  Google Scholar 

  3. Rassweiler J, Rassweiler M, Kenngott H, Frede T, Michel M, Alken P, Clayman R (2013) The past, present and future of minimally invasive therapy in urology: a review and speculative outlook. Minim Invasive Ther Allied Technol 22:200–209. https://doi.org/10.3109/13645706.2013.816323

    Article  Google Scholar 

  4. Finelli A, Gill I (2004) Laparoscopic partial nephrectomy: contemporary technique and results. Urol Oncol Semin Orig Investig 22:139–144. https://doi.org/10.1016/j.urolonc.2004.01.004

    Article  Google Scholar 

  5. Valero R, Ko Y, Chauhan S, Schatloff O, Sivaraman A, Coelho R, Ortega F, Palmer K, Sanchez-Salas R, Davila H, Cathelineau X, Patel V (2011) Robotic surgery: history and teaching impact. Actas Urológicas Españolas 35:540–545. https://doi.org/10.1016/j.acuro.2011.04.005

    Article  Google Scholar 

  6. Turchetti G, Palla I, Pierotti F, Cuschieri A (2012) Economic evaluation of da Vinci-assisted robotic surgery: a systematic review. Surg Endosc 26:598–606. https://doi.org/10.1007/s00464-011-1936-2

    Article  Google Scholar 

  7. Okamura A (2009) Haptic feedback in robot-assisted minimally invasive surgery. Curr Opin Urol 19:102–107. https://doi.org/10.1097/MOU.0b013e32831a478c

    Article  Google Scholar 

  8. Hamed A, Tang S, Ren H, Squires A, Payne C, Masamune K, Tang G, Mohammadpour J, Tse Z (2012) Advances in haptics, tactile sensing, and manipulation for robot-assisted minimally invasive surgery, noninvasive surgery, and diagnosis. J Robot 2012:412816. https://doi.org/10.1155/2012/412816

    Article  Google Scholar 

  9. Fu Y, Li K, Pan B, Zhang J, Wang S (2014) A survey of force sensing and force feedback technology for robot-assisted minimally invasive surgical system. Jiqiren/Robot 36:117–128. https://doi.org/10.3724/SP.J.1218.2014.00117

    Article  Google Scholar 

  10. Hong M, Jo Y (2012) Design and evaluation of 2-DOF compliant forceps with force-sensing capability for minimally invasive robot surgery. IEEE Trans Robot 28:932–941. https://doi.org/10.1109/TRO.2012.2194889

    Article  Google Scholar 

  11. Desai J, Valdevit A, Ritter A (2015) Development of a real-time simulink based robotic system to study force feedback mechanism during instrument–object interaction. Int Sch Sci Res Innov 9:332–337

    Google Scholar 

  12. Lee D, Kim U, Gulrez T, Yoon W, Hannaford B, Choi H (2016) A laparoscopic grasping tool with force sensing capability. IEEE/ASME Trans Mechatron 21:130–141. https://doi.org/10.1109/TMECH.2015.2442591

    Article  Google Scholar 

  13. Burkhard N, Cutkosky M, Steger J (2018) Slip sensing for intelligent, improved grasping and retraction in robot-assisted surgery. IEEE Robot Autom Lett 3:4148–4155. https://doi.org/10.1109/LRA.2018.2863360

    Article  Google Scholar 

  14. Tholey G, Pillarisetti A, Green W, Desai J (2004) Design, development, and testing of an automated laparoscopic grasper with 3-D force measurement capability. Int Symp Med Simul 3078:38–48. https://doi.org/10.1007/978-3-540-25968-8_5

    Article  Google Scholar 

  15. Li Y, Miyasaka M, Haghighipanah M, Cheng L, Hannaford B (2016) Dynamic modeling of cable driven elongated surgical instruments for sensorless grip force estimation. In: IEEE international conference on robotics and automation 2016, pp 4128–4134. https://doi.org/10.1109/ICRA.2016.7487605

  16. Takizawa T, Kanno T, Miyazaki R, Tadano K, Kawashima K (2018) Grasping force estimation in robotic forceps using a soft pneumatic actuator with a built-in sensor. Sens Actuators A 271:124–130. https://doi.org/10.1016/j.sna.2018.01.007

    Article  Google Scholar 

  17. Li H, Kawashima K, Tadano K, Ganguly S, Nakano S (2013) Achieving haptic perception in forceps’ manipulator using pneumatic artificial muscle. IEEE/ASME Trans Mechatron 18:74–85. https://doi.org/10.1109/TMECH.2011.2163415

    Article  Google Scholar 

  18. Haraguchi D, Kanno T, Tadano K, Kawashima K (2015) A pneumatically driven surgical manipulator with a flexible distal joint capable of force sensing. IEEE/ASME Trans Mechatron 20:2950–2961. https://doi.org/10.1109/tmech.2015.2415838

    Article  Google Scholar 

  19. Miyasaka M, Matheson J, Lewis A, Hannaford B (2015) Measurement of the cable-pulley Coulomb and viscous friction for a cable-driven surgical robotic system. In: IEEE/RSJ international conference on intelligent robots and systems 2015, pp 804–810. https://doi.org/10.1109/IROS.2015.7353464

  20. He C, Wang S, Sang H, Li J, Zhang L (2014) Force sensing of multiple-DOF cable-driven instruments for minimally invasive robotic surgery. Int J Med Robot Comput Assist Surg 10:314–324. https://doi.org/10.1002/rcs.1532

    Article  Google Scholar 

  21. Sang H, Yun J, Monfaredi R, Wilson E, Fooladi H, Cleary K (2017) External force estimation and implementation in robotically assisted minimally invasive surgery. Int J Med Robot Comput Assist Surg 13:e1824. https://doi.org/10.1002/rcs.1824

    Article  Google Scholar 

  22. Xue R, Ren B, Huang J, Yan Z, Du Z (2018) Design and evaluation of FBG-based tension sensor in laparoscope surgical robots. Sensors 18:2067. https://doi.org/10.3390/s18072067

    Article  Google Scholar 

  23. Su Y, Huang K, Hannaford B (2018) Real-time vision-based surgical tool segmentation with robot kinematics prior. In: International symposium on medical robotics 2018, pp 1–6. https://doi.org/10.1109/ISMR.2018.8333305

  24. Kim W, Seung S, Choi H, Park S, Ko S, Park J (2012) Image-based force estimation of deformable tissue using depth map for single-port surgical robot. In: International conference on control, automation and systems 2012, pp 1716–1719

  25. Faragasso A, Bimbo J, Noh Y, Jiang A, Sareh S, Liu H, Nanayakkara T, Wurdemann H, Althoefer K (2014) Novel uniaxial force sensor based on visual information for minimally invasive surgery. In: IEEE international conference on robotics and automation 2014, pp 1405–1410. https://doi.org/10.1109/ICRA.2014.6907036

  26. Haouchine N, Kuang W, Cotin S, Yip M (2018) Vision-based force feedback estimation for robot-assisted surgery using instrument-constrained biomechanical three-dimensional maps. IEEE Robot Autom Lett 3:2160–2165. https://doi.org/10.1109/LRA.2018.2810948

    Article  Google Scholar 

  27. Su Y, Huang I, Huang K, Hannaford B (2018) Comparison of 3D surgical tool segmentation procedures with robot kinematics prior. In: IEEE/RSJ international conference on intelligent robots and systems 2018, pp 4411–4418. https://doi.org/10.1109/IROS.2018.8594428

  28. Thananjeyan B, Garg A, Krishnan S, Chen C, Miller L, Goldberg K (2017) Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning. In: IEEE international conference on robotics and automation 2017, pp 2371–2378. https://doi.org/10.1109/ICRA.2017.7989275

  29. Kelouwani S, Agbossou K (2004) Nonlinear model identification of wind turbine with a neural network. IEEE Trans Energy Convers 19:607–612. https://doi.org/10.1109/TEC.2004.827715

    Article  Google Scholar 

  30. Kwon D, Woo K, Song S, Kim W, Cho H (1998) Microsurgical telerobot system. In: IEEE/RSJ international conference on intelligent robots and systems 1998, pp 945–950. https://doi.org/10.1109/IROS.1998.727421

  31. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  32. Yin X, Zhao X (2019) Big data driven multi-objective predictions for offshore wind farm based on machine learning algorithms. Energy 186:115704. https://doi.org/10.1016/j.energy.2019.07.034

    Article  Google Scholar 

  33. Schuster M, Paliwal K (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45:2673–2681. https://doi.org/10.1109/78.650093

    Article  Google Scholar 

  34. Bilgic B, Chatnuntawech I, Fan A, Setsompop K, Cauley S, Wald L, Adalsteinsson E (2014) Fast image reconstruction with L2-regularization. J Magn Reson Imaging 40:181–191. https://doi.org/10.1002/jmri.24365

    Article  Google Scholar 

  35. Zeiler M (2012) ADADELTA: an adaptive learning rate method. arXiv:1212.5701

  36. Yin X, Li P (2018) Direct adaptive robust tracking control for 6 DOF industrial robot with enhanced accuracy. ISA Trans 72:178–184. https://doi.org/10.1016/j.isatra.2017.10.007

    Article  Google Scholar 

  37. Yin X, Li P (2018) Enhancing trajectory tracking accuracy for industrial robot with robust adaptive control. Robot Comput Integr Manuf 51:97–102. https://doi.org/10.1016/j.rcim.2017.11.007

    Article  Google Scholar 

Download references

Acknowledgements

The paper is supported by the Natural Science Foundation of Heilongjiang Province (Grand No. F2015034). We also greatly appreciate the efforts of the reviewers and our colleagues.

Author information

Authors and Affiliations

  1. College of Mechanical and Electrical Engineering, Harbin Engineering University, Harbin, 150001, People’s Republic of China

    Lingtao Yu & Xiaoyan Yu

  2. Tianjin Institute of Aerospace Mechanical and Electrical Equipment, Tianjin, 300301, People’s Republic of China

    Yongqin Zhang

Authors
  1. Lingtao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoyan Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yongqin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoyan Yu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix: Dynamic model detailed results

Appendix: Dynamic model detailed results

$$ \begin{aligned} & \boldsymbol{\tau } = \boldsymbol{H}\left( \boldsymbol{q} \right)\boldsymbol{\ddot{q}} + \boldsymbol{C}\left( {\boldsymbol{q},\dot{\boldsymbol{q}}} \right)\dot{\boldsymbol{q}} + \boldsymbol{G}\left( \boldsymbol{q} \right) + \boldsymbol{M}_{f} + \boldsymbol{J}^{\text{T}} \boldsymbol{F} \\ & \left[ {\begin{array}{*{20}c} {\tau_{1m} } \\ {\tau_{2m} } \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {{}^{m}h_{11} } & {{}^{m}h_{12} } \\ {{}^{m}h_{21} } & {{}^{m}h_{22} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {\ddot{q}_{1} } \\ {\ddot{q}_{2m} } \\ \end{array} } \right] + \left[ {\begin{array}{*{20}c} {{}^{m}c_{11} } & {{}^{m}c_{12} } \\ {{}^{m}c_{21} } & {{}^{m}c_{22} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {\dot{q}_{1} } \\ {\dot{q}_{2m} } \\ \end{array} } \right] + \left[ {\begin{array}{*{20}c} {g_{1m} } \\ {g_{2m} } \\ \end{array} } \right] + \left[ {\begin{array}{*{20}c} {M_{f1} } \\ {M_{f2m} } \\ \end{array} } \right] + {}^{m}\boldsymbol{J}^{\text{T}} \left[ {\begin{array}{*{20}c} {F_{mx} } \\ {F_{my} } \\ {F_{mz} } \\ \end{array} } \right] \\ \end{aligned} $$

where m = a, b

$$ \begin{aligned} \boldsymbol{J}_{i} & = \left[ {\begin{array}{*{20}c} {\frac{{ - {}^{i - 1}I_{x} + {}^{i - 1}I_{y} + {}^{i - 1}I_{z} }}{2}} & {{}^{i - 1}I_{xy} } & {{}^{i - 1}I_{xz} } & {m_{i} {}^{i - 1}x_{ci} } \\ {{}^{i - 1}I_{xy} } & {\frac{{{}^{i - 1}I_{x} - {}^{i - 1}I_{y} + {}^{i - 1}I_{z} }}{2}} & {{}^{i - 1}I_{yz} } & {m_{i} {}^{i - 1}y_{ci} } \\ {{}^{i - 1}I_{xz} } & {{}^{i - 1}I_{yz} } & {\frac{{{}^{i - 1}I_{x} + {}^{i - 1}I_{y} - {}^{i - 1}I_{z} }}{2}} & {m_{i} {}^{i - 1}z_{ci} } \\ {m_{i} {}^{i - 1}x_{ci} } & {m_{i} {}^{i - 1}y_{ci} } & {m_{i} {}^{i - 1}z_{ci} } & {m_{i} } \\ \end{array} } \right] \\ {}^{m}h_{11} & = {}^{0}I_{y} + {}^{1m}I_{x} + 2a_{1} m_{1} {}^{0}x_{c1} + a_{1}^{2} m_{1} + c_{2m}^{2} \left( {{}^{1m}I_{y} - {}^{1m}I_{x} } \right) - 2s_{2m} c_{2m} {}^{1m}I_{xy} + 2c_{2m} \left( {a_{2} c_{2m} + a_{1} } \right)m_{2m} {}^{1m}x_{c2m} \\ & \quad - 2s_{2m} \left( {a_{2} c_{2m} + a_{1} } \right)m_{2m} {}^{1m}y_{c2m} + \left( {a_{2}^{2} c_{2m}^{2} + 2a_{1} a_{2} c_{2m} + a_{1}^{2} } \right)m_{2m} \\ {}^{m}h_{12} & = {}^{m}h_{21} = - c_{2m} {}^{1m}I_{yz} - s_{2m} {}^{1m}I_{xz} - a_{2} s_{2m} m_{2m} {}^{1m}z_{c2m} \\ {}^{m}h_{22} & = a_{2}^{2} m_{2m} + 2a_{2} m_{2m} {}^{1m}x_{c2m} + {}^{1m}I_{z} \\ {}^{m}c_{11} & = - \left[ {\left( {c_{2m}^{2} - s_{2m}^{2} } \right){}^{1m}I_{xy} + \left( {2s_{2m} c_{2m} a_{2} + s_{2m} a_{1} } \right)m_{2m} {}^{1m}x_{c2m} + \left( {\left( {c_{2m}^{2} - s_{2m}^{2} } \right)a_{2} + c_{2m} a_{1} } \right)m_{2m} {}^{1m}y_{c2m} } \right. \\ & \quad \left. { + \left( {s_{2m} c_{2m} a_{2}^{2} + s_{2m} a_{1} a_{2} } \right)m_{2m} + s_{2m} c_{2m} \left( {{}^{1m}I_{y} - {}^{1m}I_{x} } \right)} \right]\dot{q}_{2m} \\ {}^{m}c_{12} & = - \left[ {\left( {c_{2m}^{2} - s_{2m}^{2} } \right){}^{1m}I_{xy} + \left( {2s_{2m} c_{2m} a_{2} + s_{2m} a_{1} } \right)m_{2m} {}^{1m}x_{c2m} + \left( {\left( {c_{2m}^{2} - s_{2m}^{2} } \right)a_{2} + c_{2m} a_{1} } \right)m_{2m} {}^{1m}y_{c2m} } \right. \\ & \quad \left. { + \left( {s_{2m} c_{2m} a_{2}^{2} + s_{2m} a_{1} a_{2} } \right)m_{2m} + s_{2m} c_{2m} \left( {{}^{1m}I_{y} - {}^{1m}I_{x} } \right)} \right]\dot{q}_{1} + \left[ {s_{2m} {}^{1m}I_{yz} - c_{2m} {}^{1m}I_{xz} - a_{2} c_{2m} m_{2m} {}^{1m}z_{c2m} } \right]\dot{q}_{2m} \\ {}^{m}c_{21} & = \left[ {\left( {c_{2m}^{2} - s_{2m}^{2} } \right){}^{1m}I_{xy} + \left( {2s_{2m} c_{2m} a_{2} + s_{2m} a_{1} } \right)m_{2m} {}^{1m}x_{c2m} + \left( {\left( {c_{2m}^{2} - s_{2m}^{2} } \right)a_{2} + c_{2m} a_{1} } \right)m_{2m} {}^{1m}y_{c2m} } \right. \\ & \quad \left. { + \left( {s_{2m} c_{2m} a_{2}^{2} + s_{2m} a_{1} a_{2} } \right)m_{2m} + s_{2m} c_{2m} \left( {{}^{1m}I_{y} - {}^{1m}I_{x} } \right)} \right]\dot{q}_{1} \\ {}^{m}c_{22} & = 0 \\ g_{1m} & \quad = - \left( {c_{1} a_{1} + c_{1} {}^{0}x_{c1} + s_{1} {}^{0}z_{c1} } \right)m_{1} g - \left( {c_{1} a_{1} + c_{1} c_{2m} a_{2} + c_{1} c_{2m} {}^{1m}x_{c2m} - c_{1} s_{2m} {}^{1m}y_{c2m} + s_{1} {}^{1m}z_{c2m} } \right)m_{2m} g \\ g_{2m} & \quad = \left( {s_{1} s_{2m} a_{2} + s_{1} s_{2m} {}^{1m}x_{c2m} + s_{1} c_{2m} {}^{1m}y_{c2m} } \right)m_{2m} g \\ {}^{m}\boldsymbol{J}^{\text{T}} & \quad = \left[ {\begin{array}{*{20}l} { - s_{1} a_{1} - s_{1} c_{2m} a_{2} } \hfill & {c_{1} a_{1} + c_{1} c_{2m} a_{2} } \hfill & 0 \hfill \\ { - c_{1} s_{2m} a_{2} } \hfill & { - s_{1} s_{2m} a_{2} } \hfill & {c_{2m} a_{2} } \hfill \\ \end{array} } \right] \\ \end{aligned} $$

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, L., Yu, X. & Zhang, Y. Microinstrument contact force sensing based on cable tension using BLSTM–MLP network. Intel Serv Robotics 13, 123–135 (2020). https://doi.org/10.1007/s11370-019-00306-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-019-00306-6

Keywords

Search

Navigation