DaedalusData: Exploration, Knowledge Externalization and Labeling of Particles in Medical Manufacturing — A Design Study
Authors: 
Alexander Wyss, Gabriela Morgenshtern, Amanda Hirsch-Hüsler, Jürgen BernardAuthors Info & Claims
Pages 54 - 64
Abstract
In medical diagnostics of both early disease detection and routine patient care, particle-based contamination of in-vitro diagnostics consumables poses a significant threat to patients. Objective data-driven decision-making on the severity of contamination is key for reducing patient risk, while saving time and cost in quality assessment. Our collaborators introduced us to their quality control process, including particle data acquisition through image recognition, feature extraction, and attributes reflecting the production context of particles. Shortcomings in the current process are limitations in exploring thousands of images, data-driven decision making, and ineffective knowledge externalization. Following the design study methodology, our contributions are a characterization of the problem space and requirements, the development and validation of DaedalusData, a comprehensive discussion of our study's learnings, and a generalizable framework for knowledge externalization. DaedalusData is a visual analytics system that enables domain experts to explore particle contamination patterns, label particles in label alphabets, and externalize knowledge through semi-supervised label-informed data projections. The results of our case study and user study show high usability of DaedalusData and its efficient support of experts in generating comprehensive overviews of thousands of particles, labeling of large quantities of particles, and externalizing knowledge to augment the dataset further. Reflecting on our approach, we discuss insights on dataset augmentation via human knowledge externalization, and on the scalability and trade-offs that come with the adoption of this approach in practice.
References
[1]
E. Angel and D. Shreiner. Interactive computer graphics with WebGL. Addison-Wesley Professional, 2014.
[2]
J. Attenberg and F. Provost. Inactive learning? Difficulties employing active learning in practice. SIGKDD Explor. Newsl., 12(2):36–41, 6 pages, 2011. 3.
[3]
C.-M. Barth, J. Schmid, I. Al-Hazwani, M. Sachdeva, L. Cibulski, and J. Bernard. How applicable are attribute-based approaches for human-centered ranking creation? Computers & Graphics (CAG), 114:45–58, 2023. 2.
[4]
B. C. Benato, A. C. Telea, and A. X. Falcão. Semi-supervised learning with interactive label propagation guided by feature space projections. In 2018 31st SIBGRAPI Conf. on Graphics, Patterns and Images (SIBGRAPI), pp. 392–399, 2018. 3,9.
[5]
J. Bernard, E. Dobermann, A. Vögele, B. Krüger, J. Kohlhammer, and D. Fellner. Visual-interactive semi-supervised labeling of human motion capture data. In Visualization and Data Analysis (VDA 2017), 2017. 3.
[6]
J. Bernard, M. Hutter, M. Lehmann, M. Müller, M. Zeppelzauer, and M. Sedlmair. Learning from the Best-Visual Analysis of a Quasi-Optimal Data Labeling Strategy. In EuroVis 2018-Short Papers. The Eurographics Association, 2018. 3.
[7]
J. Bernard, M. Hutter, M. Zeppelzauer, D. W. Fellner, and M. Sedlmair. Comparing visual-interactive labeling with active learning: An experimental study. IEEE Trans. Visual Comput. Graphics (TVCG), 24(1):298–308, 2018. 3,9.
[8]
J. Bernard, C. Ritter, D. Sessler, M. Zeppelzauer, J. Kohlhammer, and D. Fellner. Visual-interactive similarity search for complex objects by example of soccer player analysis. In Jt. Conf. Comput. Vis. Imaging Comput. Graph. Theory Appl. (VISGRAP), vol. 3, pp. 75–87, 2017. 3.
[9]
J. Bernard, D. Sessler, A. Bannach, T. May, and J. Kohlhammer. A visual active learning system for the assessment of patient well-being in prostate cancer research. In VIS Workshop on Visual Analytics in Healthcare, article no. 1,8 pages, pp. 1–8. ACM, 2015. 2.
[10]
J. Bernard, M. Zeppelzauer, M. Lehmann, M. Müller, and M. Sedlmair. Towards user-centered active learning algorithms. Comput. Graphics Forum (CGF), 37(3):121–132, 2018. 3.
[11]
J. Bernard, M. Zeppelzauer, M. Sedlmair, and W. Aigner. VIAL: a unified process for visual interactive labeling. Vis. Comput., 34(9):1189–1207, 2018. 3.
[12]
A. Blandford. Semi-structured qualitative studies. In The Encyclopedia of Human-Computer Interaction, 2nd Ed. The Interaction Design Foundation, 2013. 3.
[13]
M. Blumenschein, M. Behrisch, S. Schmid, S. Butscher, D. R. Wahl, K. Villinger, B. Renner, H. Reiterer, and D. A. Keim. Smartexplore: Simplifying high-dimensional data analysis through a table-based visual analytics approach. In R. Chang, H. Qu, and T. Schreck eds, IEEE Conf. Visual Analytics Science & Technology (VAST), pp. 36–47. IEEE, Berlin, 2018. 3.
[14]
F. Bouali, A. E. Guettala, and G. Venturini. Vizassist: an interactive user assistant for visual data mining. Vis. Comput., 32(11):1447–1463, 2016. 2.
[15]
N. Boukhelifa, W. Cancino, A. Bezerianos, and E. Lutton. Evolutionary visual exploration: evaluation with expert users. In Comput. Graphics Forum (CGF), vol. 32, pp. 31–40. Wiley Online Library, 2013. 2.
[16]
S. Budd, E. C. Robinson, and B. Kainz. A survey on active learning and human-in-the-loop deep learning for medical image analysis. Medical Image Analysis, 71:102062, 2021. 3.
[17]
J. C. Chang, S. Amershi, and E. Kamar. Revolt: Collaborative crowd-sourcing for labeling machine learning datasets. In ACM Conf. Human Factors in Computing Systems (CHI), pp. 2334–2346. ACM, 2017. 3.
[18]
J. C. Chang, A. Kittur, and N. Hahn. Alloy: Clustering with crowds and computation. In ACM Conf. Human Factors in Computing Systems (CHI), pp. 3180–3191. ACM, 2016. 3.
[19]
S. Chatani, Y. Ma, H. Zhang, Y. Chen, and W. Du. Interactive labeling system for lung nodules with ct images. In Int. Symp. Comput. Consum. Control, pp. 529–532, 2020. 3.
[20]
M. Chegini, J. Bernard, P. Berger, A. Sourin, K. Andrews, and T. Schreck. Interactive Labelling of a Multivariate Dataset for Supervised Machine Learning Using Linked Visualisations, Clustering, and Active Learning. Visual Informatics, 3(1):9–17, 2019. 3, 9.
[21]
S. F. Chipman, J. M. Schraagen, and V. L. Shalin. Introduction to cognitive task analysis. In Cognitive task analysis, pp. 17–38. Psychology Press, 2000. 2.
[22]
D. H. Chung, M. L. Parry, I. W. Griffiths, R. S. Laramee, R. Bown, P. A. Legg, and M. Chen. Knowledge-assisted ranking: A visual analytic application for sports event data. IEEE Comput. Graphics Appl., 36(3):72–82, 2015. 2.
[23]
L. Cibulski and T. May. Revisiting PAVED: Studying Tool Adoption After Four Years. In EuroVis-Short Papers. The Eurographics Association, 2024. 9.
[24]
A. Coscia, H. M. Sapers, N. Deutsch, M. Khurana, J. S. Magyar, S. A. Parra, D. R. Utter, R. L. Wipfler, D. W. Caress, E. J. Martin, J. B. Paduan, M. Hendrie, S. Lombeyda, H. Mushkin, A. Endert, S. Davidoff, and V. J. Orphan. Deepsee: Multidimensional visualizations of seabed ecosystems. In ACM Conf. Human Factors in Computing Systems (CHI). ACM, 2024. 3.
[25]
A. Crisan, B. Fiore-Gartland, and M. Tory. Passing the data baton: A retrospective analysis on data science work and workers. IEEE Trans. Visual Comput. Graphics (TVCG), 27(2):1860–1870, 2021. 3.
[26]
B. Danchilla and B. Danchilla. Three. js framework. Beginning WebGL for HTML5, pp. 173–203, 2012. 6.
[27]
C. developers. Chart.js. github.com/chartjs/Chart.js, 2015. 6.
[28]
D. Duhaime. Pixplot. github.com/YaleDHLab/pix-plot, 2016. 2.
[29]
J. Eirich, J. Bonart, D. Jäckle, M. Sedlmair, U. Schmid, K. Fischbach, T. Schreck, and J. Bernard. IRVINE: A design study on analyzing correlation patterns of electrical engines. IEEE Trans. Visual Comput. Graphics (TVCG), 28(1):11–21, 2022. 3, 9.
[30]
A. Endert, L. Bradel, and C. North. Beyond control panels: Direct manipulation for visual analytics. IEEE Comput. Graphics Appl. (CGA), 33(4):6–13, 2013. 2.
[31]
C. Felix, A. Dasgupta, and E. Bertini. The exploratory labeling assistant: Mixed-initiative label curation with large document collections. In ACM Symposium on User Interface Software and Technology, pp. 153–164, 2018. 3.
[32]
O. Friard and M. Gamba. Boris: a free, versatile open-source event-logging software for video/audio coding and live observations. Methods in Ecology and Evolution, 7(11):1325–1330, 2016. 7.
[33]
B. Fruchard, S. Malacria, G. Casiez, and S. Huot. User preference and performance using tagging and browsing for image labeling. In ACM Conf. Human Factors in Computing Systems (CHI), pp. 358:1–358:13. ACM, 2023. 3.
[34]
T. Fujiwara, X. Wei, J. Zhao, and K.-L. Ma. Interactive dimensionality reduction for comparative analysis. IEEE Trans. Visual Comput. Graphics, 28(1):758–768, 2021. 2,3.
[35]
L. A. Garrison, J. Müller, S. Schreiber, S. Oeltze-Jafra, H. Hauser, and S. Bruckner. Dimlift: Interactive hierarchical data exploration through dimensional bundling. IEEE Trans. Visual Comput. Graphics (TVCG), 27(6):2908–2922, 2021. 3.
[36]
N. Grossmann, J. Bernard, M. Sedlmair, and M. Waldner. Does the layout really matter? A study on visual model accuracy estimation. In IEEE Vis. Conf., pp. 61–65. IEEE, 2021. 9.
[37]
H. Han, R. Faust, B. F. K. Norambuena, J. Lin, S. Li, and C. North. Explainable interactive projections of images. Mach. Vis. Appl., 34(6):100, 2023. 3.
[38]
S. G. Hart. Nasa-task load index (nasa-tlx); 20 years later. In Proc. Hum. Factors Ergon. Soc. Annu. Meet., vol. 50, pp. 904–908. Sage publications Sage CA: Los Angeles, CA, 2006. 8.
[39]
S. C. Hoi, R. Jin, and M. R. Lyu. Large-scale text categorization by batch mode active learning. In World Wide Web, pp. 633–642. ACM, 2006. 3.
[40]
B. Höferlin, R. Netzel, M. Höferlin, D. Weiskopf, and G. Heidemann. Inter-active learning of ad-hoc classifiers for video visual analytics. In IEEE Conf. Visual Analytics Science & Technology (VAST), pp. 23–32, 2012. 3.
[41]
P. Jeongeon, K. Eun-Young, S. P. Yeon, Y. Jinyeong, and K. Juho. Dynamiclabels: Supporting informed construction of machine learning label sets with crowd feedback. In ACM, 2024. 3.
[42]
M. S. Keller, I. Gold, C. McCallum, T. Manz, P. V. Kharchenko, and N. Gehlenborg. Vitessce: a framework for integrative visualization of multi-modal and spatially-resolved single-cell data. OSF Preprints, Oct. 2021. 2, 3.
[43]
G. A. Klein, R. Calderwood, and D. Macgregor. Critical decision method for eliciting knowledge. IEEE Trans. Syst. Man Cybern. Part B Cybern., 19(3):462–472, 1989. 2.
[44]
A. Kuffel, A. Gray, and N. N. Daeid. Impact of metal ions on per inhibition and RT-PCR efficiency. Int. j. leg. med., 135(1):63–72, 2021. 2.
[45]
P. Kumar and A. Gupta. Active learning query strategies for classification, regression, and clustering: A survey. J. Comput. Sci. Technol., 35(4):913–945, 2020. 3.
[46]
A.-P. Lohfink, S. D. D. Anton, H. Leitte, and C. Garth. Knowledge rocks: Adding knowledge assistance to visualization systems. IEEE Trans. Visual Comput. Graphics, 28(1):1117–1127, 2021. 2.
[47]
E. Lughofer. Hybrid active learning for reducing the annotation effort of operators in classification systems. Pattern Recognition, 45(2):884–896, 2012. 3.
[48]
L. McInnes, J. Healy, N. Saul, and L. Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018. 6.
[49]
L. G. Militello and R. J. Hutton. Applied cognitive task analysis (acta): a practitioner's toolkit for understanding cognitive task demands. Ergonomics, 41(11):1618–1641, 1998. 2.
[50]
G. Mistelbauer, A. Köchl, R. Schernthaner, I. Baclija, R. Schernthaner, S. Bruckner, M. Sramek, and M. E. Gröller, Smart super views-a knowledge-assisted interface for medical visualization. In 2012 IEEE Conf. on Visual Analytics Science and Technology (VAST), pp. 163–172. IEEE, 2012. 2.
[51]
G. Morgenshtern, A. Verma, S. Tonekaboni, R. Greer, J. Bernard, M. Mazwi, A. Goldenberg, and F. Chevalier. RiskFix: Supporting Expert Validation of Predictive Timeseries Models in High-Intensity Settings. In EuroVis-Short Papers. The Eurographics Association, 2023. 3.
[52]
D. Moritz, C. Wang, G. L. Nelson, H. Lin, A. M. Smith, B. Howe, and J. Heer. Formalizing visualization design knowledge as constraints: Actionable and extensible models in draco. IEEE Trans. Visual Comput. Graphics, 25(1):438–448, 2019. 2.
[53]
T. Mühlbacher and H. Piringer. A partition-based framework for building and validating regression models. IEEE Trans. Visual Comput. Graphics, 19(12):1962–1971, 2013. 5.
[54]
T. Munzner. Process and pitfalls in writing information visualization research papers. In Information Visualization: Human-Centered Issues and Perspectives, vol. 4950, pp. 134–153. Springer, 2008. 3.
[55]
K. Nie, P. Baltzer, B. Preim, and G. Mistelbauer. Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging. Comput. Graphics Forum (CGF), 2020. 2.
[56]
T. Ohm, M. C. Solà, A. Karjus, and M. Schich. Collection space navigator: An interactive visualization interface for multidimensional datasets. CoRR, abs/2305.06809, 2023. 2.
[57]
N. Pielawski, A. Andersson, C. Avenel, A. Behanova, E. Chelebian, A. Klemm, F. Nysjö, L. Solorzano, and C. Wählby. Tissuumaps 3: Improvements in interactive visualization, exploration, and quality assessment of large-scale spatial omics data. Heliyon, 9(5):e15306, 2023. 2.
[58]
A. Qian, C. Li, X. Dong, S. Chen, Y. Zhang, and G. Li. Slamvis: An interactive visualization approach for smart labeling on multidimensional data. In Parallel & Distributed Processing with Applications, pp. 19–26. IEEE, 2021. 9.
[59]
S. Raadnui. Wear particle analysis-utilization of quantitative computer image analysis: A review. Tribol. Int., 38(10):871–878, 2005. 4.
[60]
P. Ren, Y. Xiao, X. Chang, P. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang. A survey of deep active learning. ACM Computing Surveys (CSUR), 54(9):1–40, 2022. 3.
[61]
C. Ritter, C. Altenhofen, M. Zeppelzauer, A. Kuijper, T. Schreck, and J. Bernard. Personalized Visual-Interactive Music Classification. In EuroVis Workshop on Visual Analytics (EuroVA). The Eurographics Association, 2018. 3.
[62]
U.-P. Rohr, C. Binder, T. Dieterle, F. Giusti, C. G. M. Messina, E. Toerien, H. Moch, and H. H. Schäfer. The value of in vitro diagnostic testing in medical practice: A status report. PLOS ONE, 11:1–16, 03 2016. 2.
[63]
M. Sachdeva, J. Burmeister, J. Kohlhammer, and J. Bernard. LFPeers: Temporal similarity search and result exploration. Computers & Graphics (CAG), 2023. 2.
[64]
T. Sainburg, L. McInnes, and T. Q. Gentner. Parametric UMAP Embeddings for Representation and Semisupervised Learning. Neural Computation, 33(11):2881–2907, 2021. 9.
[65]
J. Schmidt, H. Piringer, T. Mühlbacher, and J. Bernard. Human-Based and Automatic Feature Ideation for Time Series Data: A Comparative Study. In EuroVis Workshop on Visual Analytics (EuroVA). The EurographicsAssociation, 2023. 9.
[66]
S.-M. Schröder, R. Kiko, and R. Koch. Morphocluster: Efficient annotation of plankton images by clustering. Sensors, 20(11), 2020. 3.
[67]
N. Schwarz. Self-reports: How the questions shape the answers. American Psychologist, 54(2):93–105, 1999. 2.
[68]
M. Sedlmair. Design study contributions come in different guises: Seven guiding scenarios. In Proc. Sixth Workshop Beyond Time Errors Nov. Eval. Methods Vis. (BELIV), pp. 152–161. ACM, 2016. 3.
[69]
M. Sedlmair, M. D. Meyer, and T. Munzner. Design study methodology: Reflections from the trenches and the stacks. IEEE Trans. Visual Comput. Graphics (TVCG), 18(12):2431–2440, 2012. 3.
[70]
C. Seifert and M. Granitzer. User-based active learning. In IEEE Int. Conf. on Data Mining Workshops, pp. 418–425, 2010. 3.
[71]
R. Sevastjanova, W. Jentner, F. Sperrle, R. Kehlbeck, J. Bernard, and M. El-Assady, Questioncomb: A gamification approach for the visual explanation of linguistic phenomena through interactive labeling. ACM Trans. Interact. Intell. Syst., 11(3–4):19:1–19:38, 2021. 9.
[72]
A. Somarakis, V. Van Unen, F. Koning, B. P. F. Lelieveldt, and T. Höllt. Imacyte: Visual exploration of cellular micro-environments for imaging mass cytometry data. IEEE Trans. Visual Comput. Graphics (TVCG), 27(1):98–110, 2021. 3.
[73]
H. Stitz, S. Gratzl, H. Piringer, T. Zichner, and M. Streit. Knowledgepearls: Provenance-based visualization retrieval. IEEE Trans. Visual Comput. Graphics (VAST’18), 25(1):120–130, 2018. 2.
[74]
J. J. Van Wijk. The value of visualization. In IEEE Comput., pp. 79–86, 2005. 2.
[75]
M. Wagner, D. Slijepcevic, B. Horsak, A. Rind, M. Zeppelzauer, and W. Aigner. Kavagait: Knowledge-assisted visual analytics for clinical gait analysis. IEEE Trans. Visual Comput. Graphics (TVCG), 25(3):1528–1542, 2019. 2.
[76]
H. Wang, Y. Ouyang, Y. Wu, C. Jiang, L. Jin, Y. Cao, and Q. Li. Kmtlabeler: An interactive knowledge-assisted labeling tool for medical text classification. IEEE Trans. Visual Comput. Graphics (TVCG), pp. 1–18, 2024. 2.
[77]
X. Wang, D. H. Jeong, W. Dou, S. Lee, W. Ribarsky, and R. Chang. Defining and applying knowledge conversion processes to a visual analytics system. Computers & Graphics (CAG), 33(5):616–623, 2009. 2.
[78]
Z. J. Wang, F. Hohman, and D. H. Chau. Wizmap: Scalable interactive visualization for exploring large machine learning embeddings. CoRR, abs/2306.09328, 2023. 2.
[79]
J. Wenskovitch and C. North. An examination of grouping and spatial organization tasks for high-dimensional data exploration. IEEE Trans. Visual Comput. Graphics (TVCG), 27(2):1742–1752, 2021. 3.
[80]
X. Xu, W. Lu, J. Lei, P. Qiu, H.-B. Shen, and Y. Yang. Sliceprop: A slice-wise bidirectional propagation model for interactive 3d medical image segmentation. In Medical Artificial Intelligence (MedAI), pp. 414–424, 2023. 3.
[81]
Yang Samuel and R. E. Rothman. Per-based diagnostics for infectious diseases: uses, limitations, and future applications in acute-care settings. The Lancet infectious diseases, 4:337–348, 06 2004. 1, 2.
[82]
X. Yuan, D. Ren, Z. Wang, and C. Guo. Dimension projection matrix/tree: Interactive subspace visual exploration and analysis of high dimensional data. IEEE Trans. Visual Comput. Graphics (TVCG), 19(12):2625–2633, 2013. 3.
[83]
Z. Zhang, E. Strubell, and E. H. Hovy. A survey of active learning for natural language processing. CoRR, abs/2210.10109, 2022. 3.
[84]
L. Zhu, P. Karasev, I. Kolesov, R. Sandhu, and A. R. Tannenbaum. Guiding image segmentation on the fly: Interactive segmentation from a feedback control perspective. IEEE Trans. Autom. Control., 63(10):3276–3289, 2018. 3.
[85]
Y. Zhu, B. Carragher, F. Mouche, and C. S. Potter. Automatic particle detection through efficient hough transforms. IEEE Trans. Medical Imaging, 22(9):1053–1062, 2003. 4.
Index Terms
- DaedalusData: Exploration, Knowledge Externalization and Labeling of Particles in Medical Manufacturing — A Design Study
Index terms have been assigned to the content through auto-classification.
Recommendations
Interactive knowledge externalization and combination for SECI model
K-CAP '07: Proceedings of the 4th international conference on Knowledge captureOne of the subjects in knowledge management is knowledge externalization and combination in which people express their tacit knowledge and formulate it. Communication among people is important for this process. This paper proposes a model that helps two ...
Externalization of knowledge through Group Storytelling: a case study in online tutoring
SBSI '16: Proceedings of the XII Brazilian Symposium on Information Systems on Brazilian Symposium on Information Systems: Information Systems in the Cloud Computing Era - Volume 1One of the great challenges of knowledge management is the externalization of knowledge (the conversion of tacit in explicit knowledge) effectively. Tacit knowledge is associated with personal experiences and represents the subjective knowledge that is ...
Augmenting Knowledge-Based Medical Systems with Tacit Healthcare Expertise: Towards an Intelligent Tacit Knowledge Acquisition Info-Structure
CBMS '01: Proceedings of the Fourteenth IEEE Symposium on Computer-Based Medical SystemsAbstract: The knowledge-rich nature of the healthcare domain has made it an ideal environment for the application of knowledge-based techniques. However, the abstract nature of tacit healthcare knowledge has resulted in the under-utilization of such a ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
1077-2626 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 01 January 2025
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 28 Feb 2025