An approach to automatic process deviation detection in a time-critical clinical process

Prior research has shown that minor errors and deviations from recommended guidelines in complex medical processes can accumulate to increase the likelihood that a major error will go uncorrected and lead to an adverse outcome. Real-time automatic and accurate detection of process deviations may help medical teams better prevent or mitigate the effect of errors and improve patient outcomes. Our goal was to develop an approach for automatic detection of errors and process deviations in trauma resuscitation.

Using video review, we coded activity traces of 95 pediatric trauma resuscitations collected in a Level 1 trauma center over two years (2014–2016). Twenty-four randomly selected activity traces were compared with a knowledge-driven model of trauma resuscitation workflow using a phase-based conformance checking algorithm for detecting true and false deviations (alarms). An analysis of false alarms identified three types of causes: (1) model gaps or discrepancies between the model (“work as imagined”) and actual practice (“work as done”), (2) errors in activity traces coding, and (3) algorithm limitations. We repaired the system to remove model gaps, reduce coding errors, and address algorithm limitations. The repaired system was first evaluated with another 20 traces and then applied to the entire dataset of 95 traces.

During the training, we detected 573 process deviations in 24 activity traces that include 1099 activities. Among these deviations, only 27% represented true deviations and the remaining 73% were false alarms. This initial deviation detection accuracy was only 66.6%, with a F ₁-score of 0.42. Detection accuracy of the repaired system increased to 95.2% (0.85 F ₁-score) during system validation and to 98.5% (0.96 F ₁-score) during testing. After deploying the repaired deviation detection system to all 95 activity traces, we detected 1060 process deviations in 5659 activities (11.2 deviations per resuscitation). Among the 5659 activities in these traces, 4893 fit the repaired knowledge-driven workflow model, 294 were errors of omission, 538 were errors of commission, and 228 were scheduling errors.

Our approach to automatic deviation detection provides a method for identifying repeated, omitted and out-of-sequence activities that can be included in the design of decision support systems for complex medical processes. Our findings show the importance of assessing detected deviations for repairing a knowledge-driven model that best represents “work as done.”