Putting ECD into Practice: The Interplay of Theory and Data in Evidence Models within a Digital Learning Environment

In this paper we describe the development and refinement of evidence rules and measurement models within the evidence model of the evidence-centered design (ECD) framework in the context of the Packet Tracer digital learning environment of the Cisco Networking Academy. Using Packet Tracer learners design, configure, and troubleshoot computer networks within an interactive interface. This leads to product data, which result from the students' final submitted network configurations, and process data, which are log file entries detailing how they got to the final configurations. We discuss how an iterative cycle of empirical analyses and discussions with subject-matter experts is essential for identifying and accumulating evidence about skill profiles of learners and their development. We present results from descriptive, exploratory, and confirmatory diagnostic modeling analyses for both data types, which required bringing to bear a diversity of tools from multivariate statistics, modern psychometrics, and educational data mining. We close the paper with a discussion of the implications of this work for evidence-based argumentation guided by ECD principles within digital learning environments more generally.