Teaching Variability in a Core Systems Course: (Abstract Only)

Computer systems form the backbone of computing from very small, mobile devices to the huge datacenters that power the digital economy. These systems often exhibit large degrees of variability in their performance that is little understood, but such variability threatens to severely diminish the effectiveness of critical systems upon which society relies. Funded by a large NSF grant, the VarSys project at Virginia Tech researches the sources of variability in computer systems and develops methods to overcome it. We believe it is crucial to raise awareness of the phenomena surrounding variability in computer systems at the undergraduate level. Towards this end, we are connecting the research techniques developed as part of this NSF award to ongoing classroom projects in a core systems course. Our key insight is to expose students to the phenomenon as it occurs in the systems software modules (e.g. a memory allocator, a fork-join thread pool) they are themselves developing in the course. We have implemented a web-based system that allows students to submit their own systems-level code to a specialized cluster which then benchmarks it while systematically varying a number of ordinal and categorical variables. These variables reflect environmental factors that can influence the performance of complex systems. Students are then presented with a visual statistical analysis of the results and asked to interpret those results. We have successfully deployed this system in 2 semesters to over 250 students and collected student data about their experience with this system and are documenting our progress towards these important learning objectives.