research-article

Public Access

(Re)Validating Cognitive Introductory Computing Instruments

Authors:

Stephen Cooper,

Jonathan Gratch,

Mohsen DorodchiAuthors Info & Claims

SIGCSE '19: Proceedings of the 50th ACM Technical Symposium on Computer Science Education

Pages 552 - 557

https://doi.org/10.1145/3287324.3287372

Published: 22 February 2019 Publication History

Abstract

\beginabstract Cognitive tests have been long used as a measure of student knowledge, ability, and as a predictor for success in engineering and computer science. However, these tests are not without their own problems relating to priming, difficulty (resulting in test fatigue) and time on exam. This paper discusses efforts to modify Parker et al.'s Second CS1 aptitude test (SCS1) \citeParker16 to reduce the time spent on the exam, provide greater customization to match concepts taught across three universities, and reduce redundancy of test questions all while maintaining the instrument's reliability. This instrument was modified for use on an ongoing grant investigating whether spatial abilities impact the success of students in introductory CS courses. The instrument developed in this paper is a revised shortened version of Second Computer Science 1 (SCS1) aptitude test, designated as SCS1R. \endabstract

References

[1]

{n. d.}. Computer programming aptitude test. https://www.kent.ac.uk/careers/tests/computer-test.htm. ({n. d.}). Accessed: April 1, 2018.

[2]

{n. d.}. CS education evaluation tools. https://csedresearch.org/tools/. ({n. d.}). Accessed: April 1, 2018.

[3]

R. Alexandrowicz. 2012. "Ganz Rasch": A free software for categorical data analysis. Social Science Computer Review 30, 3 (2012), 369--379.

Digital Library

[4]

E. Andersen. 1973. A goodness of fit test for the Rasch model. Psychometrika 38, 1 (1973), 123--140.

[5]

F. Baker and S. Kim. 2004. Item Response Theory: Parameter estimation techniques (2nd. ed.). CRC Press.

[6]

College Board. 2010. 2009 ap computer science a released exam. The College Board, New York.

[7]

T. Bond and C. Fox. 2001. Applying The Rasch Model: Fundamental measurement in the human sciences. Laurence Erlbaum Associates.

[8]

S. Cooper, K. Wang, I. Maya, and S. Sorby. 2015. Spatial skill training in introductory computing. ICER '15 (2015), 13--20.

Digital Library

[9]

A. Field. 2009. Discovering statistics using SPSS. Sage publications.

Digital Library

[10]

D. Harris. 1989. Comparison of 1-, 2- and 3-Parameter IRT Models. Educational Measurement: Issues and Practice 8, 1 (1989), 35--41.

[11]

L. Lacher, A. Jiang, Y. Zhang, and M. Lewis. 2017. Aptitude and previous experience in cs1 classes. Int'l Conf. Frontiers in Education: CS and CE 17 (2017), 87--91.

[12]

C. Lance, M. Butts, and L. Michels. 2006. The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods 9, 2 (2006), 202--220.

[13]

M. Parker, M. Guzdial, and S. Engleman. 2016. Replication, validation, and use of a language independent cs1 knowledge assessment. ICER '16 (2016), 93--101.

Digital Library

[14]

Ayala R.J. de. 2009. The theory and practice of item response theory. The Guilford Press.

[15]

A. Sahin and A. Duygu. 2016. The effects of test length and sample size on item parameters in item response theory. Educational Sciences: Theory Practice 17, 1 (2016), 321--335.

[16]

Student. 1908. The Probable Error of a Mean. Biometrika 6, 1 (1908), 1--25.

[17]

A. Tew, B. Dorn, and O. Schneider. 2012. Toward a validated computing attitudes survey. ICER '12 (2012), 135--142.

Digital Library

[18]

A. Tew and M. Guzdial. 2011. The FCS1: A language independent assessment of cs1 knowledge. SIGCSE '11 (2011), 111--116.

Digital Library

[19]

R. Wasserstein and N. Lazar. 2016. The ASA's Statement on p-Values. The American Statistician 70, 2 (2016), 129--133.

[20]

S. Y. Yoon. 2011. Revised Purdue Spatial Visualization Test: Visualization of rotations (Revised PSVT:R) {Psychometric Instrument}. (2011).

Cited By

Nitta CEiselt KStephenson BStone JBattestilli LRebelsky SShoop L(2024)The First Five Years of a Dual Track Programming SeriesProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630947(965-971)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630947
Parkinson JDziallas SMcNeill FWilliams J(2023)Exploring Models and Theories of Spatial Skills in CS through a Multi-National StudyProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600129(122-133)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600129

Index Terms

(Re)Validating Cognitive Introductory Computing Instruments
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Student assessment

Recommendations

Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment
ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research

Computing education lags other discipline-based education research in the number and range of validated assessments available to the research community. Validated assessments are important for researchers to reduce experimental error due to flawed ...
Uses, Revisions, and the Future of Validated Assessments in Computing Education: A Case Study of the FCS1 and SCS1
ICER 2021: Proceedings of the 17th ACM Conference on International Computing Education Research

Validated instruments of knowledge are important for creating an accepted and shared measurement for the most important part of education – student learning. The Force Concept of Inventory from Physics Education Research has had a significant impact ...
The FCS1: a language independent assessment of CS1 knowledge
SIGCSE '11: Proceedings of the 42nd ACM technical symposium on Computer science education

A primary goal of many CS education projects is to determine the extent to which a given intervention has had an impact on student learning. However, computing lacks valid assessments for pedagogical or research purposes. Without such valid assessments, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCSE '19: Proceedings of the 50th ACM Technical Symposium on Computer Science Education

February 2019

1364 pages

ISBN:9781450358903

DOI:10.1145/3287324

General Chairs:
Elizabeth K. Hawthorne
Union County College, USA
,
Manuel A. Pérez-Quiñones
University of North Carolina at Charlotte, USA
,
Program Chairs:
Sarah Heckman
North Carolina State University, USA
,
Jian Zhang
Texas Woman's University, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 February 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

SIGCSE '19

Sponsor:

SIGCSE

SIGCSE '19: The 50th ACM Technical Symposium on Computer Science Education

February 27 - March 2, 2019

MN, Minneapolis, USA

Acceptance Rates

SIGCSE '19 Paper Acceptance Rate 169 of 526 submissions, 32%;

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE Virtual 2024

Sponsor:
sigcse

1st ACM Virtual Global Computing Education Conference

December 5 - 8, 2024

Virtual Event , NC , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
498
Total Downloads

Downloads (Last 12 months)79
Downloads (Last 6 weeks)8

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nitta CEiselt KStephenson BStone JBattestilli LRebelsky SShoop L(2024)The First Five Years of a Dual Track Programming SeriesProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630947(965-971)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630947
Parkinson JDziallas SMcNeill FWilliams J(2023)Exploring Models and Theories of Spatial Skills in CS through a Multi-National StudyProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600129(122-133)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600129

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents