Abstract
Assessment in education is an important task. Albeit being relatively old techniques, the adoption of adaptive (CAT) instead of classical testing (FIT) still poses several questions. According to the literature, a CAT test should be faster to complete, more precise in estimating the student’s ability, and more engaging to take than FIT. On the other hand, a FIT test should be easier to prepare and complete, and less prone to anxiety than CAT. In this context, the paper initially reports on a study conducted with students (that used CAT and FIT) about different aspects of their assessments’ experiences. The results show that students seem to prefer CAT, considered fairer, more practical and faster than FIT, independently of the different attitudes towards technology. Furthermore, students did not report more anxiety or more workload with CAT. Moreover, we also evaluate different methods to convert the ability measured with CAT into a grade, so to verify if the adopted method could have influenced the results of the questionnaire.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
For completeness, the range for \(\theta \) that nullify the average difference between grades calculated through CTT and the proposed linear conversion is \([-4.15,4.15]\).
References
Alrifai, M., Gennari, R., Vittorini, P.: Adapting with evidence: the adaptive model and the stimulation plan of TERENCE. In: Vittorini, P., Gennari, R., Marenzi, I., de la Prieta, F., Rodríguez, J. (eds.) International Workshop on Evidence-Based Technology Enhanced Learning. AISC, vol. 152, pp. 75–82. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28801-2_9
Angelone, A.M., Vittorini, P.: A report on the application of adaptive testing in a first year university course. In: Uden, L., Liberona, D., Sanchez, G., Rodríguez-González, S. (eds.) LTEC 2019. CCIS, vol. 1011, pp. 439–449. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20798-4_38
Athaya, H., Dwi, R., Nadir, A., Sensuse, D.I., Suryono, R.R.: Moodle implementation for e-learning: a systematic review. In: 6th International Conference on Sustainable Information Engineering and Technology 2021, vol. 14 (2021). https://doi.org/10.1145/3479645.3479646
Bernardi, A., et al.: On the design and development of an assessment system with adaptive capabilities. In: Di Mascio, T., et al. (eds.) MIS4TEL 2018. AISC, vol. 804, pp. 190–199. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-98872-6_23
Birnbaum, A.: Some latent trait models and their use in inferring an examinee’s ability. In: Statistical Theories of Mental Test Scores, pp. 395–479 (1968)
Brown, G.T.L., Abdulnabi, H.H.A.: Evaluating the quality of higher education instructor-constructed multiple-choice tests: impact on student grades. Front. Educ. 2, 24 (2017). https://doi.org/10.3389/feduc.2017.00024
Chalmers, R.P.: mirt: a multidimensional item response theory package for the R environment. J. Stat. Softw. 48(6), 1–29 (2012). https://doi.org/10.18637/jss.v048.i06
Colwell, N.M.: Test anxiety, computer -adaptive testing, and the common core. J. Educ. Train. Stud. 1(2), 50–60 (2013). https://doi.org/10.11114/JETS.V1I2.101
Conejo, R., Guzmán, E., Trella, M.: The SIETTE automatic assessment environment. Int. J. Artif. Intell. Educ. 26(1), 270–292 (2016). https://doi.org/10.1007/s40593-015-0078-4
De Champlain, A.F.: A primer on classical test theory and item response theory for assessments in medical education. Med. Educ. 44(1), 109–117 (2010). https://doi.org/10.1111/j.1365-2923.2009.03425.x
DeVellis, R.F.: Classical test theory. Med. Care 44(11), S50–S59 (2006). https://doi.org/10.2307/41219505
Di Mascio, T., Gennari, R., Melonio, A., Vittorini, P.: The user classes building process in a TEL project. In: Vittorini, P., Gennari, R., Marenzi, I., de la Prieta, F., Rodríguez, J. (eds.) International Workshop on Evidence-Based Technology Enhanced Learning. AISC, vol. 152, pp. 107–114. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28801-2_13
Dunn, L., Morgan, C., O’Reilly, M., Parry, S.: The Student Assessment Handbook. Routledge, Abingdon (2003). https://doi.org/10.4324/9780203416518
Eleje, L.I., Onah, F.E., Abanobi, C.C.: Comparative study of classical test theory and item response theory using diagnostic quantitative economics skill test item analysis results. Eur. J. Educ. Soc. Sci. 3(1), 57–75 (2018)
Embretson, S.E., Reise, S.P.: Item Response Theory for Psychologists. Lawrence Erlbaum Associates, Mahwah (2000)
Erdt, M., Fernández, A., Rensing, C.: Evaluating recommender systems for technology enhanced learning: a quantitative survey. IEEE Trans. Learn. Technol. 8(4), 326–344 (2015). https://doi.org/10.1109/TLT.2015.2438867
Fritts, B.E., Marszalek, J.M.: Computerized adaptive testing, anxiety levels, and gender differences. Soc. Psychol. Educ. 13(3), 441–458 (2010). https://doi.org/10.1007/S11218-010-9113-3
Galassi, A., Vittorini, P.: Automated feedback to students in data science assignments: improved implementation and results. In: 14th Biannual Conference of the Italian SIGCHI Chapter (CHItaly 2021), New York, NY, USA. ACM, Bolzano (2021). https://doi.org/10.1145/3464385.3464387
Gikandi, J., Morrow, D., Davis, N.: Online formative assessment in higher education: a review of the literature. Comput. Educ. 57(4), 2333–2351 (2011). https://doi.org/10.1016/J.COMPEDU.2011.06.004
Hambleton, R.K., Jones, R.W.: Comparison of classical test theory and item response theory and their applications to test development. Educ. Meas. Issues Pract. 12(3), 38–47 (1993). https://doi.org/10.1111/j.1745-3992.1993.tb00543.x
Ling, G., Attali, Y., Finn, B., Stone, E.A.: Is a computerized adaptive test more motivating than a fixed-item test? Appl. Psychol. Meas. 41(7), 495–511 (2017). https://doi.org/10.1177/0146621617707556
Magis, D., Barrada, J.R.: Computerized adaptive testing with R: recent updates of the package catR. J. Stat. Softw. 76(Code Snippet 1), 1–19 (2017). https://doi.org/10.18637/jss.v076.c01
Mulwa, C., Lawless, S., Sharp, M., Arnedillo-Sanchez, I., Wade, V.: Adaptive educational hypermedia systems in technology enhanced learning: a literature review. In: Proceedings of the 2010 ACM Conference on Information Technology Education - SIGITE 2010 (2010). https://doi.org/10.1145/1867651
Qasem, M.A.N.: A comparative study of classical theory (CT) and item response theory (IRT) in relation to various approaches of evaluating the validity and reliability of research tools. IOSR J. Res. Method Educ. (IOSR-JRME) 3(5), 77–81 (2013)
R Core Team: R: A Language and Environment for Statistical Computing (2018). https://www.R-project.org/
Riffenburgh, R.H.: Statistics in Medicine. Elsevier/Academic Press, Cambridge (2012)
Rindskopf, D.: Reliability: measurement. Int. Encycl. Soc. Behav. Sci. 13023–13028 (2001). https://doi.org/10.1016/B0-08-043076-7/00722-1
Scalise, K., Allen, D.D.: Use of open-source software for adaptive measurement: concerto as an R-based computer adaptive development and delivery platform. Br. J. Math. Stat. Psychol. 68(3), 478–496 (2015). https://doi.org/10.1111/BMSP.12057
Tonidanel, S., Quinones, M.A.: Psychological reactions to adaptive testing. Int. J. Sel. Assess. 8(1), 7–15 (2000). https://doi.org/10.1111/1468-2389.00126
Vispoel, W.P., Rocklin, T.R., Wang, T.: Individual differences and test administration procedures: a comparison of fixed-item, computerized-adaptive, and self-adapted testing. Appl. Measur. Educ. 7(1), 53–79 (1994). https://doi.org/10.1207/s15324818ame0701_5
Vittorini, P., Menini, S., Tonelli, S.: An AI-based system for formative and summative assessment in data science courses. Int. J. Artif. Intell. Educ. 1–27 (2020). https://doi.org/10.1007/s40593-020-00230-2
Wainer, H., et al.: Computerized Adaptive Testing. Routledge, Abingdon (2000). https://doi.org/10.4324/9781410605931
Weiss, D.J., Kingsbury, G.G.: Application of computerized adaptive testing to educational problems. J. Educ. Measur. 21(4), 361–375 (1984). https://doi.org/10.1111/j.1745-3984.1984.tb01040.x
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Questionnaire
Questionnaire
1 | 2 | 3 | 4 | 5 | ||
|---|---|---|---|---|---|---|
Personal attitude | ||||||
\(Q_{1.1}\) | My attitude towards technology is good | |||||
Computerised Adaptive Test (CAT) | ||||||
\(Q_{2.1}\) | I am in favour of using CAT | |||||
\(Q_{2.2}\) | I perceived CAT as fair | |||||
\(Q_{2.3}\) | CAT is practical and fast | |||||
\(Q_{2.4}\) | I think that CAT needs more preparation than FIT | |||||
Fixed-Item Test (FIT) | ||||||
\(Q_{3.1}\) | I am in favour of using FIT | |||||
\(Q_{3.2}\) | I perceived FIT as fair | |||||
\(Q_{3.3}\) | FIT is practical and fast | |||||
\(Q_{3.4}\) | I think that FIT needs more preparation than CAT | |||||
General | ||||||
\(Q_{4.1}\) | The sequential way of asking questions allowed me | |||||
to focus more on the right answer | ||||||
\(Q_{4.2}\) | Regardless of the subject, taking an exam with | |||||
CAT, rather than FIT, made me feel more anxious | ||||||
Each question is answered with the following 5-points Likert scale:
“1 = Completely disagree”, “2 = Disagree”, “3 = Unsure”, “4 = Agree”, “5 = Completely agree”.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Angelone, A.M., Vittorini, P. (2023). A Case Study on Students’ Opinions About Adaptive and Classical Tests. In: Temperini, M., et al. Methodologies and Intelligent Systems for Technology Enhanced Learning, 12th International Conference. MIS4TEL 2022. Lecture Notes in Networks and Systems, vol 580. Springer, Cham. https://doi.org/10.1007/978-3-031-20617-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-20617-7_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20616-0
Online ISBN: 978-3-031-20617-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)