An Investigation into the Construct Validity of an Academic Writing Test in English with Special Reference to the Academic Writing Module of the IELTS Test

An Investigation into the Construct Validity of an Academic Writing Test in English with Special Reference to the Academic Writing Module of the IELTS Test Submitted by Ahmad Abdulrahman Alsagoafi to the University of Exeter as a thesis for the degree of Doctor of Philosophy in Education In February 2013 This thesis is available for Library use on the understanding that it is copyright material and that no quotation from the thesis may be published without proper acknowledgement. I certify that all material in this thesis which is not my own work has been identified and that no material has previously been submitted and approved for the award of a degree by this or any other University. Signature: ………………………………………………………….. Abstract The International English Language Testing System (IELTS) is the world’s leading high stakes test that assesses the English Language Proficiency of candidates who speak languages other than English and wish to gain entry into universities where English is the language of instruction. Recently, over 3000 institutions in the United States accepted the IELTS test to be an indicator of language proficiency (IELTS, 2012a). Because of this preference for the IELTS test, and its worldwide recognition, there has been an increase in the number of students who are taking the test every year. According to the IELTS website, more than 7000 institutions around the world trust the test results and, not surprisingly, more than 1.7 million candidates take the test every year in one of the 800 recognised test centres across 135 countries (IELTS, 2012a). These candidates include people who seek not only to obtain admission to universities, but also for immigration authorities, employers of certain companies and government agencies. Acknowledging this popularity and importance to learners of English as a Foreign Language (EFL), this qualitative study has investigated the construct validity of the academic writing module in the IELTS test from the perspectives of the stakeholders (i.e. candidates, lecturers and markers). The aim was to understand why some Saudi students fail to cope with demands of the university despite the fact that they have achieved the minimum requirements in IELTS. In this study, data was collected in two phases in two different settings through open-ended questionnaires, semi-structured observations and semi-structured interviews. Phase I was carried out in the Department of English Language (DEL) at King Faisal University in Saudi Arabia, while Phase II was conducted in one university in the UK. The sample of the study included: 8 students, 6   2   university lecturers and one marker. In this study, data were analysed and coded into themes by using NVivo 9. The results of this case study have shown that the stakeholders were doubtful about the issue of readiness of students, which is claimed by IELTS, and they wanted the test to be clearer about how the students were going to cope with university demands upon gaining entry. In addition, with respect to the content validity of the test, this study found that the tasks in the academic writing test to a large extent do not reflect the kind of tasks candidates are likely to encounter at university. Furthermore, this study pointed out that response validity, on the part of students who may not have understood the rubric of the tasks, is another important factor affecting the students’ performance. Also, the findings of this study suggested that scoring validity could have a significant effect on the students’ scores because of the inconsistency of markers during the scoring process as they may have sometimes failed to assign the students to their corresponding level of proficiency. Consequently, the study provided a set of implications as well as recommendations for future research.   3   Acknowledgements All thanks and praise is to ALLAH the Almighty the one who guided me throughout the process of this research or else without His guidance and blessings I could have never completed this dissertation. I am deeply indebted to a number of people for their inspiration, encouragement, cooperation and contribution to the completion of this dissertation. First of all, I would like to express my deepest gratitude and appreciation to my first supervisor Dr. Li Li for her endless encouragement, critical insights, great efforts to explain things clearly and simply, for her full attention and invaluable feedback during the research project. She has never been hesitant to offer her advice to me even when she was my second supervisor. I wish also to express my sincere appreciation to my second supervisor Dr. Susan Jones whose supervision is also enormously valuable and I am deeply touched by her insightful suggestions and helpful feedback, her patient guidance, and her enthusiasm in research. I also owe a debt of thanks and gratitude to Dr. Malcolm MacDonald (Warwick University) who was my former supervisor. When he was my first supervisor, he used to give me support and guidance with respect to the shaping, reading and commenting on the whole manuscript. I owe every step that I have made in this doctoral exploration to all of my supervisors who generously spent numerous hours advising and helping me to come up with a very carefully designed methodology and refine the thesis drafts. Secondly, my heartfelt thanks go to the academics and friends who helped me throughout my research project. I am really grateful to the university lecturers, students and IELTS examiner who participated in my study. Without their help and support, I would not have   4   been able to collect information for my study. For the sake of confidentiality, I am not going to mention their names. I would also like to acknowledge the efforts of Assoc. Prof. Keith Postlethwaite (Deputy Head of Graduate School of Education, University of Exeter) and Dr. Dhaif Allah Alzaidi (Assistant Professor, Taiba University) who provided their informative feedback on the face validity of the questionnaires, to Prof. Debra Myhill (Associate Dean, College of Social Sciences and International Studies, University of Exeter) for her insightful comments and sustained encouragement to use the sociocognitive framework in my study, to Dr. Liu Yongcan (University of Cambridge) with whom I had a fruitful discussion about the Socio-cultural Theory and Activity Theory, to Dr. Anthony Green for his incisive comments during our discussion about the IELTS test, to my mentor Prof. Wendy Robinson (Head of the Graduate School of Education, University of Exeter) for her emotional and academic support during my research project, to Prof. Rüdiger Zimmermann (Philipps University of Marburg) for his reply to my email regarding the models of second language writing, to Prof. Cyril Weir and Lynda Taylor for sharing their knowledge and expertise with me in the Introductory Course to Language Testing in Cambridge university. I am really very appreciative of a number of academics and experts, especially to those in the field of Language Testing and Education, for their useful comments and conversations in emails or conferences. These include: Prof. Alan Davies (University of Edinburgh), Mr. Alan Addison (IELTS Stakeholder Relations Manager, British Council), Miss Carolyn Walker (Director of INTO Language Centre, University of Exeter), Prof. Charles Alderson (Lancaster University), Miss Chrissi Florides (Director of Studies at   5   Globe English Centre), Dr. Durrant Philip (Graduate School of Education, University of Exeter), Dr. Guoxing Yu (Graduate School of Education, University of Bristol), Dr. Nasser Mansour (Graduate School of Education, University of Exeter), Dr. Ros Fisher (Director of Doctoral Studies, University of Exeter), and Dr. Vicki Feast (University of South Australia). Thirdly, my sincere thanks are also due to the Saudi Arabian Cultural Attaché in UK for tracking my progress and encouraging me to complete my PhD, and to my employer King Faisal University (KFU) who granted me this scholarship and offered their full support (academic and financial) for me to collect the data for this study. I am thankful to my supervisor during the data collection process at KFU Dr. Abdullah Alfraidan (Chairman of the Department of English Language) who dedicated his time and effort to make the data collection process successful, and without whom I would not have had the proper access to my participants to collect data. I am also grateful to the staff members in the College of Arts at KFU, especially those in the Department of English Language, who participated in the study. Finally, my deepest gratitude and love go to my parents who brought me up to make me able to face big challenges. I can never ever forget how sad they were on the day when I left Saudi Arabia to study abroad, and said that “we will rejoice when you pass your PhD with flying colours”. I am truly grateful for all their unconditional love, sacrifices, and prayers. I happily dedicate this PhD to them. My grandmother also had a great positive influence on my progress with her unlimited emotional support and eloquence in saying   6   the prayers. She used to call me twice a month to make sure that I was progressing well. I thank Allah the Almighty that she is still alive to witness this achievement. Last but not least, I would like to extend my sincere appreciation and gratitude to my soul mate, my wife Fatimah, who has travelled all the way with me in this long journey. She withdrew from her course in Saudi Arabia as soon as she knew that I was going abroad. I really appreciate this, not to mention her emotional support which helped me to overcome my countless problems that I faced during my research project. I am also thankful for her care of our two lovely children Bandar and Abdullah. Surely, I will be forever thankful for what she did and does for me. I am blessed to be her husband. My special thanks also go to my three brothers, especially Fahad, who were also concerned about making this dream a reality.   7   Table of Contents Acknowledgements........................................................................................................................ 4 Abbreviations .............................................................................................................................. 15 Chapter I ...................................................................................................................................... 20 Introduction………………………………………………………………………….. 20 1.1 Overview of the chapter....................................................................................... 20 1.2 A Brief background to the IELTS Test................................................................ 20 1.2.1 The development of IELTS........................................................................... 23 1.2.1.1 The content and the format of the test………………………………….24 (a) The listening test ……………………………………………………….…25 (b) The reading test ……………………………………………….…………..25 (c) The writing test ………………………………………………….………..26 (d) The speaking test…………………………………………………….….…27 1.2.1.2 Marking and assessment………….…………………………………….27 1.3 Rationale for the study ......................................................................................... 29 1.4 Aim ...................................................................................................................... 32 1.5 Objectives ............................................................................................................ 32 1.6 An overview of the whole dissertation ................................................................ 33 Chapter II .................................................................................................................................... 36 Context of the study .................................................................................................... 36 2.1 Introduction.......................................................................................................... 36 2.2 Brief background of the educational system in Saudi Arabia ............................. 36 a- Pre-School level (2 years) .................................................................................. 37 b- Elementary level (6 years) ................................................................................. 37 c- Intermediate level (3 years) ............................................................................... 38 d- Secondary level (3 years)................................................................................... 38 - Regular secondary education ............................................................................... 38 - Vocational and technical secondary education .................................................... 39 e- Higher education ................................................................................................ 39 2.2.1 The role of assessment in the Saudi Arabian educational system................. 41 2.2.1.1 The General Aptitude Test……………………………………….……..42 2.2.1.2 The General Aptitude Test for University Graduates…………………..42 2.2.1.3 The General Aptitude Test in English………………………………….42 2.2.1.4 The Achievement Test for Science Colleges (For Men)….....................43 2.2.1.5 The Aptitude and Achievement Test for Art Students (For Women)......43 2.2.1.6 The Aptitude and Achievement Test for Science Students…….............43 2.2.1.7 The Standardised Test of English Proficiency…………………….……44 2.3 Testing/assessment and/or Educational Organisations Offering Tests in Saudi Arabia ........................................................................................................................ 44 2.3.1 The British Council in Saudi Arabia ............................................................. 44 2.3.2 The Educational Testing Service (ETS) tests in Saudi Arabia ..................... 45 2.3.3 The Pearson English Tests in Saudi Arabia .................................................. 45 2.3.4 The Graduate Management Admission Council (GMAC) Test ................... 45   8   2.4 The College of Arts ............................................................................................. 46 2.5 A brief description of IELTS candidates in Saudi Arabia………………………46 2.6 Conclusion ........................................................................................................... 48 Chapter III ................................................................................................................................... 50 Literature Review ........................................................................................................ 50 3.1 Introduction.......................................................................................................... 50 3.2 Theoretical framework......................................................................................... 51 3.2.1 Writing as a cognitive practice...................................................................... 53 3.2.1.1 Models of L1 writing…………………………………………………...53 (a) Hayes and Flower Model (1980)……………………………………….…53 (b) Bereiter and Scardamalia Model (1987)…………………………………..55 3.2.1.2 Composing processes of second language candidates………………….57 3.2.2 Writing as a social construction practice...................................................................61 3.2.3 Writing as a social interaction activity .......................................................... 64 3.2.4 Writing as a socio-cultural practice .............................................................. 66 3.3 A brief introduction to language testing .............................................................. 70 3.3.1 Types of tests ................................................................................................ 71 3.3.1.1 Placement tests………………………………………………………….71 3.3.1.2 Progress tests……………………………………………………………72 3.3.1.3 Achievement tests……………………………………………………....72 3.3.1.4 Proficiency tests………………………………………………………...72 3.3.1.5 Diagnostic tests………………………………………………………....73 3.4 Essential qualities in language tests ..................................................................... 74 3.4.1 Validity.......................................................................................................... 74 3.4.1.1 Types of validity ........................................................................................ 76 3.4.1.1.1 Content validity……………………………………………………….79 -Authenticity…………………………………………………………………....80 3.4.1.1.2 Response validity……………………………………………………..85 (a) Macro-planning…………………………………………………………….86 (b) Organisation…………………………………………………………….….86 (c) Micro-planning…………………………………………….……………….87 (d) Translation………………………………………………………………….87 (e) Monitoring………………………………………………………………….87 (f) Revising…………………………………………………………………….88 3.4.1.1.3 Scoring validity………………………………………….…………….89 (a) Criteria/rating scale…………………………………………………………90 (1) Primary trait scoring……………………………………………………....90 (2) Holistic scoring…………………………………………………………...92 (3) Analytic scoring……………………………………………………..……93 -Empirical studies on the application of rating scales……………………...…..96 (b) Rater training………………………………..………………………...……99 3.5 Empirical studies on IELTS............................................................................... 102 3.5.1 Predictive studies on IELTS as a preparation to academic study ............... 102 3.5.2 Studies on tasks design for IELTS academic writing ................................. 104 3.5.3 Studies on assessment of IELTS academic writing tasks ........................... 109 3.5.4 Impact and washback studies on IELTS and IELTS preparation courses .. 110 3.6 Research Questions ............................................................................................ 116   9   3.7 Summary and Conclusion .................................................................................. 117 Chapter IV ................................................................................................................................. 122 Methodology............................................................................................................... 122 4.1 Introduction........................................................................................................ 122 4.2 Ontology ............................................................................................................ 122 4.3 Epistemology ..................................................................................................... 124 4.4 Rationale for choosing the qualitative approach ............................................... 126 4.4 Researcher’s role ............................................................................................... 128 4.5 Research design ................................................................................................. 134 4.6 Sampling ............................................................................................................ 136 4.7 Data collection methods .................................................................................... 138 4.7.1 Questionnaires ............................................................................................. 139 -Construction of the questionnaires …………………………………………….140 4.7.1.1 Students’ questionnaires………………………………………………140 a) Phase I……………………………………………………………………..141 -Pre-writing test Questionnaire………………………………………...…..142 -Post-writing test Questionnaire………………………………………..….143 b) Phase II……………………………………………………………………143 -IELTS Writing test tasks and University Tasks Questionnaire……….…..144 c) Pilot Study…………………………………………………………………144 -Development and validation of the students’ questionnaires……………..146 d) Administration of Students’ Questionnaires………………………………147 4.7.1.2 Marker’s Questionnaire………………………………………………..148 -The Questionnaire…………………………………………………………..149 -Administration of Marker’s Questionnaire…………………………………149 4.7.2 Interviews .................................................................................................... 150 -Transcription and validation of the interviews………………………………..151 -Administration of Students’ Interviews……………………………………….152 -Administration of Lecturers’ Interviews………………………………….…..152 4.7.3 Observations................................................................................................ 153 -Administration of observation………………………………………………...153 4.8 Triangulation...................................................................................................... 154 4.9 Type of data ....................................................................................................... 155 4.10 Data analysis .................................................................................................... 157 4.10.1 Content analysis ........................................................................................ 157 4.11 Ethical issues ................................................................................................... 158 3.12 Summary of the chapter ................................................................................... 160 Chapter V................................................................................................................................... 162 Findings ...................................................................................................................... 162 5.1 Introduction........................................................................................................ 162 5.2 Response validity question ................................................................................ 162 5.2.1 Pre-writing Test Questionnaire ................................................................... 163 5.2.1.1 Personal information, prospective course and place of study…………164 5.2.1.2 General & specific knowledge questions about the AWM in the test...166 5.2.2 Post-writing Test Questionnaire.................................................................. 173 5.2.2.1 Time allotment……………………………………………………..….174   10   5.2.2.2 Guidelines and Instructions………………………………………...…175 5.2.2.3 Before the test…………………………………………………………176 5.2.2.4 Difficulty of the questions………………………………………….....176 5.2.2.5 Grammar mistakes, style and organisation……………………………177 5.2.2.6 Graph/chart/table in task 1………………………………………….…178 5.2.2.7 Familiarity with the content of the task……………………………….179 5.2.2.8 Advantages and disadvantages of the IELTS test…………………..…181 5.2.3 Observations with students in the mock test ............................................... 182 5.2.4 Interviews with candidates (Phase I) .......................................................... 183 5.2.4.1 Time to make drafts and complete each task………………………….183 5.2.4.2 Task guidelines………………………………………………………..187 5.2.4.3 Brainstorming…………………………………………………………190 5.2.4.4 Strategies………………………………………………………………193 5.2.4.5 Task difficulty…………………………………………………………198 5.2.4.6 Task order……………………………………………………………..202 5.2.4.7 The questions………………………………………………………….203 5.2.4.8 Students’ insights on the current testing practice of the AWM……… 210 5.3 Content validity question ................................................................................... 213 5.3.1 IELTS writing test tasks and University Tasks Questionnaire ................... 214 5.3.1.1 Students’ expectations of university tasks…………………………….214 5.3.1.2 Similarities & differences between IELTS writing tasks and university tasks…………………………………………………………………………...215 5.3.1.3 Suitability of IELTS writing tasks for university students……………217 5.3.2 IELTS Writing test tasks and university tasks Interview ............................ 218 5.3.2.1 The relationship between IELTS writing tasks and university tasks….218 5.3.2.2 The role of IELTS in preparing students to university tasks………….220 5.3.2.3 Students' perspectives on the level of authenticity of IELTS writing tasks………………………………………………………………………...…222 5.3.3 Interviews with lecturers (Phase II) ............................................................ 225 5.3.3.1 Lecturers’ perceptions of IELTS tasks………………………………..226 5.3.3.2 Lecturers’ expectations of students who took the IELTS test………...230 5.4 Scoring validity question: .................................................................................. 234 5.4.1 Markers’ Questionnaire............................................................................... 234 5.5 Conclusion ......................................................................................................... 237 Chapter VI ................................................................................................................................. 239 Discussion ................................................................................................................... 239 6.1 Introduction........................................................................................................ 239 6.2 The key findings of the study ............................................................................ 239 6.2.1 Phase I ......................................................................................................... 240 6.2.1.1 Key findings for response validity question…………………………..240 6.2.1.2 Key findings for scoring validity question……………………………241 6.2.2 Phase II ........................................................................................................ 242 6.2.2.1 Key findings for content validity question……………………………242 6.3 The stakeholders’ perceptions about the test ..................................................... 243 6.3.1 The students’ perceptions of the value of the test ....................................... 243 6.3.2 The stakeholders’ perceptions of their test awareness ................................ 244   11   6.3.3 Stakeholders’ doubts about IELTS correspondence to university tasks ..... 250 6.3.4 Stakeholders’ doubts about IELTS as a measure ........................................ 255 6.4 The students’ engagement with the IELTS writing test .................................... 263 6.4.1 The students’ perceptions of test difficulty and duration ........................... 263 6.4.2 The students’ perceptions of test prompts and instructions ........................ 266 6.4.3 The perceived test taking (Students) and scoring strategies (Markers) ...... 268 6.4.4 Expertise in second language (L2) writing ................................................. 273 6.5 The connection between perceptions and engagement ...................................... 274 6.5.1 Change of perspectives ............................................................................... 274 6.6 Summary of the chapter ..................................................................................... 277 Chapter VII ............................................................................................................................... 281 Conclusion:................................................................................................................. 281 7.1 Introduction........................................................................................................ 281 7.2 Summary of the study ........................................................................................ 281 7.2.1 The stakeholders’ perceptions about the mismatch between IELTS tasks and university tasks ..................................................................................................... 282 7.2.2 Stakeholders’ doubts about IELTS as a measure ........................................ 284 7.2.3 Tracking the change of perspectives ........................................................... 287 7.3 Significance of the study………………………………………………………288 7.4 Limitations of the study ..................................................................................... 290 7.5 Implications of the study ................................................................................... 291 7.5.1 Stakeholders ................................................................................................ 291 7.5.2 Test providers .............................................................................................. 293 7.5.3 Methodological implications of the study ................................................... 295 7.6 Recommendations for future research ............................................................... 296 7.7 Final conclusion to the study ............................................................................. 298 References .................................................................................................................................. 300 Appendices ................................................................................................................................. 335 Appendix I .................................................................................................................. 336 Data Collection Tools & Key Findings Of The Study ............................................ 336 Phase I ...................................................................................................................... 336 IELTS Awareness Form for Candidates (Pre-Writing Test Phase I ) .................. 336 IELTS Awareness Form for Candidates (Post-Writing Test Phase I) ................. 338 IELTS Post-Writing Test Interview (Candidates) ............................................... 340 IELTS Questionnaire for Markers ....................................................................... 342 Phase II .................................................................................................................... 344 IELTS Tasks and University Tasks Form for Candidates (Phase II) ................... 344 Interview Questions for Candidates ..................................................................... 345 Interview Questions for Lecturers........................................................................ 346 Sample of a transcription of one interview .......................................................... 347 Key Findings (according to data collection tools)………………………………...352 Timeframe of Research…………………………………………………………….361 Consent Form........................................................................................................... 362 Appendix II ................................................................................................................ 363 IELTS Tasks & Scoring Guides ............................................................................... 363 Sample of IELTS Writing Task 1 ............................................................................ 363   12   Model answer for IELTS Writing Task 1 ............................................................ 364 Sample of IELTS Writing Task 2 ............................................................................ 365 Model answer for IELTS Writing Task 2 ............................................................ 366 Scoring Criteria for Task One .............................................................................. 367 Scoring Criteria for Task Two ............................................................................. 369 Appendix III............................................................................................................... 371 Weir’s Socio-cognitive Framework for Writing ..................................................... 371 Appendix IV ............................................................................................................... 372 NVivo Samples ........................................................................................................... 372 Appendix V………………………...………………………………………………...374 Mock IELTS Test Observation Schedule………………………………………….374 List of Tables Table 1: History of the IELTS Test .................................................................................. 22 Table 2: IELTS Band Scores ............................................................................................ 28 Table 3: The educational ladder in Saudi Arabia . ............................................................ 40 Table 4: Summary of empirical research on writing processes of L2 learners. ................ 61 Table 5: Task Characteristics. ........................................................................................... 82 Table 6: A comparison of holistic & analytic scales on 6 qualities of test usefulness ..... 94 Table 7: Tasks and Skills in IELTS versus those in Research Degrees .......................... 106 Table 8: Summary of research paradigm (ontology, epistemology & methodology), design & methods .................................................................................................... 134 Table 9: Participants, context of the study, time and duration ........................................ 138 Table 10: Framework of the study .................................................................................. 154 Table 11: An overview of questions, participants and methods. .................................... 156 Table 12: Personal information, prospective course and place of study ......................... 165 Table 13: Students’ knowledge of the test ...................................................................... 169 Table 14: Advantages and Disadvantages of the IELTS test .......................................... 181 Table 15: Summary of the students’ views on the similarities and differences between IELTS tasks and University tasks. ........................................................................... 216   13   List of Figures Figure 1: Cognitive validity parameters in writing ………………………………..…….86 Figure 2: Interactions in performance assessment of speaking skills………………..…..99 List of Diagrams Diagram 1: Test Structure ................................................................................................. 24 Diagram 2: Theoretical Framework of the Study ............................................................. 52   14