MEASURING IMPACT OF USING E-LEARNING PORTALS ON EDUCATIONAL SYSTEMS Masoud Fakhimi, Munir Abbasi , Lampros K. Stergioulas, Georgios Xydopoulos, University of Surrey, United Kingdom; Maria Fragkaki, Computer Technology Institute, Greece; Sofoklis Sotiriou, Ellinogermaniki Agogi, Greece; Luis Anido Rifon, University of Vigo, Spain; Ruxandra Margineanu, UNISOFT, Romania; and Ferenc Tatrai , EDEN, United Kingdom Abstract: Technology Enhanced Learning (TEL), Open Educational Resources (OER), innovative learning paradigms, educational scenarios and the associated digital technologies, environments, applications and content are continuously changing and rapidly evolving nowadays, thus affecting not only the effectiveness, the impact and the cognitive outcomes of teaching, but also the quality of the learning process as a whole. The impact of TEL environments and portals using OER can be evaluated in terms of usability, effectiveness and pedagogical use. Impact evaluation has always carried great potential for improving learning outcomes and tailoring learning toward the users’ needs, motivation, satisfaction and expectations. However, assessment of the impact of the use of teaching and learning resources, environments and portals in a systematic manner has proven to be difficult. The research work described in this paper attempts to assess the impact of using Open Learning Resources in a shared and distributed way, on an innovative portal, namely the “Open Discovery Space” (ODS) portal. In this European example case, there are different dimensions of impact to consider, including impact on teachers, students, stakeholders, schools and the educational system at large. The focus of the impact evaluation in this research is to understand how best such TEL environments are to be accommodated, adopted and used within the educational processes, and how well they can fit within the current national/regional curricula and educational systems. 1- Introduction: The role of Information and Communication Technology (ICT) in Education has undergone drastic changes since the beginning of the Internet era. The modern web-based open learning and education tools have transformed mechanisms of teaching and learning (Stergioulas et. al., 2014, Alian and Al-Akhras, 2010; Finlayson et al., 2006). Together with changes in teaching practices, there is also an increase in the demands of the stakeholders (schools, teachers, learners and their parents, society, etc.) involved in such educational processes (Haddad and Draxler, 2002; Harrison et al., 2002) for high-quality teaching and a constructivist and critical learning process . Another current issue is that TEL has overlooked the need for developing the learners’ skills in order to constructively build on the mass of information that they can easily access by using technology, as well as the possibility to apply critical-reflective and creative thinking (Fragkaki, Stergioulas 2014). Therefore, evaluating the impact of ICT-based educational tools has become crucial for satisfying stakeholders’ needs and for modernising the educational institutions (Kozma, 2005). In the context of educational ICT systems and portals, many studies have been limited to showing the efficiency of ICT-based teaching and learning tools only compared to traditional classroom-based teaching and learning approaches (Comber et al., 2002). However, apart from a very small number of studies (e.g. Ga-Jin, 2007), there is a scarcity of research carried out on the assessment/evaluation of the impact of the systems themselves or their efficiency (Alkhalaf et al., 2012). This paper aims to discuss the evaluation of the impact of using an innovative and open-source portal, namely Open Discovery Space [18], on teachers, learners, schools and educational systems, and show some preliminary results. It is organised as follows: Section two introduces the impact assessment of ODS. Section three briefly outlines the impact assessment methodology applied for the purpose of this research. Section four shows some of the recent preliminary impact measurement results. Section five summarises the research and discuss the results of the research so far. 2- Impact Assessment of the ODS Portal: The ODS EU project aims to provide a socially empowered multilingual innovative eLearning portal, based on an Innovation model that contains three core phases of activities: stimulating, incubating and accelerating. Educational design in ODS is performed by teachers and educational professionals who aim to facilitate the interaction between learners, resources, scenarios and tools for educational purposes. They are typically employed at primary or secondary schools, or work in organizations with an educational mission such as museums and science centres. Parents have a somewhat related role as they are responsible for the general upbringing of the learners. A consortium of 51 participant organizations is responsible for the co-design and codevelopment of the ODS pedagogical and instructional framework1. Impact assessment of an eLearning portal, such as ODS, necessarily involves evaluation and assessment of educational aspects that directly relate to the context of the teaching and learning activities. For this reason, the results of such assessment can be used as a basis to improve the educational system. This type of assessment contrasts with traditional approaches, which focus on a limited number of factors and are usually used for grading and accountability purposes. Thus, our impact assessment methodology considers various impact factors including motivation factors, engagement, empowerment, involvement, satisfaction, accessibility, availability, usability, effectiveness, achievement, performance, sharing, interaction, communication, collaboration, on-going support, innovation, training, creativity, search facility, socialising, culturally and linguistically appropriate features, etc. The objectives of the impact assessment activities are to discover the nature and extent of effects and changes, and demonstrate the outcomes on the micro-level stakeholders, including on individual teachers, learners and schools and the macro-level stakeholders, including regional, national, and EU-level stakeholders, and also policy makers. In this context, it is expected that identifying and measuring the actual impact that a portal like ODS has on educational systems will help stakeholders to fully exploit the strength of such portals and will facilitate improvements in terms of the application of digital technologies in educational systems and in their everyday educational processes. 3- Summary of the Research Methodology This section introduces the research methodology of this research. This methodology is grounded on the ODS impact assessment framework developed as part of this project. This methodology has been thoroughly explained in recently published research work (Fakhimi et al., 2014). The ODS impact assessment methodology involves a mixed (Quantitative-Qualitative) approach that has been adapted mainly from the studies conducted by (Scheuermann, F., and Pedro, 2009; Wang et al., 2007), the EFI framework (Stracke, 2012), as well as from other EU and international projects. The knowledge gathered from existing literature has been used for the formation of an impact assessment methodology, which including indicators and questionnaires specifically tailored to the ODS portal community’s impact assessment needs. More specifically, indicators and parameters to measure the impact of ODS on educational systems are derived from existing literature in an empirically validated instrument and from the objectives and activities of ODS. The methodology will also be of great value to any online learning portal stakeholders by enabling them to identify various aspects of the system and investigate the link between the impact of eLearning systems and its drivers. The indicators selected for measuring the impact of ODS portal follow the SMART criteria (Doran, 1981) and include “Key Performance Indicators” (KPIs) (Ishizaka, 2013). A questionnaire/survey was prepared based on impact indicators. The impact assessment quantitative survey includes a five-point scale/Likert based questionnaire with equal scores for each question and different weight factors for each choice. The presented research results were derived from responses to a questionnaire survey from 390 teachers across 23 countries across the Europe engaged in various ODS activities. Questionnaires were translated into 13 different European languages spoken across the Europe. Among the participants in the surveys, approx. 52% were female, and 27% were Male teachers, while the rest of the participants (approx. 21%) were not willing to disclose their gender in the survey. The surveys were conducted from January 2014 till June 2014 - almost two years after the start of ODS project – and the data was analysed using basic descriptive 1 http://opendiscoveryspace.eu/sites/ods/files/d4.1_the_open_discovery_space_educational_design.pdf statistics and factor analysis, and relationships within the dataset were examined using logistic regression techniques. 4- Results 4.1. Impact of using ODS on Educational System and Learning processes There are similarities between teachers’ use of ODS and their perceptions of its impact on students and learners. 58% of participants have mentioned that they agree/strongly agree that using ODS has enabled them to provide a better learning experiences for their students. Approximately 60% of teachers also agreed/strongly agreed ODS had helped make students more effective at strengthening their knowledge. The large majority of the participants (81%) felt that introducing ODS motivated their students to apply more digital and ICT-based tools and resources for their studies (Table 1). Table 1 Teachers view on the impact of ODS portal on educational system Strongly Disagree % Disagree % Neither Agree or Disagree % Agree % Strongly Agree % Cannot Answer this question % ICT-based or ICT-integrated learning is integrated in my school. 2% 11% 14% 45% 25% 3% Using the ODS portal enables me to provide a better learning experience for my students 2% 3% 14% 40% 18% 23% The added value of using ODS for my students are high. 0.2% 2% 19% 37% 22% 19.8% It is possible to integrate ODS in my current curriculum 2% 4% 15% 37% 23% 19% ODS offers resources which help me in personalising learning for diverse learners 1% 2% 23% 34% 15% 25% The students in my classes are motivated to use digital educational resources 2% 1% 7% 44% 37% 9% There are enough national institutions and departments (in ministries, education centres, school boards, universities etc.) supporting and advising you in the use of innovative teaching approaches and in using Digital Educational Resources 4% 20% 27% 31% 9% 9% My students use ICT tools at home e. g. for homework 1% 4% 15% 42% 27% 11% Improved my teaching approaches. 3% 4% 16% 30% 38% 9% Digital educational resources on ODS fit to my local condition 9% 12% 26% 30% 19% 4% Questions Table 1 Teachers view on the impact of ODS portal on educational system ODS has also had a positive impact on some aspects of learners’ ability to independently and autonomously manage their own learning. Nearly 70% of respondents felt that students were more motivated to work independently using various ICT based resources and tools from home, as a result of ODS use. The majority of teachers who were surveyed (73%) mentioned that they had sufficient access to and familiarity with using ICT-based tools like ODS to plan, prepare, share materials and communicate with learners and other teachers. They were also satisfied with the support provided in relation to technical support and training. They were also pleased with the dependability of ICT equipment and infrastructure and, even more satisfied with the trainings provided to familiarize them with incorporation of ODS into teaching and learning (figure 1). Chart Title Strongly Disagree % Disagree % Neither Agree or Disagree % Agree % Strongly Agree % Cannot Answer this question % 3% 4% 9,00% 23% 19,80% 9% 11% 9% 19% 25% 9% 25% 19% 27% 37% 18% 22% 23% 15% 38% 31% 30% 45% 34% 40% 37% 42% 37% 30% 27% 26% 44% 14% 14% 19% 15% 23% 20% 15,00% 16% 12% 11% 7,00% 4% 9% 4% 5,00% 1,00% 2% 4,00% 3% 2% 2% 2% 1% 1,00% 0,20% ICT-based or ICT-Using the ODSThe added valueIt is possible toODS offersThe students inThere areMy students useImproved myDigital integrated portal enablesof using ODS forintegrate ODS inresources whichmy classes areenough nationalICT tools atteaching educational me inmotivated to useinstitutions andhome e. g. forapproaches. resources on learning isme to provide amy students aremy currenthelp personalising digital departments (inhomework ODS fit to my integrated in mybetter learninghigh. curriculum experience for learning foreducational ministries, local condition school. my students diverse learners resources education centres, school boards, universities etc.) supporting and advising you in the use of innovative teaching approaches and in using Digital Educational Resources 2% 3% 2% Figure 1 : Impact of ODS 4.2. Measuring the ODS relevance to National/Regional Curricula Measuring the relevance of any ICT based or ICT assisted learning tool to the national/regional curricula is a complex task, since the term national curriculum is used in different meanings. The national curriculum might be a “core curriculum”. The “core curriculum” (in some countries “frame curriculum”) is some form of an official, compulsory skeleton of the necessary knowledge base of a pupil completing upper secondary education: the whole vertical primary / secondary education system of a region/state/country. This defines the necessary skills and competencies the pupils have to attain at different levels of education in the different subjects. In general, the subjects (or fields of knowledge) are also defined in the core (frame) curricula (West et al. 1999). 1) In other cases, the national curriculum also specifies the content. 2) In both cases, the school might have some freedom in preparing the local school curriculum. If the school has got this freedom the educational authorities often define how many percentage of the curriculum could deviate from the core (state-wide/national/regional) curriculum. 3) In extreme cases the national curriculum specifies the competence levels, the content and even the textbooks to be used at different grades. Therefore, we first run a short survey asking the teaching community about the structure of the national /regional curricula, with questions concerning the regulatory environment controlling the competence and content requirements embodied in the national/regional curricula of the specific country /region. The general conclusions showed that: a. national curricula exists in all 13 countries responding. In a few cases, however, it is to be adhered to only in state schools. b. The majority of countries use some form of a pyramid like control structure, in which the lower level should satisfy the requirements of the upper level, but might supplement it with additional content/requirements. In some countries the national curricula defines also the content and the textbooks to be used. If the curriculum is that detailed, the school has no flexibility at all, i.e. the school curricula is the same as the national/regional curricula. c. The control structure (in the countries following the pyramid-structure) is multilevel, typically with 3 levels: national (controlled at government level), a medium level, controlled by a relevant education authority and a lower level, controlled by the school / municipality/ council. d. If the school curricula is set at a lower level, the flexibility varies between 10 to 40 %, depending on the country and the subject. 77% of the respondents answered positively to the flexibility question, although the typical flexibility was only 10%. This gives space for innovative solutions in education. In parallel to the above pre-survey, the results of the survey on the impact of using ODS on educational system and learning processes was also evaluated, yielding the following conclusions: In addition to teacher’s personal experience of the impact of ODS on their teaching approaches, majority claimed that ODS is aligned with their curricula and it had potential to contribute to teaching and learning in future. In particular, teachers had positive approach towards portals like ODS to increase flexibility in learning provision in order to prepare learners for future employment. Over 60% of teachers were positive (agree/strongly agree) toward the possibility of ODS integration with their current curricula. Additionally, 70% of participants also agreed or strongly agreed that ICTbased or ICT-integrated learning is integrated in their school. In summary, most teachers participating in the survey indicated that they were confident in their pedagogical use of ODS to plan and prepare lessons, to communicate with learners and other internal and external teachers. The majority also claimed that resources and services provided by ODS are aligned with their current curricula. However, relatively fewer (less than 50%) indicated that they were confident using ODS to personalise their teaching methodologies. Our findings shows that the extent and nature of ODS impact on educational system and measuring its relevancy with curricula were associated more with teachers’ attitude and use of ODS than with contextual factors. The evidence suggested that there was a relationship between the teachers attitude towards using and accommodating ODS with their current curricula and teaching methodology. This was the main reason to motivate and stimulate their students to use ODS more effectively and hence to improve the ODS impact on their school and ultimately on the educational system. Last but not least, taking into account the structure of the national/regional curricula of the different countries/regions, in following survey we will try to differentiate between the impacts on various curricula levels, and on the education system of that specific country/region. We have to keep in mind that education in Europe is a national competence, but the equivalence of the grades attained by the students requires some harmonisation. Figure 2 shows the competence level of ODS teachers. The teachers understanding ICT in education and professional development is significant high. Figure 2 Competence level of ODS teachers 5- Discussion This research is aimed at evaluating the impact of an innovative e-learning portal, namely Open Discovery Space (ODS), on educational systems in the schools across the Europe. To achieve this, this research aimed to evaluate how best the portals and services like ODS are to be accommodated within existing teaching and learning processes as they are seen by teachers and learners. This research was just covering part of the assessment and the final assessment results are expected to be made available to the research community on a continuous basis. Taking into account the early results of this research, It seems that the use of an e-learning portal like ODS, in EU Educational Systems has a great impact on the whole system and on the learning processes, and has the potential to greatly improve the quality of the work in schools. Such an initiative is generally suitable for a large area of teaching practices, while it enables the accomplishment of the daily tasks easily and it offers a wide range of useful and updated learning resources. Furthermore, ODS is aligned with the learners’ curricula and it has the potential to contribute to teaching and learning in the future. Therefore, to be able to evaluate the impact of the portal on educational systems across the EU, requires further data collection of various Key Performance Indicators (KPIs) from different stakeholder’s perspectives Further work is currently planned in order to benchmark the results with previous studies, in order to conduct the qualitative research effectively and detect trends/patterns, and improve the reliability of ODS impact assessment survey results over time and across multiple assessment tools and instruments. Acknowledgement This work is partially supported by Open Discovery Space Project, which is funded by the EU’s programme CIPICT-PSP-2011-5, Theme 2: Digital Content, Objective 2.4: eLearning. The authors gratefully acknowledge the financial support of the European Commission for the presented work, as part of the ODS project. 6- References 1. Alian, M. and Al-Akhras, M.T. (2010) AdaLearn: an adaptive e-learning environment‖, In Proceedings of ISWSA., 2010, pp. 21-21. 2. Alkhalaf, S., Drew, S., Alhussain, T. (2012) Assessing the Impact of eLearning Systems on Learners: A Survey Study in the KSA. Pro-cedia - Social and Behavioral Sciences, Volume 47, pp.98104, ISSN 1877-0428, 10.1016/j.sbspro.2012.06.620. 3. Comber, C., Watling, R., Lawson, T., Cavendish, S., McEune, R. and Paterson, F. (2002). ImpaCT2 – Learning at Home and School: Case Studies (ICT in Schools Research and Evaluation Series No. 8). London: DfES. 4. Doran, G. T. (1981) There's a S.M.A.R.T. way to write management's goals and objectives. Management Review, Volume 70, Issue 11(AMA FORUM), pp.35–36. 5. Fakhimi.M, Stergioulas. L. K, Abbasi. M, Xydopoulos.G, Athanasiadis.N , Damigos.M. Kolovou. L, Sofoklis.S, Rifon, L.A, Fernández.M, Hoel.T, Bissinger. K, Arnold, Stracke, C.M. (2014) Methodology and Evaluation Framework For Measuring Impact Of An E-Learning Portal. 6th International Conference on Education and New Learning Technologies, Barcelona, Spain, 7-9 July, 2014. PP: 2150-2155. 6. Finlayson, H., Maxwell, B., Caillau, I. and Tomalin, J. (2006). Impact of e-Learning on student Immediate and EndPoint Outcomes in Further Education. 7. Fragkaki. M, Stergioulas, L. (2014). ICT in Emancipatory Education: An approach to crisis alleviation, in Proceedings of EDEN, 2014 (forthcoming). 8. Ga-Jin, I., (2007). Developing evaluation tool for elearning. Proceeding of the Portland International Center for Management of Engineering and Technology, Aug. 5-9, IEEE Xplore Press, Portland, OR., pp: 1526-1537. DOI:10.1109/PICMET.2007.4349475. 9. Haddad, W. D., & Draxler, A. (2002). The dynamics of technologies for education. In W. D. Haddad & A. Draxler (Eds.), Technologies for education: potentials, parameters, and prospects (pp. 3-17). Paris: UNESCO. 10. Harrison, C., Comber, C., Fisher, T., Haw, K., Lewin, C., Lunzer, E., McFarlane, A., Mavers, D., Scrimshaw, P., Somekh, B. and Watling, R. (2002) ImpaCT2: The Impact of Information and Communication Technologies on Pupil Learning and Attainment, London: Department for Education and Skills (DfES)/Becta. 11. Ishizaka, A. (2013) Multi-criteria Decision Analysis: Methods and Software. John Wiley & Sons. 12. Kozma, R. B. (2005). Monitoring and evaluation of ICT for education impact: a review, in: Wagner, D.A., Day, B., James, T., Kozma, R.B., Miller, J. and Unwin, T. (Eds), Monitoring and Evaluation of ICT in Education Projects: A Handbook for Developing Countries. Worldbank/InfoDev.Retrieved from http://www.infodev.org/en/Publication.9.html. 13. Scheuermann, F., and Pedro, F. (Eds.). (2009) Assessing the effects of ICT in education: Indicators, criteria and benchmarks for international comparisons. Publications Office of the European Union. 14. Stergioulas, L., Abbasi, M., Xydopoulos, G., Margineanu, R., Fakhimi, M., Rifon, L. A., and Fernández, M. (2014) Evaluating e-learning portals for Schools: Use and usability, user acceptance, and impact on learning. The 14th IEEE International Conference on Advanced Learning Technologies ICALT2014 Advanced Technologies for Supporting Open Access to Formal and Informal Learning. Greece, Athens, July 7-9, 2014. 15. Stracke, Chr. M. (2012) The Evaluation Framework for Measuring the Impact through Quality and Outcome Indicators for Strategic Objectives and Products. eLC / QLET, Essen. 16. Wang, Y., Wang, H. & Shee, D., (2007) Measuring eLearning systems success in an organizational context: Scale development and validation. Computers in Human Behavior, http://linkinghub.elsevier.com/retrieve/pii/S0747563205000890. 23(4), pp. 17921808. Available at: 17. West, A., Edge, A., and Stokes, E. Secondary education across Europe: Curricula and school examination systems, Centre for Educational Research Clare Market Papers No. 14. http://www.leeds.ac.uk/educol/documents/00001195.html 18. Open Discovery Space portal, www.openddiscoveryspace.eu, accessed on 1 September 2014.