Pedagogical Considerations in Use of Online Problem Sets in Technical Courses

Paper ID #17753 Pedagogical Considerations in Use of Online Problem Sets in Technical Courses Dr. Carmine C. Balascio P.E., University of Delaware Carmine C. Balascio, Ph.D., P.E., is an Associate Professor in the departments of Plant and Soil Sciences and Civil and Environmental Engineering at the University of Delaware. He earned bachelor’s degrees in agricultural engineering technology and mathematics from UD. He earned an M.S. in agricultural engineering and a Ph.D. double-major in agricultural engineering and engineering mechanics from Iowa State University. He has taught engineering and engineering technology courses in surveying, soil mechanics, site engineering, and stormwater management for over 30 years. He has research interests in urban hydrology, stormwater management, and enhancement of student learning. He is in his thirteenth year of service on Delaware’s Engineering Licensing Board, the DAPE Council, and has been active on several NCEES committees. c American Society for Engineering Education, 2017 Pedagogical Considerations in Use of Online Problem Sets in Technical Courses Abstract This paper discusses the author’s experience with using the LON-CAPA learning management system (LMS) for on-line problem set delivery in seven civil and environmental engineering, engineering technology, landscape architecture, and plant science courses offered at the University of Delaware over more than 14 years. Courses are delivered in-person but with a substantial “flipped classroom” component. LON-CAPA 1 is a free, open-source, LMS developed and supported by Michigan State University. LON-CAPA includes sophisticated testing and assessment components that provide a variety of formats for online homework (HW) problems. A review of literature and the author’s experience form the basis for discussion of the pedagogical considerations associated with use of online problem sets that count for a significant portion of student grades. Advantages of online problem sets include: significant reduction in if not elimination of cheating, better more individualized student/instructor interaction, and increased motivation for students to engage actively with the material and complete assignments successfully. Online problem set characteristics promote these results through immediate feedback, support of problem-based and collaborative learning, and facilitation of mastery-based learning and grading. Keywords: Online problem sets, active learning, collaborative learning, immediate feedback, pedagogy Introduction and Background LON-CAPA (Learning ONline – Computer Assisted Personalized Approach) 1 is web-based open-source course management software that has features comparable to those of well-known proprietary software packages such as Canvas 2, Sakai 3, or Blackboard 4, but with additional capabilities in the form of sophisticated testing and assessment components to support asynchronous learning. Asynchronous learning 5 allows the student to work independently without synchronizing his/her schedule with those of other people or events. LON-CAPA integrates testing and assessment features designed to provide individualized HW assignments, quizzes, and examinations for each student. Distribution of the open-source LON-CAPA software is free through Michigan State University under a GNU license 1. Crucially, all servers running LON-CAPA are interconnected and searchable metadata are available for all resources existing on the linked LON-CAPA servers. Any users with author privileges can search for and use any of the resources on the numerous LON-CAPA servers in their own courses. This paper focuses on the pedagogy associated with implementing the LON-CAPA online HW system in the author’s courses. The results are not limited to use with the LON-CAPA system, however. Most of the strategies and practices discussed will be applicable in the use of some competing online learning management systems such as WileyPLUSTM 6 or Pearson’s MasteringEngineeringTM 7 that have some similar features. LON-CAPA runs on the LINUX operating system. Perl, a powerful open-source interpreted language native to LINUX and UNIX operating systems is used to code problems within the author role of LON-CAPA. The authoring role in LON-CAPA is very flexible. Problems can range in pedagogical complexity from simple numerical or quantitative solutions to responses that require sophisticated conceptual understanding and reasoning. Authors may code a wide variety of problem types that include single or combined numerical, symbolic, logical, graphical, matching, multiple choice, and essay features. Links to external resources can be included in the problem keys, as well as dynamic animations and clickable graphics for student responses. Students are given individualized problems through computer randomization of parameter values and even the set of parameters themselves. Author-programmed hints can provide students with immediate feedback on their erroneous submissions. Kashy et al.8 examined use of CAPA, an online but non-web-based predecessor of LON-CAPA, for HW problem delivery in large introductory physics classes. They investigated the impact of CAPA on student exam performance while factoring in the effects of student characteristics such as gender, grade point average (GPA), and ACT scores. They found that the ‘technology can have a profound impact on learning if it is used in a way that capitalizes on its unique ability to “interact” with students, provide them with immediate feedback, and facilitate interactions among students and between students and teaching staff.’ There was evidence that female students benefited even more than males. Hedgcock and Rouwenhorst 9 looked at feedback in the context of using student response systems (or “clickers”) in business and marketing classes. A key component of such systems is the immediate feedback they provide and the way immediate feedback promotes student engagement and understanding. Hedgcock and Rouwenhorst 9 found that with the use of clickers and immediate feedback “students reported a better understanding of the materials, read more chapters before class, were more likely to recommend the course to others, and had higher exam scores” than when immediate feedback was not available. Use of immediate feedback is not without potential problems, however. Pascarella10 discussed the importance of immediate feedback and aspects of web-based homework systems that may encourage detrimental student behaviors. Simple correct/incorrect feedback with no indication as to what the student did wrong is not particularly helpful and may promote trial and error solution attempts. Black and Wiliam 11 reviewed the literature on classroom formative assessment and also provided a detailed discussion of feedback. Black and Wiliam 11 emphasize the quality of feedback. They maintained that feedback is most effective when it promotes thoughtful consideration and correction of errors within the total context of the problem and key principles it is meant to reinforce. As noted previously, LON-CAPA does give the problem author the ability to program hints for erroneous solutions; but additionally, the hints can vary in complexity from non-adaptive generic responses specific to a particular problem to responses that can be programmed to address adaptively different types of errors that might be expected in the student submissions for a given problem item. Problems that include the capacity for adaptive responses to error types are considerably more difficult to develop and program, however. This author has chosen not to write problem items that include provision of adaptive feedback. Instead, pedagogical best practices are employed that minimize any potential negative impacts of immediate feedback with respect to student use of trial and error solution strategies. Since the origins of the LON-CAPA system were in physics applications, a number of studies have examined use of LON-CAPA in that context. Pascarella 10 considered student learning styles in large introductory physics classes and how those learning styles were associated with student solution strategies for HW problems delivered in two different formats: traditional handgraded HW and online HW with CAPA. Pascarella 10 found that a relatively small number of allowable attempts for the online HW would promote deeper learning for all learning styles and would prevent students from using trial and error solution strategies. Kortemeyer 12 considered the effect of multiple tries for online homework in large calculusbased introductory physics courses. Some key conclusions from his work were: • “The probabilities of successfully completing or abandoning problems during a particular try are independent of the number of tries already made.” This may indicate that students are not learning from earlier tries 12. • The likelihood of successfully completing a problem during any given try decreases as the number of tries allowed increases. This result is likely caused by the decreased “cost” of additional incorrect tries. Carelessness and guessing are less expensive as the number of allowed tries increases 12. • The likelihood that a student will give up after any particular try “is largely independent of the number of allowed tries.” 12 In light of these findings, Kortemeyer 12 developed a mathematical model that pointed to five as the optimal number of multiple tries – at least for those problems in the introductory Physics courses he investigated. This author previously examined pedagogical difficulties associated with conventional HW assignments 13. Before the author began using LON-CAPA in his courses, he would assign four to six HW problems per week that were not collected for immediate grading. In an attempt to provide immediate feedback to students, the problem answers were posted with the problem keys. Students were advised that a few of the HW problems assigned since the last exam would be collected during the next exam for grading. Since the students already had they answers to the problems, they were required to show detailed solutions to demonstrate their understanding of the material. HW problems collected during an exam would typically amount to approximately 30% of the exam grade. Balascio 14 noted a number of undesirable outcomes from this policy: • Students would put off doing the HW problems until immediately before the exams and, hence, did not get the benefit of working with the material in a timely manner right before the related ideas and principles were to be applied in class as a springboard to more advanced concepts. • Students would get hung up on small misunderstandings or misperceptions while working on problems by themselves and simply give up. Many would not bother to seek help from the instructor even with constant encouragement to do so. • Students would collaborate on solving the problems, which in itself is not bad; but typical student behavior would short-circuit the intended learning process. (Unauthorized collaboration was evident when, in particular, identically erroneous solutions were occasionally found for several students amongst the HW collected for exams.) When collaboration degenerates into simply disseminating the HW solutions, the students receiving the information derive no benefit; and standards of academic integrity are violated. • Since students knew the answers to the assigned HW problems, they became adept at “reverse engineering” the solutions. While reverse engineering requires some ingenuity, it bypasses the learning the student is intended to derive from working the problems the “right way.” • A surprising number of students were willing to “take their chances” by doing only some of the assigned problems or just the easier ones. They were willing to gamble that either the problems they skipped would not be on the exam; or if the skipped problems did appear on the exam, they would be able to muddle their way through during exam administration. • Because students who were not motivated to do the HW problems had not actively engaged with the technical material for a sufficient amount of time, they were unprepared to work new exam problems similar to the HW. 14 These issues instigated a search for a more effective way to promote active learning through assignment of HW problems. Implementation of the LON-CAPA System As noted previously, CAPA and later LON-CAPA were originally developed by physicists for use in large lecture introductory physics courses; but their use quickly spread to other math and science disciplines 1. In addition to its free distribution, the interconnectedness of the LONCAPA servers and the availability of resources to all author-users have made it particularly attractive to instructors who may not have the programming skills or time to develop their own problem items. This feature has encouraged use of the LON-CAPA system at over 160 institutions world-wide 1 that range from middle school to graduate colleges, but published resources are still primarily in math and science subject areas. There has been a paucity of engineering-related resources. Out of necessity, this author has written and published on the LON-CAPA system over 200 HW problems in subjects ranging from surveying, hydraulics, and hydrology to site engineering and soil mechanics that are now available to all authors on the LON-CAPA system. The original audience for the author’s courses was primarily students in the Department of Bioresources Engineering’s 4-yr engineering technology (ET) program. Students in the surveying courses often included a mix of engineering, ET, and plant science majors. A large proportion of the ET students were non-traditional older students who had 2-yr ET degrees from nearby community colleges and who often were working full- or part-time. The author’s stormwater management course subsequently began to attract significant numbers of civil and environment engineering students. For the last five or so years, the stormwater management course has been required for the Department of Civil and Environmental Engineering’s (CIEG) environmental engineering students doing the water resources and water quality concentration. With the eventual demise at the University of Delaware of the Department of Bioresources Engineering and the ET program, and the transfer of the author to the Department of Plant & Soil Sciences (PLSC), the profile of students in the author’s classes has shifted considerably. For the past several years, teaching of ET students has phased out and the author’s courses have shifted to support PLSC students in plant science and landscape horticulture and design majors. The stormwater course now serves environmental engineering students almost exclusively, but that will change. Recently, the Department of PLSC has developed a professional program in Landscape Architecture that began admitting its first students in fall of 2016. The author’s courses in surveying, site engineering, and stormwater management are all required for the new landscape architecture major. For landscape architecture majors, the site engineering course is the prerequisite for a revised version of the stormwater management course; while a new course in urban hydrology and drainage design was created to serve that function for environmental engineering students. The new urban hydrology and drainage design course replaces the old stormwater management course in the environmental engineering curriculum. All the author’s courses, now housed within the department of PLSC, are cross-listed as CIEG courses and may be taken as technical electives by students enrolled as majors in the CIEG department’s programs. The author initiated use of LON-CAPA for his courses in the Department of Bioresources Engineering at the University of Delaware in fall 2002 with support from a grant awarded by the university’s Center for Teaching Effectiveness. The objective of the change to online HW sets was to address the problems with conventional HW identified in the previous section. LONCAPA allows for a rich mix of problem formats. Different problem formats may be combined within single or multi-part problems. Balascio 13 examined strategies for designing combinations of problem formats to promote problem complexity and student understanding of concepts. Balascio 14 describes the evolution of HW problem development that started with simpler-toprogram surveying problems and progressed through more difficult problems in hydraulics and hydrology for his stormwater and site engineering courses through the most difficult problems in soil mechanics. Hydraulics problems in particular frequently require use of subroutines employing induction, iteration, and recursion to solve for quantities such as normal and critical depths in openchannels and to calculate water surface profiles. Problems were added in 2008 for a soil mechanics course first taught in 2006. Soil mechanics problems for some of the material in that course were never converted to online delivery because of the programing complexity involved in coding the empirical graphical solutions required for some topics. Unlike in years prior to the use of LON-CAPA, the conventional HW problems were all collected for hand grading. In spring of 2014, the author taught another new course, Site Engineering, in which a mix of handgraded HW and LON-CAPA HW was utilized. Kashy et al. 8 did not have a way to make direct comparisons with conventional hand-graded HW. In the author’s courses, examination of conventionally delivered HW and LON-CAPA HW problems in his soil mechanics and site engineering courses allowed for some limited comparisons of the characteristics of the two approaches. Balascio 14 looked at several years of these data for indications of the effectiveness of online problem sets in comparison to conventional hand-graded homework. Positive relationships were found between completion of problem sets, whether online or conventional, and performance on exams. There was some indication that online problem sets provided a boost in raw exam scores over that associated with conventional problem sets; but while the relationships were statistically significant, predictive value was poor because of low R2 values typical of the highly scattered data that were in evidence 14. Figure 1 is the student view of a LON-CAPA problem that integrates graphical content within the problem statement. Note that every student would have somewhat different numbers and graphics. In this instance, a numerical solution is submitted, and the computer would grade the Figure 1. Student view of soil mechanics problem for particle grain size distribution illustrating integration of graphical and numerical components. problem correct or incorrect based on the submitted answer falling within an error bound set by the problem’s author. For example, the coefficient of uniformity in Figure 1, part b, the value of which is approximately 7.29, would be graded correct for any entry between 6.93 and 7.65. That range corresponds to an error bound of + 5% chosen for this particular problem. To address the concerns of Pascarella 10 about students attempting trial and error solutions to guess the answers and in keeping with the recommendations of Kortemeyer 12, students are initially allotted a maximum of five attempts to get the problems correct. Note in Figure 1 how each part of the problem displays the number of tries already used – 0 being displayed initially. Figure 2 shows the computer response to an incorrect submittal for part b of a different problem. An instructor-programmed hint is provided in the grey box for part b that has been marked as incorrect. One out of the five tries is shown as used. The student may click on the “Previous Tries” hyperlink to see all previous (incorrect) submittals. Note that as indicated in Figure 2, units can be required for the numerical submittals for any problem if desired. If no units, incompatible, or non-standard units had been submitted, the system would have responded appropriately with a “units required”, “incompatible units”, or “unable to interpret units” message. No tries would be assessed. Figure 2. View of LON-CAPA student interface with computer responses to correct and incorrect submittals. If a student exhausts the quota of five tries for a particular problem, the problem can be reset by the instructor or teaching assistant. Before problems are reset, the students are required to communicate with the instructor (or a teaching assistant) about the approaches they are using to solve the problems; so the instructor can assist the students in determining what they might be doing wrong. Following CAPA/LON-CAPA developer Edwin Kashy’s recommendations (personal communication circa 2002), there is no penalty associated with the number of tries a student uses to solve the problems, thus giving the student an incentive to continue working on the problems until they are correct. Discussion Balascio 13 examined use of the LON-CAPA system as a component for delivery of hydraulics and hydrology content in a problem-based learning (PBL) format – an approach that was gradually adapted for use in all the author’s courses. Balascio 13 discussed the pedagogical considerations involved in the design and programming of the LON-CAPA problems and identified benefits from using LON-CAPA coupled with PBL methodology that included: • Students have unique numbers for their problems. Cheating is difficult. Students can be encouraged to discuss with one another the concepts required to solve the problems but cannot simply give one another the answers. To counteract the possibility that students might circulate solution algorithms, small variations in the problem statement or supplied information can be randomized so that development of a generalized solution algorithm is much more difficult. Boland and Jacobs 15 provide additional discussion of this point. • The system provides immediate feedback by indicating whether submitted answers are correct or incorrect. Immediate feedback is not possible with hand-graded assignments unless answers are posted beforehand. As noted previously, with posted answers, the danger is that students become adept at "reverse engineering" the solutions without really mastering the thought process required to solve the problems from scratch. The LON-CAPA system can provide further feedback by displaying instructor-programmed hints that can help guide the student in the right direction. An anonymous student survey about the advantages of the LON-CAPA system yielded these typical responses asking for a comparison between LONCAPA and conventional HW: o The LON CAPA problems were more beneficial to my learning. They gave you hints and told you if you were right or wrong. o If I were to [do the] textbook problems, I would probably not know the correct answer and move on to the next problem without thinking too much about it, where with the LON CAPA you know when you’re wrong and can work it through…. • Computer grading makes mastery based learning feasible by not requiring significant instructor grading time. Multiple attempts encourage perseverance so that students will work on the problems and remain engaged in active learning until the solutions are obtained; whereas with conventional hand-graded assignments, students may simply give up or be unaware they are doing problems incorrectly until graded problems are returned. Students may or may not make an effort to determine the correct solution procedures for hand-graded problems marked wrong. • Increased student/instructor interaction allows the instructor to “coach” students with appropriate assistance. Socratic questioning can be used to help students correct their mistakes. Students spend less time in unproductive (and frustrating) confusion. LON-CAPA gives the instructor the ability to check values of intermediate calculations required to arrive at the solutions of each student’s problems. This feature makes it possible quickly to identify the concepts with which the student is experiencing difficulty. Misconceptions and gaps in knowledge can be readily addressed. • Collaborative learning is encouraged. Kortemeyer 16 found that the collaborative learning features of online homework systems were particularly advantageous to female students who were more likely to discuss solution strategies with peers before submitting answers. With LON-CAPA, all students have somewhat different problems, so they can discuss amongst themselves the approaches to solving the problems but can't give each other the answers. Such mutual learning interaction between students is beneficial because students will either be required to articulate their knowledge of a subject in ways that another student can understand or will profit from getting an alternative perspective from a peer on how to approach a problem. For the author’s courses, students are provided a variety of ways to demonstrate learning of the course material. The HW problem sets have typically counted for around 25% of the course grade. All courses include a laboratory and/or project component that counts for about 25% of the course grade, while two exams and a cumulative final count for the remaining 50%. Since the LON-CAPA HW problems are mastery based and students can get assistance from the instructor, students can realistically expect to get full credit for all the LON-CAPA problems if they persevere and complete the problems by their deadlines. With the problems counting as a quarter of the course grade, student should have ample motivation to do so. Historically, there has been much variation in percentages of HW problem completion with some students completing all problems while others do few if any. There can be considerable variation in completion percentage of HW with quality of the student cohort. At the author’s institution with a class composed of motivated students from a highly selective environmental engineering major, HW completion percentage have been averaging about 93% for the past several years. For the same class composed of students from a less selective non-engineering major, HW completion percentages were on the order of 80%. Latest data indicate an overall average HW completion percentage of around 84% for all courses and all majors taught by the author. Over the years, the author’s pedagogy has moved in the direction of the “flipped classroom” 17 with more classroom time devoted to active learning as opposed to lecture. Typically, there is significant class time allotted for solution of the online problem sets. The instructor is freed to devote high quality personalized time with one or a few students at a time providing focused individualized instruction that helps the students overcome misconceptions or gaps in learning. With increased class time devoted to problem solving, an upward trend in completion percentage of the homework sets has been seen. As noted earlier, one LON-CAPA system tool that is particularly helpful in providing individualized assistance to the students shows values of intermediate results. The instructor has access to a screen that displays the values of all the script variables used in the Perl-coded problem solution. If descriptive variable names are used in the Perl code, it is easy to determine the values of intermediate results that are needed for the problem solution. The instructor can quickly pinpoint where students have erred. Figure 3 lists the script variable values for the problem displayed in Figure 2. There were some issues initially confronted by the author that were not expected. It was quickly found that students must be encouraged or even required to record their solutions to the problems. Some students will treat the problem sets as a type of game with the only objective being to complete as many problems as quickly as possible. The five-try limit on attempts addresses this dysfunction to some extent with respect to discouraging trial-and-error attempts, but students may still be tempted to punch out the numbers rapidly on their calculators without recording the solution procedures. Students, will, however, want to have the solutions recorded so they can return to the problems for review when studying for exams, for example. The instructor must stress the importance deliberation plays in the learning process and the need to have a record of one’s thought process. Figure 3. List of Perl-code script variables for problem displayed in Figure 2. Another issue is enforcement of attendance. If during a surveying laboratory, for instance, the field work is done after the first two hours of a three-hour lab, students are required to use remaining laboratory time to work on the problem sets. The same issue can arise during lecture periods that may have, say, the last half of the class devoted to work on the problem sets. Some students will try to leave before the end of class even though they have not completed the problem sets. It was observed that such students, who were, perhaps, academically weaker or less disciplined, would struggle with finishing the problem sets on their own – even if they had actually devoted much out-of-class time to that effort. As a remedy, a policy was implemented that allows students to leave class early only if all assigned problems are completed. The instructor stresses that he is there as a resource, one that is not conveniently available outside of class, and that he is willing, indeed expects, to answer questions and provide advice that can help the student learn the material. In end of semester student course evaluations, the author’s classes are typically ranked high in work load; enforcement of the attendance policy makes full use of class time and helps minimize out-of-class demands on student time. For the author, the shift away from time spent lecturing to more personalized time coaching individual students has been particularly rewarding. It becomes possible to get a feel for each student’s thought processes, strengths, and weaknesses and to tailor Socratic responses to student questions that are uniquely beneficial. While some aspects of this approach could be incorporated using traditional hand-graded homework, the advantages are greatly enhanced with an online HW system such as LON-CAPA. Hand-graded HW puts considerable demands on instructor time without the advantages of immediate feedback. In a “workshop” type of class period designed for the instructor to work individually with students doing traditional offline HW, the instructor must take valuable time verifying correctly-done hand-graded assignments and is, thus, sidetracked from more productively providing coaching to individual students who have made errors and most need the instructor’s assistance. Posting answers to assigned traditional HW problems for immediate feedback and requiring detailed solutions is not a realistic approach. Besides the problems previously described with posting answers to traditional HW problems, Black and Wiliam 11 noted that effects “of feedback were reduced if students had access to the answers before the feedback was conveyed.” The promotion of collaborative learning is another positive aspect of the online problem sets the author has witnessed frequently and has utilized to advantage. The instructor can be spread quite thin in answering individual questions from a large class. When one question or issue is repeatedly brought up by several or more students, it is most efficient for the instructor to address that topic with the whole class, but collaborative learning can have an important role to play. The author encourages his students to talk with one another about the problems and solution strategies. Peer coaching is an important supplement to what the instructor is able to provide; and as noted previously there are important benefits to both the peer coach and the coached. This opportunity may be particularly beneficial to females who are more inclined to engage in collaborative learning activities 16. Conclusion There are numerous benefits in use of an online tool such as LON-CAPA for delivery of course problem sets and no perceived negatives when combined with good pedagogy. Among the benefits are • Minimized potential for cheating, • Immediate feedback to students, • Feasibility of mastery-based learning and reduction in grading demands on instructors, • Increased student/instructor interaction with the instructor in the role of coach, and • Increased opportunity for collaborative learning among students that may be particularly beneficial to females. When used in conjunction with active learning strategies such as the flipped classroom, online problem-set delivery such as that provided by the LON-CAPA system is clearly beneficial. While the availability of resources published on the LON-CAPA system has historically centered on math and science fields, there is no reason more engineering-related resources should not be developed, thus allowing use of online problem sets in engineering courses to spread. References 1. Michigan State University. The LearningOnline Network with CAPA. The LearningOnline Network with CAPA (2013). Available at: http://www.lon-capa.org/. (Accessed: 18th July 2016) 2. Instructure. Learning Management System | LMS | Canvas by Instructure. Canvas (2016). Available at: https://www.canvaslms.com/?lead_source_description=instructure.com_&__hstc=19918808 0.1f67a4149cf116b339992468f494d75a.1468857318684.1468857318684.1468857318684.1 &__hssc=199188080.1.1468857318684&__hsfp=2361671978. (Accessed: 18th July 2016) 3. The Sakai Project. Home | Sakai. Sakai (2016). Available at: https://sakaiproject.org/. (Accessed: 18th July 2016) 4. Blackboard, Inc. Blackboard | Reimagine Education | Education Technology & Services. Blackboard (2016). Available at: http://www.blackboard.com/. (Accessed: 18th July 2016) 5. Mayadas, F. Asynchronous Learning Networks: A Sloan Foundation Perspective. J. Asynchronous Learn. Netw. 1, 1–16 (1997). 6. John Wiley & Sons, Inc. WileyPLUS. (2016). 7. Pearson Education, Inc. MasteringEngineering / MasteringComputerScience | Pearson. Available at: http://www.pearsonmylabandmastering.com/northamerica/masteringengineering/index.html. (Accessed: 19th September 2016) 8. Kashy, D. A., Albertelli, G., Kashy, E. & Thoennessen, M. Teaching with ALN Technology: Benefits and Costs*. J. Eng. Educ. 90, 499–505 (2001). 9. Hedgcock, William H. & Rouwenhorst, Robert M. Clicking Their Way to Success: Using Student Response Systems as a Tool for Feedback. J. Adv. Mark. Educ. 22, 16–25 (2014). 10. Pascarella, A. M. The Influence of Web-Based Homework on Quantitative Problem–Solving in a University Physics Class. in Proceedings of the NARST 2004 Annual Meeting (2004). 11. Black, Paul & Wiliam, Dylan. Assessment and Classroom Learning. Assess. Educ. Princ. Policy Pract. 5, 7–74 (1998). 12. Kortemeyer, G. An empirical study of the effect of granting multiple tries for online homework. Am. J. Phys. 83, 646–653 (2015). 13. Balascio, C. C. Use of Web-Based Testing Software for Problem-Based Learning in Hydraulics and Hydrology. in Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition (ASEE, 2004). 14. Balascio, C. C. A Dozen Years of Asynchronous Learning: Using LON-CAPA for Online Problem Sets. in Proceedings of the 2015 ASEE Annual Conference & Exposition (ASEE, 2015). 15. Boland, C. & Jacobs, F. How Personalized Assignments Can Increase Learning and Reduce Cheating in a Large-Lecture Introductory Accounting Class. (2014). doi:10.2139/ssrn.2413195 16. Kortemeyer, G. Gender differences in the use of an online homework system in an introductory physics course. Phys. Rev. Spec. Top. - Phys. Educ. Res. 5, 010107 (2009). 17. Millard, E. 5 Reasons Flipped Classrooms Work | University Business Magazine. (2012). Available at: http://www.universitybusiness.com/article/5-reasons-flipped-classrooms-work. (Accessed: 8th July 2016)