Predicting Learner Performance Using a Paired Associate Task in a Team-Based Learning Environment

The Paired Associate Task [1] is a learning task in which participants have to study and recall a list of paired associates (a noun and a digit). Response time and accuracy are measured as a function of the number of exposures to each pair. Over time, accuracy increases while response time for correct answers decreases.

Variants of this task have been simulated in the ACT-R cognitive architecture to create fMRI predictions [2], study the involvement of visual search and associative memory [3], study the fan effect, and the effect of practice [4].

The current study uses variants of this task and an ACT-R model to investigate the benefits of a teaching method: team-based learning (TBL) compared to other more classical teaching methods (lecture and interactive lecture conditions).

Team-based learning [5] is an instructional strategy in which students learn in groups and which has been shown to enhance participants’ performance in certain conditions [6] and increase student engagement [7]. TBL has also been shown to improve the speed of learning [8] with the TBL group scoring higher in early stages of a learning task. Sharing answers and discussing among their group has shown to be beneficial to participants involved in a TBL setting [9]. The cognitive mechanisms that underlie those effects are however yet poorly understood. In an upcoming study, we will try to elucidate the different cognitive mechanisms that are at play in TBL and that might explain the improvement of learning and retention in this condition. In this paper, we use a computational cognitive model to make a priori predictions for an upcoming human study on the performance of human participants to a Paired Associate task in lecture, interactive lecture (IL), and TBL condition.

In the Paired Associates Task [1], subjects have to study and recall a list of 20-paired associates (a noun and a digit). Response time and accuracy are measured as a function of the number of exposures to each pair. The paired associates task has been used to study the relationship between response latency and accuracy. Over time, accuracy increases while response time for correct answers decreases. In its original version, two experiments using classic paired-associate paradigms were performed. The first experiment was a recall task, the second a recognition test. The two experiments illustrated an interesting result: even, when recall accuracy in interference conditions were raised to the level of control conditions by extra study trials, the deficit in response time in the interference condition didn’t change. Response time was thus not varying in the same way as recall, a result that can only be accounted for by theories postulating a difference in encoding and retrieval mechanisms. ACT theory [10] is such a theory, and, in his 1981 paper [1], Anderson demonstrated that its predictions reproduced the human results pattern. While the theory postulates that memory items are encoded completely or not at all, their retrieval relies on continuous quantities that are subject to interference, thus explaining the still affected response time (an indicator of interference), even when the accuracy is not affected. Interference has an effect on retrieval but not encoding.

Variants of this task have since been simulated in the computational cognitive architecture, which implements a theory of human cognition, ACT-R [11]. Borst [2] used a classical paired associate task and a variant, a generated condition to create fMRI predictions. In the generated condition, representational manipulations were involved. Instead of showing the paired-associate directly, a word phrase was given: ‘b-nd –id = adhesive strip’. While manipulation of delay led to a clear difference in response times, there was no great difference in response time between the two conditions. There was also no effect on accuracy. In this specific setting, behavioral data (accuracies and response times) didn’t provide the information necessary to define and restrict the model. fMRI predictions made by ACT-R, however, in accordance with human results showed a clear difference between these two conditions. The improvement of the model with fMRI predictions allowed to improve its significance. The classical condition led to the activation of the prefrontal region (reflecting declarative memory retrievals). The generated condition (involving representational manipulation) led to the activation of posterior parietal cortex (reflecting the update of problem representation). Those two predictions were in agreement with the existing data.

In Halverson and Gunzelman’s work [3], a different type of paired associate task is used: the Digit Symbol Substitution Test (DSST) where a symbol and digit are used (instead of a noun and a digit in the original task), to study the involvement of visual search and associative memory in the task. Two conditions with static (pair-memory variant) and moving pairs further allow the study of the tradeoff between complementary perceptual and cognitive processes and its evolution with learning. Results of the cognitive model here showed that the pair-memory variant relied mainly on “explicit” learning whereas the moving-pair variant relied more on “implicit” learning.

Finally, Pavlick and Anderson [4] used a paired associate to study the fan effect, and the effect of practice. The experiment they performed allowed to investigate the effects of practice and spacing on retention of Japanese–English vocabulary paired associates. Results showed that wide spacing of practice proved increasingly beneficial as the subject accumulated more practice. Different models of practice, forgetting, and the spacing effect were produced and compared to the data. The ACT–R’s activation equation was augmented to make the forgetting rate of a memory chunk be a function of its activation at the time of the presentation. The produced model was compared to existing model of the literature [12] and fitted to existing results. The model required less parameters manipulation to fit the data than other existing models.

In this paper, we also use variants of this task and an ACT-R model. But, on top of an individual Paired Associate task (in lecture and IL condition), we investigate a team version of the task (the TBL condition). We are thus able to compare the benefits of TBL against those other teaching methods (simple and interactive lectures) and study the cognitive mechanisms underlying the performance in TBL and more generally a team setting.

Team-based learning [5] is an instructional strategy in which students learn in groups. In TBL, a high amount of class time is spent in teams, completing team assignments. Students are organized in small groups (that remain permanent during the length of the semester of the course) and the course is organized in major units (five to seven).

TBL occurs in three sequences of learning activities: a preparation phase, an application phase, and an assessment phase. In the preparation phase, students have to study on their own the unit before class. A short test (readiness assessment test) assesses their retention of the key concepts. Application phase has the students apply the course content to solve problems that get increasingly complex. The instructor provides feedback about their responses. Finally, in the assessment phase, the instructor gives a score.

TBL courses are structured around four principles: immediate feedback, accountability, simultaneous reporting, and team development promotion. Immediate feedback is necessary for learning and retention. This aspect is not unique to TBL, it is well documented in the literature of learning in education [13]. Students must be accountable for the quality of their work. Answers have to be reported simultaneously, to avoid the potential impact of a first response in a sequential report, “answer drift”, [14] on all the others students. The chosen assignment must promote both learning and team development.

TBL has been shown to enhance participants’ performance on four aspects: engagement, retention, critical thinking and general performance increase (increased retention for students who usually have lower performance).

Kelly et al. [7] reported student engagement in a seven lectures medical course in which four of the lectures were problem-based learning (PBL) and three were TBL. Student engagement was assessed with the STROBE [15], a test that relies on observational data. Learner to learner engagement was similar in PBL and TBL conditions (but still greater than in classical lectures). Similarly, Clark et al. [16] reported the attitudes of student during a course using traditional lectures and a course using TBL. According to the classroom engagement survey (FIPSE, 2003), student participation was higher in TBL conditions. Dinan and Frydrchowski [17] also reported that students participated more during the courses. And also, that preparation, participation, and attendance all increased in a TBL environment.

Retention is also improved by TBL. McInerney [6] used TBL to improve retention in an undergraduate biology course. The performance of students at the final examination in a TBL condition was compared to a control group results obtained in previous years for the same course content but using classical lectures. TBL condition students performed better in the final exam. In a biology class, observed with two groups (one with TBL one with normal lecture), while not observing the same results in final examination, Carmichael [8] noticed that TBL class scored higher on all tests during the semester than the traditional class, except the final exam where students performed equally well. Kreie, Headrick, and Steiner [18] applied TBL to introductory Information Systems (IS) course (2 groups traditional lecture-based instruction and TBL). Retention and attendance of the course was significantly higher in TBL, and the drop out lower.

Courses given with TBL also seem to improve critical thinking and higher order reasoning in students. Students self-reported that sharing their answers and discussing them among their group has shown to be beneficial to participants involved in a TBL setting [9]. In McInerney’s study [6], notably, a large number of teams offered creative solutions to the problem, and found a solution to solve problem that was neither discussed in lecture nor in the text of the problem indicating deeper learning. Students who usually perform well also reported that they gained new problem solving skills discussing with their teammates.

Similarly, Drummond [19] found that students enrolled in an engineering entrepreneurship course for two semesters (one with classical lectures and one in TBL) had improved critical skills (according to the Critical Thinking Skills (CTS) -Washington State University). And, Carmichael [8] observed that in TBL condition students got better at data interpretations (which rely on critical thinking).

A final benefit of TBL is to reduce performance gap between the students who usually perform well and students who usually have a lower performance [6]. The authors reported that the grade distribution had changed, fewer students had scored under 70 % and more students had scored between 70 % and 90 % than in the control group (course given in the previous year without TBL).

In this paper, we use a computational cognitive model to make a priori predictions for an upcoming human study. Model predictions are generated in conditions identical to those that human participants will be placed in: a classical lecture condition, an IL condition, and a TBL condition. As reported in the literature [6, 8, 18], we expect that the more interactive the lecture setting is the better the performance should be. IL leading to better performance in retention than classical lecture, and TBL leading to better performance than IL (TBL > IL > L). Similarly to what is seen in the literature [6], we also expect our TBL condition to reduce the gap between our distinctly simulated players (differing in the time they allocate to practicing the task at home). Finally, we wish to propose, through our models, hypotheses as to which cognitive mechanisms might explain the differences in performance in the different condition.

The model was built in the computational cognitive architecture, which implements a theory of human cognition, ACT-R (Adaptive Control of Thought - Rational) [11]. In ACT-R, different modules, including two memory modules (procedural and declarative) interact to complete a cognitive task. The modules are accessed via their associated buffers. ACT-R has been used to model several tasks (see [20] for a review).

Declarative memory stores facts about the environment (know what). The procedural memory, thanks to procedural rules (know how) allows for action selection. ACT-R is a hybrid cognitive architecture. Symbolic components are combined to subsymbolic components: the retrieval of a fact from declarative memory depends on subsymbolic retrieval equations (pondering the context and history of retrieval of the fact), and, the selection of a rule depends on utility subsymbolic equations (which computes costs and benefits associated to the rule). The memory elements (chunks) are reinforced through reward patterns that occur within the environment. Learning processes act at both subsymbolic and symbolic levels.

In our model, in the lecture condition, when the model attempts to read an element on the screen during “school time”, its value is held in a chunk pair in the imaginal buffer of the imaginal module. The pair with the value read on screen is held, and, an answer that is later presented will be associated to the value in the pair. The rule that guided this behavior will thus clear the imaginal buffer, leading the pair to enter the declarative memory. If a pair chunk that had the same values associated with is already present in the declarative memory, then it will merge with the existing chunk: its base-level activation will increase. The base level-activation will decrease during the task as a function of time since the last merge.

In the IL condition, the model has the opportunity to attempt to generate a response before being given the right answer. When the pair is stored in the imaginal buffer (as described in the lecture condition), a request will be made to retrieve that pair from declarative memory and eventually give an answer. When the right answer is thus displayed, as in the lecture condition, activation of that pair will increase (due to its merging with an existing pair in declarative memory). But, an additional mechanism is involved: retrieval. Recall that retrieval of an element from declarative memory depends on the subsymbolic components. Every chunk from the declarative memory is associated with an activation, which reflects past experiences. When a retrieval request is made, the chunk that matches the requirement of the rule and has the highest activation will be retrieved. The retrieval of a chunk pair will increase its activation. Therefore, the probability of recall of this pair will increase with every retrieval and merging. Because of retrieval, it should be easier to recall a pair that has been presented in the IL condition than in the lecture condition thanks to the additional retrieval that the model had to perform.

Performance in the TBL condition is supported by one additional mechanism: utility learning. Utility learning in ACT-R supports the choice of the strategy with the best probability of success. While there might initially be a set of competing strategies, as the task goes and the model gains more experience, the utility of the competing strategies will change. For the TBL condition we initiated a set of additional eight rules to govern the reading of another player’s answer and the selection of another player’s answer (one of each of those rules per player, e.g., ‘attend-first-answer-of-player1’ and ‘select-response-of-player1). ‘attend-first-answer-of-player-X’ enables a player to attend the screen location of the answer of a specific player. ‘select-response-of-player-X’ allows the player to chose the answer of that specific player as its own. Initially, those rules were given an equal utility value, thus indicating no initial preferences. But, when the player select an answer which is the correct answer a reward is propagated. As the experiment goes on, since the model will adjust its behavior depending on the success of him following the choice of another player, we expect the utility values to adjust and reflect a preference towards the selection of the answer of the player who usually perform the best. While this mechanism will improve the performance of all players during the lecture phase, it does not account for a better performance in the final session (i.e., exam). A mechanism that was already at play in the two previous conditions will account for the increased performance in the final session. When the player selects the solution of another player, this solution is associated to the word that was placed in the imaginal module. When the imaginal module is cleared, similarly than we previously described, the pair will either enter the declarative memory if it was not already learned by the player or be merged with an existing pair in declarative memory, thus increasing its current activation. Afterwards, similarly than in the lecture and IL condition, the player sees the correct answer which he similarly will place in the imaginal module that will be cleared later on and again the correct answer entered in the declarative memory. In lecture condition and IL condition, this also occurs when the player sees the correct answer but since the player doesn’t look at the answer of other players beforehand, there is one less addition to the declarative memory.

The chunk pair is potentially reinforced in three different ways during lecture in the TBL condition: when the pair is displayed, when the model has to provide a first answer and thus retrieve the pair, and when the model can see and select the answer of another player before he provides a second answer. Utility learning ensures that, over time, the player will select the answer of the better player, who is most often the correct answer. As the player has had one more chance to memorize the correct answer, we expect performance to be better in the TBL condition than in the IL condition.

The TBL condition necessitates that we run the simulations in parallels to allow the models to interact. The procedure used was the one defined by Bütner [21], which allows communication between the ACT-R model in LISP and the Java program that administrates the test over a network using the TCP/IP connection. In each of the three variants (lecture, IL, and TBL) of the Paired Associate task, four ACT-R models run simultaneously (but interacted only in the third variant).

The 40 words used in the paired associate task are selected from the study of Paivio, Yuille, and Madigan [22]. Responses assigned to the nouns were the digits 0-9. The digits are randomly assigned to the 40 words stimuli. Equal duration time is spent in front of the word stimuli during “school time” in all conditions. Each player in each condition has 10 “school time” sessions to test and acquire its knowledge. Models are given the opportunity to practice the task on their own during “home time” sessions. During “home time”, participants can choose to study or play. During “school time”, participants can only study and take tests. Individual differences in the time they allocate to study the task individually were assigned to the models. We assigned 4 different profiles to the players, which determined the amount of time they could study at home. Players 1 to 4 practice the task 100 %, 80 %, 60 % and 40 % of the home time, respectively. Thus, player 1 practices more than twice as much as player 4. This variety of profiles allowed us to emulate the different types of students in a classroom: from students who study more during “home time” to those who study less.

In the lecture condition the four players only receive pairs of words and numbers without the opportunity to give answers. Afterwards, they take a test during which their recall of the pairs is evaluated (final test session).

The IL condition allows the model to test its past learning while learning new pairs in an interactive setting (giving a tentative answer and getting the correct answer immediately).

In the third condition (TBL), an additional stage allows the models to work in teams by interacting with each other. At each round, each model answers and then has the opportunity to see the answers of the other players and change its answer. As in classical TBL, the first answers of the other players are displayed simultaneously. The most frequent second response is automatically chosen as the group response, and feedback (i.e., correct response) is displayed. Players are thus rewarded for picking the right answer and eventually picking the answer of the players who gets the right answer most often. An individual final test is also administered.

We computed the mean accuracy of the players during each round in every condition over 32 artificial players. Figure 1 shows the IL and TBL (first answers and second answers) conditions and the 95 % confidence intervals during “school time” (10 lecture sessions). Figure 2 shows results for final evaluation session (session 11).

Mean accuracy by session for IL and TBL conditions with confidence intervals of 95 %

Full size image

Mean accuracy in final session for lecture, IL and TBL conditions with confidence intervals of 95 %.

Full size image

We were able to confirm our two hypotheses in regards to TBL: final performance was better in this condition and the gap in performance between players was reduced.

As we predicted the model performed the best in TBL condition. Performance of players, as measured by the mean accuracies, was significantly lower during the IL condition than during the TBL condition for the second answer (p < 0.001), the difference was also significant between TBL condition for first answer and IL condition (p < 0.05). Also, during the final session, performance of the players at lecture condition (as measured by mean accuracy) was lower than performance at IL condition, with the difference between the accuracies in final session in the two conditions being significant (p < 0.05). Performance at IL was itself lower than performance at the TBL condition with the difference between the accuracies in final session in the two conditions being highly significant (p < 0.001). With this, we also confirmed that the more interactive the lecture was (TBL and IL condition), the better the final results were, a result that has been similarly observed in literature [13]. Players during lecture condition had lower performance at the final session than during both IL and TBL conditions. This may be attributed to the immediate feedback, which is present in both conditions.

The accuracies in the TBL condition - second answers (Fig. 1) were the highest. It illustrates the player’s increasing ability to pick the answer of another player who usually succeeds as second answer as the experiment goes on. Accuracy in the final session for TBL being higher than IL condition also demonstrates that the player not only does learn to pick the player that has the better performance but also that through this process he reinforces his knowledge (the pair) in the declarative memory of the model. This reinforcement occurs when the player selects the answer of another player, places it in the buffer of its imaginal module. The imaginal module is cleared before the player observes the correct answer, the pair selected is then either entered or reinforced in declarative memory.

Players perform initially lower at lecture sessions in TBL condition than IL condition. There are two explanations for this result. While players in IL have a specific amount of time to provide an answer, players in TBL have this same amount of time to provide a first answer, and select the answer of another player. Players in TBL have thus less time to retrieve their answer. Additionally, players don’t select initially the answers of the better player, they have to learn which player is the better one. In the model, players are rewarded in TBL when they pick the correct answer, this ensures that slowly the model will learn to pick the player that lead him to answer correctly the most. That’s why, it takes them more sessions to perform as well, and even perform a bit higher than players in IL in the 10th lecture session. In the final, evaluation, session however TBL players perform better, indicating a better retention than IL players, a result similar to what has been observed in [6].

Standard deviation was the lowest in the TBL condition. TBL condition is the one where there are the less differences in performance between players of different profile. This is also another result from the literature: TBL reduces the performance gap between the lowest player and the better one.

Through this model we were also able to formulate some preliminary hypothesis concerning the cognitive mechanisms that are involved in the difference of performances in simple lecture courses and TBL courses.

Immediate feedback, which we described earlier, is one of the principles of TBL. The students get frequent and immediate feedback during courses. Similarly, our player gets an immediate display of the correct answer. Our model shows how immediate feedback can support learning and recall by either inserting or reinforcing the knowledge in memory. Another aspect of TBL, simultaneous report of answer forces the student and in this paper our player to recall the knowledge and either create or reinforce its knowledge by eventually afterwards looking at the answer of other players.

Having to recall the answer to produce its own answer before he sees the answer of others also trains the student to develop its own knowledge rather than just apply the strategy of picking the answer of the “better student”. This explains notably why our model did not just learn to always pick the answer of the better player but was also able to perform better than all the other conditions in the final session in which interacting with other players was not permitted.

With this model we were able to make a priori predictions. However, several elements need to be taken into account to make it a complete model of the different cognitive phenomena underlying TBL.

This model provides a gross simulation of the patterns of human results we could observe in our upcoming study. And, we already have some hypotheses on which parameters we will have to work on to fit our model to data. Several parameters could be modified. The rate at which the model learns which players to follow is one of them. It is currently impacted by two parameters, alpha which is the utility learning rate and the reward given to the player after selection. Changing those two parameters would have a strong impact on which players are followed by the others, how quickly the player learns, which player to follow and indirectly the accuracy in the final evaluation session.

Our experiment included four distinct profiles of players (with different engagement degree). This engagement degree was set beforehand and determined the time the player would allow to study during his “home time”. In the future we wish to make this parameter a variable and to study the influence of the different conditions (lecture, IL, TBL) on engagement. We predict that engagement would be the highest in the TBL condition.

A final parameter that we predict will affect performance in the TBL condition is trust. Trust has been shown to play a role in games of strategic interaction [23, 24] and we think that trust is also a parameter which affects the participants choices in the TBL condition (during the selection of the answer of an other player). Trust could notably be used to influence the reward attributed to the player after the selection of another player’s answer.

The Paired Associates Task [1] is a task in which subjects have to study and recall a list of paired associates of a noun and a digit. Response time and accuracy in this task are a function of the number of exposures to each pair and the task has been used to study the relationship between response latency and accuracy.

TBL is an instructional strategy in which students spend a significant amount of class time in teams. Enhancing motivation and engagement, TBL has been shown to lead to better student performance than classical non-IL.

A computational cognitive model was built to test those three conditions and predict results of an upcoming study with human participants. Our model reproduced the results from the existing literature comparing classical lectures and TBL. Performance of players was significantly lower during the IL condition than during the TBL condition throughout the course. And, in final session, performance of the players at lecture condition was lower than performance at IL condition which itself was lower than performance at the TBL condition. TBL also reduced the standard deviation in accuracies between players of different profiles (difference in the times allocated to work during “home time”).

Simulating TBL allowed us to observe interactions at the intra group level and their effect on accuracy during the course and in the final test session. In the future, it will also allow us to make predictions with regard to strategic interactions and the effect of interpersonal dynamics (e.g., trust) in this setting and its influence on student’s engagement.