Edited by: Xiaoxun Sun, Australian Council for Educational Research, Australia
Reviewed by: Charles Crook, University of Nottingham, United Kingdom; Yen-Teh Hsia, Chung Yuan Christian University, Taiwan
This article was submitted to Digital Education, a section of the journal Frontiers in Education
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
The research team designed and evaluated a mobile game to promote rapid retrieval of arithmetic facts among a group of children aged 7–8 years (
This article describes the principles underlying the design of a mobile game designed to promote arithmetic fact retrieval among primary school children, and an evaluation of the game in a classroom setting. The game was developed as part of the Research Adaptivity to Individual Differences in Number Games (RAIDING) project. This work draws on research in mathematical cognition, in game-based learning, and in game design, to create an engaging game that develops learners' performance in solving simple arithmetic problems. In this section, we review the literature that contributed to the rationale for the game design and its evaluation. We focus on those areas most strongly related to mathematical learning and so discuss two main bodies of work; research relating to arithmetic fluency and fact-retrieval, and research relating to game-based learning. Readers with an interest in the game design aspects of the project should see Mees et al. (
A fluent number sense involves the rapid retrieval and manipulation of stored information relating to number. For example, when presented with the number “6,” people who have had the opportunity to learn about such relationships will automatically think of 5, 7 (due to proximity on the number line) and 12 (double 6). When presented with “5+2,” most people will automatically think of “7.” Many of these fact-retrieval processes are unconscious, and outside the control of learners, but appear to be important components of children's learning of mathematics (Jordan et al.,
Adults typically solve single-digit multiplication problems by retrieval. The proportion of such problems that children solve by retrieval gradually increases with age. Lemaire and Siegler (
The research reported in this article is partly motivated by concerns that significant numbers of children are being “left-behind” at the point where their number knowledge can no longer keep up with the demands of the National Curriculum in England. The UK government has recently introduced a pilot of times-tables screening in Year 4 of primary school (8 to 9-year olds). They say that, “As well as being critical for everyday life, knowledge of multiplication tables helps children to solve problems quickly and flexibly, and allows them to tackle more complex mathematics later on in school” (DfE,
Verguts and Fias (
“
Verguts and Fias use the example of someone perceiving the problem “7 × 4.” As well as triggering representation of “28,” studies show that this problem will also activate representations of the problem's “neighbor” including “7 × 3,” “7 × 5,” “6 × 4,” and “8 × 4.” Galfano et al. (
Alongside rehearsal of multiplication tables, teachers in primary schools also rehearse number bonds; children are encouraged to learn the pairs of numbers that add to give 10, or 20 (Department of Education,
Despite its prevalence in classrooms in England, the learning of number bonds has been the subject of far less research than the learning of multiplication tables.
Video games offer an extremely promising environment for promoting number sense, as they can provide a high volume of number combinations and manipulations to the learner in a short time. Furthermore, they can adapt in their complexity to the current level of understanding and performance of the learner, while providing continuous and patient feedback (Butterworth et al.,
Video games also have great promise as a tool for researching children's developing numerical cognition, due to their ability to record high volumes of data relating to children's responses to numerical tasks. An appropriately designed game can be used as an ideal context for the development and extension of methods capturing very small changes in children's learning—so-called microgenetic methods (Siegler and Crowley,
The game was designed around touch-screen controls on (7-inch) tablet devices. It is set in outer space, and the aim is to collect and preserve alien life within the biodomes of a central mothership. The player controls a small flying robot, which can explore an area of space around its base in search of seeds and eggs to nurture in the biodomes. To support the mission, the player must earn credits to buy new components for their mothership, including biodomes, power sources and engines. These “building and collecting” mechanics fuel the economies of the game and motivate the player to continue playing. Fundamental to all of these activities is the “mining” mechanic which provides the player with credits, seeds and eggs. The player's robot must mine asteroids for resources that are converted into credits, but mining also provides the chance to find alien seeds and eggs within the asteroids. There are 100 different aliens to find in total, and each alien prefers a biodome with a particular combination of climate and plants. The inclusion of a variety of collectable elements within the game allowed for multiple reward schedules, including both linear and non-linear performance/reward contingencies; there is a linear relationship between mining activity and credits (for spending) but non-linear relationships between mining activity and receipt of both seeds and eggs, with eggs being received less often than seeds. Previous research has provided strong psychological evidence that non-linear, or uncertain, rewards are a particularly motivating aspect of games for learning (Howard-Jones and Jay,
Each time an asteroid is approached within the game, the player has the option to begin one of the minigames which requires them to “mine” numbered rocks according to a particular mathematical rule. The minigame will focus either on multiples of a target number or number bonds to a target number, and the player has the option to accept or reject the game. In
Screenshot from RAIDING game, 2-times-table mini-game.
The mathematical content of each minigame is determined by a dynamic learner model within the game. A key part of the game design is the adaptivity provided by this learner model (Butterworth et al.,
A key goal of the game design was to ensure that mathematical tasks were an intrinsic component of the “game mechanics” (the repeated activities of a game from which motivating gameplay is derived), rather than using the gameplay as a sugar-coating for mathematical content delivered outside of the gameplay (Habgood and Ainsworth,
The metagame does contain some opportunities for mathematical thinking and learning—in making decisions about how to spend credits earned, for example—but its main purpose is to promote and support players' investment in, and engagement with, the game. We argue that this is not the same as a “sugar-coating” for the mathematical content of the mining minigames because even though mathematical thinking does not pervade every aspect of the game, where mathematical thinking does occur it is intrinsically part of the game mechanics rather than a “bolt-on” component with no relation to the rest of the game.
Mathematical content is an intrinsic component of gameplay.
The game is
Sessioning, return triggers, and multiple linear and non-linear reward schedules are employed to increase engagement.
The learning content is designed according to theory in mathematical cognition, training associations between single digits rather than training responses to arithmetic problems.
We carried out a first trial of the RAIDING game, to assess its effectiveness in improving arithmetic performance among a group of 7 to 8-year-old children. We worked with two schools, and all children within the target age-range in both schools took part in the intervention (two classes in each school). Children played the game for 20 min per day for 2 weeks.
The study employed a between-groups crossover quasi-experimental design. In each school, the two classes were randomly chosen to be group A or group B (see
Experimental design.
The crossover design employed in this study allows all participants in the study to experience the game-playing intervention. This was important to the research team for ethical reasons. The comparison between groups at mid-test stage was the primary analysis as this was a fair test between the game-players and controls. In the second stage of the study, after crossover, group A became the controls. However, having played the game for 2 weeks, group A were not a perfect control group. Analyses comparing the groups at post-test were still carried out, in order to see whether group B improved by the same degree and to see whether group A retained any improvement for the first stage—but these should be interpreted with some caution.
Participants were 97 children aged 7 to 8 years-old in two primary schools. In each school, children were in two mixed-ability classrooms of 30. Participants were those children from these classrooms that met the criteria for eligibility—that they attended for pre- and mid-tests, that they did not have a learning difficulty that prevented them from accessing the game or the pre-/mid-/post-tests, and that they could use a touchscreen device and access the game. Both schools were in broadly average localities with regard to socio-economic status, and children in both schools had broadly average levels of mathematics attainment in national tests.
All 97 participants (49 in the game-playing group, and 48 controls) were included in analyses of comparisons for the first stage (comparisons between pre- and mid-tests). Seven participants were excluded from comparisons between mid- and post-tests due to absence, leaving a total 90 participants for the second stage (45 in each group).
The Westwood 1-min basic facts tests (Westwood,
Additional data were collected during game-play. Actions carried out by players within the game were automatically recorded by the software, including: total duration of play; number of games played; number of rocks selected; and response time and accuracy for each rock selection. These data were used in order to assess the relationships between in-game actions and progress and improvement in arithmetic fluency assessed through pre-/mid-/post-test comparisons.
The pre-tests were administered during the week before the start of the intervention. The explanation of the trial, and pre-testing took ~20 min with each group.
In the week before the study began, the tablets used for the project were numbered. A register sheet for the class was generated, based upon the class data gathered at pre-test, assigning each child a tablet. The game was installed, and the tablets were then placed on charge in preparation.
The children began playing the game a week after the pre-test. Each session of gameplay lasted 20 min and there was one session each day over a period of 2 weeks. This resulted in 10 sessions and a total of 200 min of gameplay.
In one school, the gameplay session occurred before the lunch break. The second school had their gameplay sessions at 9 a.m., just after the morning register had been taken and before the mathematics lessons for that day. If a child missed a session, for whatever reason, then they could complete a catch-up session the next day. If a child missed too many consecutive sessions, they would be noted down to allow for the removal of their incomplete data, and then allowed to just play the game at their own pace.
In the initial session, all members of the research team were present. We again briefly explained the project. We then handed out the tablets to the group, with each child receiving their designated tablet. In the event of children being missing from the pre-test, we had allowed for extra room on each register sheet to allow these children to be added. We ensured that additional tablets were available, in the event that we did need more for the group or in case of equipment failure. After the end of the first 20-min session, there was a short debrief to make sure they understood the overall goals of the game.
During the course of the 2 weeks, a researcher from the team attended in order to provide technical support and assistance including maintaining and charging the tablets. The researcher attended on the first, second, third and fifth days of the first week, and on the first, third and fifth day of the second week.
At the end of the final session of gameplay, in which all members of the team attended, we repeated the testing procedure as in stage 1 with both groups. Testing of the gameplaying group took place after they had finished playing the game that day.
The research team returned to the schools on the immediate Monday after the mid-test. At this point, group roles were switched and group B were introduced to the game. For the next 2 weeks the same procedure as in stage 2 was followed.
After the final session of gameplay, the same testing procedure as in stages 1 and 3 was followed.
In this project, we referred to the ethical guidelines of the British Educational Research Association (BERA) (
Data analyses were carried out in three stages. Firstly, the improvement in arithmetic performance between pre- and mid-test was compared between pupils in the game-playing group and pupils in the control group. This analysis tested the primary hypothesis, which predicted that playing the game was associated with improvements in arithmetic performance. In the second stage, the two groups' improvements were compared following crossover—between mid- and post-test. Finally, in-game data were analyzed in order to explore possible reasons for individual differences in improvement between pre- and post-test.
To assess changes in arithmetic performance over the first 2 weeks of the trial, scores from the Westwood 1-min tests from pre- and mid-test were summed.
Mean improvement in arithmetic fluency by group for 2-week intervention (error bars represent ±2 SE).
The control group improved by an average 6.20 points (
Lord's (
After the first 2 weeks of the trial, and the mid-test, the groups swapped roles so that the original control groups became the game-playing groups and the original game-playing groups became controls.
Mean improvement in arithmetic performance following crossover (error bars represent ±2 SE).
One of the core design principles for the game was that it should involve players making as many decisions about number as possible in the time that they were playing. Analysis of in-game data showed that participants made an average 2,954 judgements about numbers during the 2-week trial (
The game was designed to be as engaging for children with an already high level of knowledge of number facts as it was for children with a low level of knowledge. To test whether this aim had been achieved, an analysis of correlation between pre-test arithmetic performance and engagement with the game was undertaken. Engagement with the game was measured as the total number of rocks (numbers) seen by the player over the 2-week intervention. This is because players only see rocks to mine once they have chosen to enter into the mining minigame. Results showed a very small correlation coefficient of −0.07, indicating that children engaged with the game to approximately the same extent, regardless of initial levels of ability.
A further two correlational analyses were undertaken in order to determine whether there was a relationship between pre-test score and the number of rocks (numbers) selected, and between pre-test score and the number of rocks selected
The final analyses presented here explore relationships between players' gameplay and the improvements they experience in arithmetic. Within the game, players had some choice about what type of minigame to play, focusing on either multiples or number bonds. The analysis of differences in improvement across the four arithmetic operations led to the question of whether such differences were correlated with participants decisions to play a greater proportion of “multiples” vs. “number bonds” minigames.
Pearson correlation coefficients and significance for relationships between games played and improvement in performance for the four operations (n in all cells = 93).
Number of “Multiples” rocks seen | ||||
Number of “Number bonds” rocks seen | ||||
The more surprising results in
The findings show that playing the game for 2 weeks led to significant improvements in arithmetic performance, equivalent to ~7 months' progress. The other analyses described in the results section support the claim that improvements were due to the design of the game, as opposed to Hawthorne effect or similar. Improvement in arithmetic fluency was positively correlated with the amount of effort that children put in to progressing within the game. Further to this, children who played more times-tables tasks, as opposed to number-bonds tasks, improved more on multiplication- and division-fluency.
As a successful application of theory to game design, this study provides indirect support for the connectionist theory of multiplication described by Verguts and Fias (
It is important to note that this study has not shown the training multiples is a more effective way to support arithmetic development than the normal classroom approach of training responses to multiplication problems. Further studies will be required in order to directly compare the training of multiples with the training of multiplication triples using a game-based learning platform. Ensuring a fair comparison, including balancing engagement and volume of practice across conditions, will be challenging, but the large effect size observed in this study suggests that such further investigation would be worthwhile.
We argued in the introduction to this article that training of associations between single-digit numbers and their multiples may also lend itself to production of answers to division questions. The results show that increased gameplay in the multiples mini-game was associated with increased improvement in both multiplication and division fluency. Thus, this study has provided evidence that the mechanism by which children rapidly or automatically solve simple division problems may be closely aligned with that for solving multiplication problems. We propose that when 28 ÷ 4 is presented, for example, that a set of numbers is activated for both “28” and “4.” Following the training experienced with the game, participants will associate “28” with both 7 and 4, as it appears in both sets of multiples. The effect of the game on division fluency is more novel and surprising than the effect on multiplication fluency, and so will benefit from further research and study.
The observed effect on division performance may be an affordance of the decision to train associations between single-digit numbers and their multiples as non-directed associations. The game trains a connection from 16 to 4, for example, equally as it trains a connection from 4 to 16. This decision removes “x,” “÷,” “multiply,” and “divide,” from training stimuli, and so may allow players of the game to more easily learn associations in a way that allows retrieval in both multiplication and division contexts.
It is surprising that the “number bonds” minigame appears to have had much less of an effect on participants' learning. The correlation between the number of number bonds trials and improvement in addition fluency was close to zero. In fact, experience of the “multiples” minigame appeared to be more effective in improving addition and subtraction fluency than the “number bonds” minigame. We suggest that this finding may be due to the fact that automaticity in simple additions is due to retrieval of an association between a whole problem (e.g., “2 + 3”) and its solution (e.g., “5”), rather than associations with the two addends (Ashcraft and Fierman,
Evidence from this study supports the adage, “practice makes perfect,” but emphasizes the value of using video games to make practice engaging. Indeed, informal conversations with participants toward the end of the intervention suggest that they did not see the game as being about mathematics practice at all—their main focus was on the way that they had built their space station, and the number of aliens that they had successfully hatched. Although in some mathematical games we may want children to be actively aware of the mathematics that they are engaging with, where the aim is to train associative links between numbers this may not be necessary. Analysis of gameplay data showed that children were making ~300 decisions about numbers in each 20-min period of play. It would not be easy to achieve this volume of practice using traditional classroom methods.
Findings concerning the relationship between gameplay and outcomes suggest that it may be possible to use the game both for training and for assessment of arithmetic performance. There was a moderate to large correlation between pre-test scores and the number of correct answers given within the game. With further refinement, it should be possible to derive measures within the game that are strongly predictive of performance outside the game.
The findings of the game raise a question of the value of number bonds training. The mathematics curriculum in England requires that children learn the pairs of numbers that add to 10, or 20, for example (Department of Education,
While we did not collect systematic qualitative data on participants' and teachers' responses to the game, members of the research team were able to make some informal observations and to speak to children and teachers during the intervention. Teachers confirmed our observation that children engaged with the game well for the 2 weeks of the intervention; teachers were happy to see that the game was engaging for children with very different levels of mathematics achievement. Despite some evidence in the literature regarding differences in game-playing behaviors outside of school (Lowrie and Jorgensen,
A key limitation of this research is that the game included multiple innovations and so it is not possible to know which aspects of the game's design were more or less effective in improving arithmetic performance. It is also not possible to know from this study whether 20 min per day for 2 weeks is the optimum duration for this intervention. These limitations lead to the obvious suggestion that further work could now be carried out to isolate particular features of the game's design and determine the contribution of each feature to its apparent success. However, it is also possible that there may be complex interactions among different design components that mean it is not easy to examine them independently of one another. Further study could also vary the length of the intervention to determine how this corresponds to any improvement in arithmetic.
This intervention took place in the classroom. This allowed us to ensure that all participants in the study played the game for 20 min each day, for the 2 weeks of the trial. However, this also meant that children were potentially missing out on other learning that they could have been doing during this time. If children were to play the game at home, they could potentially experience the benefits observed in the trial reported here, without taking time out of the school day. Therefore, it would be useful to test the effectiveness of the game when played at home, outside of school time. A home trial would represent a more sensitive test of children's engagement with the game, as children who did not enjoy the game would not play for the same amount of time as they did in the classroom in the current study.
The iterative approach employed during the design of the game was very helpful in ensuring that it was both engaging for children and contributed to learning. However, it was very time consuming and labor intensive. The positive results from this trial open up the possibility of “live” testing of different versions of the game at scale, to further refine and test various aspects of the design. Different versions of the game could be released via app stores, and data collected via the cloud, in order to measure players' responses in terms of engagement and learning. This data-driven model of development and testing could provide some exciting insights into game-based learning for numeracy and mathematics.
Finally, we hope that the design and evaluation of this game represents evidence for the value of interdisciplinary collaboration in this field. Without expertise from each of the three contributing fields, mathematical cognition, game-based learning, and game design, this project would not have been possible. This kind of interdisciplinary collaboration brings risks—not least that no single member of the team can fully understand all of the detail of the project—but we argue these approaches are vital for future understanding and application of findings relating to children's learning (Jay,
The datasets generated for this study are available on request to the corresponding author.
The study was carried out in accordance with the recommendations of the British Educational Research Association and was approved by the Development & Society faculty ethics committee at Sheffield Hallam University. Written informed consent was gained from the head of school of each participating school.
TJ led the overall design of the research and wrote the initial draft of this article. JH made substantial contributions to the design of the game and to the research design and contributed to edits and rewrites of the article before submission. MM coded the game and carried out much of the data collection for the study. PH-J contributed to the research design and made edits to the article prior to submission.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The work reported here was funded by a Research Project Grant from the Leverhulme Trust. We are very grateful for this support, and for the participation of schools, teachers, and children in the project.