Tuesday, 11 Jun 2024
Over the past decade there has been noticeable, positive change in the ways educators speak and think about assessments, data, and the data literacy of PAT users and educators in general. At ACER this is evident in the questions and feedback we receive, showing the sophistication with which educators are using data. Questions around PAT scale scores are often raised, usually framed with phrases such as, ‘How much improvement…?’ and ‘What is the expected…?’. The answers are: it depends. A recent webinar facilitated by Toby Newton and Daniel O’Loughlin discussed an ACER study on predicting scale score gain. The webinar also took a closer look at the new feature in ACER Data Explorer.
What was the goal of ACER's study?
ACER wanted to take a statistical approach to understanding what scale score gain looks like in PAT over time, and analyse the effects of 3 variables: prior scale score (the initial measurement), year level (the year level of the student at the time of the prior scale score), and time (the interval between the 2 measurements). ‘Our psychometric experts looked at [these 3 variables] to judge “What effect do these have?” and “Can we use that information to model predictions based on these factors for students more generally?”’, Toby explained. The goal was to see whether the scale score gain that a student might achieve at different year levels, scoring at different levels on the scale, had similar trends or identifiable, predictive qualities.
What data were used?
ACER psychometricians analysed a large set of test results from PAT Maths 4th edition and PAT Reading 5th edition, completed by Australian students in 2020 and 2021. The 3 variables were chosen because the data was reliably available and it was reasonable to expect that these variables may have had an effect on the second score, that is, the gain. All case studies were of individual students who had completed 2 tests within a period of 6.5 and 24 months – anything outside this timeframe was excluded.
The 3 variables used in modelling did not include reference to curriculum standards or achievement levels that teachers might be working with, or the goals and targets that might have been set at the individual, class or school level. Other than year level, the study did not address demographic context, or longer-term trends and prior achievements on PAT or other assessments.
What does ‘predicted’ gain mean?
The study asked ‘For students in the same year level who had similar prior scale scores, what was the typical scale score gain achieved after a given time interval?’ The caveat here is that predicted gain is a statistical prediction not an expectation. As Toby noted, ‘What we’ve got is a prediction based on a relatively contained set of factors, but ones that we’re confident in having an effect and a predictive quality … these should not be used as a benchmark or a standard or something to set students up against, but [it is there] more to provide context and increase your understanding of how students perform’.
What does predicted gain look like?
Using an example, Toby showed how predicted gain can vary based on prior scale score or year level. In Table 1, you can see how the predicted gain of students who started at the same prior scale score varies at different year levels. Typically, the students at the lower year levels are not predicted to show as much growth over 12 months as students at higher year levels.
In Table 2, predicted gain can vary within the same year level at different prior scale scores. Typically, at the lower end the predicted gain is largest, which also aligns with what we know about patterns of learning and progress. That is, we tend to see much more growth at the early year levels and at the lower ability levels, and learning progress tends to slow down as students become more able and their learning grows.
Table 1 12-month predicted gain by year level for students with the prior PAT Maths scale score
|
Year level |
Prior scale score |
Predicted gain |
Predicted scale score |
|
2 |
125
|
+0.5 |
125.5 |
|
6 |
+4.8 |
129.8 | |
|
10 |
+3.7 |
128.7 |
Table 2 12-month predicted gain by prior PAT Maths scale score for students in the same year level
|
Year level |
Prior scale score |
Predicted gain |
Predicted scale score |
|
6
|
80 |
+13.0 |
93.0 |
|
100 |
+9.4 |
109.4 | |
|
120 |
+5.7 |
125.7 | |
|
140 |
+2.1 |
142.1 |
How should predicted gain be used?
The study and the ACER Data Explorer reporting feature answer questions around students’ predicted gain, which should be considered in conjunction with the understanding that each student’s learning journey is unique. ACER’s role as a research organisation and provider of PAT means we can analyse a multitude of assessment results and take a psychometric, statistical approach to that data. And we unpack what scale score gain students actually show in the real world, according to real data. ‘We’re talking about predictions that are statistical as opposed to expectations or targets, which obviously come with a whole lot of extra information and assumptions and goals, curriculum standards, benchmarks – those kinds of considerations that are not possible to consider when we talk about predicted gain,’ Toby explained.
While the predicted scale score gain offers additional context to student progress over time, professional expertise, experience, interpretation and understanding is central to identifying student needs and therefore navigating the support they need to progress.