The effect of year level on PISA achievement

PISA – the OECD Programme for International Student Assessment – measures whether students can apply what they have learned to solve ‘real world’ problems. To achieve this, PISA uses an age-based sample of 15-year-old students, an age-group who are nearing the end of compulsory schooling in most participating countries. Specifically, the target population is students aged between 15 years 3 months and 16 years 2 months.

The age-based sample aims to provide ‘a fair comparison’ that is not biased by differences in the structure of school systems, the meaning of year levels and year level retention policies.

In many countries, including Australia, the PISA samples of 15-year-old students are spread across several year levels and higher achievement is associated with higher year levels. In all PISA cycles, the average achievement of Australia’s 15-year-old students in Year 11 was significantly greater than for 15-year-old students in Year 10, and the achievement of 15-year-old students in Year 10 students was significantly greater than for 15-year-old students in Year 9.

Changes in Australia’s PISA sample

The size of Australian PISA sample has increased significantly over the last 20 years. In PISA 2000, the first cycle of PISA, Australia’s sample consisted of 5176 students from 231 schools. Following a 240 per cent student increase in the second cycle and subsequent gradual increases, by 2018 a total of 14 273 students from 734 Australian schools participated in PISA. The increase in size reflects the interest in using PISA results to inform policy at the sub-national level; for example, accurately estimating achievement differences between states.

In PISA 2000, when reading literacy was the major assessment domain, 6 per cent of Australia’s participants were in Year 9, 77 per cent were in Year 10 and 17 per cent were in Year 11. In 2003, when mathematical literacy was the major assessment domain, 8 per cent of Australia’s participants were in Year 9, 72 per cent were in Year 10 and 19 per cent were in Year 11.

Between 2003 and 2012, the proportion of Year 9 students in Australia’s PISA sample increased slightly, the proportion of Year 10 students declined slightly and the proportion of students in Years 11 remained the same.

Since 2012, the proportion of Year 10 students has reversed its trend, increasing to a level above its starting point of 2000, while the proportion of Year 11 students has sharply declined and the proportion of Year 9 students has remained relatively stable.

As a result, in the most recent cycle of PISA, 12 per cent of Australia’s participants were in Year 9, 81 per cent were in Year 10 and 7 per cent were in Year 11 – representing changes of +6, +4 and –10 percentage points respectively between 2000 and 2018.

This change in the year level distribution of Australia’s PISA participants was largely driven by earlier changes to school entry policies in Queensland and Western Australia.

Between 2000 and 2018, the proportion of Year 9 and Year 10 students in Queensland’s PISA sample increased by 3 and 20 percentage points respectively,  while the proportion of Year 11 students decreased by 23 percentage points. This change was mostly between the 2015 and 2018 cycles, as a result of the introduction of a foundation year in 2007.

In Western Australia, the proportion of Year 10 students increased by 34 percentage points between 2000 and 2018 while the proportion of Year 11 students decreased by the same amount. This change was mostly between 2012 and 2015, and similarly stemmed from the introduction of a foundation year in 2003.  

Changes in achievement by year level

At the same time that the year-level spread of Australia’s PISA sample has changed, there has been a decline in Australia’s average achievement of 25 score points in reading and 33 score points in mathematics.

In reading, Year 11 students’ average achievement declined 37 scale points between 2000 and 2018, while the average achievement of Year 10 students declined by 21 points. Even though the overall drop for Year 11 appears to be higher than that for Year 10, the difference between them is not statistically significant. For Year 9 students, there was no significant change in reading achievement between 2000 and 2018.

In mathematics, between 2003 and 2018 the overall decline in the average achievement of students in Year 11 (33 points) was similar to that for students in Year 10 (26 points). For Year 9 students there was no significant change in mathematics achievement between 2003 and 2018.

While the scores of Australia’s Year 10 and 11 PISA participants declined at a similar rate, students in Year 11 still achieved significantly higher on average than students in Years 10 and 9 in both reading and mathematics. The reduction in the percentage of these higher achieving 15-year-olds in Year 11 therefore contributes partly to the overall decline in Australia’s PISA achievement.

Changes in national achievement and year level distribution

The declines in Australia’s PISA achievement cannot be wholly explained by decreasing proportions of students in the higher-achieving Year 11 group, as the changes in achievement did not always correspond to the times when there were shifts in the year level distributions of students.

In reading:

• Between 2000 and 2012 there was a decline in Australia’s average reading achievement even though there was very little change in percentage of students in Year 11.
• From 2012 to 2015 there was a decline in both average reading achievement and the percentage of students in Year 11.
• Between 2015 and PISA 2018 there was a drop in the percentage of 15-year-old students in Year 11 but no decline in reading achievement.

In mathematics:

• Between 2003 and 2012 there was a decline in mathematics achievement even though there was little change in the percentage of 15-year-old students in Year 11.
• Between 2012 and 2015, and between 2015 and 2018, there were declines in mathematics achievement at the same time as there were drops in the percentages of students in Year 11.

Even in Queensland and Western Australia, the two jurisdictions with the largest shifts in year level distributions, the changes in the proportion of Year 11 students and the changes in overall achievement do not follow the same pattern. This suggests that other factors such as curriculum changes and changes in approaches to teaching may have taken place around the time of the changes in the year distribution of students.

Differences in the year level distributions of 15-year-old students in PISA have, however, contributed to differences in average achievement between Australia’s states and territories – until 2015. By 2018, differences in age-year distributions between states and territories had largely disappeared (except for Tasmania) and therefore no longer have a great influence on differences in achievement.

Implications for policy and practice

Nationally, the differences in reading and mathematics achievement by year level suggest that factors such as teacher support and classroom climate might be more positive in higher compared to lower year levels. It might also be possible that students in higher compared to lower year levels might experience richer, more complex curricula that foster the types of skills measured by PISA. When exploring factors associated with the decline in achievement in PISA reading and mathematics it might therefore be fruitful to examine Year 11 curricula and teaching. ■

Read the full article:
‘Does student grade contribute to the declining trend in Programme for International Student Assessment reading and mathematics in Australia?’ by John Ainley, Dan Cloney and Jessica Thompson, Australian Journal of Education (October 2020).

Join the webinar:
On Wednesday 4 November 2020, the Australian Journal of Education will host a panel discussion with the several of the authors of the special issue, including Dr John Ainley. Register for the free webinar '20 Years of PISA in Australia: an AJE special issue'.