Ranking universities on student equity performance
Research 4 Feb 2020 8 minute readWhile the study shows it is possible to construct a ranking of higher education student equity, it also demonstrated that such a tool will only ever measure relative performance within the group being ranked, rather than provide insight into actual performance. Alternative options for measuring and reporting higher education equity performance should therefore be explored.
In 2020, Australia’s public universities will move to a performance-based funding model that will include a 20 per cent weighting for student equity measures. Of interest in this changing policy climate, is new research by Curtin University, ACER and the National Centre for Student Equity in Higher Education which has investigated the feasibility of developing a student equity ranking of Australian universities.
The purpose of this Australian Department of Education, Skills and Employment-funded study was to determine whether it was possible to measure and rank higher education equity performance among Australian universities.
Equity groups of focus in this ranking were students from low socio-economic backgrounds; students from regional/remote areas of Australia; Indigenous students; students with disability; and students from non-English speaking backgrounds. Given the substantial differences in the size, context and needs of each of the equity groups, the study constructed separate rankings for each equity group.
Higher education equity encompasses both ‘commutative’ equity (i.e. treating people fairly) and ‘distributive’ equity (i.e. treating people differently, according to their needs). It involves both raising the absolute number of equity-group students in higher education as well as their proportional representation.
But increasing participation in higher education is not the only important issue – equity of outcomes is also important, both in terms of achievement and also ‘downstream’ through employment outcomes or post-graduate study. Similarly, the relative ‘equity performance’ of an institution should account for:
- both national and state/territory population distributions for specific equity groups (e.g. nationally the low socio-economic status (low SES) population is, by definition, 25% of the overall population, however in some states/territories it is higher and in some it is lower);
- how the institution performs in relation to the equity group compared to how other institutions perform for the same measure; and
- how the institution performs in relation to the equity group, compared to how its non-equity group students perform for the same measure.
The study team explored a range of scenarios and areas for measuring and ranking. A peer-reviewed article in the journal Higher Education provides detail of the process of construction, including scenarios based on four ‘domains’ relevant to student equity that had sufficient data on which to develop a ranking: Access and Participation; First Year Experience, Progress during Higher Education, and Graduate Outcomes. A range of indicative weighting combinations were then used to generate a vast number of ranking scenarios to test.
Below, further scenarios for the ranking are explored based on measures covering three of the key domains identified in the research. Within these domains, three core indicators and six measures have been used to test a ranking of Australian universities’ low SES student group equity performance based on currently available (2018) data from the Department of Education.
Domains, Indicators and Measures used in these low SES equity rankings
| Domain | Indicator | Measure |
|---|---|---|
| Access & Participation | Higher Education enrolments | National-based Participation Rate (by equity group) |
| State-based Participation Rate (by equity group) | ||
| First Year Experience | First-year retention (adjusted measure) | Equity group compared to national rate for equity group |
| Equity group compared to rate for non-equity group students in the same institution | ||
| Graduate Outcomes | 'Earning or learning' within the first year after graduation | Equity group compared to national rate for equity group |
| Equity group compared to rate for non-equity group students in the same institution |
We used a stacked bar-chart graph to display the contribution of each measure to the overall performance of each university. The results in these charts show four rankings – each using the same measures, but with different weights applied to them in each instance. In these rankings, the ‘top 10’ institutions differ depending on the measure which is given prominence.
Overall, most institutions’ final rank varied significantly depending on: which equity group was being measured, which aspect of equity performance was weighted the most heavily, and whether equity-group performance was compared to the corresponding performance for other students in the same institution or against the national average for the same equity-group’s performance.
For example, when participation is prioritised (as shown in the second scenario), universities enrolling high proportions of equity-group students benefit, even when the subsequent higher education student experience is sub-optimal. Conversely, universities that limit their engagement to a smaller number of relatively high-achieving equity group students rise higher in a ranking system focussing on retention (as shown in the third scenario).
The study concludes that, while it is currently feasible to construct a higher education equity ranking system for the purpose of monitoring higher education performance, it will only ever measure relative performance. In other words, rankings only provide comparisons within the group being ranked, they do not provide insight into actual performance.
Given the identified limitations of a ranking system, the study team recommends that alternative options for measuring higher education equity performance be explored. Such options for measuring and reporting higher education equity performance might include approaches like benchmarking, which ‘rate’ outcomes based on an agreed standard.
Ideally, any future approach would also involve moving from a position of ‘valuing what one can measure’ to ‘measuring what one values’. In the case of higher education equity, this means capturing actions and outcomes that occur before and after higher education studies, such as outreach, academic preparation and transition to employment or further studies. And – rather than reverting to the relativity of ranking, more nuanced measurement of outcomes should be used, incorporating benchmarks and targets so as to encourage genuine improvement across the whole university sector in this area.
Read the journal article:
‘Constructing a ranking or higher education institutions based on equity: is it possible or desirable?’, by Tim Pitman (Curtin University), Paul Koshy (National Centre for Student Equity in Higher Education), and Daniel Edwards, Liang-Cheng Zhang and Julie McMillan (Australian Council for Educational Research), Higher Education, 2020.