Data for quality improvement in VET

Participants at the summit on data for quality improvement in vocational education and training (VET) have highlighted the importance of data and effective uses of data to drive improvement in the sector, but also significant issues with management information systems and staff capacity and capability to use data efficiently and effectively.

The Summit on Data for Quality Improvement in VET, a joint initiative of the Australian Council for Educational Research (ACER), the Australian Workforce and Productivity Agency (AWPA) and the National Centre for Vocational Education Research (NCVER), brought together senior decision makers and stakeholders in VET policy, industry, training and research.

Data for accountability and improving outcomes

Opening the Summit, AWPA Chief Executive Officer Robin Shreeve observed that the VET sector maintains data at the system level in terms of indicators such as enrolments and graduations, and more recently in terms of information about providers. Mr Shreeve said that one key question is whether the best kinds of data are being collected for accountability purposes, but a second key question is whether the right kinds of data are being collected and used in the best possible ways in terms of improving outcomes. Throughout the day, Mr Shreeve emphasised the value of using quality data continuously at an institutional and course level to monitor performance and drive quality improvement.

‘When I was an institute director in the UK, I could ask any one of my lead trainers how their courses were going, what their performance indicators looked like, including completion rates, and what work they were doing to address the risks and improve outcomes,’ Mr Shreeve said. ‘Our systems required and supported staff to monitor, understand and use data relevant to their teaching and learning goals, to meet institutional and national targets and drive continuous improvement.’

Performance indicators in the VET sector

Speaking on performance indicators in the VET sector, NCVER Managing Director Tom Karmel presented a paper that analysed current data sets collected, gaps in our data collection systems and the kinds of properties that data should have in order to function well as performance indicators. Dr Karmel noted the traditional focus of data collection for the purposes of regulation, managing risk, and accountability for public investment in the VET sector, and more recent needs and interest in performance indicators at the provider level, but also the use of performance indicators to support consumer choice.

‘Regulators, employers, students and program administrators all have a need for performance data for RTOs,’ Dr Karmel said. ‘But naive comparisons are dangerous because student characteristics differ by RTO, not to mention the statistical issues which arise when indicators are calculated from a relatively small student base.’

Data for 21st century skill development

ACER’s Manager of Vocational and Workforce Assessment Services Dave Tout, addressed the ways data can enhance 21st century skill development in VET, with a particular focus on developing the key core skills of individual learners. Mr Tout quoted from a speech made by Rob Bluer of Innovation and Business Skills Australia at last year’s National Adult Language, Literacy and Numeracy Assessment Conference to argue that assessment of student learning in VET needs to broaden from a singular focus on competency to a focus on learning outcomes.

He highlighted evidence from international surveys that show a significant core skills gap in the Australian workforce and general population, and the relationship between core skills and employment, productivity, health and other outcomes. This included discussion of the 2011 Skills Australia (now the Australian Workforce and Productivity Agency) report, Skills for prosperity a roadmap for vocational education and training, which noted the increased need for workers with core skills, including language, literacy and numeracy (LLN) skills, as the economy moves from low-skilled work to greater knowledge-based work. That report also links the development of core skills like LLN to a range of positive education, work and general life outcomes.

‘While substantial investments have been and continue to be made in the VET system in data collection and analysis in regard to enrolments, costs, completions and employment outcomes and in relation to monitoring, compliance and risk management, there has been less emphasis on collecting empirical evidence about the quality of teaching and broader learning outcomes, and using such evidence to support local and system-wide continuous improvement strategies,’ he said.

‘It is now understood that best practice training providers collect and use evidence of student learning, including core skills of literacy and numeracy, to drive continuous improvement in learning outcomes and training practices.’

Key observations

Key observations from the roundtable discussion at the Summit included:

Governments, systems, providers and trainers need quality data to monitor outcomes, make effective decisions and drive quality improvement agendas.
Students need quality data in order to make informed decisions about enrolment and course selection, and employers need quality data in order to make informed recruitment and training decisions.
Since collecting and reporting data in a timely way is costly, difficult and time consuming, and changes may require costly modifications to major embedded systems, any data collected should be high quality, meaningful and purposeful.
Data need to be designed to fit the different purposes of government, regulators, RTOs, students and employers.
Care needs to be taken in developing data collection regimes to address the circumstances of the full range of RTOs – public and private, large and small, institution based and workplace based.
RTOs require better quality and more timely and accessible data for use for continuous improvement in terms of analysis of course level data on completions, student satisfaction, core skill development and the like, and better systems and capacity to understand and use the data for monitoring and improvement planning.
LLN data are critical for institutional planning, improved teaching and learning practice, and improved student learning outcomes.
VET trainers need improved skills in understanding and using assessment, including assessment of LLN, and in pedagogical strategies to respond to student-level data.

Overwhelmingly, participants to the Summit, held in Melbourne on 9 April 2013, agreed on the importance of developing high quality and purposeful data and the need to develop further capacity at a system, institution and VET workforce level to use data to drive quality improvement.

List of participants

Anne Jones, Deputy Vice Chancellor, Victoria University
Ben Vivekanandan, National Policy Manager, ACPET
Berwyn Clayton, Director of the Work-based Education Research Centre, Victoria University
Dr Chrissie Berryman, General Manager, Skills Tasmania
Dave Tout, Manager, Vocational and Workforce Education Assessment Services, ACER
Denise Stevens, CEO, TAFE Development Centre
Dorothy Rao , Training Packages Manager, CS&HISC
Professor Erica Smith, Professor of Education, University of Ballarat
Francesca Saccaro, Associate Director, OTEN and Organisational Capability, Western Sydney Institute
Jenny Dodd, Acting Chief Executive, Canberra Institute of Technology
Kym Peake, Deputy Secretary, Higher Education and Skills Group, Victorian Department of Education and Early Childhood Development
Liz Stafford, Manager Governance & Policy, ASQA
Marilyn Hart, Manager, Research and Tertiary Education Reform, AWPA
Martha Kinsman, Consultant, TAFE Directors Australia
Martin Powell, National VET Equity Advisory Council, NATESE
Megan Lilly, Director, Education and Training, Australian Industry Group
Ralph Saubern, Director, Professional Services, ACER
Robin Shreeve, CEO, Australian Workforce and Productivity Agency
Robwyn Miller, Director, Industry Skills Unit ,TAFE NSW
Sandra Pattison, General Manager, Statistics, NCVER
Sharyn Sturgeon, Acting General Manager, DEEWR
Sue Fergusson, General Manager, Research, NCVER
Tessa Bachmayer, Director Strategic Planning, TAFE NSW Sydney Institute
Tom Karmel, CEO, NCVER

Data for quality improvement in VET