Capturing the contemporary postgraduate research experience

The revised Postgraduate Research Experience Questionnaire (PREQ) has been implemented, following a review conducted by the Australian Council for Educational Research (ACER) for the Commonwealth Department of Education and Training.

Questions had been raised about PREQ’s appropriateness in the contemporary context after more than 15 years in use. The PREQ was established in 1999 to collect information on the higher degree research (HDR) experience, and administered as part of the Graduate Outcomes Survey – a census of recent graduates from Australian universities and other higher education institutions.

Why review PREQ?

Governments, universities and other stakeholders use the results of PREQ to improve the quality of HDR training at a disciplinary, institutional and national level. However, market conditions have changed considerably since its inception.

In 2015, more than 60 000 candidates were enrolled in HDR in Australia (DET, 2017a), and enrolments have increased by 50 per cent in a decade. Additionally, the diversity of the candidate population has changed significantly; the proportion of international student enrolments have more than doubled between 2001 and 2015, from 14 per cent to 32 per cent, and today’s candidates are often older and more likely to be working while studying. The changes in size and constitution of the candidate cohort, in combination with other key factors such as policy adjustments, increasing completion rates and a greater emphasis on cooperation between universities and industry, supported the case for a review of the PREQ.

The review process

ACER undertook a review program over two phases: a detailed assessment of the existing instrument, followed by the redevelopment, testing and piloting of a revised PREQ tool.

Phase one consisted of a review of literature, current practice and trends, statistical analyses and consultation with key stakeholders from universities, peak bodies and government. While the statistical analysis found that PREQ items and scales were performing satisfactorily, stakeholder feedback indicated solid grounds for redeveloping the tool. Several problems were identified, including low response rates, issues with reporting methods and content gaps, with items about candidates’ interactions with industry and professional development experiences in particular deemed to be missing. Five options resulted from phase one, and option one – to keep the current tool but update content and include several new measures – was selected.

Phase two involved piloting a revised PREQ tool that included questions about industry engagement and the development of broader generic and employability skills – key gaps identified during phase one. The revised instrument was designed in close consultation with stakeholders and piloted with more than 1100 recent HDR graduates from 20 universities between May and June 2017.

The report

Statistical analysis showed that the revised questionnaire was successfully measuring the appropriate factors. Stakeholders gave detailed feedback on both tactical factors – the wording of individual items, for example – and strategic issues like suggestions for future improvements, but broadly reported finding the revised instrument robust, useful and well-written.

Accordingly, the review report recommended retaining the new and revised items with a few small revisions, such as the addition of more information to allow for contextual analysis of the results. The Department adopted these recommendations for implementation in the 2017 PREQ. Further recommendations from ACER will be considered in the future, including investigating alternative methods of delivery – such as a combination of telephone and online surveying, modifying the current online-only system – in order to improve response rates, and exploring the possibility of a survey of current HDR candidates, rather than recent graduates.