Improving assessment processes and quality in medical specialisations

Improving assessment processes and quality in medical specialisations

The Australian Council for Educational Research (ACER) has been engaged by a number of specialist colleges in medicine to evaluate the quality of assessment and data collection processes, and of test item development.

In undertaking this work, ACER has found that there is considerable knowledge and effort in some colleges to collate psychometric data about their assessment, but less practice in using this evidence to improve test items, exams and development processes.

Through workshops with colleges and their Fellows, ACER helps to provide that link – illustrating how to make improvements to these areas based on what the data shows.

Many of the colleges ACER works with have long established regimes for running their exams and, while these have served them well up until now, advances in technology for storage and ongoing development of item banks and more streamlined methods for reporting are now widely accessible. With some expert advice assessment in these colleges can be transformed.

ACER’s researchers are well equipped for providing advice and recommending processes in these very high stakes assessments, with prior experience working in medical admissions for universities, the Australian Medical Assessment Collaboration and in a range of school-level tests.

Questions explored by ACER on behalf of the medical colleges include:

• Are the current assessments assessing what they are intended to assess?
• Are pass marks realistic and defensible?
• Are assessments at a standard that is considered best practice?
• What tools and processes are needed to enable ongoing evaluation and review of assessments?

ACER’s work with the colleges is leading to higher quality assessments, a closer match between examinations and curriculum, and substantial efficiencies in the development of test items and reporting of results.

These projects significantly help the colleges in streamlining processes, produce much higher quality examinations and result in better and fairer outcomes for students.