AHELO fieldwork goes live

Interesting new questions lie at the core of the OECD’s Assessment of Higher Education Learning Outcomes (AHELO) Feasibility Study, led internationally by ACER: Is it possible to undertake an international assessment of final-year students’ capacity to use, apply and act on the knowledge and reasoning they have gained from their degrees? Is it possible to assess these outcomes in an efficient and internationally comparable way? Can policymakers, institutional leaders, faculty and students be convinced that the assessment of higher education learning outcomes is an essential checkpoint in the educational process?

In previous editions of Higher Education Update, we have reported on preparatory work underway by ACER and partner organisations to create AHELO assessment frameworks and instruments in three testing strands—generic skills, economics and engineering—and context instruments for students, teaching staff and institutions.

From February 2012 preparatory work will be put to the test as fieldwork commences. Up to 40 000 students and 10 000 teaching staff across more than 250 higher education institutions in 16 countries will participate in fieldwork between February and May 2012. Ten institutions will participate in Australia, each testing up to 150 final-year students.

In addition to multiple choice items, all three assessment strands incorporate constructed response tasks which invite students to type an extended response. These responses must be scored by domain experts in each participating country. Ensuring that scoring standards are replicated and monitored across countries is one of the challenges involved in the AHELO Feasibility Study. This calls for the development of detailed scoring rubrics and the intensive training of those responsible for scoring in each country.

In November 2011, ACER’s Hamish Coates, Sarah Richardson and Julian Fraillon led a three-day international training session for national project managers from all countries involved in AHELO’s fieldwork. The lead scorers for each country (mostly university professors) were provided with an introduction to the scoring approach used in AHELO. They also scrutinised the scoring rubrics created by test developers and made suggestions for their revision.

Follow-up international training from ACER and partners is scheduled for March 2012 in Paris. The lead scorers from each country will be provided with rigorous training in the application of scoring rubrics. They will have extensive opportunities to practice scoring genuine student responses under the guidance of test developers. They will be trained in using the AHELO Test System to monitor between-scorer reliability. After this meeting, lead scorers will return to their countries and train other members of the national scoring teams.

The approach to the training of scorers is indicative of the attention to detail and international cooperation required for all facets of operations in a large cross-national study such as AHELO. Similar approaches have been taken to the training of national project managers in the translation of assessment instruments, the sampling of students and teaching staff and the use of the AHELO Test System. In addition to training, all those involved in AHELO are provided with ongoing support and advice to help them implement AHELO activities in their countries.

Already, much has been learned through AHELO of the capacity to develop assessment definitions and instruments that are valid across languages and cultures. Results from the main testing round will yield new knowledge about higher education students’ learning. Creating strategies for higher education institutions and stakeholders to make good use of this information is the key challenge ahead.

Find out more:
Further information about the AHELO Feasibility Study is available from <http://www.oecd.org/education/imhe/theassessmentofhighereducationlearningoutcomes.htm>