ACER: developing statistical software for more than 25 years

Developed in-house by ACER to support research and assessment programs, ACER ConQuest is also available for the public to use. Version 5 brings new features like the ability to use simulation methods to test item fit, the ability to treat item parameters as random effects, and to estimate models using Markov chain Monte Carlo (MCMC) estimation.

ACER ConQuest 5 is the culmination of more than 25 years of statistical software research and development. Along the way, many large-scale assessments and other research studies have benefited from ACER software.

The story of ACER ConQuest and its predecessor, QUEST, begins in 1983 with the arrival of lead author, Ray Adams. In 1983, there were three key pieces of software being used by ACER: Itan, an in-house piece of software developed by Lindsay Mackay and Steve Farish to conduct classical test theory (CTT) analysis; BICAL, software for fitting the Rasch model to dichotomous data; and CREDIT, developed by Geoff Masters for fitting partial credit and similar models to polytomous data. This software was run on a single DEC VAX-11 minicomputer. Analytic runs were input via a user at a terminal connected to the machine, and output was printed by a line printer.

Figure 1. Digital Equipment Corporation (DEC) VAX-11/750 minicomputer. CC BY-SA 3.0 Retro-Computing Society of Rhode Island

As ACER grew with more people undertaking project work, more computing power was required and it became clear that new software development could streamline projects. This included large projects like the NSW Basic Skills Testing Program (1989) – one of the state-based assessment programs that was a precursor to the National Assessment Program (NAP). QUEST was developed in 1990 by Ray Adams and Khoo Siek Toon. QUEST brought CTT and items response theory (IRT) analysis into one application that was interactive and ran on desktop machines. This increased the number of people who would work simultaneously. QUEST also expanded the kind of models that could be fit including the Partial Credit and Rating Scale Models using joint maximum likelihood (JML) estimation.

As the diversity of the analytic work being undertaken increased, there was a need to fit more varied models. To do this a new piece of statistical software, ACER ConQuest, was developed that could fit a highly generalised item response model that could be parametrised to fit specific models, including the classically named models like Partial Credit Model. ACER ConQuest also implemented marginal maximum likelihood (MML) estimation, allowing the drawing of plausible values – a method now central to all large-scale assessments – and could fit multidimensional items response models. ACER ConQuest was first written in the FORTRAN programming language (and at that stage was called MATS: Multi Aspect Test Software) and was later re-written in C++. MATS and ConQuest was written in collaboration with Mark Wilson at the University of California, Berkeley.

Figure 2. ACER ConQuest was developed by the Australian Council for Educational Research (ACER) in partnership with the University of California, Berkley.

The use of ACER ConQuest grew further when ACER partnered with the International Association for the Evaluation of Educational Achievement (IEA) and conducted the IRT scaling of the first Trends in International Mathematics and Science Study (TIMSS 1995). ACER ConQuest 2 focused on the user experience. After being re-written by Ray Adams and Margaret Wu, ACER ConQuest 2 could do user-friendly things like automatically produce design matrices by iterating over the data. ACER ConQuest 2 ran on the windows OS, including windows NT, but remained compatible with VAX 32 bit machines for large jobs.

Version 3 of ACER ConQuest emerged from ACER’s work on the OECD's Programme for International Student Assessment (PISA). ACER led the international consortium undertaking PISA from 1998 until 2012 and continues to conduct the Australian component of the study, including the most recent cycle in 2018. With the biggest team of analysts yet, features like a graphical user interface (GUI) and the ability to generate valid syntax through the use of point-and-click menu commands were developed to support as many users as possible. In addition, native plotting functions were added to export a range of plots including the eponymous Wright Maps.

ACER ConQuest 4 emerged after this, and added new features, including the ability to estimate paired comparison models (Bradley-Terry-Luce (BTL) model), directly manipulate matrices, and added more CTT analysis including the estimation of Mantel–Haenszel tests of differential item functioning (DIF). Perhaps the most notable assessment program to use ACER ConQuest 4 is NAPLAN, which uses a multidimensional item response model with latent regression model to draw Plausible Values (PVs). This expansion of features was in part conducted to support new areas of research, including enabling ACER’s Centre for Global Education Monitoring  to assist countries to report on their progress of Sustainable Development Goal 4. ACER ConQuest 4 was also used to scale a new family or regional assessments, including the Pacific Islands Numeracy and Literacy Assessment, Southeast Asia Primary Learning Metrics, and Monitoring Trends in Educational Growth in Afghanistan.

Despite having only just been released, ACER ConQuest 5 has already supported important research. The OECD International Early Learning and Child Well-being Study took advantage of the ability to estimate high-dimension IRT models to test the veracity of simultaneously estimating abilities on ten outcome domains. ACER is also undertaking research by applying a wider variety of flexible and descriptive models (e.g., explanatory IRT models, tree-based IRT models). Looking ahead, a new generation of assessments will be scaled using version 5, while the next progression in ACER’s research agenda will explore the intersection of IRT and new frontiers in analysis, including machine learning.

Find out more:
For further information about ACER ConQuest 5 visit www.acer.org/conquest

If you’d like to delve deeper into education data analysis, our Understanding Rasch Measurement Theory course will teach you everything you need to know. Find out more: https://www.acer.org/au/professional-learning/postgraduate/rasch.