Scoring system
ACER uses the Intellimetric® automated essay scoring system (AES) from Vantage Learning to automatically score the OWA essays.
The Intellimetric® system is ‘trained’ with a set of scripts and their scores, known as the 'training set'.
The 'training set' for the OWA consisted of writing scripts drawn from a representative sample population. Expert ACER human markers, trained specifically on each essay type, provided the scores and marking guidelines to accompany the scripts.
For each OWA essay topic (writing prompt), the Intellimetric® system was trained to apply scores in a way that is similar to a human marker. This involved a 'training set' of over 300 high quality scripts for each prompt.
Training the Intellimetric® system works inductively: the system uses the human scores to ‘learn’ the marking rubric and the way it has been applied by the human markers. The system ‘trains’ through the systematic interaction of the set of scores and the features of the scripts, and by the accumulation of relationships, as more scripts from the training set are analysed: the system ‘builds’ itself as it progressively analyses material from the 'training set'.
The reliability of automated essay scoring
Extensive research into the use of computer-based marking of candidate writing has been undertaken. Computer-based marking has been proven to be as, if not more consistent than traditional hand scoring. View the article 'An Overview of Automated Scoring of Essays' (external link) recently published in the Journal of Technology, Learning and Assessment.