Developed by ACER and the United Nations Institute of Statistics (UIS), the Pairwise Comparison Method Toolkit describes how countries can estimate the proportion of students meeting Sustainable Development Goal (SDG) indicator 4.1.1 using their national assessments.
SDG 4.1.1. indicates the proportion of girls and boys achieving at least a minimum proficiency level in reading and mathematics at 3 stages of schooling – end of lower primary, primary and lower secondary.
About the Pairwise Comparison Method
ACER and the UIS have developed a toolkit that describes how pairwise comparison can be used to locate the global minimum proficiency level (MPL) benchmarks on a learning assessment scale. Pairwise comparison is an empirical method that combines expert judgement with statistical linking using Item Response Theory.
By setting the global benchmarks on a learning assessment scale, countries can establish the proportion of students meeting the minimum proficiency levels in reading and mathematics. This data can then be used to report progress towards achieving SDG indicator 4.1.1. This evidence is also relevant for understanding learning levels of student populations at the local and regional level, and how these develop over time and across stages of schooling. Being able to benchmark such education monitoring data against global standards provides an important additional perspective on the quality of education systems.
The Pairwise Comparison Method Toolkit is designed for national governments, assessment agencies and their partners.
Benefits
The Pairwise Comparison Method has several benefits over other methods used to report progress towards SDG 4.1.1.
- It is cost- and time-effective
- It is relatively simple to implement
- It can be run remotely using a specifically designed platform
- It provides valuable psychometric information about large-scale assessments
The toolkit
Contact us for any questions related to the Pairwise Comparison Method Toolkit.
Implementation stages
National governments, assessment agencies and their partners follow 6 key stages to implement the Pairwise Comparison Method.
Stage 1 Initial engagement
Contact the UIS to determine whether the Pairwise Comparison Method is the appropriate approach to report against SDG 4.1.1 for a specific assessment.
Stage 2 Evaluation of eligibility criteria
Evaluate whether the assessment meets the first 5 UIS eligibility criteria for SDG 4.1.1 reporting to determine if it is sensible to proceed with the Pairwise Comparison Method.
Stage 3 Workshop preparation
Identify and confirm facilitators, invite subject matter experts and prepare materials for the Pairwise Comparison Method workshop.
Stage 4 Workshop implementation and analysis
Compare pairs of items to determine which is most difficult. Use the resulting scale to set benchmarks and document details regarding the process followed.
Stage 5 Evaluation of outcomes
Evaluate whether the Pairwise Comparison Method was implemented appropriately to meet criterion 6 of the UIS eligibility criteria for SDG 4.1.1 reporting.
Stage 6 Results reporting
Report results to the UIS so that they can be included in global reporting against SDG indicator 4.1.1.
Frequently asked questions
-
The Pairwise Comparison Method is cheaper than other statistical linking methods that require additional assessment administration with either common items or common learners.
-
The duration of the pairwise comparison needs to be calculated on a case-by-case basis. Based on experience, we estimate about 12 weeks for the whole process.
-
Separate workshops should be planned for each minimum proficiency level (SDG 4.1.1a, SDG 4.1.1b and SDG 4.1.1c), for each subject (reading and mathematics), and for each language of assessment used in the Pairwise Comparison Method.
-
Between 15 and 20 in-country experts are required for each workshop. This allows for representation across different genders, geographies and languages. It also ensures that the process obtains a replicable outcome, but is also practical and manageable.
-
In-country subject matter experts must have strong content knowledge and teaching skills in reading or mathematics. They must also be perceived as experts in their field to foster the confidence of host governments in their decisions.
-
The Pairwise Comparison Method utilises the Learning Progression Scales for reading and mathematics, developed by ACER. An International Standard Setting Exercise was carried out by ACER in 2022 to establish the minimum proficiency level benchmarks for reading and mathematics on the Learning Progression Scales.
In applying the Pairwise Comparison Method, multiple subject matter experts make comparative judgements of pairs of items from the Learning Progression Scales and the respective assessment, generating a rank order of items. Through statistical linking the minimum proficiency level benchmarks are then placed on the respective assessment scales for reading and mathematics. The benchmarks can then be used to establish the proportion of learners meeting the relevant minimum proficiency levels in reading and mathematics using the respective assessments.
More detailed information is provided in the Pairwise Comparison Toolkit.