Introducing item-response theory to learning assessment systems of Bihar and Uttar Pradesh

The Australian Council for Educational Research (India) has been a technical partner in UNICEF’s project Strengthening Systemic Capacities in Learning Assessment. Under this initiative, ACER has been building systemic capacities in four areas − robust assessment design, high quality assessment tools, data analysis, and reporting – across five Indian states including Bihar and Uttar Pradesh.

In 2019-2020, ACER conducted workshops for state officials in four phases focusing on two major components:

• data management starting from data collection to analysis
• psychometric analysis of data including Classical Test Theory (CTT) and Item Response Theory (IRT).

IRT is widely used to analyse data from large-scale student assessments around the world. This cutting-edge psychometric analysis enables links between tests allowing comparison of student performance over the years.

IRT is now considered best practice for analysis of student performance in large-scale international assessments. The Programme for International Student Assessment (PISA), the Trends in International Mathematics and Science Study (TIMSS), and the Progress in International Reading Literacy Study (PIRLS) rely on IRT to provide an accurate understanding of student learning. 

In India, the use of IRT in learning assessments has been less explored. Only recently, the National Achievement Survey (NAS) has used IRT to analyse test scores. For many state-level assessments and resource persons managing them, IRT is rather a completely new method for student data analysis. In Bihar, for instance, state officials who were nominated for capacity development workshops came with a near blank slate but nonetheless were enthusiastic about learning the data analysis technique. The situation was similar for Uttar Pradesh, where only two participants had prior theoretical understanding of inferential statistics.

The Research and Assessment Director for ACER India Dr Mee Young Han said, ‘Our facilitators are well trained to explain advanced statistical concepts in a simple way. We understand that participants come from diverse backgrounds and have different needs and resources. Some may have basic understanding of statistics whereas others have no knowledge of the subject, thus the pace of the workshop needs to be managed accordingly. It is through discussions that we build a common understanding and agreement on large-scale data analysis techniques.’  

‘Our workshops provide hands-on experience to participants so that they are able to analyse data on their own after undergoing training. In our workshops, they get a first-hand experience of the tools such as Excel, Statistical Package for the Social Sciences (SPSS), R, and ACER’s ConQuest software used for statistical analysis.’ 

‘Building the capacity of state level institutions to conduct learning assessment and use findings to evidence based planning is a key area of UNICEF support. State level functionaries found the workshops delivered by ACER very useful, increasing their technical knowledge and building their confidence in supporting large-scale learning assessment at the state level. The Initiative contributed to state’s capacity to establish state learning assessment cell as envisaged in the National Education Policy, 2020,’ said Terry Durnnian, Chief of Education, UNICEF, India Country Office.

Dr Han added, ‘One of the ways to know if participants have understood a concept is through feedback. Participants in Bihar highlighted that IRT analysis through ConQuest and interpretation of its output, and the section on scaling methodology were valuable for them. We have also collected feedback from participants in Uttar Pradesh and other states.’

ACER has designed, managed, and implemented several large-scale international, regional, and national assessments and provided capacity building support to governments around the world. In South Asia, we will continue to support education stakeholders in building robust assessment systems that help them to address learning gaps and improve learning in the region.