Strengthening capacities for state-level learning assessment: A case of Uttar Pradesh in India

Under the UNICEF-funded Strengthening Systemic Capacity on Learning Assessment (SSCLA) programme, the Australian Council for Educational Research (India) is engaged in building capacities of four Indian states − Bihar, Chhattisgarh, Jharkhand, and Uttar Pradesh (UP) – and the Union Territory of Jammu and Kashmir. In consultation with the Ministry of Education (MoE) and state governments,  UNICEF India is contributing to an extensive assessment capacity building exercise for 15 Indian states through SSCLA.

Student learning data is considered to be of utmost importance for objectively understanding achievement gaps and systematically addressing inadequacies through policy intervention. The report Use of learning assessment data in education policy-making highlights a number of issues that can arise between data generation and its use as there is a long series of steps involved; for example, shortcomings in assessment instruments or sampling can render data unusable. In addition, lower technical capability in data analysis might lead to drawing misguided inferences or insufficient understanding of a situation. Instances such as failure to differentiate between correlation and causation might spawn misleading inferences and misguided policies. Thus, all steps have to be reliable to ensure that the evidence generated can be used for policy intervention.

Test scores and ranks used by education systems to quantify student learning are not sufficient by themselves to serve the fundamental purpose of assessment, which is to establish and understand where learners are in their learning. Education systems are ubiquitously facing these challenges in capturing valid student learning data and drawing well-founded inferences to guide policy and delivery choices.

Several stages of training have been designed to fortify state capacities in developing robust and responsive learning assessment systems and using data for educational policy and planning. To provide exposure to an international standard assessment development process, participants are introduced to 14 key elements of a robust assessment cycle. Subsequent sessions include clearly defining the domains being tested, developing tests according to frameworks of particular states, validating the tests through various means including empirical data, assessment data analysis, using scale scores in the right context, mapping scale scores to progression in the domains, making inferences about student learning, and reporting for policy formulation.

Over a period of two years, participants selected from various education agencies including State Council of Educational Research & Training (SCERT), Samagra Shiksha, and District Institute of Education and Training (DIET) in each state are expected to participate in 53 days of training over 11 weeks. The training workshops would lead to the development of assessment frameworks and item banks for numeracy and language. Each state in India runs a different learning assessment programme, therefore, training workshops are customised to improve the ongoing initiatives and every state can complete the training at its own pace. 

Demographics and educational achievement in India vary across states. According to the 2011 Census, 40 million children between the ages of 6 to 13 years lived in Uttar Pradesh, of which 31.2 million were enrolled in nearly 246 000 schools. As per the National Achievement Survey (NAS) 2017, the average scores of the students of UP were lower than the national average in all subjects across grades.

The UNICEF state office in UP initially shortlisted 62 participants from diverse educational backgrounds on the basis of their job profiles and interviews. The participants included assistant teachers, headmasters, lecturers from DIETs, and state resource persons. The participants have so far completed a zonal training, three weeks of training on assessment material development, and four weeks of data analysis training. Two weeks of further training on reporting will conclude the programme in the state.

During assessment material development training, participants developed an assessment framework to be incorporated in Mission Prerna – a flagship program under the Department of Basic Education, UP. The team also created competency-based items and participated in panelling of test items. From this team, UNICEF shortlisted 25 participants with knowledge of computers for the data analysis workshops. According to a pre-workshop quiz, the participants had limited knowledge of assessment and data analysis, but facilitators observed them as enthusiastic and engaged throughout the workshops.

The participants reported a high degree of satisfaction from the sessions. The most valuable sessions according to participant reporting included learning about robust assessment cycle, creating assessment tools and items, data management, application of Rasch modelling, differential item functioning (DIF) analysis, equating, and scaling methodology.

The workshops focused on methods of assessing student ability on a measurement scale instead of percentages and ranks – a rather novel approach for the participants. The journey was not without challenges, as many participants were engaged in multiple training programs and projects simultaneously. Nevertheless, participants shared learnings with each other in the evenings to cover missed courses from the training. As participants gained more knowledge their interest also gained momentum.

Although the workshops were temporarily hampered due to the COVID-19 crisis, the momentum continues. During lockdown, participants reached out to teachers across the state through district-wide online training in assessment with the help of the Department of Basic Education. After consultation with the state, the remaining workshops have been resumed online.