Lessons from international assessments

While policy makers in the vocational, adult and workplace education (VAWE) sector can learn much from international assessments, as ‘The value of international assessments of adults’ foundation skills’ illustrates, educators and others in the sectors can also learn much about teaching and learning.

The theoretical frameworks and research that sits behind international assessments like the Adult Literacy and Life Skills (ALLS) survey and the Program for the International Assessment of Adult Competencies (PIAAC) survey provide rich information for educators about teaching and learning.

Frameworks, descriptions and research about literacy

The theoretical frameworks underpinning the ALLS and PIAAC surveys describe what literacy and numeracy in the 21st century incorporates, while each also results in the publication of comprehensive reports and analyses of each of the domains being assessed.

PIAAC describes literacy as understanding, evaluating, using and engaging with written texts to participate in society, to achieve one’s goals, and to develop one’s knowledge and potential. The PIAAC Literacy Framework elaborates a range of characteristics including:

medium – pen and paper, digital
text type – argumentation, description, exposition, instruction, narration, records
social contexts – work, personal, community, education, and
task aspects – access and identify, integrate and interpret, evaluate and reflect.

Reading components

PIAAC for the first time incorporated a lower level assessment of reading, for adults who are identified by a screener test to have low levels of literacy. This aspect of PIAAC is called the Reading components assessment. In previous surveys, the information on the reading abilities of adults with poor skills was often insufficient to get an understanding of their difficulties and build a description of their abilities. The assessment of component skills is intended to provide a greater level of information about the skills of individuals with low levels of literacy proficiency. The reading components assessment assesses:

word meaning – the meaning of everyday words
sentence processing –judging whether a sentence makes sense – and
passage comprehension – the time taken to read a passage and choose between correct and incorrect words to gain meaning.

The assessment frameworks and the material sitting alongside them provide insights into what can be taught and what components of teaching reading are crucial for the successful development of literacy skills. These can also be utilised in the development of curriculum and assessment frameworks.

Task and text complexity in reading

Research based on such international assessments suggests that a number of variables interact to determine the level of difficulty of reading tasks. The variables relate to the structure and complexity of the text, to the nature of the task – in other words, the relationships between the text and the question being asked, and to the nature of the processes or strategies by which readers relate the information in the question to information in the text.

For example, three variables in particular have been found to be highly related to the difficulty of a reading task: type of match, plausibility of distractors and type of information.
Type of match relates to the relationship between what the reader is asked to do in the question and the presentation of the required information in the text. For example, the reader may simply need to locate the information, or may need to cycle through the text to identify two or more pieces of conditional information, or compare or contrast pieces of information.

Plausibility of distractors concerns the extent to which information in the text shares one or more features with the information requested in the question, but does not fully satisfy what has been requested. Tasks are judged to be easiest when no distractor information is present in the text. They tend to become more difficult as the number of distractors increases, as the distractors share more features with the correct response, and as the distractors appear in closer proximity to the correct response.

Type of information relates to the concreteness or abstractness of the requested information. Tasks are found to be easier when the information is more concrete and more difficult when the information is more abstract.

There are several lessons for teachers and trainers, including the need to teach students how to ‘read’ the task or question; the need to help students develop strategies to access and identify information, and interpret and relate parts of texts to each other; and the need to work with students on drawing on knowledge, ideas and values external to the text.

Frameworks, descriptions and research about numeracy

In PIAAC, numeracy was described as ‘the ability to access, use, interpret and communicate mathematical information and ideas, in order to engage in and manage the mathematical demands of a range of situations in adult life’.

Like literacy, this description is elaborated more fully in the framework, and highlights the complexity and extent of numeracy, and the range of factors that impact on someone becoming numerate. This can be utilised to help describe curriculum and assessment frameworks, alongside providing guidance about the range of factors to be addressed in numeracy teaching and learning.

Task complexity in numeracy

ALLS developed a complexity scheme to predict the difficulty or complexity of a numeracy assessment task, was empirically validated by ALLS, and further developed in PIAAC. The complexity scheme identifies five factors to account for the difficulty of different tasks, two relating to textual aspects and three to mathematical aspects of tasks. The textual factors address type of match or problem transparency to do with how difficult it is to identify and decide what action to take, and the plausibility of distracters to do with how many other pieces of mathematical information are present. Mathematical factors address the complexity of the mathematical information to be manipulated, the type of operation that is required, and the expected number of operational steps and types of steps that are required.

Again, there are several lessons for teachers and trainers, including – since a numeracy teacher is also a teacher of literacy and language – the need to explicitly teach students how to ‘read’ the text, and excavate the maths from the context, the need to create tasks and teach across the full range of cognitive operations and content areas. This also includes explaining how the complexity of the mathematical information, the type of operations and number of steps involved can increase the difficulty of numeracy tasks.

Deepening our understanding

International assessments of adult literacy and numeracy skills reveal much about what adults can do and cannot do. In particular, adults find solving numeracy problems to be more difficult when questions address more formal aspects of school-type mathematics than equivalent questions phrased in more contextually based formats.

International assessments provide useful evidence to support the upskilling of the language, literacy and numeracy skills and knowledge of educators in the VAWE sector. Building on the empirical research from such international assessments strengthens the links between testing, research, and practice.

Further information:

A longer version of this article was published in Fine Print, the journal of the Victorian Adult Literacy and Basic Education Council.

Lessons from international assessments