The hard work of improvement
Geoff N Masters
Australian Council for Educational Research
A common strategy for promoting improved employee or organisational performance is to place a strong focus on organisational results. For example, in commercial businesses, it is common to focus on results such as sales volumes, total business revenue, annual company profit or share price. With desired results clearly identified, results metrics are then established to measure existing performance levels, set targets for improvement, monitor improvement over time and hold employees accountable for achieving better results.
The perceived advantage of focusing on results is that it clarifies and concentrates effort on the main game: the key purpose of the organisation’s work. It also provides a basis for evaluating the performances of employees and the organisation as a whole, while giving employees freedom to find and create strategies for achieving improved results.
As part of their drive for improvement, organisations also sometimes attach incentives to results, either in the form of rewards – for example, increased pay for increased sales – or sanctions such as the threat of dismissal, transfer or closure. These forms of extrinsic motivation usually are based on the assumption that the key to improved results is greater employee effort.
However, there is growing evidence that focusing on results alone is an ineffective improvement strategy in many contexts and often leads to unintended and undesirable behaviours. Exhortations and incentives to improve are of limited value if equal attention is not paid to the guidance and support employees need to make improvements in their practice.
It is now widely recognised that, when performances are evaluated only in terms of measurable results, employees and organisations find ways to ‘game’ the system. Hospitals improve patient survival rates by taking fewer high risk patients; companies maximise short-term returns to shareholders by not investing in long-term growth strategies.[i ] And in extreme cases, a narrow focus on results produces corrupt behaviour – for example, manipulating a company’s financial results to make its performance look better than it is.
There are obvious lessons in this experience for current efforts to improve educational outcomes. Following the model adopted in business, education systems in a number of countries are now attempting to drive improved performance by placing a strong focus on results such as student test scores, participation levels and school completion rates. These results metrics are being used to set targets for improvement and to hold teachers and schools accountable for producing better results, often with accompanying incentive schemes.
For example, following the US No Child Left Behind Act of 2001, which required all American schools to demonstrate ‘adequate yearly progress’ in improving reading and mathematics results, more than thirty US states now provide rewards and/or sanctions to schools based on changes in student test performances. Some of these systems link performance pay for teachers to improved test scores, provide financial rewards to schools for improving their performances and transfer principals and close schools when test results decline or fail to improve.
Such ‘high-stakes’ uses of test results often undermine the purposes for which tests were designed in the first place: namely, to clarify educational standards, ensure that all students achieve essential skills and knowledge, monitor trends over time and evaluate the effectiveness of educational initiatives and programs.
And, as in business settings, efforts to drive improvement by focusing on results alone have produced a range of unintended and undesired behaviours. Under results-driven incentive schemes, there is evidence of schools assigning their best teachers to the grades in which high-stakes testing occurs; teachers spending minimal time on untested aspects of the curriculum; large amounts of time being spent on drilling students in test-taking strategies; lower-achieving students being withheld from testing; and schools making surface-level responses to achieve short-term test gains at the expense of deeper, longer-term improvements in classroom teaching.[ii]
A consequence of gaming behaviours of this kind is ‘score inflation’ – increases in measured results that are not reflective of real improvement. A number of studies have shown that apparent improvements on high-stakes tests have not been matched by improvements on low-stakes tests of the same content. For example, large gains on fourth-grade reading tests in the high-stakes Kentucky state assessment in the early 1990s were not matched by reading gains on the low-stakes National Assessment of Educational Progress (NAEP) in that state.[iii] Teachers often describe the practices they adopt in response to results-driven improvement efforts as inconsistent with their own understandings of good teaching.
And, not surprisingly, when strong incentives are attached to results, instances of corrupt practice also emerge. These practices include exposing students to test papers prior to testing, placing answers to test questions on classroom walls and altering students’ answers following testing.
At a more fundamental level, questions are being asked about the effectiveness of incentives as a way of improving performance. Research in psychology has shown how performances can be reduced if results-driven incentives take the place of intrinsic motivation. In other words, rewarding people for behaviour that they would have engaged in anyway can sometimes reduce levels of performance.
A recent US review of the effectiveness of results-driven school improvement efforts not only highlighted how incentive programs are producing unanticipated and unintended consequences, but also questioned the extent to which incentive programs have produced measurable improvements in student results. The review concluded: ‘The research to date suggests that the benefits of test-based incentive programs over the past two decades have been quite small… The guidance offered by this body of evidence is not encouraging about the ability of incentive programs to reliably produce meaningful increases in student achievement.’[iv]
In business, too, there is growing recognition that a focus on results alone is not the answer to improved performance. Research into the practices of successful companies shows that, in addition to focusing on results, high-performing companies invest heavily in building internal capacity and long-term organisational health. These investments often have no obvious benefit for measurable results, but enhance the capacity of the organisation to adapt to changing circumstances. In companies that are successful in the long term, there is a strong focus on promoting positive cultures, a shared sense of mission, effective leadership, continuous learning, teamwork, staff commitment, openness, honesty, innovation and creativity. Recent research suggests that focusing on organisational health in combination with results is twice as effective as focusing on organisational health alone, and nearly three times as effective as focusing on results alone.[v]
In general, qualities and practices such as culture and teamwork are harder to define and to measure than sales volumes and test scores. But research is clear that a lop-sided focus on results alone not only drives unintended ‘gaming’ behaviours on the part of employees and organisations, but also is less effective in achieving genuine, long-term improvement.
Some organisational efforts to focus attention on work practices and internal processes go no further than a relatively superficial focus on compliance. These efforts attempt to drive improvement by ensuring that employees are doing the jobs expected of them and that organisations have in place the processes and practices believed necessary for success. Compliance approaches sometimes are adopted to minimise risk by specifying minimally acceptable standards of practice and behaviour.
A feature of compliance approaches is that they usually involve relatively straightforward observations. Does the school have a behaviour management plan? Is the school’s annual report available on its website? Does the annual report show progress against goals for the current review period? Has the entire Year 5 curriculum been covered? Has this teacher participated in the requisite hours of professional development? Does the teacher comply with relevant legislative, administrative and organisational requirements? Has the teacher participated in assessment moderation activities? Does the teacher maintain an orderly classroom environment? Is there evidence of the teacher using a range of teaching strategies?
Compliance metrics usually involve some kind of dichotomy – for example, yes/no; present/absent; completed/not completed. In general, if observations of practices and processes are recorded using checklists and tick-a-box formats, then the focus of attention probably has not moved beyond compliance.
A further feature of compliance approaches is that they tend to have the same expectations of all organisations and of all employees in the same role in an organisation. Expectations are not differentiated because the purpose is to ensure that minimum standards are being met.
Although compliance approaches have a role to play in improvement efforts, genuine improvement invariably requires a deep engagement with the quality of employee practice. Improvement depends not only on ensuring minimally acceptable practice, but also on understanding and promoting best practice and harnessing employees’ intrinsic motivation to maximise the effectiveness of their work.
In education it has been common to argue that teachers, as professionals, should be left alone to make their own judgements about appropriate teaching interventions and strategies. But it is interesting to contrast this argument with practice in other professions such as medicine where substantial work has been done to capture accumulated professional knowledge about best practice. Although professional judgement has an obvious place in the practice of medicine, there also are ‘standards of care’ that practitioners are expected to follow – agreed best practices for the handling and treatment of particular medical conditions based on accumulated professional experience.
Sustained, long-term improvements in educational outcomes similarly depend on studying, understanding, describing and promoting best practice throughout the profession. Such work goes well beyond mapping minimal expectations of schools, teachers and school leaders. It goes to the detail of highly effective teachers’ pedagogical practices and highly effective leaders’ day-to-day leadership work. It involves understanding the expert knowledge and skills that underlie best practice. And it probably involves the eventual development of ‘standards of practice’ – agreed best ways of professionally intervening and addressing particular kinds of educational problems and challenges.
Unlike the minimal expectations and compliance requirements of employers and governments, highly effective practices of this kind can be identified only through the systematic study of professional practice. What is it that expert mathematics teachers know and do that less able teachers do not? What are the distinguishing features of highly effective school leadership? What does it mean to become more expert in the assessment of student learning and the provision of effective feedback? The hard work of improvement begins with research-based understandings of the nature of excellent practice, whether of classroom teachers, school leaders or education systems.
And because excellence is developed incrementally over time, quality metrics always are based on a developmental view. They describe increasingly deep knowledge, understandings and practices in specific aspects of professional work, and so provide a framework for establishing where employees and organisations are at any given time in their ongoing development and what actions and learning may be required for further improvement.
Much is now known about what it means to become a more expert teacher. The development of pedagogical expertise includes becoming better at creating supportive learning environments in which all students are emotionally engaged and motivated to learn; establishing starting points for teaching by exploring where individuals are in their learning and development; making explicit to students what they are expected to learn; designing learning opportunities to address the needs of students who are at different points in their learning; connecting new material to past learning and assisting students to see continuity in their learning over time; promoting deep learning by emphasising underlying principles, concepts and big ideas; demonstrating explicitly what students are to do and checking that learning is occurring; taking advantage of teaching and learning opportunities as they arise; providing ongoing feedback to students on their learning; and promoting positive student beliefs about their own capacity to learn.
Much is also known about the nature of school improvement. Schools usually become more effective places of learning by developing and implementing improvement strategies to which all staff are committed; systematically monitoring improvements in student outcomes and sharing this information across the school community; setting and communicating high expectations of all learners; identifying student needs and deploying staff and school resources in ways that best address those needs; creating a professional teaching team with high levels of subject knowledge and pedagogical expertise; ensuring whole-school curriculum clarity and vertical alignment to provide continuity of student learning across grades; and promoting highly effective, evidence-based teaching practices throughout the school, including differentiated teaching to ensure that every student is engaged and learning successfully.
Studies of education systems that have achieved significant gains in student performance over time are providing insights into the nature of system improvement. These studies are suggesting that education systems become more effective by aligning effort at all levels of the system around the core goal of improving student learning. Such systems diagnose and study the details of student, school and system performance and target effort and resources on underperforming parts of the system. They build professional capacity by attracting more able people into teaching and by improving the effectiveness of initial and continuing teacher education, and they work to ensure that excellence is distributed throughout the system. High-performing education systems understand the essential importance of improving pedagogical practices and take a long-term perspective on changing the culture of the system – the values, understandings, skills, practices and relationships necessary for significantly enhanced performance.[vi]
Whether at the level of teachers and leaders, whole schools or entire systems, significant and sustained improvements in performance require more than a focus on results and more than compliance with standards and minimal expectations. The hard work of improvement requires deep engagement with the quality of practice.
In this context, research-based elaborations of what improving practice looks like – in the form of developmental frameworks and rubrics – provide quality metrics that enable individuals, organisations and systems to identify and reflect on current levels of practice, design improvement strategies and monitor improvements in their practice over time.[vii]
Improved performances can be achieved by promoting greater attention to the results an organisation was established to deliver; by confirming that employees are performing the roles and tasks expected at their levels; and by ensuring organisational compliance with minimal standards of practice and behaviour. But deep and lasting improvements depend on studying and understanding highly effective professional practices and providing support and creating the conditions that make these practices part of ongoing day-to-day work.
References
[i ] Prendergast, C (1999). The provision of incentives in firms. Journal of Economic Literature, 37, 7-63.
[ii] Hamilton, L (2004). Assessment as a policy tool. Review of Research in Education, 27, 25-68.
[iii] Koretz, D, & Barron, SI (1998). The Validity of Gains on the Kentucky Instructional Results Information System (KIRIS). MR-279-PCT/FF. Santa Monica, CA: RAND.
[iv] Hout, M & Elliott, SW (Eds.) (2011). Incentives and Test-Based Accountability in Education. Washington, DC: The National Academies Press.
[v] Keller, S & Price, C (2011). Beyond Performance: How Great Organizations Build Ultimate Competitive Advantage. Hoboken, NJ: John Wiley & Sons.
[vi] Fullan, M. (2011). Choosing the wrong drivers for whole school reform. Seminar Series 204. Melbourne: Centre for Strategic Education.
[vii] Masters, GN (2010). Teaching and Learning School Improvement Framework. Melbourne: Australian Council for Educational Research.
© Australian Council for Educational Research 2011
19 Prospect Hill Road
Camberwell, VIC 3124
AUSTRALIA
