EdTech in the service of learning: The importance of evaluating education technology

International comparisons show that Australian schools are among the highest users of computer-based technologies in the world, both in terms of numbers of devices in schools and the amount of time spent using those devices.

A couple of years ago, my daughter invited a French exchange student from her class to have dinner at our place. I asked her what she thought the main differences were between her school in France and school in Australia. Her immediate response was - computers. ‘In my school in France we have a room with computers where we learn programming and other skills,’ she said. ‘In Australia, computers are everywhere in the school and everything you do, you do with computers.’

Where is the evidence?

Given the ubiquity of digital technologies in our schools, it might surprise people to learn that there is little evidence that the adoption of these technologies is associated with improvements in school effectiveness. Reviews of the research literature tend to conclude that there is little or no evidence that computer-based instruction leads to improvements in learning outcomes or student engagement.

Nor is there evidence that adoption of technology in schools leads to efficiency gains at the school or system level. That is not to say that educational technologies have not had an impact on school education, but as Michael Fullan reminds us, change is not the same as progress. The ubiquity of computer-based technologies in classrooms, staff rooms, offices and homes has certainly altered the daily experience of students, teachers, educational leaders and administrators, situated within policies which have driven billions of dollars of spending on hardware, software, infrastructure, support services and training.

The challenge with implementation

We know that there are success stories out there and we know that some of the biggest companies in the world are intensely focused on technology in education. We also know that proponents of technology have linked edtech to precisely the kinds of approaches that research tells us should result in improved learning outcomes, such as personalised learning, differentiation, and student-centred approaches.

The answer from research is that while technology has the potential to lead to improved outcomes, the way it is implemented in systems, schools and classrooms is not always optimal and this results in a mix of good, bad and neutral outcomes. This would go some way to explaining why there are many studies of particular education technology programs that demonstrate improvement at a local level but little evidence of technology adoptions making a mark at scale. It might also explain why there is relatively good evidence of the usefulness of technology in addressing the needs of students for whom traditional and mainstream educational approaches and systems have been less effective, including students with special educational needs.

These observations have led to a stronger focus in research on trying to understand and develop the skills and knowledge that teachers and school leaders need to choose and use technology effectively. While there has been some progress in this in research, unfortunately, a focus on teacher professional learning and capacity development has not always been reflected in the priorities of education systems and schools or the approaches of technology companies.

In high performing systems, edtech tends to be clearly situated within broader national goals

As part of a recent review of a national education system undertaken by the Australian Council for Educational Research (ACER), I helped develop an international benchmark on effective use of education technology. This was based on a review of the research literature and evidence from studies of international best practice.

The first takeout from that review was that in high performing systems, the use of educational technology in schools tends to be clearly situated within broader national goals around the development of an advanced information technology sector and knowledge-based economy. In high performing systems, there are long term plans and commitments around the adoption and integration of educational technologies and strategies for integrating technologies effectively in the learning system.

The purposeful integration of technology to optimise teaching and learning outcomes

For technology to be effective in schools, it must be purposively directed at supporting improvements in learning outcomes. In adopting technologies, we should be able to define their purpose, understand their role in the learning system, identify metrics for success and processes for evaluation.

Crucially we need to understand that in most cases technology adoption represents the replacement of some existing educational provision with something new. We must always frame adoption of technology in education in terms of improvement.

The Technology Evaluation Value Framework

I have been working with colleagues at ACER to develop a framework for schools to evaluate the impact of technology at a school level. The Technology Evaluation Value Framework is being designed as a capacity development program for schools, teachers and school leaders. It helps focus their attention on crucial questions of adoption and evaluation, including:

• What is this technology intended to do?
• How will it do it?
• What are the costs (and opportunity costs)?
• How is success defined?
• What is the decision-making process for continuing, modifying or abandoning the program?

It is hoped that using this framework in decision-making will help build schools’ capacity to think purposefully about technology adoption and use and become more discerning – and demanding – consumers of education technology.

The future role of edtech

Twenty years ago, Larry Cuban bemoaned how large amounts of money were being spent on technology without any evidence of its impact on student learning while there were so many other urgent, unmet needs. A decade ago, Emeritus Professor Phillip Hughes turned this around and argued that there were too many problems in education for us to ignore the potential for technology to make a difference.

More recently, Professor Neil Selwyn has drawn our attention to what he calls technological determinism in education: the sense that technology is progressive, advanced and inevitable, and if education doesn’t come to grips with it, it will be left by the wayside. He argues that this helps explain why so much technology is being adopted in educational settings without clarity of purpose or a means for evaluating its impact.

The big challenges for education identified in the early 2000s remain: how can we best prepare students for life, work and further study in the 21st century; and, how can we ensure every learner achieves their best?

It remains to be seen what, if any role technology can play in addressing those challenges. If it is to make a positive contribution, we must all – educators, systems managers, technology companies, researchers – reorient our thinking and expectations: technology in the service of learning, not education racing to keep up with technology.