How do you start to build an evidence base?
We all know that it’s important to know the evidence behind particular practices, both for what we do in the classroom and what we do in interventions. In education, we used to be subject to fads and fashions, but that is beginning to change. Now, we are much more likely to hear the question, “Where is your evidence?”
This can only be a good thing – as long as people have a sound understanding of what constitutes evidence. Because randomised controlled trials have been touted as the “gold standard”, that is now all that “evidence” means for some. In reality, there is a hierarchy of types of evidence, and all of it is useful in some way. There is a process of collecting evidence prior to developing a randomised controlled trial, so that variables are understood and practices are validated as being worthy of investigation before the investment of time and money that RCTs require.
But there’s an obvious problem here for curriculum and intervention designers. How do you build an evidence base for a new intervention? The answer is that we build on the basis of converging findings in the research literature. Stanovich and Stanovich (2006) point out that where direct experimental evidence is lacking,
“. . . [teachers should consider] proof of reason-based practice that converges with a research-based consensus in the scientific literature. This type of justification of educational practice becomes important when direct evidence may be lacking (a direct test of the instructional efficacy of a particular method is absent), but there is a theoretical link to research-based evidence that can be traced.”
If you are in the process of deciding on a new intervention, remember that there should be a sound basis in the research literature for all the critical components of the programme. As an illustration, we have outlined the research base for four key elements of Thinking Reading below. (We have also outlined our own direct evidence here.)
Systematic Synthetic Phonics
“[The National Reading Panel] concluded that the results of a meta-analysis of the results of 66 comparisons from 38 different studies indicated “solid support for the conclusion that systematic phonics instruction makes a bigger contribution to children’s growth in reading than alternative programs providing unsystematic or no phonics instruction.” (ibid)
More broadly, Hattie’s Visible Learning meta-analysis (2009) found that phonics had an average effect size of 0.6 from 425 studies . “Overall, phonics instruction is powerful in the process of learning to read – both for reading skills and for reading comprehension.”
Hattie’s summary of the research on Direct Instruction across 304 studies: “The effects of Direct Instruction are similar for regular (d=0.99) and special education and lower ability students (d=0.86), . . . for reading (d=0.89) . . .” Direct Instruction was also the clear winner in a large-scale, longitudinal study called Project Follow Through in the United States, achieving far higher gains in basic skills, problem-solving and self-esteem than the other eight programmes involved. Direct Instruction is based on detailed theoretical work which was field-tested multiple times in every respect by Engelmann and Carnine (1991). These principles are integral to the design of Thinking Reading.
Precision Teaching is not actually a method of teaching: it is a method of assessment which yields powerful information for teachers about the rate of student learning. It is based on the principle that fluency is a more sensitive and useful performance criterion than percentage correct.
“Since the late 1960’s, Precision Teaching practitioners have been developing and refining estimated fluency standards for a wide range of skills, based on observation of thousands of students and how they have been able to perform after achieving (or not achieving) specific performance levels.” (Binder, Haughton & Bateman, 2002).
Kubina (2003) discusses the importance of oral reading fluency:
Research has shown that oral reading fluency, or ORF, serves as one of the best measures of basic reading competence. Fuchs, Fuchs and Hosp (2001) conducted a literature review and found ORF predicted comprehension better than direct measures of reading comprehension such as questioning, retelling, and cloze. ORF measures involve recording the number of words read aloud correctly and incorrectly per minute. While oral reading fluency has received considerable attention in the reading literature, Precision Teaching further enhances its use by including performance standards.
In Thinking Reading, we use Precision Teaching performance standards (fluency criteria) to assess student progress and to make teaching decisions on a daily basis.
Applied Behaviour Analysis
Applied Behaviour Analysis is “the science in which tactics derived from the principles of behaviour are applied systematically to improve socially significant behaviour and experimentation is used to identify the variables responsible for behaviour change.” (Cooper, Heron, & Heward, 2007, p. 21). Applied Behaviour Analysis has been a distinct field of research since the 1930s. Note that the ‘behaviour’ in Applied Behaviour Analysis can refer to a complete range of human behaviours, including cognitive processes, emotional responses, students’ approach to tasks, their task persistence, how they respond to praise or having errors pointed out. There are over 25 journals devoted exclusively to research and discussion on Applied Behaviour Analysis. For a discussion on its utility, you may wish to study this lecture by Professor Carl Hughes.
A key element of Applied Behaviour Analysis is the use of the single-subject study design to measure the impact of teaching decisions on student progress. In Thinking Reading, we use these principles to ensure that each element of students’ performance is able to be tracked, and their progress represented visually; the programme also contains numerous distinct practices to ensure that students develop high levels of intrinsic motivation.
In addition to the many case studies and the aggregated data reviewed by Professor Greg Brooks in “What works for children and young people with literacy difficulties?”, our evidence base also consists of the combination of powerful teaching approaches outlined above. Such combinations have been shown to be additive in their effect (see for example, Vargas et al, 2002) and this is why we expect – and can show evidence for – typical gains in students’ reading averaging two months’ progress per lesson. As we’ve said before, RCTs are very useful if carefully designed, but you don’t need RCTs to know what works
You may also wish to read this article by Kozloff, Lanunziata, and Cowardin (1999), who summarise the links between Direct Instruction, Precision Teaching and Applied Behaviour Analysis in more detail than we can here.
Brooks, G. (2016). What works for children and young people with literacy difficulties? (5th edition). Retrieved from http://www.interventionsforliteracy.org.uk/assets/What-Works-5th-edition.pdf The Dyslexia-SpLD Trust
Binder, C., Haughton, E, & Bateman, B. (2002). Fluency: Achieving True Mastery in the Learning Process. Retrieved from http://www.fluency.org/Binder_Haughton_Bateman.pdf 16 October 2016.
Camilla, G., Vargas, S. & Yurecko, M. (2003). Teaching Children to Read: The Fragile Link Between Science and Federal Education Policy. Education Policy Analysis Archives, 11(15). Retrieved from http://epaa.asu.edu/ojs/article/view/243
Cooper, J. O., Heron, T. E & Heward, W.L. (2007). Applied Behaviour Analysis. (2nd edition). Harlow: Pearson
Engelmann, S. & Carnine, D. (1991). Theory of Instruction: Principles and Applications (revised edition). Oregon: ADI Press
Hattie, J. (2009). Visible Learning. Oxon: Routledge
Hughes, C. (video) (2012). J Carl Hughes Presentation to Second QUART conference, Belfast. Teaching Academic Curriculum: Effective Inclusion: Queen’s University, Belfast
Kubina, R. M. Jr. & Starlin, C. M. (2003). Reading with Precision. European Journal of Behaviour Analysis. 4. 13-22.
Stanovich, Paula J. and Stanovich, Keith, E. (2003) Using Research and Reason in Education: How Teachers Can Use Scientifically Based Research to Make Curricular & Instructional Decisions. New Hampshire: RMC Research Corporation.