Research
Comments 6

You don’t need RCTs to know what works

In recent times a great deal has been made of the importance of using RCTs in education in order to build a body of evidence about which practices are effective. This is very laudable, but as others have pointed out, there are many implementation issues with RCTs. My concern here is not so much with the implementation, as with their usefulness – or possible lack thereof. While the statistical power of a randomised controlled trial potentially provides greater confidence in the outcome, interpretation of the data needs to be approached cautiously.

Take, for example, a trial using a randomised control group. The experimental group receives a reading intervention and the other does not. The mean of the intervention group improves more than the control group. How much does this tell us? Who gained the most benefit? Can we be confident that the intervention alone was responsible for the raised average? Did most students move a little, or did a few move a lot. Overall, it can be argued, on an average basis the intervention is more effective than no intervention. But for how many of the students? What were their characteristics? A good discussion in the research paper would attempt to identify the reasons why the intervention worked for some students more than others, but unless the experiment is set up carefully this information may be hard, if not impossible, to extract. In fact, to make a meaningful interpretation, unless the impact was very uniform, we would get close to having to compare individual results in order to make sense of the data.

median_or_mean_height

There is a much more efficient, and often more illuminating, way of tracking the impact of an intervention.  It also allows us to see quickly if results can be replicated. It is known as the single-subject design. It is well suited to the needs of teachers in the classroom, enables them to identify useful information that they can immediately employ in their teaching with particular students, and allows for individual variations while enabling the results of multiple students to be compared.

In the single-subject design, a baseline of current student performance is taken (the A phase). After this information is gathered, the intervention is introduced (the B phase). In the B phase, the impact on the student’s performance is tracked. When the B phase has been completed, we can conclude the experiment (also called a ‘case study’ design). But at this stage it will not really prove that the intervention was the reason for any change in the student’s performance. To confirm that, we need to go back to the baseline conditions (the A phase again) and after a suitable period we then reinstate the intervention (the B phase). If the same impact on performance occurs again, we have replication. This replication suggests that the intervention is the factor that is making the difference. And if we compare the results of individual students, and see a similar pattern of replication, our inference is strengthened. This design is often referred to as ABAB.

Screen Shot 2014-11-08 at 17.32.04

The famous example above was published way back in 1968, and shows how the mathematics errors and corrects of a student (‘Bob’) rose and fell depending on which his teacher gave the most attention to. At first (the A phase) Bob was making lots of digit reversals in his addition, and the teacher spent time patiently correcting each one with him. In the B phase she paid no attention to his errors but did give time to commend him on his corrects. His errors rapidly reduced and his corrects increased proportionately. Then the researchers moved back to the A phase. When the teacher gave more attention to Bob’s errors, the errors increased. Repeating the B phase conditions produced a replication of the same pattern as before – Bob’s errors decreased when his teacher gave them no attention.  (Note to avoid any confusion: the study did not suggest that teacher attention caused the errors, but that Bob was making errors in order to get the teacher’s attention. You can read the full study here).

This article by John O. Cooper (unfortunately behind a paywall!) explains how behavioural methodologies including the variations of single-subject design were developed, and how these developments made the approach increasingly useful for teachers in the classroom. There is also a short outline of the increasing usefulness of single-subject design in this comprehensive text, Research Methods in Education, by Cohen, Manion and Morrison (p.284).

The charts of students in Thinking Reading programmes look more like case study designs. Because they need to catch up to their peers as quickly as possible, we haven’t been reverting to baseline to see if progress stalls. This format is called ‘multiple baseline’, and does not require the subject to return to baseline because replications with other subjects will confirm the relationship between the intervention and the student’s performance.  The repeated replications provide strong evidence that the reading programme is responsible for the improvements in reading scores. You can see the charts on the Case Studies and Sustained Progress pages on the Thinking Reading website.

mean

Visit our website

You may also be interested in:

Treasures Old and New

A Matter of Sensitivity

The Writing on the Wall

Teaching Reading is Rocket Science

6 Comments

  1. Thanks for the post – it’s an interesting and important question. I agree that RCTs are sometimes a bit over-hyped as a research methodology, but I would argue that they offer us the strongest protection against bias when evaluating interventions, particularly where researcher blinding is used. I don’t think any of us would be happy to accept medical treatments derived from single-case studies, as there’s too much risk of individual variations creating idiosyncratic findings that can be difficult to generalise to a wider, more heterogeneous clinical population. The same applies withe educational research – we need representative samples of a range of learners.

    A couple specific comments re the post – I don’t think it’s correct to say that an education RCT compares a group receiving an intervention with a “no intervention” group. The children in the control group receive an intervention that is known as “standard practice” – because everyday classroom experience is of course an intervention in itself.

    You state: “The mean of the intervention group improves more than the control group. How much does this tell us? Who gained the most benefit?”
    It can tell us quite a lot actually – like whether the difference is statistically and clinically (“real world”) significant, and we can assess the magnitude of the effect of the intervention (broadly small, medium, large – a la the work of John Hattie). It’s also very easy to look at subgroups (e.g. boys Vs girls; children from ethnic minorities; children with identified special learning needs, etc) and see if there is a differential degree of benefit – this is often an important aspect of the data analysis in fact.

    You also ask “Can we be confident that the intervention alone was responsible for the raised average?” That’s an important question for all interventions, including single case methodology – and any researchers worth their salt will consider other possible intervening variables in their analysis of the findings. Finally, you ask “Did most students move a little, or did a few move a lot?” Again, this is a question that is easy to tackle at the analysis stage.

    Personally, I think where RCTs have under-delivered has been in situations (mainly in medical research) where researchers have been overly selective about inclusion/exclusion criteria. For example, when researchers are examining a new medical treatment, it’s all well and good for experimental rigour to recruit patients with good English language skills who will be compliant with the treatment, but that’s not who GPs see walking into their clinics each day. So an intervention might be efficacious (work under experimental conditions), but not effective (work under real-world conditions).

    So I think we need to give RCTs a good go in education, but we need to closely scrutinise the methodology, analysis and interpretation – as indeed we should with all research.

    Like

    • Thanks, Pam. I really appreciate you balancing out the discussion by elaborating on the issues around RCTs, including their strengths. My post is primarily focused on introducing people to single subject design as a more accessible approach to what is going on in our own classrooms, but thank you for clarifying why RCTs remain an important part of the research repertoire.

      With regard to the control group, I should perhaps have clarified that a control group can also receive an alternative intervention. However, in the case of reading at secondary school, I certainly think that students do not receive reading help as part of their every day experience.

      I guess my over-riding concern regarding RCTs is the extent to which they are accessible to practitioners, whereas developing a multitude of single subject replications can be much more achievable. Thank you again for such a thorough response.

      Like

  2. I definitely agree that the RCT is not a technique for practitioners to employ – however hopefully over time the results from RCTs that others have conducted will be – and ideally in a form that is readily accessible – both in terms of locating papers and also deriving take-home messages. I agree that practitioners should be encouraged to use single-case methodology – and in fact reading interventions probably lend themselves well to this. Anything that encourages a “scientist-practitioner” model is a good thing!

    Like

  3. Pingback: Building on the Evidence | thinkingreadingwritings

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s