7.4 Other design considerations
Many issues must be kept in mind when designing studies, such as selecting the sample and managing confounding. In this section, some specific design considerations are discussed:
Not every design consideration will be relevant to every study.
7.4.1 Carry-over effect
In the Himalaya study, what if patients spent two weeks on the Himalaya 292 diet, then the next two weeks on the refined cereal diet?
Potentially, the influence of the first diet could still be impacting the subjects’ faecal weight for a little while after stopping the first diet. This is called the carryover effect.
The impact of the carryover effect can be minimized by using:
- A washout period or similar; for example, after finishing one diet, the participants spend four weeks on their usual (before study) diet, and then revert to the second diet being used.
- Random allocation: if possible, the order in which the treatments (i.e. the diets) are used should be randomly allocated. That is, some participants start by spending four weeks on the Himalaya 292 diet, then (after a washout period) four weeks on the refined cereal diet; meanwhile, other participants start by spending four weeks on the refined cereal diet, then (after a washout period) four weeks on the Himalaya 292 diet.
Example 3.7 (Carry-over effect) In the Himalaya 292 study, the authors report:
That is, subjects were randomly allocated to a diet: some subjects began the study on the Himalaya 292 diet while others started on the refined cereal diet, No washout period was used; however, since the response variable was recorded after four weeks on the diets, no washout period was necessary.Subjects were allocated randomly to […] dietary treatments according to a cross-over study design with each intervention phase lasting 4 weeks. There was no washout period between phases.
— Bird et al. (2008), p. 1033
FIGURE 7.5: Using a ‘washout’ period to minimize the carry-over effect
7.4.2 Hawthorne effect
What if the patients in the Himalaya 292 study were being watched (or waited for) while defecating?
People often behave differently (either positively or negatively) if they know (or think) they are in a study or are being watched. This is called the Hawthorne effect (McCarney et al. 2007).
Example 3.8 (Hawthorne effect) People are more health-conscious if they know they will be followed up on a regular basis. For example, a study aiming to increase fruit and vegetable intake in young adults (Clark et al. 2019) noted that
The changes that did occur could be explained by the Hawthorne effect […] the intervention […] can inherently cause participants to change behavior because they know they are being observed…
— Clark et al. (2019)
The impact of the Hawthorne effect can be minimized by using:
- Good planning: Sometimes, efforts can be made to reduce the chance that individuals know they are being observed. Often, however, this is not possible when studying people (as ethics usually requires individuals’ informed consent).
- Blinding of participants: Blinding individuals may be possible; that is, the individuals do not know which treatment they are receiving. Note, however, that this is only a partial solution: the participants will almost certainly still know that they are in a study (especially when the individuals are people, as ethics usually requires individuals to be informed).
Example 7.3 (Hawthorne effect) In the Himalaya 292 study, the authors report:
The study was explained fully to the subjects, both verbally and in writing, and each gave their written, informed consent before participating.
— Bird et al. (2008), p. 1033
That is, the subjects knew they were in a study. As is usual, this was an ethics requirement (in this case, from the Ethics Committee of the CSIRO). The Hawthorne effect may influence the results.
However, the subjects did to know which diet they were on:
Volunteers were not told the identity of the test cereal in the foods provided to them.
— (Bird et al. (2008), p. 1033)
Example 7.4 (Hawthorne effect) In an experimental study (Lorenz et al. 2019) to compare the efficacy of a new type of toothpaste, participants were given two types of toothpaste to use (a new type, and an exisiting type), and evaluations of plaque remaining on the teeth were taken. The authors state that:
… a plaque-reducing effect was seen not only in the test group but also in the control group. This phenomenon is due to the so-called Hawthorne effect that can lead to an overestimation of the effect and false positive results.
— Lorenz et al. (2019), p. 5
That is, since all participants knew they were being assessed after brushing their teeth, there may have been a tendency to brush their teeth better than usual. The authors then state:
To minimize the Hawthorne effect, longer study durations of more than 6 months were suggested.
— Lorenz et al. (2019), p. 6
7.4.3 Placebo effect
What if people thought they were on the wholegrain diet, but they weren’t? Perhaps surprisingly, individuals in a study may report effects of a treatment (either positive or negative), even if they have not received an active treatment. This is called the placebo effect.
Managing the placebo effect is difficult! However, impact of the placebo effect can be minimized using:
- Good planning: Experts may be able to suggest other ways exist to reduce this effect (e.g. measuring long-term effects rather than just short-term effects (Example 7.4) or using different designs (Kessels, Mozer, and Bloemers 2019)).
- Controls: Use units of analysis without the treatment applied, but as similar as possible in every other way to those units of analysis receiving the treatment. This allows the effect of the treatment to be ssessed, over and above the placebo effect.
Sometimes the control group receives a placebo. A placebo is a non-effective treatment. Those who receive the placebo should be selected through random allocation when possible. Sometimes, using a placebo is unethical. The Wikipedia entry about placebos is intriguing.
Example 3.10 (Placebo effect) In the Himalaya 292 study, the authors report
That is, the subjects were blinded to the diet they were exposed to. However, some may think they are on the refined cereal or Himalaya diet, and respond accordingly (perhaps unconsciously).On each day of the intervention periods, volunteers were asked to consume a combination of bread, breakfast cereal, muffins and crackers that would supply in total 103g of the test cereal. The aim was for each volunteer to consume 60g cereal flakes (or puffed rice for the refined cereal diet), two slices of bread, one muffin and six savoury crackers each day. Volunteers were not told the identity of the test cereal in the foods provided to them
— (Bird et al. (2008), p. 1033)
FIGURE 7.6: Pain relief, for various pain relief medicine
7.4.4 Observer effect
What if the researchers assessing the outcomes knew the diet allocated to each patient? Perhaps surprisingly, this can have an (unconscious) impact on the values of the response variable. This is called the observer effect (or, in experiments, sometimes called the experimenter effect).
The impact of the observer effect can be minimized using:
- Good planning: Think about reasonable means for minimising the observer effect.
- Blinding of the researchers: Blinding the researchers may be possible: that is, the people giving the treatment and the people evaluating the treatment do not know what treatment has been given. Instead, a third party can be used. For example, the researchers may give an assistant two drugs labelled A and B. The assistant then administers the drug and evaluates the participants’ response to the treatments. Later, the assistant tells the researchers whether Drug A or Drug B performed better, but only the researchers know what drugs the labels A and B refer to.
The observer effect does not just apply to situations where people are used as participants.
Example 7.6 (Observer effect) ‘Clever Hans’ (https://en.wikipedia.org/wiki/Clever_Hans) was a horse that seemed to be able to perform simple mental arithmetic. After much study, Carl Stumpf realised that the horse was responding to involuntary (and unconscious) cues from the trainer. This was discovered, in part, by using an experiment where the people interacting with the horse were blinded.
The same effect has been observed in narcotic sniffer dogs (Bambauer 2012), who may respond to their handlers’ unconscious cues.7.4.5 Describing blinding
Blinding is when those involved in the study do not know which comparison group the study individuals are in:
- A study can blind the researcher to knowing what comparison group the study individuals are in.
- A study can blind the participants to knowing what comparison group they are in.
- A study can blind the analysts to knowing what comparison group the inidividuals are in during analysis.
If only the participants are blinded, the study is called single blind. If both the researchers and participants are blinded, the study is called double blind. If the researchers, participants and the analyst are blinded, the study is called triple blind. However, for clarity, explicitly stating who or what is blinded is best. Blinding should be considered in all studies, when possible (and it is not always possible).
Blinding of participants does not just apply to people; it is also relevant with animals (Example 7.6 about Clever Hans).