A guide to experimental design

An experiment is a type of research method in which you manipulate one or more independent variables and measure their effect on one or more dependent variables. Experimental design means creating a set of procedures to test a hypothesis.

A good experimental design requires a strong understanding of the system you are studying. By first considering the variables and how they are related (Step 1), you can make predictions that are specific and testable (Step 2).

How widely and finely you vary your independent variable (Step 3) will determine the level of detail and the external validity of your results. Your decisions about randomization, experimental controls, and independent vs repeated-measures designs (Step 4) will determine the internal validity of your experiment.

Step 1: Define your research question and variables

You should begin with a specific research question in mind. You may need to spend time reading about your field of study to identify knowledge gaps and to find questions that interest you.

We will work with two research question examples throughout this guide, one from health sciences and one from ecology:

Example question 1: Phone use and sleep

You want to know how phone use before bedtime affects sleep patterns. Specifically, you ask how the number of minutes a person uses their phone before sleep affects the number of hours they sleep.

Example question 2: Temperature and soil respiration

You want to know how temperature affects soil respiration. Specifically, you ask how increased air temperature near the soil surface affects the amount of carbon dioxide (CO2) respired from the soil.

To translate your research question into an experimental hypothesis, you need to define the main variables and make predictions about how they are related.

Start by simply listing the independent and dependent variables.

Research question Independent variable Dependent variable
Phone use and sleep Minutes of phone use before sleep Hours of sleep per night
Temperature and soil respiration Air temperature just above the soil surface CO2 respired from soil

Then you need to think about possible confounding variables and consider how you might control for them in your experiment.

Confounding variable How to control for it
Phone use and sleep Natural variation in sleep patterns among individuals. Control statistically: measure the average difference between sleep with phone use and sleep with phone use rather than the average amount of sleep per treatment group.
Temperature and soil respiration Soil moisture also affects respiration, and moisture can decrease with increasing temperature. Control experimentally: monitor soil moisture and add water to make sure that soil moisture is consistent across all treatment plots.

Finally, put these variables together into a diagram. Use arrows to show the possible relationships between variables and include signs to show the expected direction of the relationships.

Diagram of the relationship between variables in a sleep experiment

Here we predict that increasing phone use is negatively correlated with hours of sleep, and predict an unknown influence of natural variation on hours of sleep.
Diagram of the relationship between variables in a soil respiration experiment

Here we predict a positive correlation between temperature and soil respiration and a negative correlation between temperature and soil moisture, and predict that decreasing soil moisture will lead to decreased soil respiration.

Step 2: Write your hypothesis

Now that you have a strong conceptual understanding of the system you are studying, you should be able to write a specific, testable hypothesis that addresses your research question.

Null (H0) hypothesis Alternate (Ha) hypothesis
Phone use and sleep Phone use before sleep does not correlate with the amount of sleep a person gets. Increasing phone use before sleep leads to a decrease in sleep.
Temperature and soil respiration Air temperature does not correlate with soil respiration. Increased air temperature leads to increased soil respiration.

The next steps will describe how to design a controlled experiment. In a controlled experiment, you must be able to:

  • Systematically and precisely manipulate the independent variable(s).
  • Precisely measure the dependent variable(s).
  • Control any potential confounding variables.

If your study system doesn’t match these criteria, there are other types of research you can use to answer your research question.

What is your plagiarism score?

Compare your paper with over 60 billion web pages and 30 million publications.

  • Best plagiarism checker of 2019
  • Plagiarism report & percentage
  • Largest plagiarism database

Scribbr Plagiarism Checker

Step 3: Design your experimental treatments

How you manipulate the independent variable can affect the experiment’s external validity – that is, the extent to which the results can be generalized and applied to the broader world.

First, you may need to decide how widely to vary your independent variable.

Soil-warming experiment

You can choose to increase air temperature:

  • just slightly above the natural range for your study region.
  • over a wider range of temperatures to mimic future warming.
  • over an extreme range that is beyond any possible natural variation.

Second, you may need to choose how finely to vary your independent variable. Sometimes this choice is made for you by your experimental system, but often you will need to decide, and this will affect how much you can infer from your results.

Phone-use experiment

You can choose to treat phone use as:

Step 4: Assign your subjects to treatment groups

How you apply your experimental treatments to your test subjects is crucial for obtaining valid and reliable results.

First, you need to consider the study size: how many individuals will be included in the experiment? In general, the more subjects you include, the greater your experiment’s statistical power, which determines how much confidence you can have in your results.

Then you need to randomly assign your subjects to treatment groups. Each group receives a different level of the treatment (e.g. no phone use, low phone use, high phone use).

You should also include a control group, which receives no treatment. The control group tells us what would have happened to your test subjects without any experimental intervention.

When assigning your subjects to groups, there are two main choices you need to make:

  1. A completely randomized design vs a randomized block design.
  2. An independent measures design vs a repeated measures design.

Randomization

An experiment can be completely randomized or randomized within blocks (aka strata):

  • In a completely randomized design, every subject is assigned to a treatment group at random.
  • In a randomized block design (aka stratified random design), subjects are first grouped according to a characteristic they share, and then randomly assigned to treatments within those groups.
Completely randomized design Randomized block design
Phone use and sleep Subjects are all randomly assigned a level of phone use using a random number generator. Subjects are first grouped by age, and then phone use treatments are randomly assigned within these groups.
Temperature and soil respiration Warming treatments are assigned to soil plots at random by using a number generator to generate map coordinates within the study area. Soils are first grouped by average rainfall, and then treatment plots are randomly assigned within these groups.

Sometimes randomization isn’t practical or ethical, so researchers create partially-random or even non-random designs. An experimental design where treatments aren’t randomly assigned is called a quasi-experimental design.

Independent vs. repeated measures

In an independent measures design (also known as between-subjects design or classic ANOVA design), individuals receive only one of the possible levels of an experimental treatment.

In medical or social research, you might also use matched pairs within your independent measures design to make sure that each treatment group contains the same variety of test subjects in the same proportions.

In a repeated measures design (also known as within-subjects design or repeated-measures ANOVA design), every individual receives each of the experimental treatments consecutively, and their responses to each treatment are measured.

Repeated measures can also refer to an experimental design where an effect emerges over time, and individual responses are measured over time in order to measure this effect as it emerges.

Counterbalancing (randomizing or reversing the order of treatments among subjects) is often used in repeated-measures design to ensure that the order of treatment application doesn’t influence the results of the experiment.

Independent measures design Repeated measures design
Phone use and sleep Subjects are randomly assigned a level of phone use (low, medium, or high) and follow that level of phone use throughout the experiment. Subjects are assigned consecutively to low, medium, and high levels of phone use throughout the experiment, and the order in which they follow these treatments is randomized.
Temperature and soil respiration Warming treatments are assigned to soil plots at random and the soils are kept at this temperature throughout the experiment. Every plot receives each warming treatment (1, 3, 5, 8, and 10C above ambient temperatures) consecutively over the course of the experiment, and the order in which they receive these treatments is randomized.

Experiments are always context-dependent, and a good experimental design will take into account all of the unique considerations of your study system to produce information that is both valid and relevant to your research question.

Frequently asked questions about experiments

What is experimental design?

Experimental design means planning a set of procedures to investigate a relationship between variables. To design a controlled experiment, you need:

  • A testable hypothesis
  • At least one independent variable that can be precisely manipulated
  • At least one dependent variable that can be precisely measured

When designing the experiment, you decide:

  • How you will manipulate the variable(s)
  • How you will control for any potential confounding variables
  • How many subjects or samples will be included in the study
  • How subjects will be assigned to treatment levels

Experimental design is essential to the internal and external validity of your experiment.

 

What are independent and dependent variables?

You can think of independent and dependent variables in terms of cause and effect: an independent variable is the variable you think is the cause, while a dependent variable is the effect.

In an experiment, you manipulate the independent variable and measure the outcome in the dependent variable. For example, in an experiment about the effect of nutrients on crop growth:

  • The independent variable is the amount of nutrients added to the crop field.
  • The dependent variable is the biomass of the crops at harvest time.

Defining your variables, and deciding how you will manipulate and measure them, is an important part of experimental design.

What is a confounding variable?

A confounding variable, also called a confounder or confounding factor, is a third variable in a study examining a potential cause-and-effect relationship.

A confounding variable is related to both the supposed cause and the supposed effect of the study. It can be difficult to separate the true effect of the independent variable from the effect of the confounding variable.

In your research design, it’s important to identify potential confounding variables and plan how you will reduce their impact.

What is the difference between a control group and an experimental group?

An experimental group, also known as a treatment group, receives the treatment whose effect researchers wish to study, whereas a control group does not. They should be identical in all other ways.

What is the difference between internal and external validity?

Internal validity is the degree of confidence that the causal relationship you are testing is not influenced by other factors or variables.

External validity is the extent to which your results can be generalized to other contexts.

The validity of your experiment depends on your experimental design.

What’s the difference between reliability and validity?

Reliability and validity are both about how well a method measures something:

  • Reliability refers to the consistency of a measure (whether the results can be reproduced under the same conditions).
  • Validity refers to the accuracy of a measure (whether the results really do represent what they are supposed to measure).

If you are doing experimental research, you also have to consider the internal and external validity of your experiment.

Is this article helpful?
Rebecca Bevans

Rebecca is working on her PhD in soil ecology and spends her free time writing. She's very happy to be able to nerd out about statistics with all of you.

7 comments

Josender
September 4, 2020 at 8:38 AM

Thank You so much for the helpful document.

Reply

Ricky Tan
August 23, 2020 at 1:34 AM

This blog was very helpful for applying Lean Startup experimentation methodology. Thank you!

Reply

Saad Muhammad
August 21, 2020 at 9:56 PM

I am a freshman and i want to make a experimental design for my class can anyone please help me make a whole experimental design about phone usage before bed including the stats and all the variables please.

Reply

Ange
July 8, 2020 at 1:25 PM

What are the sources of natural variability in field experiment?

Reply

Pauline AJ
June 14, 2020 at 8:31 PM

Very helpful Rebecca.
in writing up my study, do you suggest I put as heading, Sampling design or Experimental groups, when i am describing the intervention and control groups -- number of subjects in each

Thank you

Reply

William Peters
May 9, 2020 at 3:36 PM

Good source to augment understanding of research.

Reply

Lwande Omondi
March 10, 2020 at 7:20 PM

Very helpful. Thanks

Reply

Comment or ask a question.

Please click the checkbox on the left to verify that you are a not a bot.