Systematic Review | Definition, Example & Guide

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

Example: Systematic review
In 2008, Dr. Robert Boyle and his colleagues published a systematic review in the Cochrane Database of Systematic Reviews.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

They used systematic methods to find, select, and synthesize all available evidence, and they described these methods in detail in their article. Based on the evidence, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema.

What is a systematic review?

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias. The methods are repeatable, and the approach is formal and systematic:

  1. Formulate a research question
  2. Develop a protocol
  3. Search for all relevant studies
  4. Apply the selection criteria
  5. Extract the data
  6. Synthesize the data
  7. Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative (qualitative), quantitative, or both.

Systematic review vs. meta-analysis

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis. A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size.

Prevent plagiarism, run a free check.

Try for free

Systematic review vs. literature review

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Systematic review vs. scoping review

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

When to conduct a systematic review

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention, such as a medical treatment.

To conduct a systematic review, you’ll need the following:

  • A precise question, usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
  • A team of at least three people. Best practices require three people for certain steps of the systematic review process. Ideally, in addition to your research team you’ll also have an advisory group of about six people.
  • Access to databases and journal archives. Often, your educational institution provides you with access.
  • Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
  • Bibliographic, word-processing, spreadsheet, and statistical software. For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

Pros and cons of systematic reviews

A systematic review has many pros.

  • They minimize bias by considering all available evidence and evaluating each study for bias.
  • Their methods are transparent, so they can be scrutinized by others.
  • They’re thorough: they summarize all available evidence.
  • They can be replicated and updated by others.

Systematic reviews also have a few cons.

  • They’re time-consuming.
  • They’re narrow in scope: they only answer the precise research question.

Step-by-step example of a systematic review

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

  • Allow you to more effectively communicate your research to other researchers and practitioners
  • Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO:

  • Population(s) or problem(s)
  • Intervention(s)
  • Comparison(s)
  • Outcome(s)

You can rearrange these four components to write your research question:

  • What is the effectiveness of I versus C for O in P?

Sometimes, you may want to include a fifth component, the type of study design. In this case, the acronym is PICOT.

  • Type of study design(s)
Example: Formulate a research question (PICO)
Boyle and colleagues were interested in:

  • The population of patients with eczema
  • The intervention of probiotics
  • In comparison to no treatment, placebo, or non-probiotic treatment
  • The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
  • Randomized control trials, a type of study design

Their research question was:

  • What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

  • Background information: Provide the context of the research question, including why it’s important.
  • Research objective(s): Rephrase your research question as an objective.
  • Proposed methods
    • Selection criteria: State how you’ll decide which studies to include or exclude from your review.
    • Search strategy: Discuss your plan for finding studies.
    • Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

    If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee. This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

    It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov.

    Example: Develop a protocol
    In 2006, Boyle and colleagues published a protocol that included background information, a research objective, and proposed methods.

    Step 3: Search for all relevant studies

    Searching for relevant studies is the most time-consuming step of a systematic review.

    To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

    • Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
    • Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD). In medicine, clinical trial registries are another important type of gray literature.
    • Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

    At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator.

    Example: Search for all relevant studies
    Boyle and colleagues searched the following sources for relevant studies:

    • Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
    • Handsearch: Conference proceedings and reference lists of articles
    • Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
    • Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

    Step 4: Apply the selection criteria

    Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol. The third person’s job is to break any ties.

    To increase inter-rater reliability, ensure that everyone thoroughly understands the selection criteria before you begin.

    If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

    You should apply the selection criteria in two phases:

    1. Based on the titles and abstracts: Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
    2. Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

    It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram.

    Example: Apply the selection criteria
    Boyle and Tang independently read the titles and abstracts of the studies that were identified during their search. They excluded any study that didn’t refer to a randomized control trial of probiotics for eczema.

    Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

    When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

    After applying the selection criteria, they were left with 12 studies involving 781 participants.

    Step 5: Extract the data

    Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

    1. Information about the study’s methods and results. The exact information will depend on your research question, but it might include the year, study design, sample size, context, research findings, and conclusions. If any data are missing, you’ll need to contact the study’s authors.
    2. Your judgment of the quality of the evidence, including risk of bias.

    You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group.

    Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

    Example: Extract the data
    Boyle and Tang independently extracted the data using data collection forms. They collected data about the studies’ populations, methods, and outcomes.

    They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

    If Boyle and Tang disagreed on anything, then Varigos broke the tie. Boyle and Bath-Hextall entered the data from the forms into a computer spreadsheet.

    Step 6: Synthesize the data

    Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

    1. Narrative (qualitative): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
    2. Quantitative: Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis, which allows you to combine results from multiple studies into a summary result.

    Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

    Example: Synthesize the data
    Boyle and colleagues used a meta-analysis approach to synthesize the data. They combined data from different studies to calculate overall effect sizes (risk ratios and differences between means) for the outcomes. They calculated confidence intervals for these effect sizes.

    Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

    Boyle and colleagues’ meta-analysis found that there was no significant effect of probiotic treatment on eczema symptoms, either overall or in any subgroup.

    Step 7: Write and publish a report

    The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

    Your article should include the following sections:

    • Abstract: A summary of the review
    • Introduction: Including the rationale and objectives
    • Methods: Including the selection criteria, search method, data extraction method, and synthesis method
    • Results: Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
    • Discussion: Including interpretation of the results and limitations of the review
    • Conclusion: The answer to your research question and implications for practice, policy, or research

              To verify that your report includes everything it needs, you can use the PRISMA checklist.

              Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews, and/or in a peer-reviewed journal.

              Example: Write and publish a report
              Boyle and colleagues published their report in 2008. They published it in the Cochrane Database of Systematic Reviews and a peer-reviewed journal called Clinical and Experimental Allergy. They updated their report in 2018.

              In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema.

              Frequently asked questions about systematic reviews

              What is a literature review?

              A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question.

              It is often written as part of a thesis, dissertation, or research paper, in order to situate your work in relation to existing knowledge.

              What is the difference between a literature review and an annotated bibliography?

              A literature review is a survey of credible sources on a topic, often used in dissertations, theses, and research papers. Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts, with an introduction, a main body, and a conclusion.

              An annotated bibliography is a list of source references that has a short description (called an annotation) for each of the sources. It is often assigned as part of the research process for a paper.  

              Is a systematic review primary research?

              A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

              Cite this Scribbr article

              If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

              Turney, S. (2022, October 17). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved November 28, 2022, from https://www.scribbr.com/methodology/systematic-review/

              Is this article helpful?
              Shaun Turney

              During his MSc and PhD, Shaun learned how to apply scientific and statistical methods to his research in ecology. Now he loves to teach students how to collect and analyze data for their own theses and research projects.