Conducting a Systematic Review of the Literature

Identify the research question

The first step in beginning a systematic review is to clarify the aim and define the research question.⁹ Although this appears to be similar to that of a narrative review of the literature, the research question must be very specific and provide a clear focus on what needs to be answered by the review. Formulating a question in the problem (P), intervention (I), comparison (C), and outcome (O) format (PICO) will aid the investigators in making a concise search.

As an example of this process, the most recent evidence on the efficacy of lasers in dental hygiene care might be a topic that would benefit from a systematic review. The researchers would then create a PICO question about the effectiveness of laser therapy in an adult population with localized periodontal disease that will guide the selection of studies to address this topic (Table I). The inclusion criteria for a systematic review given this example would need to specifically address the parameters of the question and define which types of studies to include in the assessment. For example, the search could be limited to adults with periodontal disease, using laser therapy in comparison to scaling and root planing treatment protocols. The researchers might also place limitations on the languages of the published studies, time frames of the publications, types of populations studied, locations, methods or additional exclusion or inclusion criteria.

Create the protocol

The researchers should register the review protocol before searching the literature to ensure the systematic review has not been previously performed or published. The registration process requires background information on the topic, research question, inclusion and exclusion criteria, search strategy methods, quality assessments, methods for addressing bias, and how the data will be analyzed. These steps allow the researchers to be confident that the systematic review will provide new and valuable information to identify the research team with their question and search strategy.⁴

The International Prospective Register of Systematic Reviews, or PROSPERO, is an open- access online database of systematic review protocols in health and social care.¹⁰ Online training protocols through the Cochrane Library may be helpful in providing additional guidance in creating a systematic review protocol.¹¹ The Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) is an evidence-based, minimum set of items for reporting in systematic reviews and meta-analyses.¹² The PRISMA format will ask the researcher to list all of the events related to the research question, objectives, and methods. The PRISMA checklist can serve as a continual guide for all the items that must be included in the review process.¹³

Search the literature

Once the review process protocol has been approved, the research team can begin to comprehensively search the literature. The search strategy must follow the methodology presented in the protocol with clearly defined inclusion and exclusion criteria. The databases are identified to include all possible sources of information. Databases related to the research question are searched first followed by less common data sources as a means to expand the base of related studies. The search strategy must be sufficiently robust and reproducible for the chosen research question and allow for sensitivity and specificity to the question. The researchers should continue to expand their search by including more and more terms, controlled vocabulary, and Boolean operators. Collaboration with a reference librarian is highly recommended for this process.

There are three additional sources of information searches that may be used; grey literature, snowballing and hand searching.^4,6 Grey literature includes unpublished theses, conference proceedings, government reports, and unpublished trial data. Adding grey literature to the search strategy can help reduce bias of omitting related articles and information. Snowballing and hand searching allow for expansion of the number of studies that meet the inclusion criteria.¹⁴

Screening the findings

An extensive search process allows the researchers to continue to expand potential articles and reduce screening bias. This process allows the researchers to collect a significant number of articles which appear to meet the inclusion criteria. The next step is to assess the titles and abstracts to confirm whether they meet the inclusion criteria. Utilizing a team of at least two researchers/reviewers is extremely important for this process. Each researcher must independently evaluate each article and be in consensus on its inclusion in the review. The process for reaching consensus must be clearly defined in the systematic review protocol. Creating a chart or spreadsheet is critical for keeping track of the studies. Once the abstracts have been independently reviewed and agreed upon, the full text of selected articles is reviewed for inclusion in the analysis.

Quality assessments

The next step in the review process is to assess the data and select the critical appraisal tools to be used. There may be a wide range of research methods used in the studies meeting the inclusion requirements. The researchers would need to select one appraisal tool or checklist that is appropriate for the various study designs found in the selected articles. GRADE is a quality assessment tool used for rating the certainty of the evidence and helps to determine the strength of clinical practice recommendations. Researchers can use the GRADE (grading of recommendations, assessments, development, and evaluation) formula to present summaries of evidence and rate study findings from very low to high. The Cochrane organization provides helpful GRADE training.¹¹

Once an appraisal tool has been chosen, the researchers are now ready to assess the qualities of the studies related to the research question. As stated previously, this process must be performed by at least two members of the research team. Following the PRISMA flow chart, there might be a vast number of articles to be considered for the review, however a much smaller number will meet all of the inclusion and exclusion criteria. The researchers must make judgments regarding which articles to include. At this point, the team of researchers along with the librarian and statistician respond to concerns of bias regarding selection, performance, detection, attrition, or reporting.^4,7 Bias is a systematic error which could result in the results or inferences deviating from the truth. The search strategies should also be re-checked by the research librarian to confirm they are robust and reproducible (Table II).³ The PRISMA checklist also assists the research team in addressing and preventing bias.

An example of a PRISMA flow diagram is shown in Figure 1. In a study by Bradshaw et al. the oral manifestations of Lyme disease were addressed in a systematic review.¹⁵ Notice that the researchers first identified 217,381 possible articles. The screening process brought the total to 12 articles that met the inclusion parameters and withstood the tests of bias as shown in Table II.¹⁵ These studies formed the basis for the evidence for the systematic review and the researchers were then ready to extract data, analyze the results, and synthesize the findings (Table III).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

PRISMA flow chart on oral manifestations of Lyme disease¹⁵

Summarizing the findings

Data extraction refers to pulling data from each study that has meaning according to the inclusion criteria and research question. This could include a range of items such as the type of study, location of the research, randomization, variables, or standard deviations of the results. It is essential to perform this analysis according to the research question; this should not be a summary of the individual studies. Data extraction is done independently by a minimum of two researchers. Bias is reduced by gathering the same information on the included studies. Tables can be created to illustrate the selected articles and present the relevant data that has been extracted.

The next step is to determine whether a meta-analysis is appropriate for the systematic review. A meta-analysis summarizes the data by pooling the results of the studies identified in the systematic review. A statistical analysis can be used if the studies are found to be homogeneous in study methodology and sample size of effect.¹⁶ Heterogeneity should be statistically insignificant in a meta-analysis to compare similar results. However, the research team will not know whether the outcomes of the studies included in the systematic review are compatible for this type of comparison until the data extraction process is complete. A detailed description of the steps for performing a meta-analysis is beyond the scope of this report. The PRISMA checklist serves to guide the research team in addressing all necessary steps in the data extractions and tabulations. The research librarian and statistician can play a valuable role in confirming this process.

Interpreting the findings

The interpretation of the findings provides the opportunity to give meaning to the results.⁷ Using the PRISMA checklist as a guide, the researchers can provide an overview of the studies selected and the implications of the findings. As an example from the laser efficacy PICO question, the researchers might discuss the level of evidence available comparing the use of lasers with or without scaling and root planing. They would present any limitations for the evidence and the review process utilized for the included studies along with the notable findings. The researchers would then discuss the implications of the existing evidence. This might yield new clinical practice guidelines or identify the need for future research.

Reporting the results

Deciding how to assemble the results of a systematic review into a readable format can be a challenging process.⁹ Referring to the PRISMA checklist can provide useful guidance regarding how to format a systematic review suitable for publication. By reviewing each step provided in the PRISMA guidelines, researchers should have a working template of the key components of the manuscript draft. The first figure should be the PRISMA flow chart, illustrating the full number of studies identified and the exclusion steps that lead to the final number of studies included in the review.⁹ Additional tables will be based on the findings of the data assessment tools used and the data extracted from the included studies. The ultimate goal of a well conducted systematic review is to be able to share the results in the literature.