Critical Appraisal of a Systematic Review/Meta-Analysis

This study material by Desklib provides guidance on how to critically appraise a systematic review/meta-analysis. It includes 14 questions to analyze and interpret the meta-analysis depicted in the assigned forest plot diagram. The study material also provides information on the APA citation, search strategy, and results of the study. The subject, course code, course name, and college/university are not mentioned.

1
Critical Appraisal of a Systematic Review/Meta-Analysis
Students Name
Institutional Affiliations
Revised May S2017 B

2
Appraisal of A Systematic Review/Meta-Analysis
Your Name ______________________________________________Date:__________
Directions: Access the full version of this Systematic Review (SR). There are 14 questions for
the four parts of this appraisal assignment. Analytically and concisely answer each question and
support with a brief comment and the page number(s) in the SR. Use APA format (page
numbers) for all quotes and citing facts from the SR. If page numbers for supporting evidence
are not provided, the appraisal is not acceptable and will be returned for one revision.
 Write the APA Citation for this study: See Module 2 for an example.
Ciciriello, S., Johnston, R.V., Osborne, R.H., Wicks, I., deKroo, T., Clerehan, R., O’Neill,
C., & Buchbinder R. (2013). Multimedia educational interventions for consumers about
prescribed and over-the-counter medications. Cochrane Database of Systematic Reviews,
2013 (4), 1-241, doi: 10.1002/14651858.CD008416.pub2.
Explain your search strategy in obtaining the full text of this SR
Via the Foley Library Portal, I accessed the Cochrane Library after which keyed in the title of the
article assigned “Multimedia educational interventions for consumers about prescribed and over-
the-counter medications ” into the search field then searched yielding one result which matched
the citation delivered in the study case. Through the process I was able access or get the PDF of
the systematic review.
Part I. Is There a Clear Question and Are the Results of the Study Valid?
1a. Was an 'a priori' design provided? Search the “history” section of the Cochrane SR for
the information on the “protocol”.
 Yes □ No
□ Not addressed in SR □ Not applicable
Comment and supporting information:
The authors stated the primary objective of the study “To examine the impacts of multimedia
patient education interventions about prescribed and over‐the‐counter medications in people of
all ages, including children and carers” (Ciciriello et al., 2013 p. 4).Additionally, the exclusion
and inclusions are also discussed in the background.
1b. Write you translation of the primary aim/objective of this SR into PICOT format.
Revised May S2017 B

3
P People of all ages, including carers and children.
I Impacts of multimedia patient education interventions (Ciciriello et al., 2013 p. 4)
C Traditional forms of education delivery
O Examine the impacts of multimedia patient education intervention about over‐the‐
counter and prescribed medications.
T Not indicated
2. Was a duplicate study selection and duplicate data extraction method utilized?
□ Yes □ No
□ Not Addressed in SR □ Not applicable
Comment and supporting information:
Two independent authors made preliminary selections of potentially eligible research on the
basis of abstracts and titles (Ciciriello et al., 2013 p. 4). A full-text version of the research was
gotten for further review when there was qualm on the inclusion of the study at this phase. In the
subsequent phase of the review, the two independently working authors’ dag out and assembled
critical data from the selected articles and they state that the incongruities in the stages were
determined or solved through discussion.
3. Was a comprehensive literature search performed and is it unlikely that important,
relevant studies were missed?
□ Yes □ No
□ Not addressed in SR □ Not applicable
Comment and supporting information:
It is important to note that the search relied upon the outcome of the previous version of the
systemic review whose dates ranged from 1950-2011. The biographers of the present systemic
review searched many databases including MEDLINE, Central, and EMBASE from (1974-June
2011 (Ciciriello et al., 2013 p. 4). The authors detailed all the databases which the search was
Revised May S2017 B

4
done and also their search strategy as well as the results obtained. In the same token, the authors
brushed up the reference lists of the research in an attempt to establish any other research related
to the review. The authors also cited textbooks as references; however, it is not known whether
specialized experts or registers in the field were explicitly consulted.
4. Was a list of studies (both included and excluded) provided?
A list of included and excluded studies should be provided. Sometimes there are too many
excluded studies to list in the SR, but the SR authors should explain reasons or types of studies
excluded.
□ Yes □ No
□ Not addressed in SR □ Not applicable
Comment and supporting information:
The authors included the list of studies and mentioned that they identified twenty-four studies
which enrolled more than 8000 participants.
5. Were the characteristics of the included studies provided?
□ Yes □ No
□ Not addressed in SR □ Not applicable
Comment and supporting information:
The authors included tables, characteristics, and raw data and analyzed the data.
6. Was the scientific quality (methodological rigor) of the included studies sufficiently
assessed and documented? What quality criteria were identified ‘a priori’?
□ Yes □ No
□ Not addressed in SR □ Not applicable
Comment and supporting information:
The two independent authors utilized the Risk Bias Tool to examine the risk for bias in every
included research including a discussion which defined the biases (Higgins, Altman & Sterne,
2011).
Revised May S2017 B

5
7. Heterogeneity: Were the studies homogeneous? What statistical methods were used to
assess heterogeneity and to analyze the data for the meta-analysis? Explain.
□ Yes □ No
□ Not addressed in SR □ Not applicable
Comment and supporting information:
The authors used Chi-squared test to determine that the study was heterogeneous.
8. For the SR, were sources of support disclosed and declarations of interest declared?
□ Yes □ No
□ Not addressed in SR □ Not applicable
Comment and supporting information:
The authors did not provide the support sources also never declared interest.
9. For the SR (not individual studies), did the researchers assess for publication bias and
address it sufficiently?
□ Yes □ No
□ Not addressed in SR □ Not applicable
Comment and supporting information:
The two independent authors extracted information and evaluated the risk of bias of the included
researches and contacted study authors to get any missing data or information.
Part II. What are the Results of the Study?
10. In the SR, was the scientific quality (rigor) of the included individual studies used
appropriately in formulating overall conclusions/implications?
□ Yes □ No
□ Not addressed in SR □ Not applicable
Comment and supporting information:
Revised May S2017 B

6
The authors considered scientific quality and methodological rigor in the analysis and presented
them in the conclusion (Ciciriello et al., 2013 p. 142). The stated that the review provided
evidence that multimedia education about medications is more effective than usual care. They
stated that the overall quality of the research was based low quality evidence which came from
the smaller number of trials.
11. For this systematic review/meta-analysis, analyze and interpret the meta-analysis
depicted in the assigned forest plot diagram. See Module 2. Respond to the following
questions based upon examination of the assigned forest plot diagram.
Analyze the Assigned Forest Plot #_1.4 outcome_4_on page_145_of the SR
Test for heterogeneity showed a considerable heterogeneity (I2 = 52%), (Ch2 = 210) with 1
degree of freedom (Ciciriello et al., 2013 p.145). The forest plot indicates that even though there
was heterogeneity with non‐overlapping confidence interval the researches favored the
multimedia education cluster because of the 1 degree of freedom.
a) What interventions are being compared in this forest plot and what is the outcome being
measured?
a. Intervention of interest = Multimedia educational interventions for consumers
about prescribed and over-the-counter medications (Ciciriello et al., 2013 p.145).
b. Comparison intervention = Multimedia education versus usual care or no
education.
c. Outcome of interest = Compliance with medication (% who complied)
Is the listed outcome of interest (above) considered to be desirable or adverse
event? Multimedia education is desirable.
b) For this forest plot, how many studies or subgroups are being combined?
a. If there are subgroups, how are the studies divided or grouped?
The forest plot combines 2 subgroups or studies. The studies are divided into multimedia
education and usual care or no education.
c) What is the statistical measure (e.g. RR, OR, etc.) that represents the pooled effect?
The statistical measure is RR (Ciciriello et al., 2013 p. 5).
a. How is this statistical measure defined?
Revised May S2017 B

7
The measure is defined as Risk Ratio M-H Random 95% CI
Is it a binary or continuous variable?
It is a binary variable.
d. What data analysis model (Random Effects or Fixed Effects) was used for this meta-
analysis and was it appropriate? See question 7 (above).
The forest plot used the Random Effects when combining the two studies, but the
heterogeneity calculated was I2= 52%.
a. Did the researchers use the data analysis model they intended? Explain your
thinking.
Yes. Because the results the effects model are statistically significant and reflect
favor of the intervention.
e. What is the I squared value for this meta-analysis and, according to these SR
authors?
a. Does this represent a large amount of heterogeneity or diversity between the
studies when all of the studies are combined? See question 7 (above).
Explain your thinking.
The I2=52% for this forest plot’s pooled studies. These effects measure showed
considerable heterogeneity.
f. For the overall (Total) effect in this forest plot,
a. What are the values for the confidence interval range (upper and lower
limits)?
Lower limit is 0.96 while upper limit is1.08 (Ciciriello et al., 2013 p. 145).
b. Does this CI range reflect a precise (narrow confidence interval range)
overall (Total) effect or an imprecise overall (Total) effect (wide confidence
interval range)?
The CI range reflects a narrow confidence interval range.
g. By examining the vertical line in the forest plot,
a. What is numerical value for the line of no effect or the null value for the
statistical effect measure?
The null value RR =1.02
Revised May S2017 B

8
h. By examining the position of the summary diamond (overall or Total effect) to the
vertical line,
a. Is there a statistically significant difference in the outcome for the
intervention versus the comparison group as shown in the forest plot?
Yes, there is a statistically significant effect for this outcome. The diamond does not
touch the null-effects line indicating that the likelihood of the chance being responsible
for the results is minimal.
b. Is the Test for Overall Effect statistically significant? Explain your thinking and
how the position of the summary diamond and the p value for Test for
Overall Effect relates.
i. Given the above data, I am inclined to interpret the results as valid and supports that
multimedia education is more effective compared to usual in contributing to patient
education interventions. It is important to note the high heterogeneity and the p value of
0.15 shows a larger sample (Nannenga et al. 2009 p. 39).
j. PUTTING IT ALL TOGETHER: From this forest plot meta-analysis, utilize the
information about the number of studies, heterogeneity results, CI precision, and the
overall effect with its statistical significance to generate a conclusion statement about
the overall effect of this intervention versus the control group on the outcome of interest.
Translate the numerical values for the statistical measures into words (see the Stats
tip Sheet) that peers and patients can understand. Interpret the statistical measures
such as the Risk Ratio.
For this forest plot, the confidence interval range is 0.96-1.08. The CI is calculated to =
1.04. The CI for the pooled summary effect does reflect a precise (narrow) effect. I
interpret the results as valid and supports that multimedia education is more effective
compared to usual in contributing to patient education interventions
Part III. Will the Results Help Me in Caring for My Patients/Clients?
12. Are your current or future patients similar to those included in the systematic
review and would it be feasible to implement the findings in your current or future
Revised May S2017 B

9
practice setting? Considering your patients’/clients’ preferences and values about the
intervention, are there reasons why this treatment would not be acceptable? Explain.
No. I currently work at the medical ICU which requires a lot of care and attention. The
ICU department involves activities like surgical operations. It is important to note that the
findings in the systematic review are feasible to implement in both the current and future
practice setting. Multimedia education is important because it helps in improving the
patient’s knowledge. Therefore, the patient in the ICU department will be able to understand
the reasons for surgery instead of drugs prescription.
13. Were all clinically important outcomes considered including the costs, risks and
benefits of the treatment? What was missing?
Are there any contraindications or circumstances that would inhibit me from
implementing the treatment with current or future patients/clients? Explain.
The authors included analysis of how the interventions in the studies would potentially be
funded for each patient, if included, but the costs of implementing this intervention into daily
practice were not discussed. Obstructions exist in my practice for practical implementation
(Melnyk & Fineout-Overholt, 2011). Furthermore, it would have been nice had the authors
discussed some of the clinical implications of implementing this into a provider’s daily
practice.
Part IV. Your Reflections
14. As you have read and used the appraisal criteria to evaluate this meta-analysis,
what did you learn from this assignment?
It is important to note that I learnt a considerable amount about how interpretation of the
statistical results or outcome of studies or research and what that means for their reliability
and validity. Correspondingly, I learnt about how to examine biases in any given research
and how the biases can influence interpretations of the results of the research.
How will you use SRs/MAs in your future practice? Explain
I plan to use Systematic Reviews in my future practice to help me find easily and fast find the
most comprehensive information or data on a subject which may be facing my customer. I
Revised May S2017 B

10
would also use Systematic Reviews try to keep my practice up-to-as well as guide my
medical decision-making.
Adapted from the AMSTAR tool to assess the methodological quality of any systematic
review. Shea et al. (2007). BMC Medical Research Methodology 7, 10. doi:10.1186/1471-2288-
7-10. Fineout-Overholt, E. and Melnyk, B (2005), Evidence-based practice. NY: Lippincott
Williams & Wilkins.
Revised May S2017 B

11
References
Ciciriello, S., Johnston, R. V., Osborne, R. H., Wicks, I., Clerehan, R., O'Neill, C., &
Buchbinder, R. (2013). Multimedia educational interventions for consumers about
prescribed and over‐the‐counter medications. Cochrane Database of Systematic Reviews,
(4).
Nannenga, M. R., Montori, V. M., Weymiller, A. J., Smith, S. A., Christianson, T. J., Bryant, S.
C., ... & Bolona, E. R. (2009). A treatment decision aid may increase patient trust in the
diabetes specialist. The Statin Choice randomized trial. Health Expectations, 12(1), 38-
44.
Higgins, J., Altman, D., & Sterne, J. (2011). Chapter 8: Assessing risk of bias in included
studies. In: Higgins, J., & Green, S. (2011). Cochrane handbook for systematic
reviews of interventions. Version 5.1.0. The Cochrane Collaboration. Retrieved from
http://www.cochrane-handbook.org.
Melnyk, B., & Fineout-Overholt, E. (2011). Evidence-based practice in nursing & healthcare: A
guide to best practice. (2nd ed). Philadelphia: Lippincott Williams & Wilkins
Revised May S2017 B