Association for surgical education
Assessment of critical appraisal skills

https://doi.org/10.1016/j.amjsurg.2002.12.006Get rights and content

Abstract

Background

Studies have provided little evidence that critical appraisal skills improve with focused courses. However, outcome measures in these studies have been questionable. The goal of this study was to develop a feasible, reliable, and valid assessment of critical appraisal skills.

Methods

Forty-four surgery residents read three articles and then responded to short answer questions and provided 7-point ratings regarding various methodological aspects of each article. Reliability and validity of the examination were assessed.

Results

The mean score was 52.4% (SD 8.6%). Internal consistency of the 55-question examination was 0.77. Interrater reliability of clinician markers was 0.91. Mean score for residents with more intensive critical appraisal training was significantly higher than for those with little or no training (56.6% versus 49.3%, t35 = 2.31, P = 0.02), suggesting construct validity.

Conclusions

This examination has promising psychometric properties, and may be useful in evaluating critical appraisal curricula.

Section snippets

Examination

Three articles were chosen from leading medical journals by a clinical epidemiologist. They were chosen to be relatively good methodologically, and to represent different methodological topics (effectiveness of treatment, the natural history or prognosis of disease and risk factors or etiology of disease). A clinician with no formal epidemiology training also assessed the articles to ensure agreement that the articles were interesting and clinically relevant [4], [5], [6].

The examination

Results

Forty-four residents participated in the examination. All but one resident completed the examination in three hours or less. The mean score on the total examination was 52.4% (SD 8.6%).

The internal consistency of the 55-item examination was 0.77. The correlation between the clinical epidemiologist and the nonepidemiologist surgeon was 0.91, and between the clinical epidemiologist and the nurse was 0.78. Thus interrater reliability was excellent when a clinician performed the ratings, and was

Comments

Evidence-based medicine has been heralded as a new paradigm for medical education [14]. Teaching residents and practicing clinicians to be effective at critical appraisal is one of the major steps for enabling the use of EBM in clinical practice. Norman and Shannon [2] reviewed the literature on the effectiveness of instruction in critical appraisal. These authors found that studies that evaluated the teaching of critical appraisal to residents showed disappointing results, with minimal gains

References (16)

  • Muller S. Physicians for the twenty-first century: report of the project panel on the general professional education of...
  • G.R. Norman et al.

    Effectiveness of instruction in critical appraisal (evidence-based medicine) skillsa critical appraisal

    Can Med Assoc J

    (1998)
  • M.L. Green

    Graduate medical education training in clinical epidemiology, critical appraisal, and evidence-based medicinea critical review of curricula

    Acad Med

    (1999)
  • C. Mann et al.

    Epidural patient-controlled analgesia in the elderly after major abdominal surgery

    Anesthesiology

    (2000)
  • M.P. Nevitt et al.

    Prognosis of abdominal aortic aneurysms—a population-based study

    N Engl J Med

    (1989)
  • W.D. Dupont et al.

    Long-term risk of breast cancer in women with fibroadenoma

    N Engl J Med

    (1994)
  • D. Sackett et al.

    Evidence-based medicinehow to practice and teach EBM

    (2000)
  • S.H. Gehlbach et al.

    Teaching residents to read the literature

    J Med Educ

    (1980)
There are more references available in the full text version of this article.

Cited by (19)

  • Acquisition of evidence-based surgery skills in plastic surgery residency training

    2011, Journal of Surgical Education
    Citation Excerpt :

    Similar findings were noted in a meta-analysis on effectiveness of instruction in critical appraisal, in which residents showed a small change in knowledge (mean gain: 1.3%; standard deviation [SD]: 1.7%), even less than medical students (mean gain: 17.0%; SD: 4.0%).23 It is possible that substantial gains in EBS knowledge might not be detected by our posttest examinations because of a lack of valid assessment of critical appraisal skills.24 Multiple obstacles are likely to inhibit the effective teaching of EBS.

  • Surgical Journal Clubs: A Bridge Connecting Experiential Learning Theory to Clinical Practice

    2007, Journal of the American College of Surgeons
    Citation Excerpt :

    Unfortunately, previous assessment tools have several methodologic problems. Very few studies have used a pretest–posttest design to confirm that the journal club increased critical appraisal ability.21 The validity and reliability of the journal club evaluative instrument was determined only in a single study.25

View all citing articles on Scopus
View full text