Abstract
Purpose: The purpose of this pilot study was to determine if the use of a video-recorded clinical session affects the accuracy of dental hygiene student self-assessment and dental hygiene instructor feedback.
Methods: A repeated measures experiment was conducted. The use of the ODU 11/12 explorer was taught to students and participating faculty through video and demonstration. Students then demonstrated activation of the explorer on a student partner using the same technique. While faculty completed the student assessment in real time, the sessions were video recorded. After completing the activation of the explorer, students and faculty completed an assessment of the student's performance using a rubric. A week later, both students and faculty viewed the video of the clinical skill performance and reassessed the student's performance using the same rubric. The student videos were randomly assigned a number, so faculty reassessed the performance without access to the student's identity or the score that was initially given.
Results: Twenty-eight students and 4 pre-clinical faculty completed the study. Students' average score was 4.68±1.16 on the first assessment and slightly higher 4.89±1.45 when reviewed by video. Faculty average scores were 5.07±2.13 at the first assessment and 4.79±2.54 on the second assessment with the video. No significant differences were found between the differences in overall scores, there was a significant difference in the scores of the grading criteria compared to the expert assessment scores (p=0.0001).
Conclusion: This pilot study shows that calibration and assessment without bias in education is a challenge. Analyzing and incorporating new techniques can result in more exact assessment of student performance and self-assessment.
Introduction
The importance of self-assessment in health care related occupations has been well established in the literature and the ability to self-assess is vital to ensure professional growth and development.1,2 One challenge that arises in the implementation of self-assessment is students' understanding of self-assessment and its purpose. Students have reported that they are unfamiliar with the concept of self-assessment or are unclear of the expectations especially in the clinical environment.1,2 In some cases it is due to a lack of exposure to self-assessment. In a dental hygiene study, Mould et al reported that 52% of students had no previous experience with self-assessment. An additional 24% of students reported minimal experience with self-assessment.2 Medical literature shows that students do not accurately self-assess and that there is a need to provide self-assessment instruction during clinical education.1
In dental hygiene, self-assessment is a necessary component of the curriculum, but can be difficult to cultivate. The Commission on Dental Accreditation (CODA) Standards for Dental Hygiene Programs outlines the responsibilities of institutions, putting the onus on educators to incorporate the self-assessment process into the curriculum.3 Critical thinking and self-assessment expectations are noted in standard two for educational programs stating “Graduates must be competent in the application of self-assessment skills to prepare them for life-long learning.”3 Consequently, dental hygiene programs must have evaluation mechanisms in place to examine student understanding as well as ability to apply self-assessment skills.3 CODA accreditation standards advise dental hygiene programs to demonstrate that students are competent in the application of self-assessment. Programs can demonstrate compliance through “evaluation mechanisms designed to monitor knowledge and performance.”3
The challenge of teaching the skills of effective self-assessment is one seen in other health professions. In addition, health profession studies have shown that the evaluation scores from student self-evaluation do not correlate with the scores given by faculty members.4,5 In providing instruction to students, educators must be able to show students discrepancies in their performance so that the student can accurately self-assess in alignment with the instructor or expert assessment. The use of video has offered the ability to record student performance for later review and critique. Video may provide the opportunity for a student to re-examine the clinical skill performance to identify deficiencies in their performance thus having the potential to improve self-assessment practices. This is already a standard educational practice in some medical programs that could be applied to dentistry and dental hygiene as well. A study involving physiotherapy students reported that a recorded simulated exam allowed students to reflect on and evaluate their performance against established standards.6 Based on those study findings, students can use video to compare their perceived performance to the actual recorded performance. This process of video review may provide the student with an opportunity to recognize areas of adequate and deficient performance. In addition, the process of video review could provide a measurable objective to discern whether or not self-assessment practices are in alignment with instructor assessment.
Clinical instructors could benefit from the utilization of video review as well. Observation is the most frequently utilized type of clinical assessment technique among dental hygiene educators7 but direct observation “is limited by high interrater score variability.”8 The use of video will provide an alternative assessment technique for clinical instructors or even enhance the technique of the direct observation. The use of video as an alternative method for clinic assessment may help to reduce rater bias. In addition, instructors can use the video-recorded performances to determine if the feedback provided is complete and identify missed opportunities to provide corrective feedback. The feedback offered during a direct observation experience may differ from the feedback offered after a video review of the clinical performance.
Dental hygiene students must learn to evaluate their clinical performance to ensure that safe and appropriate care is provided to patients, however there is currently not a systematic method of self-assessment training for these students. The utilization of a video recorded clinical simulated patient learning experience is an alternative teaching method that can be investigated for its usefulness in the training of dental hygiene students and its effect on dental hygiene clinical instructor feedback. Other health professions have used instructional media such as simulation and video to provide students with opportunities to reflect on their clinical performance.4,9,10 Students from other allied health professions have reported that the use of videos allowed them to more accurately self-assess.5,11 A medical study found that the implementation of video exercises allows for improved calibration between student self-assessment and faculty feedback.5 Although video is a widely used form of technology, its most effective use in the training of dental hygiene students has yet to be determined. Further, the use of a video-recorded clinical session used for student self-assessment has not been evaluated for dental hygiene. Because dental hygiene has a unique psychomotor skill set, it must be established whether or not the use of video is a relevant tool to evaluate the accuracy of student self-assessment practices. The purpose of this study is twofold in that it seeks to determine if the use of a video-recorded clinical session affects the accuracy of dental hygiene student self-assessment, if the use of a video-recorded clinical session influences feedback provided to students by dental hygiene clinical instructors.
Methods and Materials
This study was a repeated measures experiment that received expedited approval from an institutional review board. In preparation for the study, reading assignments, a video demonstrating the technique for using an ODU 11/12 explorer, and a self-assessment orientation were completed. First year dental hygiene students in the pre-clinical course were invited to participate in this pilot study. Clinical instructors viewed the same video that the students watched demonstrating the technique for using an ODU 11/12 explore. Review of this benchmark video before the laboratory exercise was intended to calibrate both students and clinical instructors. Students then demonstrated the proper technique, indicated in the benchmark video for using the ODU/11/12 explorer on a simulated patient. There were 4 criteria used to assess the clinical skill performance of exploring with the ODU 11/12 explorer. The first criterion assessed the choice of the correct working end, the second criterion assessed adaptation of the instrument to the teeth, the third criterion assessed instrument insertion into the gingival tissue, and the fourth criterion assessed instrument activation. These scores were compared to a standardized score established for each video based on criteria taught from the textbook by the pre-clinical faculty and video review by all faculty prior to the real time assessment. The clinical instructor observed the techniques of the student while one of the investigators video-recorded the clinical performance. Immediately following the performance, the student completed a self-assessment and the clinical instructor completed a faculty assessment of student performance. The data collected from this clinical session was classified as the pre-video assessment.
The post-video assessment was completed a week later using a secure streaming server and email contact entailing evaluation instructions and a link to the video. This link connected viewers to the video-recorded clinical performance and an electronic version of the rubric used for the pre-video assessment. The post-video assessment was completed through a link to a secure electronic evaluation form in a survey software program (Qualtrics, Provo, Utah). In addition to the faculty score, the primary investigator and the co-investigator along with 2 pre-clinic course directors established a score for each video performance. The score established by the investigators and course directors is referred to as the “expert assessment” for each video performance. The standardized or expert assessment was determined by a negotiated approach. The expert assessment was the comparison for other scores completed by the students and the faculty.
Both descriptive and inferential statistics were used to analyze the data. Statistical analysis involved a total sum score and sub categorical scores assigned to each of the rubrics received from the students, instructors and investigators. These scores were totaled from the assessment rubric using the 4 criteria described previously. The criteria on the rubric were used for students and instructors to rate the performance of the clinical skill. The criteria were each scored on a 0 to 2 scale; a score of 0 represented major errors, a score of 1 represented minor errors in performance or inability to complete the skill, and a score of 2 represented the ability to perform the skills without errors. Scores were totaled by rater (instructor or student) and by time (pre- and post-video review). A mixed model for repeated measures was used to account for the association of scores between the student pre- and post-video reviews (intra-rater comparison), clinical instructor pre and post video reviews (intra-rater comparison), as well as associations between students, instructors and the expert assessment score (inter-rater comparison). The following parameters were estimated based on the mixed model: the difference of total score and criterion scores at different time intervals for the student self-assessment, the difference of total score and criterion scores at different time intervals by the instructor, difference between instructor and student self-assessment at different time intervals, and difference of scores evaluated by the student or instructor as compared to the expert assessment scores. A power analysis of 85% was determined and an alpha level of 0.01 was used for statistical testing.
Results
Thirty-two first year dental hygiene students were invited to participate in the study. Twenty-eight of the students provided consent for an 87.5% participation rate. Due to an incomplete data set, data analysis was completed for 27 subjects (n=27). Pairwise comparisons using both parametric and non-parametric methods were analyzed. In Table I, the data was summarized by count. Students scored their performance as a 0, 1 or a 2. For criterion one, 2 students scored their performance as a zero before the video was reviewed. After the video was reviewed, 5 students scored their performance as a 0 for criteria one. Instructors assigned a 0 score to 7 students for criteria one for both pre and post-video review assessments. There were 5 zeroes assigned for criteria one by the expert assessment group. The counts for the other criteria can be reviewed in Table I. Data analysis showed an increase in mean student self-assessment scores post-video review. Average instructor scores decreased post-video review. Overall scores assessed by the instructors and by the students were rated higher as compared to the expert assessment mean scores.
Differences in scores from the expert assessment scores by rater, time and criterion can be reviewed in Table II. Analysis of variance from the summary data can be examined in Table III. No statistically significant difference was found between pairs after adjustments were made using the Tukey-Kramer method. Overall scores were statistically different by criteria from the expert assessment scores. The pairwise comparisons of scores can be reviewed in Table IV.
Discussion
This study investigated the effect of video on assessment by comparing differences in student self-assessment scores as well as differences in instructor scores. The ratings by the student and by the instructor were compared to an expert assessment rating. There were 4 criteria used to assess the clinical skill performance of exploring with the ODU 11/12 explorer. The first criterion assessed the choice of the correct working end, the second criterion assessed adaptation of the instrument to the teeth, the third criterion assessed instrument insertion into the gingival tissue, and the fourth criterion assessed instrument activation.
There was an interesting trend observed when comparing students' scores before and after video review. Student scores were higher than the expert assessment scores before review of the video and lower after review of the video for criterion one. It is possible that the students were more critical of their performance after the video review and recognized areas for improvement. A physiotherapy study lends support to this trend where students reported that the experience helped them to see how much they needed to work on.6 Conversely, students rated their performance slightly higher for criterion two after they reviewed their video-recorded performance. The students may have observed their performance on video to be better than they had perceived during the actual clinical performance. This tendency has been noted in medical literature whereby higher performing students underrate their scores.1 A similar trend related to assessment of the insertion of the instrument was seen with criterion three. One theory for this trend is that previous assessment experiences and feedback from the clinical instructors may not have been consistent with the established standards as evidenced by the differences in scores from the expert assessment means.
Data Summary Count by Rubric Category and Overall Score
Although not statistically significant, there were some numeric changes in scores by instructors after they reviewed the video. For criteria two and three the instructors decreased the scores from the ratings they would have given pre-video review (during the direct observation/real-time experience). Similar trends were reported in a study by Benson et al where higher scores were assessed when students were evaluated in real-time compared to scores assessed in a videotaped evaluation.12 The decrease in scores could also be due to the fact that instructors were blinded to which students' video they were scoring. Grading bias during the direct observation could have accounted for the higher scores. The expert assessment scores derived from blinded review of the videos by the investigators and preclinical course directors. In other words, the raters did not know which students they were assessing. It is possible that the instructors tend to rate higher because of their close interaction with the students in the clinical setting. Although the instructors intend to evaluate the students to a competent standard, the instructors may be recognizing the students as novice clinicians. In medical education, positive bias has been referred to as “generosity error.”13
Mean differences of Scores from Expert Assessment for Rater, Time and Criteria
Summary Statistics of the 3-Way Analysis of Variance
Another factor that could have influenced the instructor evaluations is a problem noted in medical education. Instructors do not want to show favoritism so they assess higher scores when evaluating students in small group.13 Without realizing, instructors may take into account the students' attitude and personality when evaluating the skill performance.12,14 For criterion one, there was no change with instructors' scores between pre- and post-video review. For criterion four, instructors increased their ratings after video review. For clinical instructors, the use of video could allow for a more accurate assessment of the student's clinical skill performance.12 Some of the differences between instructor ratings and expert assessment ratings could be attributed to the need for additional calibration. Although a video and rubric were provided before the study to demonstrate the assessment parameters, a reliability quotient was not established. In addition, the accuracy of assessment by the instructors could be improved through training of junior faculty with senior faculty. Nursing research reports that novice faculty are hesitant to assess lower grades as this may in turn effect their evaluations by the students.15 The patterns noted in this study may indicate a need for an unbiased method for grading students in the clinical setting.
Pairwise Comparisons of Scores from Expert Assessment
The comparison of instructor to student scores showed varied trends. For criterion one, student mean scores decreased after video review while the instructor mean scores remained consistent. Conversely, student scores remained consistent and instructor scores decreased after video review for criterion four. With regard to criteria two and three, the students increased their scores while the instructors decreased the scores post-video review. Overall means between instructor and student scores post video review differed by 2.07%. The relationship between clinical instructor scores and student self-assessment scores was discussed by Geissler who reported a difference of 5% between student and faculty scores.16
Overall means by criteria are significantly different from the expert assessment means. This difference suggests that criteria one and four were well understood by the students and instructors. It is also possible that the assessment criteria for two and three required additional clarification for the students and instructors. Other limitations discovered through this pilot study were the small sample size and the need for additional calibration among instructors.
Even though the use of video did not have a statistically significant effect on student self-assessment scores, it may still provide value as a teaching tool. Today's generation of students are expecting technology to be incorporated into their education.17 Instructors can consider using video to review a clinical performance with a student and compare self-assessment ratings to instructor ratings. This method could allow instructors to help students more accurately assess by reviewing performance deficiencies as well as proficiencies.18 The results of this pilot study can be used as a foundation for a full-scale study. Additional research related to the use of video-recorded patient simulations as a method for evaluating student self-assessment is warranted.
Conclusion
Self-assessment has been defined broadly as the involvement of learners in judging whether or not learner-identified standards have been met.18 Although there is not currently a systematic method for self-assessment training of dental hygiene students, accreditation requires dental hygiene programs to have evaluation mechanisms in place to examine students' understanding as well as ability to apply self-assessment skills.3 This pilot study aimed to bridge the gap by providing a basis for future investigation into the use of video to aid in the self-assessment training of dental hygiene students. In addition, the self-assessment strategy could be used in a continuum of time to indicate progression of skill and student acknowledgement of their strengths and weaknesses. Moreover this strategy could be applied to assessment of other instruments used for clinical performance such as curets or scalers. The data collected in this study also evokes a need for inquiry into the use of video for calibration of dental hygiene faculty. It is necessary to discover a valid method for self-assessment training of dental hygiene students.
Footnotes
Tammy R. Sanderson, RDH, MSDH is the senior clinic coordinator at the College of Southern Nevada. Rachel C. Kearney, RDH, MS is the graduate program director at The Ohio State University College of Dentistry Division of Dental Hygiene. Denise A. Kissell, RDH, MPH, is a Clinical Assistant Professor, Dental Hygiene, The Ohio State University College of Dentistry. Jessica Salisbury, RDH, MS, is an advisor for Press Ganey Associates, Inc.
This study supports the NDHRA priority area, Professional Education and Development: Validate measures that assess continued clinical competency.
- Copyright © 2016 The American Dental Hygienists’ Association
