Abstract
Purpose: Dental hygiene educators often provide inconsistent instruction in clinical settings and various attempts to address the lack of consistency have been reported in the literature. The purpose of this pilot study was to determine if the use of a use of a self-instructional, radiographic anatomy (SIRA) module improved DH faculty calibration regarding the identifica-tion of normal intraoral and extraoral radiographic anatomy and whether its effect could be sustained over a period of four months.
Methods: A convenience sample consisting of all dental hygiene faculty members involved in clinical instruction (N=23) at the University of North Carolina (UNC) was invited to complete the four parts of this online pilot study: a pre-test, review of the SIRA module, an immediate post-test, and a four-month follow-up post-test. Descriptive analyses, the Friedman's ANOVA, and the exact form of the Wilcoxon-Signed-Rank test were used to an-alyze the data. Level of significance was set at 0.05. Participants who did not complete all parts of the study were omitted from data analysis comparing the pre to post-test performance.
Results: The pre-test response rate was 73.9% (N=17), and 88.2% (N=15) of those initial participants completed both the immediate and follow-up post-tests. Faculty completing all parts of the study consisted of: 5 full-time faculty, 5 part-time faculty, and 5 graduate teaching assistants. The Friedman's ANOVA revealed no statistically significant difference (P=0.179) in percentages of correct responses between the three tests (pre, post and follow-up). The exact form of the Wilcoxon-Signed-Rank test revealed marginal significance when comparing percent of correct responses at pre-test and immediate post-test (P=0.054), and no statistically significant difference when comparing percent of correct responses at immediate post-test and the follow-up post-test four months later (P=0.106).
Conclusions: Use of a SIRA module did not significantly affect DH faculty test performance. Lack of statistical significance in the percentages of correct responses between the three tests may have been affected by the small number of participants completing all four parts of the study (N=15). Additional research is needed to identify and improve methods for faculty calibration.
Introduction
Faculty calibration is a means of determining a standard that can be reproduced consistently.1-3 Years of experience or educational background may contribute to the lack of consistency among faculty, which can be frustrating for students, become a distraction to learning, and impact overall satisfaction with the education experience. 1,4-11 Students have reported focusing on individual instructors' preferences in order to enhance their own grades.9-11 Knowing that certain faculty emphasize clinical evaluation in specific areas, students may focus more attention on that area while overlooking other aspects of patient care. This alteration in clinical performance has also been noted by faculty and can have a negative and potentially harmful impact on patient care.9-11 Calibration of faculty members is a means to reduce inconsistencies among instructors, especially in areas where there is room for subjectivity.1-2
Previous research has revealed low levels of agreement among dental educators in clinical decisions and performance. 1,2,5,7-15 Calibration in dentistry mainly focuses on educators working in clinical settings and calibration efforts have included a range of topics including cavity preparations, restorations, dental sealants, radiographic interpretation, and treatment planning.5-7,15-17 Dental hygiene faculty calibration has been studied in the areas of calculus detection, scaling errors, and the writing of clinical notes.1,8,12 Various efforts to calibrate dentists, dental hygienists and graduate teaching assistants in radiology have focused on radiographic interpretation, periodontal diagnosis, quantifying bone loss, and detection of dental caries; however, calibration in the use of radiographic terminology and identification of anatomical landmarks has not been assessed.4,5,7,13, 18, 19
Radiographic interpretation is part of the dental hygiene process of care.20-21 Radiographs provide significant information regarding the periodontal condition, prognosis, and long-term evaluation of treatment.21 Moreover, dental radiology is an integral part of the dental hygiene curriculum, and is incorporated into both the written national board examination and clinical examinations.20,22 Hence, radiology is of great importance to the practice of dental hygiene. Currently no studies have been identified on dental hygiene faculty calibration in radiology, specifically with respect to the identification of normal radiographic anatomy.
Dental Hygiene Faculty Challenges
Many dental and allied dental programs are utilizing distance education sites to address access issues. While these new teaching sites offer increased options in delivery of education as well as patient care, they also increase the number of faculty working with students. Increased faculty numbers and the inclusion of multiple sites can contribute to lack of consistency in teaching and student evaluation.
A shortage in dental educators has also been documented.23-24 In an attempt to overcome this faculty shortage, recruitment of part-time faculty has become a trend with dental school part-time vacancies increasing by three percent. 24-25 Dental hygiene programs also utilize part-time faculty to help fill voids. While utilizing adjunct and part-time faculty members to support clinical needs can be advantageous, it can also lead to inconsistencies in faculty calibration.
There are many barriers for delivering and achieving faculty calibration. Subjective factors include: diverse backgrounds, educational levels, and work-related experiences. Preference for delivery of calibration sessions can alter faculty reception and success; some faculty may prefer face-to-face options and are more likely to attend and participate. Moreover, while it can be assumed that faculty with increased teaching and or clinical experiences are more calibrated than part-time for junior faculty, this is not necessarily accurate. Timing, methodology, and location of faculty calibration sessions can also impact success. If schools prefer face-to-face calibration sessions, then part-time faculty may not be available to come in on the specified dates, due to other jobs or obligations. For distance education sites, significant travel may be required for instructors to attend calibration sessions. Calibration sessions should also include a meaningful agenda and provide new and relevant information. Faculty members who fail to see the relevance of the session are less inclined to attend or participate.
With the addition of distance education, online teaching and increased numbers of part-time faculty, educators must explore innovative, flexible, and creative ways for faculty calibration. The use of self-instructional modules has been discussed in the dental education literature.26 Several studies have evaluated the effects of self-instructional packages on student test performance and found them to be equal to other instructional formats. 27-32 Use of self-instructional packages for faculty development has also been explored, but to a lesser extent.33 Implementation of online calibration modules to meet the needs of increasing numbers of adjunct or part-time faculty could prove to be a useful approach.
The importance of faculty calibration in education has been demonstrated in multiple studies.2, 3, 34 The aim of this study was to identify a unique faculty calibration method. This study evaluated the effect of using a self-instructional radiographic anatomy (SIRA) module on dental hygiene faculty test performance regarding the identification of normal intraoral and extraoral radiographic anatomy and whether the effect was sustained over a period of four months. The study also assessed whether years of experience, preference of instructional method (face-to-face, and online), and faculty groupings (full-time, part-time, and graduate teaching assistants) affected test performance.
Methods
This pilot study adopted a repeated measures design that was exempt from review by the Institutional Review Board (IRB) at the University of North Carolina (UNC). A convenience sample of twenty-three clinical dental hygiene faculty members was identified through the UNC School of Dentistry online directory and consisted of six full-time, eleven part-time, and six graduate teaching assistants (GTAs). Qualtrics, a web-based survey research software program (Copyright © 2015, Version 614720.331s of the Qualtrics Research Suite, Qualtrics, Provo, UT, USA) was used to generate all emails sent to possible study participants and to administer the tests for the study. The identified clinical dental hygiene (DH) faculty members received a Qualtrics generated email informing them of the study purpose and design, and invited them to participate in the four parts of the online study: a pre-test, self-instructional radiographic anatomy (SIRA) module, immediate post-test, and a four-month follow-up post-test. Faculty implied consent to participate by using the link provided in the email invitation prompt to the Qualtrics pre-test instructions and questions. All pre and post-test questions were pilot tested prior to faculty testing by two non-clinical UNC DH faculty members.
The DH faculty members were provided a one-week time frame to complete the online pre-test. The pre-test consisted of six demographic questions, and twenty multiple-choice questions requiring participants to identify normal radiographic anatomy from multiple radiographic images (intraoral periapical radiographs and extra-oral panoramic radiographs).
One week following the pre-test, a Qualtrics generated email was sent to the DH faculty members who had completed the pre-test. This email contained links to the online SIRA module, and the post-test instructions and questions. Participants had two weeks to review the online SIRA module and it could be accessed at any time during the two-week period. The online SIRA module consisted of text and visual aids introducing the identification of normal radiographic anatomy on intraoral and extraoral radiographic images. The module allowed users to read explanations of the different anatomical landmarks, view images, and to take a self-quiz with projected images and anatomic landmarks to label. The length of time required for reviewing the module content was self-paced and allowed participants to scroll back and forth through the module text and view images as needed. Participants were instructed to complete the immediate post-test once they had completed reviewing the online SIRA module.
At the conclusion of the fall semester, four months following the initial viewing of the online SIRA module and completion of the immediate post-test, a Qualtrics generated email was sent to the DH faculty participants with a link to the Qualtrics follow-up post-test instructions and questions. Participants were instructed to complete the four-month follow-up post-test within a one-week time frame.
Questions on both post-tests (immediate post-test, and follow-up post-test) consisted of a single question regarding the preferred method of calibration (online self-instruction or face to face) and the same twenty multiple-choice questions pre-viously used on the pre-test; however, the questions were arranged in a different order.
Data Analysis
Data was downloaded from the Qualtrics server into an Excel spreadsheet and coded in numeric format. This data was then exported to a SAS program for statistical analysis (Version 9.3, SAS Institute, Cary, North Carolina, USA). Descriptive statistics of the group population were reported. The Friedman's ANOVA was used to determine whether there was a statistically significant difference in the percentage of correct responses between the three tests: pre-test, immediate post-test, and follow-up post-test. The exact form of the Wilcoxon-Signed-Rank test was used to determine whether there was a statistically significant difference in the percent change of correct responses from pre-test to immediate post-test, from immediate post-test to follow-up post-test, and from pre-test to follow-up post-test. Level of significance was set at 0.05. Participants who did not complete all parts of the study were omitted from data analysis comparing pre to post-test performance.
Results
Out of the twenty-three (N=23) UNC clinical DH faculty members identified as possible study participants, seventeen (N=17) completed the online pre-test resulting in a 73.9% initial response rate. Faculty members completing the online pre-test consisted of five full-time faculty (29.4%), six part-time faculty (35.3%), and six graduate teaching assistants (35.3%). Among the study participants, 29% or (N=4) had been practicing dental hygiene less than five years, 41% (N=7) had five to fifteen years of experience and 29% (N=5) had more than fifteen years of clinical experience. Regarding clinical teaching, 53% (N=9) of the participants had less than five years of experience. (Table I). Of the seventeen (N=17) participants who completed the online pre-test, 15 completed both the immediate and follow-up post-tests resulting in an 88.2% response rate. Faculty completing all parts of the study consisted of five full-time faculty (33.3%), five part-time faculty (33.3%), and five graduate teaching assistants (33.3%). (Table I).
Descriptive statistics of clinical dental hygiene faculty
The majority of the faculty participants, 94% (N=16), felt that calibration of DH faculty in radiology and radiographic anatomy was necessary. When asked to choose a preferred method of instruction for faculty calibration in radiology 58.8% (N=10) indicated that face-to-face instruction was their preferred method.
The results of the pre-test demonstrated that faculty had knowledge of radiographic anatomy (45-90% with a median score of 65%). (Table II) The overall median test scores improved from pre-test (65%) to immediate post-test (75%), and then decreased to 70% for the follow-up post-test. However, the Friedman's ANOVA indicated no statistically significant difference (P=0.179) in the percentage of correct responses between the three tests. An overall median percent change of 5% was noted from pre-test to immediate post-test, with a corresponding P-value of 0.054 as indicated by the exact form of the Wilcoxon-Signed-Rank test. No overall median percent change was noted from pre-test to follow-up post-test, and from immediate post-test to follow-up post-test. The exact form of the Wilcoxon-Signed-Rank test indicated no statistically significant difference when comparing percent of correct responses at pre-test and follow-up post-test (P=0.665), and when comparing percent of correct responses at immediate post-test and follow-up post-test (P=0.106).
Dental hygiene faculty test performance for all pre and post-tests
Percent change between the different tests according to faculty groupings
Table III represents the percent change between the different tests according to faculty groupings: full-time, part-time, or GTA. Only the median percent change for the graduate teaching assistant group negatively decreased (-5%) from pre-test to immediate post-test, and from pre-test to follow-up post-test. Median percent change for full-time faculty was 5% for both the pre-test to immediate post-test, and pre-test to follow-up post-test intervals. Whereas median percent change for part-time faculty was 15% from pre-test to immediate post-test, and decreased to 10% from pre-test to follow-up post-test. Therefore, test performance of the GTA group decreased from the pre to post tests, and test performance improved for the full-time faculty group.
Faculty members with less than 5 years of clinical practice had a median percent change of (-5%) from pre-test to both post-tests. This indicates that the median percent change for this faculty group decreased both times. When comparing the median percent change for pre to immediate post and pre to follow-up post-tests, faculty with over five years of clinical practice had a positive median percent change at both intervals. Hence, median percent change for faculty with more than five years of practice improved by the same amount at both test intervals. Median percent change according to years of clinical teaching experience among the study participants did not follow the same pattern however. Those with more than fifteen years of teaching experience had the lowest (5%) median percent change from pre to immediate post-test, compared to the 7.5% and 15% change in the less than five years, and five to fifteen years of teaching experience groups. This indicates that all faculty had a positive median percent change from pre to immediate post-test regardless of the amount of teaching experience, but the degree of change was not the same for each teaching experience group. Median percent change from pre to follow-up post-test according to years of teaching experience was -2.5% for less than five years, 10% for five to fifteen years, and 0% for more than fifteen years. Therefore, median percent change from pre to follow-up post-test decreased in comparison to the median percent change from pre to immediate post-test, with the less than five years of teaching experience group being the only group that had a negative percent change.
Although 60% of the faculty indicated they would prefer face-to-face instruction over online instruction for calibration, median percent change from pre to immediate post-test was equal for both instructional method choices regardless of the method they would choose (face-to-face or online). Table IV illustrates the percent change between the different tests according to the indicated preference of instructional method. The median percent change from pre to follow-up post-test was -5% for the faculty members who chose online instruction as their preferred method of calibration. This shows that preference for online instruction did not necessarily mean that faculty performed better given that they used an online module for this study.
Percent change between the different tests according to indicated preference of instructional method
DISCUSSION
Low levels of agreement among dental educators regarding clinical decisions and performance have been documented. 1,2,5,7-15 Poor faculty calibration has been shown to lead to student frustration, modification of patient care based on instructor grading patterns, and an overall decrease in effective clinical teaching and learning. Effective faculty calibration is critical to reduce teaching inconsistencies and enrich student learning.10
Attempts to reduce inconsistencies among educators through calibration or training have shown varied outcomes. Many studies that evaluated the effect of faculty calibration used face-to-face instruction, or interactive group sessions as the calibration intervention.1,7,8,15 Research evaluating the use of self-instructional modules as a means to calibrate faculty has not been widely reported in the literature. This pilot study evaluated the effectiveness of a SIRA module on the improvement of test performance for DH faculty in the attempt to identify a possible calibration method.
Use of the online SIRA module as a calibration tool posed a few advantages: it was possible to include part-time faculty who are not frequently present at the university, information was available for review at any time and place, and it accommodated the preference for online instruction. Out of the twenty-three possible participants for this study, eleven (47.8%) were part-time faculty members; therefore, use of the online SIRA module enabled the inclusion of those faculty. Disadvantages of using the online self-instructional module include no way of monitoring if all content was reviewed by the faculty, the need for internet and computer access to view the module, and no choice of instructional preference. The online self-instructional module used for this study has been used with dental and DH students at UNC in the past. Both Ludlow et al. and Fleming et al. used this web-based module and compared it to the use of slide/tape instruction of students. Both studies found that preference for web-based instruction did not necessarily mean that student test performance improved.31-32 In agreement with their findings, the current study found that even though over half of the DH faculty would choose face-to-face instruction as their preferred method of instruction, the median percent change from pre-test to immediate post-test was equal regardless of what they prefer Therefore, preference for one method or another did not seem to make a difference in the success of the calibration. Age could have influenced the choice for an instructional method; however, this study did not inquire about the age of the participating faculty.
Studies that have evaluated student education using self-instructional modules in comparison to other instructional modalities have reported no difference in test performance according to instructional format, whereas, some have found self-instructional modules to be the most effective when combined with a didactic format.27,30-32, 35 Jim et al. evaluated the use of a computer-assisted self-instructional module for continuing education of pharmacists and reported significant improvement and retention of knowledge from pre-test to immediate post-test and two-week post-test.28 Therefore, self-instructional modules could be viable modes for faculty calibration, as well as adjuncts to other calibration methods.
The Friedman's ANOVA indicated no significant difference in the percentage of correct responses between the three tests. Although the Exact-Wilcoxon-Signed-Rank test indicated no statistical significance when comparing the percent change between the tests, the P-value of 0.054 when comparing the percent change from pre to immediate post-test could be considered marginally significant. This could be a Type II error due to lack of statistical power because of the fairly small sample size.
Pre-test scores ranged between 45-90% with a median score of 65%, demonstrating that faculty did have knowledge of radiographic anatomy. The overall median test scores improved by 10% from pre-test to immediate post-test, and then decreased by 5% at the four-month follow-up. This is in contrast to some studies that reported a longer effect of a calibration exercise.7,8,15 Jacks et al investigated short and long term effects of training on the capacity of DH faculty to write patient chart entries according to a specific format. 8 Faculty were able to adhere to the desired format for approximately one year
Haj-Ali and Feil found that with calibration training, inter-rater agreement improved and was sustained for ten-weeks among educators of an operative preclinical lab when evaluating Class II amalgam preparations.15 In the current study, the follow-up post-test was administered four months after faculty reviewed the online SIRA module. The retention of information may have decreased from immediate post-test to follow-up post-test due to the extended lapse between viewing the material of the online module and taking the follow-up post-test. Perhaps a one-time intervention for calibrating faculty is not adequate. Future research should address the question of how often faculty need to be calibrated to retain consistency.
Lanning et al. evaluated the accuracy and consistency of radiographic interpretation among a group of clinical instructors in conjunction with a three-part training program.7 Faculty consecutively completed a pre-test, phase-I training, post-test 1, and phase-II training. Three months later, post-test 2 was administered and faculty attended phase-III training. The findings showed that faculty agreement improved over time, and it was concluded that lengthening a training program could result in further improvement.7 In the current study, faculty were instructed to review the content of the online SIRA module once during a two-week period and the follow-up post-test was administered four months after, contrasting the three-part training program of Lanning et al. There was a decrease in DH faculty test performance from immediate post-test to follow-up post-test. If the study protocol allowed DH faculty to review the SIRA module content several times, follow-up test performance may have improved.
The GTA group seemed to score lower from pre-test to immediate post-test and from pre-test to follow-up post-test with a median percent change of -5%. Full-time faculty test performance however, indicated improvement at both intervals: pre-test to immediate post-test, and pre-test to follow-up post-test. Median percent change for part-time faculty showed a 15% improvement from pre-test to immediate post-test, and slightly less improvement of 10% from pre-test to follow-up post-test. GTAs are considered students as well as faculty so they are in the process of learning while carrying a student workload in addition to their teaching responsibilities. This could have affected their test performance negatively, especially if they did not have sufficient time to thoroughly review the SIRA module material. In contrast with the results of the current study, Firestone et al. reported that the diagnostic accuracy of dental students diagnosing dental caries from radiographs was similar to that of experienced clinicians.36
This study compared test scores according to years of DH practice and clinical teaching experience. Results indicated that those with more than five years of DH practice performed better than those with less than five years of DH practice when comparing pre to post-test performance. Those with more years of practice may have performed better on the tests as they have been exposed to a wider variety of clinical situations that include the examination of radiographs. Hinkelman et al. researched methods of decreasing subjective evaluation in a preclinical environment and reported that reliability of examiners was not significantly affected by years of clinical experience.37 In the current study, faculty evaluation practices were not assessed in comparison to their test performance. Hellén-Halme et al. evaluated whether educational level and dental practice effect the accuracy of diagnosing dental decay from radiographs in groups of dental and dental hygiene students, and dentists with more than five years of clinical practice. Findings indicated that both practice and experience were important for diagnostic accuracy as experience accumulates during clinical practice.4
When comparing years of clinical teaching experience, all participants scored better on the immediate post-test, regardless of the amount of teaching experience. Those with 5-15 years of teaching experience (N=3) had the most improvement. Only those with less than 5 years of teaching experience (N=8) had a negative percent change when comparing the pre-test to the four-month follow-up post-test scores. Experience with teaching dental radiology or clinical courses that include radiology could possibly explain why those with more teaching experience did not have a negative percent change when comparing pre to four-month post-test scores.
Lack of statistical significance may be attributed to the small number of participants. Additionally, participants were instructed to review the content of the SIRA module once on their own time over the period of two weeks prior to taking the immediate post-test. This could be considered a limitation as there is no way of knowing how often faculty reviewed the module, and if all of the module material was indeed reviewed or not. Faculty were instructed to refrain from using a smartphone to view the module as images may be distorted. Smartphones with internet access are often at hand and people often use them to quickly access information even when clarity is compromised. It can only be assumed that faculty members did not use a smartphone to view the module, and that test performance was not affected by distortion of images due to use of a smartphone.
Another limitation of the study is the small sample size. Future studies could duplicate the current design with a larger sample of faculty and with those from different institutional settings such as universities and community colleges.
Research findings have consistently shown the importance of faculty calibration in education.2, 3,34 Identifying appropriate and affordable means for faculty calibration may have far-reaching benefits to both students and faculty. It is important to identify and research various methods for faculty calibration to adapt to different settings and reach all types of clinical teachers, including full-time, part-time, and those located at distance education sites. Implications from this research and additional studies may help to identify new and innovative ways of calibrating faculty to increase reliability, consistency, and effective teaching.
CONCLUSION
This study aimed to evaluate the effectiveness of using a SIRA module as a possible method of calibrating DH faculty in radiographic anatomy. Use of a SIRA module did not significantly affect dental hygiene faculty test performance. Test performance at four-months was lower in comparison to immediate post-test results, indicating a possible need for more frequent calibration interventions. DH faculty were receptive to using a SIRA module for knowledge enhancement; however, the preference for face-to-face instruction needs to be considered. Additional research should continue to be conducted to identify and improve methods for faculty calibration.
Footnotes
Jennifer L. Brame, RDH, MS is a clinical associate professor in the Department of Dental Ecology at the University of North Carolina, School of Dentistry, Chapel Hill, NC; Demah Salem AlGheithy, BSDH, MS is a lecturer in the Dental Health Department at the College of Applied Medical Sciences at King Saud University in Riyadh-Saudi Arabia; Enrique Platin, RTR, MS, EDD Clinical Professor in the Department of Diagnostic Sciences at the University of North Carolina, School of Dentistry, Chapel Hill, NC; Shannon H. Mitchell, RDH, MS is a clinical associate professor and dental hygiene program director in the Department of Dental Ecology at the University of North Carolina, School of Dentistry, Chapel Hill, NC.
This manuscript supports the NDHRA priority area Professional development: Education (evaluation and educational models).
- Copyright © 2017 The American Dental Hygienists’ Association
