Abstract
Purpose: This study investigated the calibration efforts of entry-level dental hygiene programs in the U.S. Four aspects were explored, including attitudes, characteristics, quality and satisfaction, to evaluate current calibration practices.
Methods: A descriptive comparative survey design was used. Directors of accredited dental hygiene programs (n=345) were asked to forward an electronic survey invitation to clinical faculty. Eighty-five directors forwarded the survey to 847 faculty; 45.3% (n=384) participated. The 37-item survey contained multiple-choice and Likert scale questions and was available for 3 weeks. Descriptive statistics were used to analyze demographic data and research questions. The Kruskal-Wallis, Spearman Correlation Coefficient and Mann-Whitney U tests were employed to analyze hypotheses (p=0.05).
Results: The demographic profile for participants revealed that most worked for institutions awarding associate entry-level degrees, had 1 to 10 years' experience, taught clinically and didactically, and held a master's degree. Clinical instructors valued calibration, believed it reduced variation and wanted more calibration. Some were not offered quality calibration. There was a difference between the entry-level degree awarded and the program's evaluation of clinical skill faculty reliability, as analyzed using the Kruskal-Wallis test (p=0.008). Additionally, full-time versus part-time educators reported more observed student frustration with faculty variance, as evaluated using the Mann-Whitney U test (p=0.001, bfp=0.004).
Conclusion: Faculty members value calibration's potential benefits and want enhanced calibration efforts. Calibration efforts need to be improved to include standards for measuring intra- and inter-rater reliability and plans for resolving inconsistencies. More research is needed to determine effective calibration methods and their impact on student learning.
Introduction
In dental hygiene education, clinical instructors with varying experience, backgrounds and education unite with the common goal of creating competent graduates prepared to care for the public. This unique expertise provides a wealth of knowledge not found in textbooks. However, this diversity might also interfere with providing quality dental hygiene education.1
While the educational goal might be unified, the teaching methods and clinical techniques of instructors might be conflicted. Faculty variation distracts the students from focusing on patient care and redirects them to satisfying the evaluating instructor.2 Dental hygiene students develop competence through didactic instruction, evaluation of clinical care and performance modeled by instructors. There are often multiple ways to perform efficacious skills. Novice students learning to think critically and problem-solve might experience difficulty sorting through instructor inconsistencies.
Students begin their careers with education as their sole foundation. Reducing variation to better meet the Standards for Clinical Dental Hygiene Practice and accreditation standards will help programs accomplish student competencies in patient care, ultimately benefiting the public.3 As programs improve instruction and assessment methods, graduates will be better prepared for ever-changing patient demands. Investigating current efforts should aid in planning and implementing effective future calibration offerings.
Previous medical education research investigated student perceptions of faculty variation, variation causes, calibration attempts and faculty development. Several studies demonstrated considerable variation in assessment and clinical judgment among health care education faculty.4-10 Dental education faculty exhibited variation in periodontitis diagnosis and treatment planning,7 cavity preparation assessment,9 calculus detection,4 radiographic interpretation,8 periodontal probing11 and student performance assessment.10
Qualitative research has described faculty and student frustration with instructor inconsistency.2,12-14 One study reported only 53% of dental students were satisfied with the consistency of clinical instruction and assessment.12 Common concerns reported were different standards and frequent disagreements among instructors.12 Students reported altering clinical performance to satisfy instructors.12 Twenty percent of perceived program weaknesses revolved around faculty inconsistency.12
In investigating possible variation causes, some researchers indicated faculty status as a culprit,10 while others suspected varying educational and/or professional backgrounds,1,11,14 personal preference differences, and aging faculty populations could be responsible.1 For example, grades for student performance were assigned differently by full-time faculty, residents and part-time clinical faculty.10 This variation was possibly due to differing calibration requirements of faculty groups; part-time faculty were calibrated yearly, whereas full-time faculty were only calibrated upon hiring.10 Variation was also linked to years of clinical experience. One study involving periodontal probing accuracy showed the highest agreement among faculty with more experience.11
Calibration methodology studies revealed varied, but promising, results.4,6,8 While calibrating faculty in explorer calculus detection, researchers concluded calibration became increasingly difficult as calculus detection became more complex.4 Studies have demonstrated short- and long-term potential for calibration efforts to reduce radiographic interpretation variation8 and cavity assessment preparations.6 Similarly, the collective literature on faculty development is optimistic, revealing high levels of faculty appreciation and desire for more professional development opportunities.15-18 Faculty members have reported altering their teaching and/or assessment methods following calibration exercises, and they perceive un-calibrated colleagues as resistant to changing teaching methodology.16 An operational definition of calibrate is “to standardize as a measuring instrument by determining the deviation from a standard so as to ascertain the proper correction factors … to measure precisely; especially to measure against a standard.”19
Available literature on clinical faculty variation and calibration might seem ample; however, dental hygiene is clearly underrepresented.4 The majority of studies have been conducted in medical and dental educational programs. The level of variation and consequences cannot be assumed to be similar among different types of health care programs. Additionally, little research is devoted to faculty development for teaching in clinical (versus didactic) settings.16
Based on literature reviewed, research questions and hypotheses were developed to answer questions regarding calibration efforts for entry-level dental hygiene clinical faculty members. The questions were:
What were the faculty attitudes regarding calibration?
What were the characteristics and quality of the current calibration efforts?
Were faculty satisfaction with their program's calibration efforts?
Methods and Materials
The voluntary electronic survey involved minimal risk and was approved as exempt from review by the Human Subjects Committee (#3706) at Idaho State University. Instructors who taught in accredited dental hygiene clinical programs during the 2011 to 2012 academic year were invited to participate, regardless of employment status, years of experience or responsibilities. A census of the entire population was used to include as many clinical instructors as possible and obtain a large sample. Supervising dentists were excluded.
The self-designed 37-question survey was developed by reviewing the literature related to calibration. This review steered the question development. Participants' demographics were collected by including 7 closed and open-ended questions. Attitude about calibration at the institution was assessed using 8 Likert type questions on a scale ranging from 1 being “strongly agree” to 5 for “strongly disagree.” Characteristics of calibration were evaluated incorporating 5 closed and open-ended questions. Quality of the calibration was examined using 7 items and satisfaction of calibration efforts with 10 items that were constructed using the 5-point Likert type scale.
The 37-item questionnaire was assessed for content validity by performing a Content Validity Index (CVI).20 Experts were asked to rank each survey item for relevancy to research questions. Questions ranked as “not relevant” or “somewhat relevant” were revised or excluded. A minimum CVI score of 0.75, indicating at least 75% of experts viewed the item as “relevant” or “quite relevant,” was required for inclusion. Reliability was analyzed using a test-retest format. An agreement of 75% or greater among 8 participants indicated acceptable reliability. Items below 75% were revised for increased clarity. The pilot study determined 92.6% reliability between the test and retest responses.
The final survey was constructed using Survey-Monkey® to reduce cost while enhancing efficiency and convenience. Participant consent was obtained in the survey introduction. Survey access was denied to non-consenting participants. To ensure anonymity and confidentiality, SurveyMonkey® did not store personal identifiers. Participants could discontinue the survey at any time prior to submitting their responses. Data were downloaded for statistical analysis and reported in aggregate form.
Dental hygiene program directors' emails were obtained from the American Dental Hygienists' Association and from the programs' websites.21 An email was sent to directors of all 345 programs in the U.S., asking them to forward a survey invitational letter and Uniform Resource Locater (URL) to all clinical instructors. Directors were asked to indicate participation by responding to the email and providing the number of clinical faculty receiving the survey invitation. An incentive drawing for one prepaid $100 Visa® card encouraged director participation. One week later, a second email was sent to non-responding directors, and a reminder email was sent to those who indicated participation, asking them to forward a reminder letter to clinical faculty. This follow-up procedure was repeated 1 week later; the survey was available for 3 weeks.
Research questions were analyzed using descriptive statistics. Mean, minimum and maximum values were calculated for Likert-style questions. Frequencies and percentages were calculated for multiple-choice items. Hypotheses involved ordinal data and were tested with non-parametric inferential statistics. The Kruskal-Wallis test was used to detect differences within variable groups, the Spearman Correlation Coefficient was used to identify relationships between ordinal variables and the Mann-Whitney U test determined differences on ordinal scales between 2 variables (p=0.05). The Bonferroni correction was utilized to control Type I statistical errors encountered when multiple analyses were performed.
Results
Eighty-five program directors (24.6% of those contacted) forwarded the survey invitation to their clinical faculty (n=847). While 393 faculty members consented to and opened the survey, 384 (45.3% of those invited) completed it. One hundred and three (26.8%) respondents were not able to answer questions regarding the characteristics of, quality of and satisfaction with calibration efforts, because they were not offered calibration during the 2011 to 2012 academic year; thus, only 281 responses were possible for the analysis of these questions. Additionally, some participants chose not to answer specific questions, resulting in differing numbers of responses (254 to 384) for the remaining survey items.
The demographic information for the sample was evenly distributed from each geographic area (Table I). The majority of respondents were faculty members who taught both clinically and didactically (55.7%, n=214) in programs awarding entry-level associate degrees (47.9%, n=178). One-third (38.2%, n=147) worked only in the clinical setting. Half of the respondents held a master's degree (50.8%, n=193) and worked full-time (53.0%, n=196).
Table II conveys the results of survey items that investigated attitude toward calibration based on the Likert scale of 1=strongly agree, 2=agree, 3=undecided, 4=disagree and 5=strongly disagree. Participants indicated a strong mean agreement (1.1) and no disagreement with viewing faculty calibration as an important aspect of educating students. Responses also revealed an overall willingness to attend non-mandatory calibration exercises. Clinical instructors perceived students were more satisfied with their clinical experiences when instructors were calibrated, and frustrated when instructors were not calibrated. There was agreement (2.1) with students changing their performance depending on their evaluator, and agreement with instructor status and varying professional judgment presenting difficulties in calibrating faculty.
The characteristics of calibration questions revealed that full-time and part-time educators were required to participate (69.0%, n=189) (Table III). Nearly one-fourth of the participants reported attendance was not required for clinical faculty. Participants could also select the answer choice of “other” and provide written responses, which included reports of calibration being required, yet not attended, or calibration only implicating specific faculty members, such as those involved with particular skills or clinics.
When asked about calibration frequency, the majority of participants (74.6%, n=200) were offered calibration every year, semester or quarter. A small portion (7.1%, n=19) was offered calibration only once every 2 to 4 years. Two-thirds (66.5%, n=169) reported their institutions offered calibration on a routine basis, although many indicated calibration was offered whenever deemed necessary (41.7%, n=106), such as when a problem arose or a new technique was introduced. “Accreditation” and “new faculty” were not significant reasons for calibrating clinical faculty. Participants who selected “other” and provided written responses (1.6%, n=4) included calibration being offered infrequently, when needed, or when external continuing education classes were available as a means of calibration. Other written responses mentioned that getting the entire faculty together for participation was challenging.
Calibration compensation was included in contracted salary/pay for about one-third (35.0%, n=95) of the respondents, while another 38.5% (n=106) received no compensation. One-fifth (19.6%, n=54) of the participants were compensated on an hourly basis. Written responses (6.5%, n=18) revealed some institutions paid part-time, but not full-time educators, as it was considered a part of contracted duties, and other programs compensated one calibration session per semester. Receiving continuing education credit for calibration participation was another form of compensation, and some also received reimbursement for travel expenses. Scheduling calibration during regular working hours prevented some institutions from paying additional wages.
Demographic Variables of Respondents (n=384)
All clinical skills questioned in the survey were included in calibration exercises. Power instrumentation was calibrated the least (54.6%, n=142). Periodontal assessment/classification was the most commonly calibrated skill (85.4%, n=222). Written responses indicated that local anesthesia, computer training, grading and professional documentation also were calibrated.
The respondents were divided about the quality of their institutions' calibration (Table IV). Most participants indicated that calibration was held in a clinical setting (2.4) but were undecided if calibration consisted of discussion rather than skill calibration (2.5). Respondents had varied attitudes when asked if calibration assessed clinical performance (3.3), a predetermined level of performance was required (2.5) or if calibration assessed reliability (3.1) and consistency (3.3). Faculty disagreed (3.8) that calibration efforts included a pre-test to determine pre-calibration performance.
Summary of Attitudes Toward Calibration (n=384)
Table V summarizes the survey questions pertaining to calibration satisfaction. Participants felt that calibration reduced variation and that they preferred more calibration (2.1). The mean values were between “agree” and “indecision” that calibration adequately addressed variation between members (2.6), calibration quality satisfaction (2.6) and individual faculty inconsistency being adequately addressed (2.7). The results were inconclusive (range 2.8 to 3.0) if faculty had been calibrated in each specified clinical skill.
The Kruskal-Wallis test (p=0.008) revealed a difference between the entry-level degree awarded and the program's evaluation of clinical skill faculty reliability. Further analysis of this finding with the Mann-Whitney U test revealed a difference between bachelor and associate entry-level programs (p=0.003, bfp=0.009). In addition, comparing certificate to bachelor entry-level programs was also suggestive of a difference (p=0.021, bfp=0.063). It was also found that full-time versus part-time faculty members reported more observed student frustration with faculty variance, as evaluated using the Mann-Whitney U test (p=0.001, bfp=0.004).
Discussion
Research shows instructors with less experience have greater levels of variation.11 One-half of respondents worked part-time and had 10 or fewer years of experience as clinical faculty. If this sample is representative of the dental hygiene faculty population, one-half of clinical instructors have not yet reached the level of expert. It is accepted among various fields of study that reaching expertise requires 10 years of experience.22 Experts view, process and react to situations differently than novices and have enhanced judgment and decision-making skills.23
For the majority of participants, all faculty members were required to attend calibration; however, participants described difficulties in getting part-time employees to attend, due to commitments to other jobs. One-half of respondents had master's degrees and were more likely to have completed advanced educational methodology coursework. More than one-third of the participants worked only in clinic (either instructors and/or administration) and might not have the same opportunities as instructors working in both the clinic and classroom for hearing student frustrations, discovering gaps between classroom theory and clinical practice, or benefiting from networking with other didactic colleagues.
The overall attitude of clinical faculty toward calibration was positive. Participants viewed calibration as very important and were willing to voluntarily participate. These findings are congruent with previous research.16,17,24 Clinical faculty also felt calibration improves student satisfaction with their educational experiences, while variance frustrates students. There was agreement that students change their performance to match the evaluating instructor, as reported in previous studies.2 It is possible for such alterations to go unnoticed by faculty; surveying students might help determine the effects of variance on their education. Participants were divided in their attitude toward the effects of professional judgment and instructor status on calibration. While some felt these factors make calibration more difficult, others did not. Further research to reveal sources of difficulty would be beneficial.
Summary of Characteristics of Calibration Exercises (n=281)
Summary of Quality of Calibration (n=281)
Summary of Satisfaction with Calibration Efforts (n=281)
Attendance for calibration efforts was mandatory for the majority of full- and part-time employees, yet some faculty did not attend, or attendance was only required for the educators involved in teaching/evaluating the skill being calibrated. True calibration evaluates the reliability of faculty; this can only be achieved if every clinical faculty member participates fostering a sense of teamwork as they work toward common goals.
More than one-third of the respondents reported calibration occurred when a specific problem or need arose. Calibration should be preventive and is necessary well before need is evident. Establishing a schedule for frequency and what is to be calibrated would ensure each clinical skill is addressed and maintained on a regular basis. Many programs acquire new part-time clinical faculty as often as every year or semester. Newer faculty might be heavily influenced by their clinical experiences and find calibration efforts personally threatening.14 Experienced dental educators view expert technical skills as an essential element for clinical faculty,1 yet might be resistant to change or unable to see the need for it.16 Program directors indicated that calibration is one of their biggest challenges; allotting ample opportunities for clinical calibration sessions, in positive, non-threatening manners, would help increase the likelihood of achieving faculty reliability.
The desire to improve reliability, consistency, and effective teaching might often be enough incentive for participation. However, many part-time instructors also work in private practice and full-time faculty work many hours to fulfill their responsibilities. Compensating faculty for time in calibration exercises would increase its appeal and help encourage attendance. More than one-third of respondents did not receive compensation, perhaps because of budget restraints. The relationship between compensation, mandatory participation and attendance should be investigated to determine if remunerating faculty or other factors might enhance participation.
For many, calibration opportunities were not used to improve reliability and consistency of clinical skills. This concept identifies the need for programs to decipher between true calibration (including an evaluation and comparison of performance), teacher in-services, educational methodology workshops and faculty meetings. Some respondents were quite positive about their experiences, while others were not. Faculty members need perceived benefits from calibration including measurable goals for faculty calibration.
Most respondents thought that calibration occurred in clinical settings, yet most also agreed that calibration consisted of discussion rather than actual calibration of skills performance. Gathering all clinical faculty members might pose an ideal time to discuss clinical issues; however, such activity does not necessarily reduce performance variability. Most calibration sessions did not include any measurement of inter-rater (consistency between faculty members) or intra-rater (consistency of each individual faculty member) reliability. Utilizing a standard to which everyone will be compared is optimal for calibrating and streamlines the process of evaluating inter-rater and intra-rater reliability.4,6 Dental hygiene programs and licensure exams use standards to measure student performance and clinical instructors should be held to the same expectations, if not greater. If everyone is compared to the same standard, all participants who agree with the standard also agree with each other, and measuring each participant multiple times would determine intra-rater reliability. After gathering reliability data, programs need a plan for resolving inconsistencies and re-evaluating outcomes to ensure reliability was established. Discovering a problem is only beneficial if an effective resolution plan has been constructed.
Previous literature suggested a connection between faculty status/years of experience and attitudes toward faculty development.15,16,24 However, this study did not. Full-time employees did voice a stronger agreement with faculty variance causing student frustration that is in agreement with previous research.2,12,13 This effect could be because full-time faculty members have more opportunities to witness frustration. Also, faculty who worked for institutions awarding an entry level bachelor's degree (as opposed to an associate's degree or certificate) had significantly lower agreement with instructors being assessed multiple times to evaluate intra-rater reliability. This finding could be attributed to these universities employing faculty or administrators with advanced degrees and strong research backgrounds, heightening the need for reliability and their programs' possible shortcomings.
Respondents were undecided about their satisfaction with calibration. If the efforts do not actually calibrate participants, the sessions are not a wise use of resources. Therefore, recommendations for administrators for improvement include establishing guidelines about attendance and remuneration and including this information in the faculty written department policies. Also, the department might involve the entire faculty in creating a calibration philosophy and publish it for existing and new faculty. A plan should be created for calibrating new faculty. If existing faculty are calibrated, a mentor could be assigned to work alongside a new instructor until calibration is achieved, as evidenced by evaluating students simultaneously to establish inter-rater reliability. Calibration efforts can be enhanced by implementing student evaluation mechanisms, by using patients during the exercises and by incorporating a standard for measuring performance. The calibration experience would also be recreated for any absence, therefore, attendance could improve knowing that additional time is involved in make-up sessions for the calibration presenter as well as for faculty. The individuals responsible for planning and implementing calibration must have ample scheduled time to ensure calibration is quality-oriented, meets outcome measures and merits the participants' time and the program's resources.
Additionally, calibration efforts need to be safe and non-threatening for participants, which include maintaining confidentiality of results. Faculty members should not feel threatened about job security or that the calibration exercise might be due to a lack of performance. The emphasis needs to be placed on improving teaching skills to enhance student learning.
This study design posed several limitations. The sampling method depended on the program directors' cooperation for eligible clinical instructor invitation. Self-selection bias presents a limitation in which subjects decide for themselves if they want to participate.25 Directors and faculty members might have decided whether or not to participate as a result of their personal attitudes, experiences or satisfaction with their institution's calibration. In addition, when writing multiple-choice questions, it is difficult to include every possible answer choice, thus soliciting forced answers.26 If there was doubt that every reasonable response was included, an “Other (please specify)” answer choice was added. Email invitations might have been disregarded by potential participants.27 Therefore, sending multiple invitations helped increase the number of faculty members who read the message.
Conclusion
Demographic data found equal distribution of respondents from the 4 regions of the U.S. Calibration characteristics, attitudes, quality and satisfaction as measured by this survey research would seem to be generalizable to most dental hygiene programs. Dental hygiene programs are encouraged to strategically plan frequent calibration events that address each clinical skill taught and assessed. Such calibration sessions need to utilize a standard measuring clinical faculty's performance and a plan for reducing unreliability.
This study's findings support past research indicating mixed yet promising results that calibration reduces variation, and that more research specific to dental hygiene is necessary, such as identifying calibration methods that effectively reduce clinical faculty inconsistencies.4,6,8 The effect of calibration on the students' learning has not yet been investigated. Determining effective calibration techniques that enhance student learning should be a focus of future research.
Acknowledgments
The authors thank Renee Thompson, Administrative Assistant, for her assistance in constructing and managing the electronic survey, the program directors for their participation, and the respondents for their contributions. Much appreciation is also extended to Teri Peterson, M.S., Statistician, Division of Health Sciences, for her statistical analyses guidance.
Footnotes
Nichole L. Dicke, RDH, MDSH, Department of Dental Education, Indiana University-Purdue University Fort Wayne. Kathleen O. Hodges, RDH, MS, Professor Emerita, Department of Dental Hygiene, Office of Medical and Oral Health; Ellen J. Rogo, RDH, PhD, Associate Professor, Department of Dental Hygiene, Office of Medical and Oral Health; Beverly J. Hewett, RN, PhD, Clinical Assistant Professor, School of Nursing. All from Idaho State University.
This study supports the NDHRA priority area, Professional Education and Development: Investigate the extent to which new research findings are incorporated into the dental hygiene curriculum.
- Copyright © 2015 The American Dental Hygienists’ Association
