Abstract
Purpose: Oral health care providers have been charged with recommending the human papillomavirus (HPV) vaccine for the prevention of HPV oropharyngeal cancers (OPC). The purpose of this study was to determine dental hygiene student competency of the application of brief motivational interviewing (BMI) and the accuracy of HPV vaccine information for the prevention of HPV OPC.
Methods: A convenience sample of 59 senior dental hygiene (DH) students from the class of 2020 (n=31) and the class of 2021 (n=28) participated in the HPV OPC curriculum and skills-based BMI training at the University of Minnesota School of Dentistry. Students completed two audio-recorded patient interactions and one Objective Structured Clinical Examination (OSCE). Student self-assessment and faculty evaluation scores were determined by a standardized BMI HPV rubric. Descriptive statistics were used to analyze the data.
Results: Class of 2021 self-assessment ratings were higher than the class of 2020 in all components of the BMI HPV rubric at all three time points. Faculty evaluation achieved statistically significant improvement for the class of 2021 from patient Interaction 1 (evocation p<0.01; summary p<0.01) to the OSCE (evocation p<0.05; summary p<0.01). Both classes rated themselves as competent (≥70%) for most BMI subcategories.
Conclusion: Dental hygiene student competence in demonstrating the components of the Spirit of MI to the accuracy of HPV and HPV vaccine information was achieved through the implementation of the skills-based BMI HPV training. Outcomes of student self-assessment and faculty evaluation highlighted the Kirkpatrick Model as a framework to evaluate BMI skills-based training.
- dental hygiene education
- brief motivational interviewing
- skills-based training
- patient education
- human papillomavirus vaccine
Introduction
Most oropharyngeal cancers (OPC) in the United States (US) are attributed to the human papillomavirus (HPV), in particular HPV16 and HPV18.1-3 Healthy People 2030 set an objective to reduce HPV-vaccine preventable infections from 15.1% to 8.7% for individuals aged 20 to 344 and the indication for the HPV vaccine was expanded by the Food and Drug Association for the prevention of HPV-related OPCs in 2020.5 Despite this information, the incidence of HPV OPCs is increasing and the HPV vaccine uptake goal is still below the projected threshold.2,4
Although health care provider communication has been identified as one of the most influential facilitators of vaccine uptake, providers have reported barriers to discussing HPV with patients, including a lack of HPV OPC knowledge, comfort in discussing a sensitive topic, and a lack of confidence in vaccine advocacy communication.6–12 Recent studies investigating HPV communication suggest that motivational interviewing (MI) may be the optimal counseling approach to HPV-OPC and HPV vaccine discussions during patient care10,13–16 Motivational interviewing is a collaborative patient-centered counseling approach to support a positive behavior change.17 Research has explored using MI as an optimal communication approach for HPV counseling followed MI guiding strategies and MI RULES during brief motivational interviewing (BMI) sessions.10,13–15 Brief motivational interviewing is intended for health care providers that have limited time (5-10 minutes) to counsel patients on their perceptions of change, provide behavior change support, and promote awareness.18,19
Oral health care providers need knowledge on general HPV information, the role of HPV in OPC, and HPV vaccination facts to develop competent skills in HPV communication and vaccine advocacy. Additionally, MI training must include coaching, role-playing, and feedback to effectively deliver information and confidently counsel patients on HPV OPC prevention. Improved communication training may be effective in promoting provider confidence in their HPV discussion skills. Furthermore, the American Dental Association (ADA) has published a policy statement to encourage oral health care providers to recommend the HPV vaccine for the prevention of HPV OPC and asserts that “comprehensive skills-based training should be integrated into existing dental education programs” for vaccine advocacy.20 A comprehensive curriculum for HPV OPC communication should include training in HPV knowledge and evidence-based communication strategies. However, formal didactic and clinical training to prepare students for HPV OPC communication has been found to be limited in dental hygiene education programs.21
In 2019, the University of Minnesota (UMN) School of Dentistry dental hygiene (DH) program launched a skills-based BMI HPV curriculum, consisting of an HPV OPC and HPV vaccination advocacy module and communication skills training. A critical aspect of implementing a training program in an education setting is the evaluation of student outcomes. The Kirkpatrick Model22 for training was used as the framework to evaluate this skills-based program. The Kirkpatrick Model includes four levels of evaluation, 1) “reaction” of the students’ training experience, 2) “learning” of knowledge and skills attained from the training, 3) “behavior” as a result of the training, and 4) “results” of the training to achieve an outcome.22,23
The first phase of evaluation of this skills-based curriculum assessed the “reaction” from students. Students’ perceptions of the skills-based curriculum to enhance their knowledge, attitudes, confidence and comfort during HPV communication and vaccine advocacy has been assessed and reported previously.16 The second phase that was studied determined students’ level of “learning” the importance of and their confidence in applying BMI strategies during HPV communication and vaccine advocacy.15 The third phase had two specific aims: 1) evaluate student competence in demonstrating the components of the Spirit of MI17 while applying BMI during HPV communication and 2) determine the accuracy of HPV and HPV vaccine information while using BMI. The purpose of this study was to measure the outcomes of the skills-based BMI HPV curriculum on student competency in alignment with the third (behavior) and fourth (results) levels of the Kirkpatrick Model.
Methods
This study was determined to be exempt by the Institutional Review Boards (IRB) of the UMN (STUDY00007617) and A. T. Still University (IRB #2019-106). A convenience sample of 59 DH students from the class of 2020 and the class of 2021 at the UMN DH program participated in the skills-based BMI HPV curriculum. The skills-based BMI HPV curriculum was developed utilizing the existing MI thread in the UMN DH program.
The MI thread begins in the communications course in semester two of a six semester program. In semester two, DH students have three 1-hour 40-minute sessions (five lecture hours) on MI content. Lecture and learning activities encompass coaching and role-playing, practicing the Spirit of MI, MI principles, MI guiding strategies, and MI RULES.17,18,24 Operational definitions are provided in Figure 1. Students learn the Elicit-Provide-Elicit (E-P-E) approach to ask evocative questions to understand patient awareness, provide information with permission to fill in gaps in understanding, and elicit patient motivation and readiness for change in semester two.17 Additionally, the MI thread includes evaluation of student competency of MI skills during Objective Structured Clinical Examinations (OSCE) with standardized patients in semesters 3-6 of their educational training. An OSCE is a gold-standard assessment tool to evaluate student clinical competence, while eliminating live patient bias.25 The class of 2020 and the class of 2021 both completed the MI thread as students in the UMN DH program.
Students’ application of MI is evaluated using the UMN standardized MI rubric. In 2018, faculty participated in calibration sessions to determine inter-rater reliability and the results were used to make modifications to develop a revised UMN standardized rubric. The UMN standardized MI rubric has been used for evaluation of students’ application in the MI thread for three cohorts.
For the class of 2021, the skills-based HPV BMI curriculum was modified after evaluation of the curriculum.15,16 Additionally, the COVID-19 pandemic resulted in distance learning in 2020, requiring a revision of learning activities. Figure II displays the skills-based BMI HPV curriculum for the class of 2020 and the class of 2021. Details on the learning modules, resources, and activities for the skills-based BMI HPV curriculum have been described previously.15,16
Evaluation instrument
Student self-assessment and faculty evaluation scores were determined by the standardized MI HPV rubric. The standardized UMN MI rubric was modified to include HPV and HPV vaccine language. The MI HPV rubric is worth 30 points and includes the Spirit of MI17 components with the following HPV vaccine language: collaboration to elicit ideas for change in HPV vaccine status (9 points possible), acceptance by asking permission before providing information about the HPV vaccine (6 points possible), compassion to understand patient perceptions with the HPV vaccine (6 points possible), evocation to support patient autonomy regarding the HPV vaccine (6 points possible) and the MI guiding strategy of summarizing (3 points possible).18,25 Within each of these categories (excluding summary) are sub-categories (3 points per sub-category). Students can score a three or a zero in each sub-category on their application of MI principles, MI strategies, MI RULES, and the application of BMI, while delivering accurate HPV and HPV vaccine information via the E-P-E approach.
Student self-assessment and faculty evaluation
Students in the class of 2020 completed two audio recorded discussions with patients in the UMN clinic demonstrating BMI during HPV communication; students completed the HPV Patient Interaction 1 and self-assessed using the standardized MI HPV rubric. Two DH faculty investigators (CS and MA) team-graded the HPV Patient Interaction 1 using the same standardized MI HPV rubric. Faculty feedback provided an opportunity for students to implement improved HPV knowledge and application of BMI to their audio recorded HPV Patient Interaction 2. Following the first patient interaction, students completed Patient Interaction 2, self-assessed, were team-graded and provided with faculty feedback. Once the two patient interactions were completed, student competency was evaluated during an OSCE. The OSCE simulations were recorded to allow students to self-assess and for faculty reference during the evaluation process. For the class of 2020, there was no faculty calibration session prior to the OSCE. It had been previously determined that the essence of MI principles had been preserved on the modified MI HPV rubric, so formal faculty calibration was deemed unnecessary for the OSCE evaluation. Dental hygiene faculty individually evaluated students during the BMI HPV OSCE in real-time and had access to the recording to confirm initial evaluation and feedback.
Students in the class of 2021 were given the option to complete the HPV Patient Interaction 1 with a first-year DH student as a mock patient. The first-year DH students had not yet been exposed to the MI thread in the curriculum. This modification was implemented to allow novice students to practice their HPV knowledge and BMI skills with a peer to gain confidence prior to a HPV Patient Interaction in clinic.15,16 The class of 2021 students completed the HPV Patient Interactions by audio recording, self-assessed, were team-graded and provided with feedback by the same DH faculty (CS and MA) as the class of 2020. The only modification was a one-hour faculty calibration session via Zoom to prepare for the OSCE. During the faculty calibration session, the MI HPV rubric criteria was discussed, faculty team-graded two BMI HPV audio recordings from the class of 2020 and discussed grading discrepancies to standardize evaluations. Team grading led by MA and CS was used for the OSCE for the class of 2021. This modification served two purposes: 1) enhance student MI coaching by providing feedback by two faculty, 2) serve as an additional faculty calibration session.
Data analysis was performed using a statistical software program (R version 3.5.2) for the Wilcoxon signed-rank test to compare the two classes (2020 and 2021). Comparisons for each class were analyzed for Patient Interaction 1 (PI 1), Patient Interaction 2 (PI 2), and OSCE/Patient Interaction 3 (PI 3); p-values less than 0.05 were regarded as statistically significant. Agreement between student and faculty ratings were quantified using the percent agreement and kappa statistic.
Results
A total of 59 DH students from two cohorts (class of 2020, n=31; class of 2021, n=28) completed the skills-based BMI HPV curriculum. Students’ self-assessment and faculty evaluation frequencies are shown in Table I. Class of 2021 self-assessment ratings were higher than the class of 2020 in all components of the MI HPV rubric. Comparing the self-assessments from the class of 2020 to the class of 2021, there was a statistically significant improvement for Patient Interaction 1 (acceptance p=0.03), Patient Interaction 2 (collaboration p=0.02; evocation p<0.01; and summary p=0.02) and for the OSCE (summary p<0.01). Faculty evaluation of students was also higher for the class of 2021 as compared to the class of 2020, except for the MI HPV rubric component of acceptance for Patient Interaction 2. Comparing the faculty evaluation for the class of 2020 to the class of 2021, there was a statistically significant improvement for Patient Interaction 1 (evocation p<0.01; summary p<0.01) and the OSCE (evocation p 0.05; summary p<0.01).
The sub-categories of students’ self-assessment, faculty evaluation frequencies are shown in Table II. Students from both classes rated themselves high (≥70%) for most of the sub-categories of the MI HPV rubric. However, lower student self-assessment ratings and faculty evaluations (<70%) were identified for both classes during Patient Interaction 1. There were statistically significant improvements for Patient Interaction 2 (“uses open-ended questions” p= 0.02) and for the OSCE (“eliciting ideas for change” p=0.01; “supports self-efficacy/patient autonomy” p=0.04) when comparing the class of 2020 to the class of 2021. Faculty evaluations demonstrated statistically significant improvement between classes for the following: Patient Interaction 1 (“ask permission” p=0.03; “uses open-ended questions” p<0.01), Patient Interaction 2 (“uses open-ended questions” p=0.05) and for the OSCE (“introduction” p=0.02; “supports self-efficacy/patient autonomy” p=0.02).
Agreement between student self-assessment and faculty evaluation across three time points is shown in Table III. Agreement ≥70% was achieved for two of the five components of the MI HPV rubric for the class of 2020 and faculty while agreement was achieved for three of the five components for the class of 2021 and faculty. The kappa statistic quantifies the relationship between the observed level of agreement and the level of agreement expected due to chance, given the distribution of observed ratings. The observed level of agreement for student self-assessment and faculty evaluation is more than chance, but not near perfect agreement (Table III).
Discussion
This study is the third phase of evaluating a skills-based curriculum to determine student “behavior” and “results” using the Kirkpatrick Model in assessing competency in applying the four Spirit of MI components (collaboration, acceptance, compassion, and evocation) to support patient autonomy in receiving HPV and HPV vaccine information. The skills-based BMI HPV curriculum has been implemented for two classes in the UMN DH program and is planned to continue for future cohorts. The UMN DH program is a competency-based educational program; students are required to achieve a minimum of 70% for competency. Accordingly, a threshold of ≥70% was set for students to demonstrate competency in the skills-based HPV BMI curriculum.
Regarding the first specific aim, many factors can inhibit student competency in demonstrating the components of the Spirit of MI while using BMI during HPV communication, including patient-provider trust and rapport and patient resistance. Patient Interactions 1 and 2 were dependent on the availability of patients meeting the assignment criteria. This resulted in unequal experiences for students. Qualifying patients who agreed to be recorded for the Patient Interactions who were engaged and interested in the HPV discussion, allowed students to practice their skills. Whereas other patients who agreed to support their student provider in completing an assignment but were not interested in receiving HPV or HPV vaccine information, demonstrated resistance to the topic. Stull et al., also reported that patient resistance may have negatively impacted students’ confidence and comfort during HPV discussions.17
A lack of confidence and comfort in HPV discussions was reflected in low (≤70%) self-assessment scores for all components of the Spirit of MI for Patient Interaction 1 for both cohorts in this study. Both Arnett et al. and Stull et al. reported that an increased number of patient encounters to practice using BMI for the delivery of HPV and HPV vaccine information resulted in higher perceptions of confidence.15,16 In this study, student self-assessment scores increased over time; the highest self-assessment scores were for the OSCE. This also aligned with faculty evaluation of student performance from Patient Interaction 1 to the OSCE. Another possible reason for this outcome was that an OSCE allows for equal, standardized experiences, improving consistency for faculty evaluation of students’ skills.
A positive finding was the improved outcomes in the categories of evocation and summary for the class of 2021. Evocation is a foundational principal of the Spirit of MI to evoke a patient’s own intrinsic motivation while building confidence and supporting autonomy.17,18,24 Since HPV OPCs and HPV vaccine acceptance are sensitive topics, students’ ability to demonstrate evocation is key in maintaining patient autonomy. Additionally, to stay within the parameters of BMI (5-10 minutes), a skill is to demonstrate summaries to close the MI session and check for accuracy. Students in the class of 2021 were able to meet competency in these two areas by their senior year compared to the class of 2020 who were not able to achieve this in their third year of the dental hygiene program.
The class of 2021 had higher overall self-assessments and faculty evaluation ratings. Unique to the class of 2021, the BMI HPV training was online and moved to semester four, whereas the class of 2020 was held in a traditional classroom environment and occurred in their third semester. Stull et al. suggested that BMI HPV skills-based training be presented late in the curriculum to allow DH students time to gain confidence in clinical protocols before approaching this potentially sensitive topic.16 This shift in the sequence of the skills-based BMI HPV curriculum produced stronger outcomes of student competency for the class of 2021 indicating a greater likelihood that these graduates will be able to demonstrate all aspects of the Spirit of MI during HPV communication as licensed professionals. These findings align with Reno et al. who found that as providers had more opportunities to implement MI during HPV discussions, the perceived usefulness of the communication technique increased.10
The learning transfer process is also noted as a determining factor of effective training.23 Learning transfer from the Kirkpatrick Model (level 2) to level 3 (behavior) as reported by Arnett et al.15 was also demonstrated in this study. A noted limitation to the Kirkpatrick Model is the training of evaluators and other outside influences that may impact the evaluation of the training program.23 To address this limitation, a faculty calibration session prior to the class of 2021 OSCE and faculty team-grading of student OSCEs was implemented to reduce contextual influences of faculty evaluation of student competency.
Each component of the MI HPV rubric included accuracy of HPV knowledge and vaccine advocacy coupled with BMI principles to assess the accuracy of the information provided. Student self-assessment and faculty evaluation frequencies ranged from 74%-100% for “eliciting accurate information about the HPV vaccine,” “understand the patient’s perceptions and/or concerns with the HPV vaccine,” and “emphasized patient autonomy regarding the HPV vaccination.” Ratings for “eliciting their ideas for change in HPV vaccination status” varied for students’ self-assessments and faculty evaluation for Patient Interactions 1 and 2; however, final ratings from the OSCE ranged 73%-100%, suggesting curricular strength.
Stull et al. reported students’ lack of HPV and HPV vaccine knowledge may have impacted their application of BMI.16 Additionally, the application of MI requires training, coaching, and feedback to retain skills.15,27,28 This was evident with low frequencies of student self-assessment scores for Patient Interaction 1 for both cohorts and supports prior evidence that students need knowledge, training, and practice applying their BMI skills during HPV discussions.15,16 The class of 2021 completed a 90-minute HPV online educational module, an online 120-minute BMI coaching and role-playing session and a 40-minute MI refresher lecture;15,16 whereas, the class of 2020 completed a 40-minute HPV online education module and only had a 50-minute BMI coaching and role-playing session.15,16 The class of 2021 had higher self-assessment scores and faculty evaluations at all three time points and may indicate that the longer duration and sequencing of the BMI HPV curriculum may provide the appropriate amount of training for DH students to effectively and confidently apply BMI for HPV and HPV vaccination discussions.
Findings from the educational literature indicate that self-assessment in the DH curricula allows students to recognize gaps in their knowledge and skills to inform consequential, self-directed learning.28 This may explain why general knowledge of HPV improved for students in this study. The curriculum provided three opportunities for students to self-assess on their HPV knowledge, which may have contributed to their demonstrated knowledge of HPV information and vaccination advocacy. Daley et al. reported dental hygienists as having discrepancies in their HPV OPC knowledge and identified a need for increased education.11 This study suggests that the learning transfer from Kirkpatrick Model level 2 (learning) to level 3 (behavior) of demonstration of accurate HPV information may have been accomplished through self-assessment. Students were able to determine inaccuracies in their provided HPV information to self-correct for Patient Interaction 2 and the OSCE. The accuracy of HPV knowledge and vaccine advocacy was evaluated with both self-assessments and faculty evaluation, with results ranging between 84%-100%, indicating that the skills-based BMI HPV training enhanced students’ knowledge, a known gap in the literature.6–8 Further, the OSCE results demonstrated that the intended goals of the program were achieved with this skills-based training and completed the final evaluation level (results) of the Kirkpatrick Model.
Dental hygiene programs interested in implementing a skills-based BMI HPV training program need to include training on the Spirit of MI, MI principles, MI guiding strategies, MI RULES and the E-P-E technique for BMI counseling. Students need patient experiences for practicing their MI skills on general oral health topics, to gain confidence and competence applying MI within the parameters of 5-10 minutes to align with BMI. It is also important for students to have MI skills before applying BMI to counsel patients on HPV and HPV vaccine advocacy. Education programs need to have a curriculum that includes the epidemiology of HPV and OPC and the role of vaccination in preventing OPC. A skills-based BMI HPV curriculum is best implemented during students’ senior year of a DH program when they are more confident in their own clinical skills. It is critical to have topic-expert faculty to develop, implement, and evaluate the curriculum in addition to faculty training in BMI and HPV.
This study had limitations. Using a convenience sample of DH students from one dental school in the Midwest does not represent the general population of DH students at other educational settings. The UMN MI and the modified MI HPV rubrics were standardized evaluation tools used in the MI thread at one institution, however neither rubric is validated to determine student competency. This is the first known evaluation of a skills-based BMI HPV training program to prepare students for HPV education and HPV vaccine advocacy. Future research should identify how much BMI HPV training is needed to be knowledgeable on the role of HPV in OPCs and improve HPV vaccine uptake and literacy of patients. Longitudinal data is needed to determine whether BMI HPV skills acquired as students are retained and applied as licensed oral health care professionals.
Conclusion
Results from this study address the call for educational institutions to provide skills-based training to prepare oral health care professionals to raise awareness of oral HPV and provide vaccine advocacy. Dental hygiene student competence in demonstrating the components of the Spirit of MI applied to the accuracy of HPV and HPV vaccine information was achieved with the implementation of the skills-based BMI HPV training. Outcomes of students’ self-assessments and faculty evaluations in this study highlighted the Kirkpatrick Model as the framework to evaluate skills-based training. Dental hygiene programs can use the findings of this study to evaluate students’ application of BMI communication to raise HPV OPC awareness and HPV vaccine advocacy.
Footnotes
This manuscript supports the NDHRA priority area, Professional development: Education (evaluation).
Disclosure
This research was supported by the National Institutes of Health’s National Center for Advancing Translational Sciences, grant UL1TR002494. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health’s National Center for Advancing Translational Sciences.
- Received July 29, 2021.
- Accepted December 23, 2021.
- Copyright © 2022 The American Dental Hygienists’ Association