Abstract
Purpose: In response to several publications drawing attention to self-assessment and revised Commission on Dental Accreditation standards that state graduates should possess and demonstrate the ability to self-assess, dental hygiene and pre-doctoral dental programs find themselves searching for ways in which to incorporate self-assessment practices into the curriculum. Research indicates that students are often not familiar with self-assessment strategies nor are they effective at self-assessment upon entering professional programs. Therefore, students must be taught strategies to self-assess and be given opportunities to practice and refine these skills. Opportunities to develop and demonstrate self-assessment skills can be incorporated across the curriculum at the classroom level and at the global level. Both the A.T. Still University Arizona School of Dentistry and Oral Health and the University of Missouri-Kansas City School of Dentistry utilize a capstone portfolio project to incorporate self-assessment throughout the curriculum. By combining artifacts from their dental and dental hygiene school experience with reflective writing essays, students can demonstrate attainment of program competencies. As more faculty members and students become involved and engaged in assessment strategies such as portfolios, they are also gaining a greater appreciation for the value of self-assessment.
Introduction
When the ADEA-CCI issued a white paper outlining core principles that should characterize dental education and inform and guide dental curricula in 2006, one of the core principles was assessment.1 The white paper aligned with one of the propositions suggested in the 1995 Institute of Medicine (IOM) report on dental education, which was the need for dental education to promote self-assessment.2 Likewise, the call for dental education to incorporate curricula that engages students in self-assessment has gained even more attention since the Commission on Dental Accreditation (CODA) revised standards for pre-doctoral dental programs. As of July 1, 2013 Standard 2-10 states that graduates must demonstrate the ability to self-assess, as opposed to the previous Standard 2-22 where graduates were required to recognize the role of self-assessment.3
The hallmark of a competent individual has been defined as the ability to accurately self-assess.4 Self-assessment has been defined in numerous ways. Gordon defines valid self-assessment as “judging one's performance against appropriate [valid] criteria.”5 He further states that accuracy in self-assessment means “gaining reasonable concurrence” between one's own self-assessment and other validated measures. The process of developing this skill involves providing the student with expected standards of professional behavior, including the capacity to recognize one's own abilities and limitations. Key to Gordon's definition of self-assessment is the necessity of having criteria or standards that are both appropriate and valid. He contends that to assess one's own performance against standards that are anything less would be futile. It is understandable then why a critical component to self-assessment in the educational arena is ensuring that students possess a clear understanding of the criteria/standards. Boud describes self-assessment in terms of involving students in the process of identifying the criteria/standards they will use to apply to their work and then making decisions about whether or not they have met these criteria/standards.6 While self-assessment requires faculty experts for the development of appropriate and valid criteria, and instructing students on the how and why of the criteria, it ultimately is the student's responsibility to learn the criteria and work at applying them to their work products. In essence, the idea is to “flip” the assessment strategy from being solely a faculty domain, to involving students in the activity of assessing. It becomes the students' responsibility to assess their work, identifying where they meet the standards and criteria and where they do not. Taking it the next step then involves students as reflective practitioners in identifying where knowledge, skill or value gaps exist and determining strategies for filling those gaps. When self-assessment is viewed and practiced through these lenses, it becomes an activity done with the students rather than to the student.
With the above as background for change (i.e., IOM report, competency-based education, ADEA-CCI and CODA revisions) dental hygiene and predoctoral dental programs find themselves exploring various strategies to ensure that their graduates meet the new standards of demonstrating the ability to self-assess. This paper provides a review of the literature on self-assessment and suggests how dental hygiene and pre-doctoral dental programs can incorporate pedagogical strategies in the form of individual course-level assignments to global strategies like portfolio assessment that require students to “demonstrate” self-assessment.
Self-Assessment in Review
Self-assessment is not new and in fact has been of interest to researchers for decades and yet adoption into educational settings, dental education in this case, has been slow at best. Literature supports the concept that making judgments about one's own learning is integral to the learning process.6 It is through the formative aspects of assessment that self-assessment motivates further learning, encourage students to take ownership for their learning, and can accommodate different learner readiness, experience and backgrounds. Interestingly, until 1989 there had been no major review of the literature on self-assessment conducted.7 In 1991 Gordon published a comprehensive review of the validity and accuracy of self-assessment in health professional training as a follow up to the review conducted by Falchikov and Boud.5 In 2010 a Best Evidence Medical Education Systematic Review was published on the topic of self-assessment.8 Collectively these reviews called for better research methodologies in studying the impact of self-assessment on learners. However, with that said, there are several themes or consistencies found in the literature that are outlined in the 2010 Best Evidence Medical Education Systematic Review and discussed below.
First, all the studies agree that self-assessment skills remain underdeveloped during the educational process. While self-assessment is recognized as integral to the development of health professionals, self-assessment skills are rarely taught and the ability to self-assess seldom tested. This has been validated in the dental hygiene literature by a study conducted by Mould et al, where it was found that students entered dental hygiene education unprepared to engage in self-assessment.9 As with any instance where change is required, changing assessment strategies to incorporate self-assessment is met with resistance. Students predominantly come from educational backgrounds where the process of increasing metacognition, self-awareness and responsibility for one's learning is atypical.10 Uncertainty and discomfort are sure to arise when students are required to assume increased responsibility for their learning. In educational environments where assessment has been the domain of faculty, it is understandable why students would initially feel uncomfortable with taking on the process of self-assessment. Likewise, faculty are products of educational experiences that also followed a faculty-centered learning environment and so much like students, they have little experience with the implementation of student self-assessment in the learning environment. Yet in professions such as dentistry that operate under the principles of self-regulation and autonomy, the competent clinician must be a self-directed, lifelong learner.
A second theme found in the self-assessment literature involves differences in self-assessment abilities between high performing students versus lower performing students. Specifically, students in the bottom quartile of class tend to overestimate their abilities while those in the top quartile tend to underestimate their performance. While students in the top quartile underestimate their performance, they have been found to be more accurate in their self-assessment abilities than those in the bottom quartile. It was found that poor performers with additional training in logical reasoning and meta-cognitive skills, could improve their ability to recognize when they performed poorly.8 In other words, it was the improved metacognitive skills that enabled the less competent to become more accurate in self-assessment.
Third, students tend to give their peers a more balanced assessment in comparison to how they rate themselves (self-assessment), in terms of the ability to identify one's own strengths and weaknesses as compared to peers. Curtis et al, who examined dental student self-assessment, suggests that the ability to more accurately assess peers than oneself could be related to societal pressures as opposed to educational needs, where students perceive mistakes as a sign of weakness, and are therefore reluctant to acknowledge them.11 Therefore, in order for self-assessment to be effective, educators need to develop an environment in which mistakes are openly shared, discussed and accepted as part of the total learning experience. Boud would agree that it is through this type of formative assessment process that students are able to develop their self-assessment skills.6
Fourth, an examination of individual characteristics which could account for differential ability in self-assessment found gender to be influential. While studies are inconclusive when it comes to gender differences, there does appear to be a trend of males expressing higher levels of confidence than their female counterparts. The researchers agree that this provides fertile ground for future research.
Fifth, research suggests that self-assessment of practical skills tends to result in higher accuracy than assessment of cognitive skills. It is not entirely clear why practical skills may be better self-assessed than knowledge. One explanation extended is that practical tasks involve criteria that are specific and when combined with feedback on self-assessment makes practical skills more amenable to self-assessment. Another explanation is that outcomes on practical skills are harder to dispute so the potential for self-deception is lessened.
Finally, there is some evidence to support that student's with more experience are able to demonstrate greater skills in self-assessment. For example, senior dental hygiene and fourth year dental students versus first year dental hygiene and dental students will have a greater knowledge foundation and skills practice to reflect on and self-assess their abilities.
One thing that is clear in the literature on self-assessment is that accurate self-assessment is very difficult. Falchicov, Boud and Gordon all found that generally, self-assessors were poor to moderate judges of their performance.6,7,12 Therefore, educators are faced with the issue of determining the most appropriate instructional strategy for students to demonstrate the ability to self-assess. More recent literature suggests a change in terminology that takes the focus away from being summative and punitive and instead focuses on the process or formative aspects of self-assessment and self-directed learning. This process allows for trial and error, it is forgiving of mistakes, and serves to reinforce the steps necessary for achieving a competent endpoint.
Self-Directed Assessment Seeking
While the value of self-assessment has been supported through many decades of research, recent publications debate the validity or stability of self-assessment.13,14 Eva and colleagues concur with the work of Falchicov, Boud and Gordon6,7,12 that in general, self-assessors are poor to moderate judges of their performance. Further Eva and colleagues contend that self-assessment is not a stable skill but rather varies by content, context and perspective.13,14 The authors conducted a theory-oriented literature review with the hope that it might refine our understanding of self-assessment. Some of their findings follow. Self-efficacy, for example, consists of “an interaction between self-concept beliefs about one's skills or abilities and the specific context in which those skills or abilities will be applied for the attainment of a particular goal.”13 Rather than worrying about the accuracy of self-assessment, researchers in the field of self-efficacy concern themselves with how self-assessment is used in problem solving situations. Research in social psychology informs a discussion of self-assessment in that it posits people believe themselves to be better than what they are, and these self-deceptions lead to less depression and more persistence. It is because of these tendencies to overestimate one's ability that social psychologists suggest the necessity of looking outward on how others react rather than simply reflecting inward. Evans et al discuss the concept of self-deception in relation to professional students.15 They suggest that because these are students who have performed well consistently in school and received positive feedback from a young age, they perhaps have gained a self-confidence that may be resistant to modification. Further, when asked to self-reflect, they tend to score based on the potential or ideal performance versus their actual performance. In total, this line of research that has borrowed from theory would suggest self-assessment is tied to self-concept and self-worth.
Recognizing that self-assessment skills can vary by content, context and perspective, improvement can only occur if students are taught how to seek out critical feedback from others and are instructed in methods used to evaluate their own performance. Specifically, Eva and Regehr state that, “…the route to self-improvement is not through becoming a more accurate self-assessor, but through seeking out feedback from reliable and valid external sources (experts, self-administered tests, etc.), and … making a special effort to take the resulting feedback seriously rather than discounting it: to reflect rather than ruminate.”13 They refer to this method of self-improvement as self-directed assessment seeking and describe it as a process, rather than viewing it as ability. They further define self-directed assessment seeking as “a habit one needs to acquire and enact rather than an ability one needs to master.”13
A later article by the same authors attempts to further elucidate the discussion of whether or not professionals can accurately self-assess by making distinctions between self-assessment (defined as an ability), self-directed assessment seeking and reflection (defined as pedagogical strategies and process), and self-monitoring (defined as immediate contextually relevant responses to environmental stimuli).14 As is the case in research, it is critical to develop operational definitions to define variables and concepts so that all people involved in the research, and those individuals that will read the research in the future, are able to stand on common ground. Whether one speaks of self-assessment, self-directed assessment seeking, reflection, or self-monitoring, clear operational definitions establish common ground.
In the end, too much literature exists validating the benefits of self-assessment to abandon the concept completely. In addition, CODA standards use the term “self-assessment.”3,16 For these reasons, the term self-assessment will be used for the remainder of this paper with the understanding that the theory behind self-assessment and self-directed assessment seeking have relevance for the following discussion on pedagogical strategies and gaining competency in pre-doctoral dental education.
Pedagogical Strategies for Promoting Self-Asssessment
One of the factors that contribute to inaccuracies in self-assessment include students not understanding what is expected of them.15 Further, faculty may also “assume” that students enter dental hygiene and dental school with the ability to self-assess. Research shows this is not the case, and as with any skill, self-assessment must be learned.8 In order for this to happen, it must be incorporated into the curriculum and taught to students.
Self-Assessment Across the Curriculum
To effectively incorporate self-assessment, programs should identify specific assignments and assessment methods that provide exposure to the process. By adopting the concept of self-directed assessment seeking, a process that allows for trial and error, is forgiving of mistakes, and serves to reinforce the steps necessary for achieving a competent endpoint, students may demonstrate improved accuracy in evaluating their performance.
Self-Assessment at the Classroom Level
Examples of incorporating self-assessment within a course can be found in Curtis et al and Gadbury-Amyot et al.11,17 A critical element of these learning experiences was the establishment of sound standards and criteria, developed by faculty content experts, for students to follow, along with a requirement that students self-assess their work using those standards/criteria. The grading rubric for student self-assessment and faculty evaluation was one in the same. Ideally, the evaluating faculty member would meet with the student to discuss areas where there was a lack of calibration and encourage self-reflection on ways to address filling those gaps with the goal of becoming more competent in that skill.
The reflection aspect is a vital component in taking self-assessment skills full circle. By providing guidance to those students that inaccurately self-assess initially, improvement can be observed. Curtis et al was able to show modest improvement in the calibration of students to faculty with multiple self-assessment trails. It was noted that student scores differed by an average of 5% from faculty assessments, but improved to a 3.3% difference after a four-month trial.11
Self-Assessment at the Global Level in the Form of Portfolio Assessment
In addition to individual assignments at the classroom level, dental hygiene and dental programs are also exploring global or capstone methods of assessment to help students develop and “demonstrate” self-assessment skills. Using portfolios within the curriculum provides a student-centered tool that encourages the student to assess his or her performance both at the classroom level and from a holistic or global perspective. The value of portfolio assessment in developing self-assessment skills is in the form of reflection and reflective writing. Brown and Glasner discuss the relationship of the 2 acts by stating that self-assessment involves the process of reflection, but not all reflection leads to self-assessment.18 By completing portfolios, students are encouraged to reflect on intentions, actions, thoughts, and feelings. Through this reflective process, students can create personal meaning from their own experience, in this instance their experience in dental school, and take a critical stance towards their meanings and interpretations.
Students need to be introduced to basic reflection strategies to allow them to reach Bain et al's higher levels of reflection.19 Likewise, faculty should be educated on prompts and questions to utilize when providing feedback that will challenge and promote deeper levels of reflection.20 High-level challenging feedback that focuses on the process and levels of reflective writing is considered most effective. Providing structure through the use of templates and rubrics can also promote thoughtful reflection. An excellent resource for faculty and programs on rubrics can be found in MedEdPORTAL.21
Reflective Writing and Self-Assessment: Two Examples
At the A.T. Still University Arizona School of Dentistry & Oral Health (ASDOH), students complete a capstone portfolio project, based on the school's 14 program competencies (previously described in Gadbury-Amyot 2012).20 As a graduation requirement, dental students assemble and submit a portfolio that demonstrates their progression and attainment of the competencies, which contains information from all four years of dental school. For each of the 14 competencies, students submit at least two pieces of evidence (artifacts) and one “reflection” essay. Students are oriented to the portfolio during their first year and are given a handbook for the Portfolio Capstone Project as well the syllabus.
The purpose of the portfolio capstone project is to demonstrate that each student has progressed from a novice to a professional, thereby theoretically meeting the A.T. Still ASDOH competencies. Since ASDOH dental students also receive a public health certificate or a master's in public health degree upon graduating with their DMD degree, they can present artifacts from either of those programs in their portfolios. Artifacts can be written assignments, exam results, pictures, case studies, treatment plans, models, videos, patient radiographs, interactions with others, or outreach trips. Choosing artifacts and reflectively writing about them is individual and personal. There is really no “wrong” artifact as long as it relates to the competency, is meaningful to the student, and the student can justify why he/she choose it and can write reflectively about it.
Since portfolios are designed to show progress and growth throughout the curriculum, it is stressed to the students that a portfolio artifact can be a thing or an experience they have had and that is not necessarily their best work. Presenting an artifact from early in their education in which they may have not performed well and then submitting a similar assignment from the end of the program in which they did perform well provides a method of demonstrating growth and progress towards attaining competency. All work cited in the portfolio must be completed while students are enrolled in these programs.
Students are assigned a portfolio mentor who guides them and helps them to identify information relevant to each competency, such as, providing input on artifacts collected and progress made throughout the portfolio creation process. Students meet with and are advised by their portfolio mentor on a regular basis throughout the portfolio process. The portfolio mentors are faculty members who have demonstrated abilities in teaching and learning and have shown an interest in the portfolio process. Additionally, all mentors are trained in portfolio methodology and reflective writing, and attend multiple calibration sessions for grading of portfolios offered by the Student Portfolio Committee.
The use of a portfolio assessment strategy provides a robust assessment methodology, using both formative and summative assessments and reflective analysis, to help determine attainment of competency at the end of a four-year dental school curriculum. The portfolio is but one piece of the school's assessment strategies utilized to determine student attainment of the program competencies.
Reflection Essays
Students write and submit Reflection Essays for each competency summarizing why the artifacts presented in the portfolio address a particular competency and how the artifacts demonstrate progression and attainment of that skill. Further, students write and submit an Integrative Essay, assessing the entire portfolio process, toward the completion of the Student Portfolio Capstone project.
Competencies and Reflection Essays for the portfolio are submitted in groupings of two or three, according to established deadlines. For example, Competencies 7 and 10 are due in December of the third year; Competencies 3 and 4 are due in February of the third year; Competencies 2, 6 and 12 are due in May of the third year, and so on. Dates of submission for the Competencies are chosen based on curricular content and clinical experience. The final portfolio is due at the end of January in the fourth year.
Reflective Writing Workshops
Before the first grouping of competencies is due, in spring of the second year, students complete a reflective writing workshop. At this workshop, reflective writing as a learning tool is discussed at length as it relates the portfolio project. Reflective writing, has been defined as the “evidence of reflective thinking” and typically involves: “looking back at something (often an event, i.e., something that happened, but it could also be an idea or object); analyzing the event or idea (thinking in depth and from different perspectives, and trying to explain, often with reference to a model or theory from your subject); and, thinking carefully about what the event or idea means for you and your ongoing progress as a learner and/or practicing professional.”22
Once the theoretical aspect of reflective writing is covered, students then complete a small group reflective writing exercise. They are first asked to write down three books they would want to have with them if they knew they were going to be stranded on a desert island for a period of time. Additionally, they are to think about and write down why they chose those three books. Their selections and justifications are then shared with the group. In the next phase of the exercise, students are told that after three years they are rescued from the desert island. At this point, students are asked to reflect upon, write down and then share with the group how the books they chose influenced them as a person.
Subsequent to that brief exercise during the workshop, well-written reflection essays along with artifacts previously submitted from earlier classes are reviewed with the group as a whole. Then, the large group is broken into smaller groups, facilitated by portfolio mentors, for the second reflection exercise. In these small groups, mentors choose one competency (e.g., Competency 2 Ethics) and encourage discussion among the group as to what artifacts students might submit for this competency and why they would submit them.
Prompts for reflective writing are discussed with the students and noted in Figure 1. A basic reflection essay format, as follows, is also presented to students:
Introduction to the competency: thoughts, ideas, theories studied
Description of artifacts and each artifact's story: what it is and why it was chosen.
Interpretation: How do the artifacts chosen and their stories, when looked at as a whole, tell the student's story of how they are competent in that particular area, e.g., ethics, leadership, communication, etc.
Conclusion: What did the student learn from this?
In the Reflection Essays, students include their rationale for inclusion of artifacts in relationship to attainment of each competency and analyze and summarize why he/she believes the competency has been met. Students may also identify for each competency, if appropriate, where they will include additional work in this area as part of future professional development goals. Reflection essays guidelines are generally that they be 500 to 1,000 words in length or 1 to 2 pages double–spaced.
In addition to the individual Reflection Essays, students also write an Integrative Essay regarding the portfolio project and their 4 year dental school educational experience. The purpose of this essay is to provide an opportunity for students to assess and comment on their growth as they've progressed through the program to finally arrive as a candidate for the Doctor of Dental Medicine degree. Students are encouraged to write their Integrative Essay not as a summary of what they learned, but as a discussion of what the journey has meant to them as an individual and how it has shaped him or her as a professional.
Reflective Writing Prompts
These essays are expected to be 1,000 to 2,000 words in length (2 to 4 pages double-spaced) and should describe the portfolio in which overall progress has been charted toward attainment of the 14 A. T. Still ASDOH competencies. In this essay, students take one last opportunity to reflect back upon and holistically consider their time in dental school, integrating their experiences with the curriculum, clinic, external rotations, and other experiences into a unified experience. It is an opportunity for the student to take time and review the portfolio from beginning to end and reflect upon their clinical skills; interpersonal and communication skills; overall professional development; and, critical thinking skills.
The portfolio capstone project and its reflective writing component have made a positive impact on the culture of the A.T. Still ASDOH and its faculty, staff, students. The movement of better and more effective means of assessing student's critical thinking has grown within the ranks of A.T. Still ASDOH faculty. More faculty are getting involved in the portfolio project and as a result, more are becoming engaged in this assessment strategy. As this project is presented to each incoming class, the interest and enthusiasm grows, resulting in a greater appreciation of self-assessment and the capstone portfolio project.
University of Missouri-Kansas City School of Dentistry (UMKC-SOD)
Similar to A.T. Still ASDOH, portfolios at the UMKC-SOD are only one piece of the total assessment strategy. In contrast to ASDOH, UMKC-SOD predoctoral portfolios are built around 6 to 10 of the program competencies. The reasoning behind this model comes from the belief that other assessment strategies within the curriculum adequately capture competency for the remaining 4 program competencies. Unlike the predoctoral portfolios, dental hygiene portfolios at UMKC SOD encompass all program competencies.
Also similar to A.T. Still ASDOH, UMKC-SOD dental hygiene and predoctoral students are introduced to portfolio assessment during their orientation to dental school. In contrast to A.T. Still ASDOH, UMKC faculty have pre-determined exactly what evidence is to be included in student portfolios. This approach was taken based on the literature around validity and reliability of performance/portfolio assessment which shows that having consistent items for review increases reliability of faculty evaluation.23-24 Review of the curriculum map outlining which competencies are tied to which courses, including assignments, projects and assessment measures, was integral to this process. In the predoctoral program, a faculty portfolio committee, including the Academic Dean and the Associate Dean of Instructional Technology and Faculty Development, determined what were some of the exemplar examples of course assignments, projects, and events in the curriculum that contribute toward student development of knowledge, skills and values to meet program competencies. The UMKC SOD Division of Dental Hygiene global portfolio assessment project has been explained in detail previously.25
Faculty development has been as crucial to the implementation of portfolio assessment as student development. In many instances assignments were tweaked/modified to strengthen student development of required knowledge, skills and values. In all instances grading rubrics were developed or revised to better capture what was expected from the assignments. The last criteria on the grading rubric requires that students reflect on what knowledge, skills and values were gained from the assignment and taking it to the next step, indicate how this will lead them toward meeting program competencies.
Similar to ASDOH, UMKC students complete a global reflection toward the completion of their coursework for inclusion in the portfolio. This global reflection is the last piece of evidence in the portfolio. This is a chance for each student to step back and view the 2 or 4 years of work holistically. It is an opportunity to take some time to review the portfolio from beginning to end rather then as individual components/evidence. Students are instructed to pay attention to their development as a professional through the complete program, recognizing growth from one year to the next and/or progression from a novice to a competent professional.
By default the faculty who have coursework included in the student portfolios serve as mentors through their feedback during the time the assignments, projects, and events are undertaken. As stated above, faculty are trained in portfolio methodology and are able to provide formative feedback as needed. Beyond these faculty, additional faculty are tapped for a summative evaluation that occurs in the students' final year; for dental hygiene this takes place in the Spring of their last semester and for predoctoral students it is their fourth year, first semester. This timing was deliberate so that those students who do not pass the summative evaluation, have time for mentoring and revision as completion of portfolios are required for graduation.
Conclusion
This paper has presented theory and research behind self-assessment in higher education and related this information directly to dental education. Examples of pedagogical strategies for implementing self-assessment in dental education at both a classroom and global level (portfolios) are described. Applied examples from 2 different dental schools using portfolio assessment where students “demonstrate” self-assessment through reflection and reflective writing were described. It is the authors' hope that these examples can be instructive for dental education programs endeavoring to meet the revised accreditation standards that require students possess and demonstrate self-assessment skills.
Footnotes
Cynthia C. Gadbury-Amyot, MSDH, EdD, is Associate Dean and Professor of Instructional Technology and Faculty Development, University of Missouri-Kansas City. Janet L. Woldt, MS, PhD, is Associate Dean for Academic Assessment, Arizona School of Dentistry and Oral Health, A.T. Still University. Kylie J. Siruta-Austin, RDH, MSDH, ECP-III, is a consultant, University of Missouri-Kansas City.
This study supports the NDHRA priority area, Professional Education and Development: Validate and test measures that evaluate student critical thinking and decision-making skills.
- Copyright © 2015 The American Dental Hygienists’ Association
