Abstract
Purpose: Adoption of portfolio assessment in the educational environment is gaining attention as a means to incorporate self-assessment into the curriculum and to use evidence to support learning outcomes and to demonstrate competency. Portfolios provide a medium for students to demonstrate and document their personal and professional growth across the curriculum. The purpose of this literature review is to discuss the drivers for portfolio education, the benefits to both students and program faculty/administrators, the barriers associated with portfolio use, and suggested solutions that have been determined through several years of “lessons learned.” The University of Missouri Kansas City School of Dentistry, Division of Dental Hygiene department has been utilizing portfolio assessment for over 15 years and has collected data related to portfolio performance since 2001. Results from correlational statistics calculated on the 312 dental hygiene students that graduated from 2001 to 2013 demonstrate a positive and significant relationship between portfolio performance and overall GPA as well as portfolio performance and NBDHE scores.
Introduction
Assessment is a critical component of the educational experience. Assessment verifies students are acquiring the necessary knowledge, skills, problem solving and critical thinking abilities.1 Research shows students view assessment as something that is done to them. Beyond the percentages and letter grades, many students have little knowledge about what is involved in evaluating their coursework.2 Because of this disconnect, there is a growing trend among accrediting bodies to incorporate student self-assessment into the curriculum.3 In addition, accrediting agencies and other stakeholders such as state governments are calling for actual evidence to support students' learning outcomes.3 The implementation of portfolio assessment provides an opportunity for institutions to encourage self-assessment and challenge students to identify sources of evidence to demonstrate and document their personal and professional growth as they progress through the curriculum and on into their professional careers.
Drivers for Portfolio Assessment
What is often referred to as the assessment movement in U.S. higher education began in the early 1980s in part because of the call for curriculum reform, including greater curricular coherence, the use of powerful pedagogies associated with high learning gains and knowledge about student outcomes and experiences.4,5 But an even stronger impetus for assessment was the growing interests of state governments in using assessment to demonstrate return on investment - in other words, make higher education more accountable. In the fall of 1988, then Secretary of Education William Bennett issued an executive order requiring all federally approved accreditation organizations to include in their criteria for accreditation evidence of institutional outcomes. Bennett's executive order specifically held higher education institutions accountable to accrediting bodies for producing and documenting outcomes.6 By the mid-1990s, higher education began to see a shift in state focus and formula funding from input (number of students, library holdings, credentials of faculty, etc.) to output (number of graduates, average time to graduation, etc.). One measure of outcome or output which received considerable attention was student competence. The Commission on Dental Accreditation (CODA), as a federally approved accrediting body, adopted standards for a competency-based curriculum for dentistry in 1998 followed by dental hygiene education in 2000. As a result, dental education has been at the forefront of competency-based (or standards-based) education at higher education institutions across the country. Competency is defined as the skills, knowledge and professional values of an individual ready for beginning independent practice.7
In response to the assessment and accountability mandates, institutions of higher education sought out instruments designed specifically to assess the effectiveness of their educational program. Most institutions used a mixture of surveys and a collection of cognitive examinations like the American College Testing or Graduate Record Examination. However, there was a push by faculty for authentic assessment strategies, where actual student work products from assignments in regular courses, or well-established surveys like the National Survey of Student Engagement, as measures of assessment of student learning. It was within this environment that the University of Missouri-Kansas City (UMKC) School of Dentistry, Division of Dental Hygiene, embarked on a portfolio assessment strategy beginning in 1998. At the time, CODA had not yet adopted competency–based curriculum for dental hygiene, so initial portfolios were based on program goals. However, in 2000 when CODA adopted a competency-based education model for dental hygiene, the division was able to redirect their portfolio assessment strategy to program competencies. Fast forward to 2013 where accreditation standards specifically require student competency in such things as critical thinking, self-assessment and ethical reasoning, and it is not difficult to see how portfolios can provide a mechanism for capturing students' knowledge, skills and values (i.e., competency) while demonstrating growth over time in a way that traditional one-shot testing and assessment cannot do. Many institutions are turning to portfolios as a strategy for demonstrating student competency along with tangible evidence for programs going through accreditation review.
On July 1, 2013, revisions to the Pre-Doctoral Dental Education Standard 2-23 went into effect to include assessment of overall competency, not simply individual competencies, in order to measure the graduate's readiness to enter the practice of general dentistry.8 In preparation for the revised standards, recent national accreditation workshops sponsored by CODA and the American Dental Education Association were conducted for dental educators.9 An emphasis area at the workshops has been the need for evidence for each student (in addition to overall program data) to support decisions about competency. Although at one time it was sufficient to state as a program that all students are meeting the standards, it will now be necessary to demonstrate competence on a student-by student basis. This shift in focus has led to more widespread use of portfolio assessment in dental education programs.10
Another change in dental education that could potentially serve as a driver for portfolio assessment is the recent change in reporting of the National Board Dental Hygiene Examination (NBDHE) scores. Where previously a numeric score was provided to the student and licensure bodies, as of January 1, 2012, the NBDHE results are reported as pass/fail. For dental hygienists who desire to go into graduate programs, portfolios have been shown to provide an advantage for the applicant. Likewise, the authors believe this scenario will be similar for dental students desiring to continue into advanced education programs.
Outcomes
Since the UMKC School of Dentistry was the first dental hygiene program in the nation to implement portfolios in 1998, longitudinal data was available for review and analysis. Faculty have documented and tracked dental hygiene student performance on portfolios, cumulative GPA, NBDHE and clinical licensure exams to create a database that includes all graduates of the program from 2001 to 2013. The following results represent the population of students for 12 years and thus provide a strong measure of confidence in the data analysis. Ongoing collection and analysis of this data is approved by the UMKC SSIRB #13-414.
Subjects
The population for the analysis consists of 312 full-time dental hygiene students that graduated from the UMKC School of Dentistry Dental Hygiene program between 2001 and 2013. The mean age at graduation for the students is 25.78 years (SD=5.49); however, ages range from 19 to 48. There are 306 females and 6 males in the sample. The ethnic breakdown of the group consists of 86.2% Caucasian, 4.5% Asian/Pacific Islander, 3.5% Hispanic, 1.9% African American and 1.0% Native American (Table I).
Correlational Analyses
According to Messick, a method for determining the external validity of an assessment is to determine if students who score high on the test also score high on other presumed indicators of the construct being measured.11,12 To help support the external validity of portfolio assessment, correlational analyses were conducted between student portfolios and traditional assessment measures of dental hygiene student competency including the NBDHE, GPA and the regional clinical examination scores (Table II). Results demonstrate a positive, moderate and significant relationship between portfolios and GPA (r=0.433; p<0.01) and portfolios and the NBDHE (r=0.339, p<0.01) as defined by Cohen's conventions to interpret effect size.13 A positive but smaller correlation was found between portfolios and regional clinical licensure examination scores (r=0.252; p<0.01). These relationships appear to be good evidence for validity of portfolio assessment.
Fifteen years of experience with a portfolio assessment strategy has illuminated various benefits and certain challenges that accompany the use of portfolio assessment. Many of the benefits and challenges experienced by the UMKC School of Dentistry faculty members and dental hygiene students parallel those that are discussed in the literature.
Benefits of Portfolio Assessment
One of the most acknowledged advantages of using portfolios in the curriculum are student-centered tools requiring the student to self-evaluate and assess their performance. The ability to accurately self-assess is considered a hallmark of competence and is therefore necessary for the development of competent graduates.14,15 Unfortunately, many students enter dental hygiene programs with little experience or awareness of self-assessment. In a study conducted by Mould et al, students openly described their unfamiliarity with self-assessment. When asked about using self-assessment as measurement to follow one's progression of skill development, responses indicated many students did not feel adequately prepared to self-assess upon entering the program. No formal orientation to self-assessment was provided to students in the study; however, when it became evident that a lack of knowledge about the purpose and process of self-assessment was a common theme throughout the analysis, it was determined that there is a need for a specific orientation at the beginning of the dental hygiene.14 Gwozdek et al confirm the benefit of portfolios in developing self-assessment as a professional skill, providing opportunities to self-assess how the course material they complete benefits them personally and professionally, and supporting the development of increased self-perception of competence and confidence over time.3
The benefits of portfolio assessment to the program itself cannot be overstated. Evaluating student reflections in a portfolio provides a method for faculty to assess critical thinking, professionalism and health promotion skills.1 Through the process of reviewing portfolios, faculty members are able to perform a 360 degree assessment of the curriculum in a holistic way, offering opportunities to determine both strengths and weaknesses of the program. Academic programs adopting portfolio assessment found it provides an opportunity to support formative assessment with qualitative data in the form of reflection, identify gaps in student understanding and concepts to clarify in the curriculum, quantify students' perceptions of achieving competency-based learning outcomes, provide summative program evaluation through reflection data, and offer valuable insight into areas for program improvement.3
Demographic Characteristics of Participants (n=312)
Correlations among Traditional and Nontraditional Measures of Dental Hygiene Student Competency as Evidence of External Validity
Challenges Associated with Portfolio Assessment
Time: One of the most commonly identified challenges to implementing and sustaining a portfolio assessment strategy is time. Portfolios require a significant time commitment from both students and faculty. Utilizing pre-existing structures within the curriculum is critical for combating this issue. For example, most programs have some type of mentor-mentee system in place with faculty and students. At the UMKC School of Dentistry, the authors tapped into that system to include portfolios, assigning the faculty mentor as the person responsible for keeping students on track with their portfolios through the curriculum.
At the end of the day, portfolio assessment cannot stand apart from the curriculum or it will be perceived as additional busy work by both students and faculty. Using existing assignments that are already part of the curriculum as evidence of competency for student portfolios or modifying existing assignments to better capture competency for inclusion in portfolios assists in less faculty resistance. If portfolio assignments are well-developed and strategically embedded across the curriculum, then the students can receive formative feedback from faculty through the courses in which assignments are required (embedded), and summative feedback toward the end of the academic program as portfolios are evaluated holistically.10 With over 15 years of experience with portfolio assessment, the authors can report with confidence that it typically takes faculty approximately 30 minutes per portfolio to provide a summative review of portfolios as the students near graduation.
Control: Control in the educational environment has also been identified as a challenge as educators have been reluctant to share control of evaluation and assessment with students. An even greater struggle has been getting students to assume responsibility for their own evaluation and assessment.16 In 2006, the American Dental Education Association's Commission on Change and Innovation in Dental Education issued a paper outlining 8 core principles that should characterize dental education and guide curriculum development.17 One of the core principles, Lifelong and Self-Directed Learning, supports this shift from teacher-focused and teacher-directed learning to student-focused and student-directed learning. The goal of this shift is for the students to become self-directed, self-disciplined, self-aware and self-corrective learners. Portfolios support this strategy by capturing the student's own appraisal, self-assessment and reflection on their performance including areas of improvement, lessons learned and insights about the learning process.10 Clearly, educators are being challenged to facilitate this shift and portfolios are an excellent vehicle for this transition in pedagogy.
Validity and Reliability: Finally, concerns related to validity and reliability of portfolio assessment have been a challenge as educators continue to question whether portfolio assessment is psychometrically sound and defensible. Variability among portfolios has been identified as a potential weakness as it can make consistent evaluation difficult for evaluators, thereby compromising reliability.1 Research has emphasized that in performance assessment, carefully constructed scoring rubrics and intensive training sessions and calibration exercises for raters are essential elements for producing ratings that are reliable.10,16,18
In 2003, Amyot et al investigated the validity and reliability of portfolio assessment in a dental hygiene program. A generalizability study followed by a decision study found that faculty accounted for very little variability or error (1.28%) in portfolio measurement, and that a generalizability coefficient (analogous to reliability coefficient) of 0.69 could be achieved. This coefficient represents the agreement between 2 different evaluators for each portfolio that was reviewed. Based on the findings from the study, the authors concluded that portfolios can serve as a valid and reliable measure for assessing student competency.16 A caveat of these findings was that 0.69 would be unacceptable if the interpretation and use of portfolio scores were high stakes, e.g., determining licensure. In the UMKC School of Dentistry program, portfolios are used in the final semester as a capstone project and account for 60% of the students total grade in that course. Therefore, we are able to validate this interpretation and use with a reliability of 0.69. In the event that portfolios were to be used as high stakes, it would be necessary to calibrate faculty further and increase reliability.
Discussion
Since the UMKC School of Dentistry has been utilizing portfolio assessment in the dental hygiene program for 15 years, several opportunities for “lessons learned” have occurred along the way. The learning issues encountered and strategies utilized to address them are discussed below.
Student-Selected Evidence
Research in performance assessment has shown that when there is standardization in portfolio entries, reliability is improved.10,18,19 In accordance with the literature, faculty have determined which assignments are to be included in the portfolio. However, students at the UMKC School of Dentistry requested the ability to self-select pieces of evidence that they felt demonstrated their competence. Therefore, an additional section was added that allowed the students to include any additional assignments or items that they felt further supported their competence with the understanding that faculty would evaluate the required items and that the self-selected pieces of evidence needed to be kept at a minimum.
Portfolios as Evidence of Curricular Quality
In 2003, following a site visit from CODA, the UMKC School of Dentistry, Division of Dental Hygiene, received a commendation on self-evaluation because of the students' performance on their portfolios. The report states, “The visiting committee noted that the dental hygiene curriculum includes multiple opportunities for students to produce scientific writings and conduct self-assessments. The students complete an evidence-based paper in the majority of dental hygiene courses. The self-assessment is best exemplified by the portfolio exercise. The dental hygiene faculty is commended for implementing a curriculum that facilitates the students' competence in scientific writing and self-assessment.”20
As stated previously, faculty members can utilize student portfolios to better see their curriculum in a holistic way and contribute to the program quality enhancement. Gwozdek et al discuss the benefits of portfolios on program and administrative assessment.3 It can serve as a tool for faculty to prepare for upcoming projects, make modifications to existing assignments, identify gaps in student understanding, and clarify any lingering misinformation.
Student Perceptions of Portfolio Assessment as Evidence of Curricular Quality
Although the benefit of portfolios in providing evidence of curricular quality is evident to faculty members and external reviewers, it is not always perceived as beneficial by the students. Prior to graduation, UMKC School of Dentistry senior dental hygiene students are given a survey to evaluate various components of the dental hygiene curriculum. Responses are provided using a 5-point Likert-scale: 1 - strongly disagree, 2 - disagree, 3 - neutral, 4 - agree, 5 - strongly agree. One statement related to portfolio assessment reads, “Development of programmatic portfolios helped me to reflect upon my educational experience and growth and feel more confident in my ability to communicate my competence as a dental hygienist.” Results from the past 5 years indicate the students often disagreed with the statement. In 2010 and 2012 the mode was 1 (IQR=2 and 3), in 2011 the mode was 2 (IQR=3), in 2009 the mode was 5 (IQR=2) and in 2008 the mode was 3 (IQR=1.25). The faculty has determined that it is important to continually remind the students why they are developing portfolios and what the portfolio represents so that they can better see the value of the exercise. If the students do not grasp the concept of the assignment, they often view it as busy work. It was also determined that in the future, the statement should be re-written to ask students if preparing a portfolio helped them to obtain skills in self-assessing, reflecting on program competencies, and capturing their personal and professional growth throughout the program.
Portfolios as a Visible Response to Domains of Legitimate External Concern
One of the current measures for determining clinical competency and subsequent licensure is clinical licensure exams. A problem related to this method of one-shot testing is the inconsistency between student performance at accredited schools and performance on clinical licensure exams.21-27 Other issues include the ethical treatment of humans when using live patients for the exam, the expense associated with traveling great distances to test and the increased stress and expenses that accompany repeating the exam upon failure.27
To address these problems associated with current clinical licensure examinations, the California Dental Board implemented a new pathway for dental students to obtain licensure upon graduating from a California dental school using portfolio assessment. The law took effect January 1, 2011 and allows students at the 6 California dental schools to complete the licensure process over the course of their final year in dental school instead of waiting until after graduation. The students that choose this option will be required to complete specific clinical experience benchmarks in seven categories and pass a final assessment in each area. Once all benchmarks and assessments have been completed successfully, the students will submit their finished portfolio to the dental board for approval and subsequent licensure.28
Gadbury-Amyot et al suggests portfolio assessment such as this is an appropriate model of assessment to use in determining qualifications for licensure to practice within the competency-based educational framework that is currently taught. The use of portfolios allows students to present multiple representations of their work evaluated by multiple faculty members.27,29 Using both theoretical and empirical evidence, a strong case for validity in portfolio assessment has been determined and could serve as a valuable tool in measuring clinical competency.
Electronic Portfolios (ePortfolios)
With the increased use of technology in dental and dental hygiene curriculums, the transition from traditional paper-based portfolios to ePortfolios is becoming more prevalent. Helen Barrett describes ePortfolios as using electronic technologies as a container to collect and organize evidence in many media types (audio, video, photos, etc.) while connecting it to appropriate outcomes, goals or standards.30 Barrett identifies enhancements such as archiving, linking, storytelling, collaborating and publishing that are possible when technology is added to traditional portfolio processes.
By requiring students to prepare a portfolio electronically, they gain important digital literacy skills. According to Jones-Kavalier et al, digital literacy is a person's ability to perform tasks effectively in a digital environment.31 Literacy includes the ability to read and interpret media, to reproduce data and images through digital manipulation, and to evaluate and apply new knowledge gained from digital environments. Digital literacy is vital for students to develop because of the increased use of technology in all aspects of dentistry.
Additional advantages of ePortfolios include the ease of access for both the students and the faculty members. If an internet connection is available, they can have 24 hour accessibility to the portfolio contents. It also allows multiple people to review the portfolio at any point in time, rather than having to share 1 paper-based traditional portfolio. Having the information available electronically also allows it to be easily organized, searchable and transferrable.32
When choosing to transition to ePortfolios, the program must chose an electronic portfolio system to manage the portfolios. Several companies provide ePortfolio systems with various technologies. Helen C. Barrett developed and maintains a website that provides an overview of ePortfolio technologies available based on the level of interactivity and the level of personal expression and creativity for the portfolio developer.32 In 2009, when the UMKC School of Dentistry switched from traditional paper-based to portfolios, they chose Foliotek as their ePortfolio system for multiple reasons, including Foliotek's willingness to customize their system towards the program's individual needs. The system also facilitated interactivity and contained a data management system that allowed for collection of evaluation data for producing reports and quantitative data.10
HIPAA and FERPA Guidelines
Over time, guidelines associated with the Health Insurance Portability and Accountability Act (HIPAA) have become stricter due to the increased use of electronic health records. On January 17, 2013, the U.S. Department of Health and Human Services modified the Privacy Rule to strengthen the standards for protecting individuals' health information. This rule modifies HIPAA by including information about electronic health care transactions. It defines and limits the circumstances for which an individual's protected health information may be used or disclosed by health groups and health care providers. Specific changes to the rule are discussed in the document titled, “Modifications to the HIPAA Privacy, Security, Enforcement, and Breach Notification Rules Under the Health Information Technology for Economic and Clinical Health Act and the Genetic Information Nondiscrimination Act; Other Modifications to the HIPAA Rules.”33
Since portfolios highlight the student's clinical experiences, there have been concerns raised related to the opportunity for potential HIPAA violations by including protected health information in case study reports, competency forms, grade sheets, etc. in student portfolios. In response to these concerns and the modifications to the privacy rule, the UMKC School of Dentistry obtained legal advice regarding the impact on portfolio assessment. Recommendations include de-identifying patient data (a requirement since the inception of portfolio assessment at the UMKC School of Dentistry) and establishing criteria for evaluating the students' compliance with HIPAA guidelines and incorporating it into the scoring rubric. The dental hygiene faculty members have since modified the rubric to include a section stating that the portfolio has been evaluated for any HIPAA or privacy concerns. Since some patients may have conditions/situations that are so unique, there is a chance the patient could be identified, even if no protected health information was disclosed. For example, if a patient with a rare genetic condition that affects the oral cavity was treated in the clinic, it is possible that others would be able to identify that patient based on those conditions. Therefore, it was suggested that a panel of reviewers be established to review situations in which the uniqueness of the case in the portfolio is in question.
In addition to the HIPAA regulations, the portfolio reviews must also comply with the Family Educational Rights and Privacy Act (FERPA). The guidelines of FERPA state that schools must have written permission from the parent or eligible student in order to release any information from a student's education record.34 Therefore, if students will have access to each other's portfolios as part of a peer-review process, documented consent must be obtained from the student prior to disclosing that information.
Academic Integrity Violations
Another area of concern is violations of academic integrity. When evaluating a portfolio recently, a dental hygiene faculty member recognized the written self-reflections were similar to a portfolio she had previously evaluated. Upon further review, it was determined that the student had “copied and pasted” from a portfolio that was submitted the previous year. Upon meeting with the faculty, the student was informed she would receive a “0” for the portfolio and would be asked to report to the UMKC School of Dentistry Honors Council. To prevent this issue from occurring again in the future, all reflections that are included in the portfolio will have to be submitted using the Turnitin software, a program designed to ensure original work by checking all submitted documents against other papers, databases and publications.35
Conclusion
In response to the assessment and accountability mandates proposed by CODA and incorporated into recent changes to accreditation standards, institutions will need to utilize assessment instruments designed to demonstrate student self-assessment and show competence in program outcomes on a student-by-student basis. Portfolios can serve as that instrument as they allow faculty members to evaluate a student's abilities in critical thinking, problem solving, self-assessment, professionalism and health promotion. Although the challenges associated with portfolio assessment are widely discussed in the existing literature, 15 years of experience in utilizing portfolios at the UMKC School of Dentistry, Division of Dental Hygiene program has provided insight into methods that can be used to combat those issues, while highlighting the benefits that accompany this instrument for assessment. With even more changes in assessment strategies for dental hygiene education programs on the horizon, portfolios are a promising option as a valid and reliable method for measuring student competency.
Acknowledgments
The authors acknowledge all the past graduates of the UMKC School of Dentistry, Division of Dental Hygiene, who have gone on this assessment journey with use to implement portfolio assessment of student competency.
Footnotes
-
Cynthia C. Gadbury-Amyot, MSDH, EdD, is an Associate Dean and Professor, Instructional Technology and Faculty Development, University of Missouri-Kansas City School of Dentistry. Kimberly Krust Bray, RDH, MS, is the Director of Dental Hygiene, University of Missouri-Kansas City School of Dentistry. Kylie J. Austin, RDH, MS, ECP-II, is a Consultant, University of Missouri-Kansas City School of Dentistry.
-
This study supports the NDHRA priority area, Professional Education and Development: Evaluate the extent to which current dental hygiene curricula prepare dental hygienists to meet the increasingly complex oral health needs of the public.
- Copyright © 2014 The American Dental Hygienists’ Association
