Abstract
The concepts of reliability and validity are explored in this short report. The importance of assuring that data collection tools are both reliable and valid are explored for use in quantitative, qualitative, and mixed method study designs. Quantitative and qualitative attributes for achieving reliability and validity are provided. Discussion of when information collected and presented is not reliable and valid impacts the body of scientific knowledge and researcher credibility.
INTRODUCTION
Researchers may begin forming research questions considering how they can most effectively capture data for answering them. When defining the research question, it is important to identify if the tool selected will provide both reliable and valid results. Study design, population and sampling methods, study aims, and research questions should all guide the selection of a data collection tool.1-8 Doing so has an impact on the quality of the data collected, and the quality of research being conducted.9 Information needs to be reliable before it can be valid.1-2,5 Additional considerations related to reliability and validity are also impacted by whether the researcher is conducting qualitative, quantitative, or mixed method research.1-8,10 If the researcher uses a flawed study design which does not use a research method or reliable and valid data collection tool, results will be flawed.1-2,8-9 This short report addresses the importance of applying research validity and reliability and why they matter.
Reliability
Simply put, reliability refers to finding the same result over time. Thus, we have consistent results time and time again.1-6 Let us consider some simple examples associated with weather. Tools demonstrating reliability may include a thermometer, rain gauge, or barometer. Thermometers provide a measure of cold or heat on a defined scale like Celsius or Fahrenheit. A rain gauge measures liquid precipitation, rain. Similarly, gauges are scaled with metric or English measures. Lastly, barometers determine atmospheric air pressure, which relates to weather predictability. Pressure changes noted with a barometer can help in forecasting the weather by predicting storms or clearing weather. While barometric pressure is more complex than this, it provides a measure of reliability in predicting atmospheric change. A leaking rain gauge, or faulty thermometers or barometers will not provide reliable results. This principle also applies to data collection; if a tool used for collecting data is not reliable it will not provide the information researchers seek to find. Study design and data collection should yield accurate, consistent results when repeated over time.1-2, 5 Reliability attributes differ depending on whether the researcher is conducting a quantitative or qualitative study.3-6 If the purposes of research include pursuing facts and evidence, reliability is a key factor in doing so.
Quantitative Reliability
Three key attributes need to be accomplished while applying the concept of reliability in quantitative research.1-2 Stability should yield consistent results when a data collection tool is used repeatedly. If a statistical comparison is made, there should be consistent results.1-2, 9 This is an example of test/retest.1-2 However if a similar, yet different data collection tool is used and the concepts tested are the same even if the wording slightly differs, this is an example of alternative or parallel form reliability.1-2 Stability is demonstrated if there are high correlations between each time individuals complete the test measures. High, moderate & weak correlation coefficients demonstrate the reliability attribute of stability.1-2 Equivalence measures inter-rater reliability.1-2 When various users apply a collection instrument, the importance of the attribute of equivalence applies.1-2 An example of this in education includes various instructors using a student process evaluation instrument. If all clinical instructors rate and use the process tool similarly, and have similar outcomes, it documents achievement of inter-rater reliability.1-2 If the scores on an assessment differ, it demonstrates the lack of this equivalence attribute.1-2 Lastly, the attributes for internal consistency as a measure of reliability can be applied by using inferential statistics documenting measuring a construct.1-2,7-8 Strong correlations found while using measures including Cronbach s alpha, Kuder-Richardson, split-half reliability, or individual item to total correlation can document the attribute of internal consistency.1-2,7-8 Of these measures, Cronbach’s alpha1-5 is used most often, where the calculated Cronbach result is between 0-1.1-5 Correlational averages resulting in an acceptable reliability score are between 0.7-1.0. Computed averages lower than 0.7 are either moderately or weakly correlated, thus, the questions being assessed are typically not demonstrating the attribute of truly being reliable.1-5
Conceptually, if the data collected is not replicable, it may not be reliable, nor can other researchers trust that the results are accurate. Thus, study replicability cannot be accomplished.1-5,7-8 Common issues associated with quantitative data collection reliability are associated with small sample sizes, inappropriate statistical analyses, and the referenced associated attributes of quantitative reliability.1-3,5,7-8
Qualitative Reliability
Broadly speaking, reliability also applies within qualitative research.1-2,4,6 Consistency of application of analytical procedures provides a foundation for qualitative research methodologies.1-2,4,6 If philosophical premises, positions, or purpose differ, however, an alternative framework for establishing attributes of rigor is appropriate.1-2,4,6 Mentoring junior researchers in applying qualitative research methodologies can result in documenting and demonstrating the rigor of the reliability attributes for using a qualitative or mixed method study design.1-2,4,6,10 The reason for this approach is because there are no accepted standards associated with qualitative research design.1-2,4,6
Qualitative study reliability is associated with applying methodologies assuring the trustworthiness of research findings.1-2,4,6 There are multiple approaches for qualitative reliability including documenting verbatim transcripts of individual accounts, scrupulous record keeping and decision tree matrices, recognition of personal biases, participant validation of the transcripted communication, data clarification associated with applied terminology, and data triangulation.1-2,4,6-8 Assuring credibility associated with qualitative research often is a construct of three key attributes: 1) applicability for other researchers use, 2) truth value, and 3) neutrality/consistency.1-2,4,6 These reliability attributes are often associated with enhancing qualitative research findings.1-2,4,6
Qualitative research should be designed using clear data collection strategies and techniques for enhancing credibility because there are no comprehensive or collective criteria and terminology for evaluating its results.1-2,4,6-9 Thus, application of the attributes shared here can increase result credibility.1-2,4,6-9 Application of a mixed method study design may require the use of an appraisal for determining if the study design is appropriate prior to identifying reliable and valid means to collect data collection.11 This appraisal tool was outlined in Gurenlian’ s short report, “Mixed Methods, or Mixed Up?”10 Mixed method study finding’s reliability and validity can impact the power of its results.10 Thus, the investment of time, expertise, and funding are not the only factors needing consideration during its application.10 If a thorough, clear, and exacting investigative process is not defined, findings may be scrutinized.9-10
Validity
Research validity relates to identifying if a measure does, or does not, produce accurate results.1-2,5 In other words, does the result represent what was measured? As in the weather instrument examples, the concept of validity applies. Data Management Association (DAMA) defines validity as “the degree data values are consistent as defined.”12 Thus, if the weather instruments provided faulty results due to manufacturing defects, results presented are not accurate. If a rain gauge leaks, it does not provide a valid measure of how much precipitation fell. If a barometer or thermometer do not have precise levels of either mercury or red liquid for measuring liquid expansion, temperature readings and barometric pressure results will be inaccurate. Barometers require atmospheric calibration to assure precision of readings, and mercury or aneroid cells for measurement are calibrated for yielding precise results. Thus, the importance of validity results in the limitation of errors,1-6,8 which is an important consideration as an application of consistency or replicability over time and as applied within research.
Quantitative Validity
The capacity to which validity can be applied is considered several ways.1,3,5,9 Construct validity describes congruence between study data collection tools and theoretical constructs, frameworks, models or questions, assuring they are aligned.1,3,5,9 If data collection and the theoretical underpinnings are not aligned, results cannot be considered as valid nor the findings accurate. Internal and external validity, subsequently establish both causal relationships between variables, or if the findings are generalizable to other circumstances or groups.1,3,5,9 These types of validity may be overlooked when researchers are not considering interrelationships between independent and dependent variables; nor if there is a recognition that the population and sample may perhaps be too small to be extrapolated to a larger, more representative, population.7-9 Lastly, another key consideration is if the statistical analyses or tests selected for measurement are the correct tools to use. This concept brings in the need for statistical validity, which assures the statistical test used in the research analysis provides data which is accurate, and the findings, reliable.1-10
Qualitative Validity
Generally speaking, qualitative validity differs from quantitative validity as there are differing frameworks and purposes applied while conducting qualitative research. However, the concepts of qualitative validity still address precision or consistency of data findings.2 Thus, qualitative validity is predicated on recognizing both researcher and subjects have individual experiences or potential biases influencing their perspectives associated with the subject being researched.2,4,6,9 Qualitative research recognizes the concepts of truth value as a measure of validity, knowing there are multiple potential realities; consistency and neutrality as measures of reliability, and applicability as a measure of if findings can be generalized to other groups.2,4,6,9 There are multiple means qualitative researchers can use to ensure study results are credible. Researchers should have the training and skills for applying any sort of research, including identifying study design, identifying data collection tools, determining reliability/validity of data collection for assuring results are accurate and a true representation of results before reporting it.1-11
DISCUSSION
There is a need for all researchers, whether they apply quantitative,1,3,5,7-9 qualitative, 2,4,6-9 or a mixed methods10 study design, for assuring results are credible for broader dissemination.9 While it is appropriate to recognize there is no unanimously accepted vocabulary for application of these universal research principles, the theoretical frameworks of truth, honesty, clarity, consistency, and authenticity1-6 are foundational for researchers to apply.
Why do reliability and validity matter? For other researchers to value and recognize the importance of study results, findings must be true, credible, and replicable. Researchers cannot replicate published studies if the methodology section of a manuscript does not provide clear study design(s), aims, research questions and appropriate data collection tools used.1-8,10 If the data collection tools selected are not gathering reliable and valid information, results may be spurious, non-replicable, and depending on statistical analytics applied, not generalizable to a population outside the initial study.7-8 Additionally, erratic study results may not align with prior research outcomes.9
Researchers have an ethical obligation to apply the research principles of reliability and validity for producing trustworthy, credible results. If research findings are not based on these two concepts and results are not dependable, researcher credibility may be questioned. Pursuing lines of scientific inquiry as provided in the framework of the National Dental Hygiene Research Agenda (NDHRA) can advance scientific investigation and evidence-based practices for the profession.15 The code of ethics of the American Dental Hygienists’ Association clearly lies out the responsibilities for scientific investigation aligning not only with the NDHRA but are also applicable to the role of researcher - societal trust, non-maleficence, beneficence, and veracity.16 Promoting public trust, doing no harm while creating or identifying information supporting public or person-centered wellbeing, and being truthful in reporting findings as evidence-based researchers are rooted in these very principles. Study design, population and sampling methods, study aims, and research questions should all guide the selection of reliable and valid data collection tools.1-8,10 All of these are strong reasons as to why the concepts of reliability and validity in research matter.
CONCLUSION
Reliability and validity are crucial concepts associated with research. If issues exist with clear study designs or methodology, data gathering tools, and poor identification or definition of study populations, samples and sample stratification, the data results are likely to lack either reliability or validity or both one or both. Reliability and validity allow for replication of consistent results over time, while also assuring that those results are trustworthy.
DISCLOSURES
The author has no conflicts of interest to disclose.
Footnotes
NDHRA priority area, professional development: education (evaluation).
- Received July 15, 2024.
- Accepted October 25, 2024.
- Copyright © 2024 The American Dental Hygienists’ Association
This article is open access and may not be copied, distributed or modified without written permission from the American Dental Hygienists’ Association.
