Valid and Reliable Tools The goal of an evaluation is to determine the success of an intervention a 2023
Valid and Reliable Tools
The goal of an evaluation is to determine the success of an intervention, a new process, the launch of a new technology, patient satisfaction, or any number of things. Surveys are a popular tool for gathering this type of information. For the results of the evaluation to be meaningful, however, the survey used must be both reliable and valid. What does that entail? A reliable instrument is one that would yield similar results when given to different groups under identical circumstances. For example, if a survey was given to nurses on the use of a certain piece of technology, all respondents would understand the phrasing of the questions the same way. Validity refers to how well the instrument actually measures what it is intended to measure. Determining the reliability and validity of a survey instrument can be complicated and involves the use of statistics. For this reason, many researchers opt to use instruments that are already developed and tested.
For this Discussion, you consider survey instruments that would be appropriate to use in specific situations.
The following scenarios will be used for this week’s Discussion:
- Scenario 1: A large hospital intends to implement a computerized provider order entry (CPOE) system. In order to get a good idea of its effects, the hospital’s leadership has asked for an evaluation of the CPOE system’s impact 90 days after its initial implementation.
- Scenario 2: Years ago, the primary hospital for a large, rural county distributed personal data assistants (PDAs) to all of its physicians in an attempt to modernize. After looking at many other more up-to-date mobile systems, physicians and hospital leaders are curious about how their current PDA-based system performs.
- Scenario 3: The informatics department of one of North America’s largest hospitals is conducting an internal review of its health information technology systems. This review will evaluate the need for any changes to its systems and may serve as justification for different budgetary allocations. Because of its sheer size and the number of personnel it affects, the hospital’s electronic health record system will be a pivotal point of the review.
To prepare:
- Review this week’s Learning Resources on reliability and validity.
- Review the AHRQ Evaluation Survey Compendium.
- Review the scenarios presented above.
- Using the “Locate a Survey for your Project” tool available on the AHRQ website, identify a survey tool that would be appropriate for use for each scenario.
- Reflect on the specific characteristics of a valid, reliable survey tool.
By tomorrow Tuesday 1/17/17, post a minimum of 550 words in APA format with 3 references that include the unique survey tool you identified for each scenario and a justification for your selections
Required Readings
Friedman, C. P., & Wyatt, J. C. (2010). Evaluation methods in biomedical informatics (2nd ed.). New York, NY: Springer Science+Business Media, Inc.
Chapter 5, “Measurement Fundamentals” (pp. 113–144)
This chapter details the importance of reliability and validity in measurement in informatics. It defines the different types of validity and introduces statistical equations for accurately determining the reliability and validity of a research finding.
Anderson, E. F. (2011). A case for measuring governance. Nursing Administration Quarterly, 35(3), 197–203.
Retrieved from the Walden Library databases.
The author of this article describes the Index of Professional Nursing Governance, which is a tool used to measure the degree and implementation of shared governance. The author presents a case in which the index was used in a hospital to assess the degree of shared governance over time.
Kaphingst, K. A., Kreuter, M. W., Casey, C., Leme, L., Thompson, T., Cheng, M. R., et al. (2012). Health literacy INDEX: Development, reliability, and validity of a new tool for evaluating the health literacy demands of health information materials. Journal of Health Communication, 17(Supp 3), 203–221.
Retrieved from the Walden Library databases.
In this article, the authors describe the development, refinement, and testing of Health Literacy INDEX, a tool designed to determine criteria for judging the quality of health information materials.
Penfold, R. B., Kullgren, J. T., Miroshnik, I., Galbraith, A. A., Hinrichsen, V. L., & Lieu, T. A. (2011). Reliability of a patient survey assessing cost-related changes in health care use among high deductible health plan enrollees. BMC Health Services Research, 11(1), 133–143.
Retrieved from the Walden Library databases.
This article describes a study that sought to determine the reliability of survey questions used to measure the behaviors and knowledge of high-deductible health plan beneficiaries. The authors of the article highlight their findings after conducting control and trial surveys among the beneficiaries.
Seto, I., Foisy, M., Arkison, B., Klassen, T., & Williams, K. (2012). The evaluation of an evidence-based clinical answer format for pediatricians. BMC Pediatrics, 12, 34–41.
Retrieved from the Walden Library databases.
The authors of this article describe a tool (Clinical Answers) that supplies the user with summaries of key medical evidence-based practices. They highlight a survey given to pediatricians to examine their use of Clinical Answers and to garner ways the product could be improved.
Agency for Healthcare Research and Quality. (n.d.). Health IT survey compendium. Retrieved from http://healthit.ahrq.gov/portal/server.pt/community/health_it_tools_and_resources/919/health_it_survey_compendium/27874
Required Media
Laureate Education (Producer). (n.d.f). Reliability and validity. Retrieved from CDN database. (NURS 6431)
This video explores the concepts of reliability and validity in measurement tools. The video stresses the importance of reliability and validity when selecting an appropriate tool to evaluate a PICO question.
