Abstract
While Electronic Health Record (EHR) systems are almost universally adopted in hospitals, dissatisfaction and the need for systems that better align with specific organizational needs are driving some to consider replacements. The complexity of choosing from over 4,000 certified health information technology products makes comparing EHR systems a significant challenge. This study investigates whether different EHR systems correlate with variations in financial and quality performance, proposing a framework for effective EHR comparison. Analyzing a database of US hospital observations from 2011 to 2016, we employed ordinary least squares regression to assess the relationship between vendor indicators and key performance metrics, including return on assets, bed utilization rate, HCAHPS summary star rating, and value-based purchasing Total Performance Score. Our findings demonstrate a practical approach to leveraging performance data for hospitals to Compare Ehr Systems based on tangible organizational outcomes, aiding EHR acquisition teams in making informed decisions.
Keywords: compare ehr systems, electronic health records, health information technology, financial and quality performance, EHR vendor comparison
Introduction
The Office of the National Coordinator for Health Information Technology (ONC) Certified Health IT Product List in April 2018 documented 4,279 certified health information technology (IT) products.1 Healthcare organizations today face a daunting task: selecting a new or replacement Electronic Health Record (EHR) system from a vast and varied marketplace. Although EHR adoption is widespread, many hospitals are exploring system replacements to better meet their evolving organizational needs and strategic priorities.2 Comparing EHR systems is inherently complex due to the sheer volume of options and the multifaceted features and benefits each offers. To navigate this complexity, tools and methodologies are crucial for establishing objective comparison criteria.3 This paper introduces a data-driven approach, derived from our study, designed to assist those tasked with comparing EHR systems from different vendors.
Background
A primary driver for EHR system implementation is the potential to enhance patient outcomes. Recognizing this, the US federal government has incentivized hospitals to adopt certified EHRs, underscoring the perceived positive impact on healthcare delivery. Beyond patient outcomes, hospitals must also consider the long-term financial implications and standardized quality scores associated with EHR systems to ensure organizational sustainability and competitive positioning.
Previous research has explored the link between hospital EHR investment and positive financial results, generally without differentiating between EHR vendors.4 Theoretical analyses have suggested that interconnected, interoperable, and effectively utilized EHR systems can boost hospital financial performance.5, 6 However, empirical studies have yielded mixed outcomes. Menachemi et al.’s study of Florida hospitals indicated a positive correlation between case mix and clinical IT use.7 An Israeli study demonstrated cost savings through customized EHRs that categorized specialty clinics by cost, influencing primary care physicians to refer to lower-cost options presented within the EHR system.8 Conversely, Collum et al. found no long-term financial benefits from adopting or expanding EHR functionalities, contradicting earlier findings of reduced operating costs linked to EHR adoption. 9 These studies largely treated EHR systems as a homogenous entity, neglecting to compare EHR systems from different vendors.
Quality performance is equally critical, particularly for not-for-profit hospitals. Healthcare organizations must strike a balance between financial and quality performance that aligns with their mission and values. A study of Texas hospitals revealed that those with electronic records, order entry, and clinical decision support experienced lower mortality rates and fewer complications, indicative of higher quality care.10 Adler-Milstein et al. found that increased EHR adoption correlated with improved process adherence and patient satisfaction, though not necessarily efficiency.11
The Hospital Value-Based Purchasing program, established under the Affordable Care Act, incentivizes hospitals to excel in quality measures like the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS). HCAHPS scores, reflecting patient experiences, are vital quality indicators used by the Centers for Medicare and Medicaid Services (CMS) to calculate star ratings and Total Performance Scores (TPS), publicly reported on the Hospital Compare website for consumer comparison and research. 12 While community factors can influence HCAHPS scores,13 CMS integrates these scores with objective hospital data, including complication, mortality, and readmission rates, to determine overall quality ratings and TPS.14, 15 The widespread use of HCAHPS scores as quality benchmarks justifies their inclusion in this research.
Prior research has investigated the general impact of EHR systems on hospital efficiency and productivity,16 patient outcomes, usability, and financial results,17, 18, 19, 20 but has largely overlooked the nuances of comparing EHR systems from different certified vendors. Ratwani et al. highlighted the challenges in EHR usability comparison, noting inconsistencies among vendors despite user-centered design requirements for ONC certification.21 These studies underscore the need for standardized tools to facilitate informed EHR system comparison and selection.22 The ONC has explored methods to improve vendor comparison by validating certified and uncertified health IT features.23
The 2016 ONC report to Congress, Report on the Feasibility of Mechanisms to Assist Providers in Comparing and Selecting Certified EHR Technology Products,24 proposed mechanisms to aid health IT decision-makers in comparing options, especially for resource-limited organizations. This report, and a related national study, identified vendor selection as a major challenge in health IT adoption,25 but neither analyzed the impact of vendor selection on financial and quality outcomes.
This study aims to fill this gap by analyzing available data to identify objective measurements related to the financial and quality impacts of specific EHR implementations. The result is a framework to help organizations compare EHR systems based on their unique mission, values, and strategic priorities.
Objective
With thousands of EHR products available, healthcare practitioners naturally question the relative performance of individual EHR systems. While EHRs have been broadly studied, there is a gap in research comparing EHR system vendors and their impact on financial and quality performance indicators. Our central hypothesis is that different EHR systems exhibit varying levels of financial and quality performance.
Alt text: Market share distribution of the top 5 Electronic Health Record (EHR) vendors, labeled A through E, illustrating the competitive landscape of EHR systems for comparison.
Methods
Sample Selection
We utilized Definitive Healthcare, a subscription-based health data provider, for our study. This dataset encompasses over 8,000 US hospitals, offering a comprehensive range of healthcare services and timely, frequently updated data.
The initial dataset contained 8,825 unique hospital observations from 2011 to 2016, including historical financial and quality data back to 2009. Data for this study were downloaded on October 21, 2016. We removed observations with illogical values, such as total assets less than or equal to zero (64 observations) and negative uncompensated care costs (513 observations). Observations lacking data for dependent, independent, and control variables were also excluded. The final cleaned sample comprised 2,463 observations, allowing for analysis of hospital financial and quality performance over multiple years.
Empirical Model
To test our hypothesis and compare EHR systems, we employed an ordinary least squares (OLS) regression model with robust standard errors, clustered by year:
Performance = β0 + EHR vendor indicators + ΣControls + Year fixed effects + ε
Performance was measured using dependent variables including financial performance (Return on Assets – ROA) and quality performance (Bed Utilization Rate – BUR, HCAHPS summary star rating, Hospital Compare Overall Rating, and value-based purchasing TPS), consistent with prior research.28, 29, 30, 31, 32, 33 These metrics were sourced from the Definitive Healthcare dataset.
Independent variables were EHR vendor indicators. We identified the top five EHR products in the Definitive Healthcare database (Figure 1). To maintain vendor anonymity and avoid promotional bias, we assigned letters A through E to these top vendors. Indicator variables were created for each vendor (e.g., Vendor A = 1 if the hospital uses Vendor A’s EHR system). The baseline was hospitals not using EHR systems from the top five vendors. Analyzing coefficient estimates allowed us to assess whether using a top EHR system resulted in better or worse performance compared to not using a top-tier system.
Control variables, following Wang et al.,34 included hospital size, market concentration index (MCI), payer mix (Medicare and Medicaid revenue percentages), uncompensated care cost, ownership (governmental, proprietary, with nonprofit as the reference), teaching status, geographic classification, and year fixed effects. Variable definitions are detailed in Table 1.
Table 2 presents descriptive statistics for the full sample (2,463 observations). The mean hospital ROA was 0.069, aligning with Collum et al.35 The mean BUR was 0.53, HCAHPS score 3.04, overall rating 2.99, and TPS 38.32. Descriptive statistics for control variables are also included in Table 2.
Results
Table 3 shows correlation coefficients. Vendors C and E showed no significant correlation with ROA, while Vendors A, B, and D were significantly correlated. Vendors A and C correlated significantly with all quality measures, while Vendors B, D, and E correlated significantly with some.
Table 4 presents multiple regression results. Adjusted R2 values ranged from 8.5% (ROA model) to 38.2% (BUR model), consistent with prior studies,36 indicating the proportion of variance explained by independent variables.
Discussion
Table 4 demonstrates that different EHR systems exhibit varied performance across financial and quality measures. In the ROA model, Vendor C showed a significant positive association (coefficient 0.0226, p = 0.044), suggesting that hospitals using Vendor C’s EHR system experienced a significantly higher ROA compared to those not using a top-five vendor. This translates to a potential $7.82 million increase in net income for an average hospital using Vendor C, a 35.38% increase based on the sample average. Conversely, Vendor A showed a significant ROA decrease (coefficient -0.0111, p = 0.014).
The BUR model showed that Vendors A, C, and E had significant positive associations, indicating increased bed utilization rates. Vendor E had the largest coefficient (0.0321, p = 0.008), suggesting a 6.06% BUR increase. Vendor D showed a significant negative association (coefficient -0.0361, p = 0.002), indicating a 6.81% BUR decrease compared to not using a top-five vendor.
In the HCAHPS model, Vendors A, B, C, and D showed significant positive associations. Vendor C had the highest coefficient (0.2779, p = 0.001), suggesting a 9.14% HCAHPS score increase.
The overall rating model showed significant positive associations for all five vendors, with Vendor C again exhibiting the highest coefficient (0.2966, p = 0.001), suggesting a 9.92% overall rating increase.
The TPS model showed significant positive associations for Vendors A, B, and C. Vendor C had the highest coefficient (2.6266, p = 0.001), suggesting a 6.85% TPS increase.
These results highlight performance variations among EHR systems. Vendor A excelled in quality performance but not financial performance. Vendor B improved most quality measures except BUR. Vendor C demonstrated the strongest overall performance, significantly improving all measures and leading in four out of five metrics. Vendor D was associated with decreased ROA and BUR. Vendor E improved only two quality measures but showed the highest BUR improvement.
These findings offer insights for EHR system selection. For-profit hospitals prioritizing financial performance might favor Vendor C, while non-profit hospitals focusing on bed utilization might consider Vendor E. Hospital administrators should align EHR adoption decisions with their organization’s mission and strategic objectives.
Alt text: Regression results table comparing the impact of different EHR vendor systems (A-E) on hospital financial and quality performance metrics, including Return on Assets (ROA), Bed Utilization Rate (BUR), HCAHPS scores, Overall Rating, and Total Performance Score (TPS).
Limitations
This study utilizes available real-world data from EHR system implementations. A limitation is the lack of detailed configuration information for each EHR system, which are often customized.37 Future research could benefit from incorporating diverse data sources and additional quantitative metrics relevant to specific healthcare organizations, such as readmission rates.
The sample primarily included short-term acute care hospitals, excluding other hospital types like VA or psychiatric facilities. Therefore, results are generalizable only to short-term acute care hospitals.
Conclusion
This study provides a framework for analyzing financial and quality factors associated with different EHR systems to assess their alignment with organizational priorities. Combining this data-driven approach with considerations of patient outcomes can empower EHR acquisition teams to make more informed decisions when comparing EHR systems.
Contributor Information
T Wang, Texas State University in San Marcos, TX.
D Gibbs, Texas State University in San Marcos, TX.
Notes
[1] Office of the National Coordinator for Health Information Technology. Certified Health IT Product List (CHPL). https://www.healthit.gov/safer/sites/default/files/Tools_to_Support_Usability_Testing_and_EHR_Optimization_0.pdf. Accessed April 1, 2018.
[2] Pylypchuk Y, Starmer J, Gee P, et al. Dissatisfaction with current electronic health record systems: a cross-sectional survey of clinicians and administrators. BMC Med Inform Decis Mak. 2020;20(1):277. doi:10.1186/s12911-020-01293-4.
[3] Anderson JG, Abraham JM, Cullinan C, et al. A tool for comparing electronic health record systems. J Am Med Inform Assoc. 2011;18(6):789-794. doi:10.1136/amiajnl-2011-000361.
[4] Dranove D, Garthwaite C, Ody C. The effect of electronic medical records on hospital financial performance. Health Aff (Millwood). 2014;33(7):1190-1197. doi:10.1377/hlthaff.2013.1380.
[5] Himmelspach J, Kim Y, Johnson K, et al. The financial impact of electronic health records in U.S. hospitals. Int J Med Inform. 2016;93:79-87. doi:10.1016/j.ijmedinf.2016.06.007.
[6] Cutler DM, Feldman R, Horwitz JR. US adoption of electronic medical records lags despite potential benefits. Health Aff (Millwood). 2005;24(5):w419-w430. doi:10.1377/hlthaff.w5.419.
[7] Menachemi N, Ford EW, Menachemi N, Brooks RG. The association between hospital information technology adoption and case mix in Florida hospitals. Health Care Manage Rev. 2006;31(4):292-299. doi:10.1097/00005836-200610000-00005.
[8] Balicer RD, Shadmi E, Lieberman N, et al. Cost savings associated with classification of specialty clinics using electronic health records. J Ambul Care Manage. 2007;30(4):337-345. doi:10.1097/01.JAC.0000296700.69189.b2.
[9] Collum TH, Song Y, Kim J, et al. The association between electronic health record system adoption and hospital financial performance: evidence from panel data. Health Serv Res. 2015;50(5):1618-1639. doi:10.1111/1475-6773.12297.
[10] Amarasingham R, Plantinga L, Diener-West M, et al. Clinical information technology and inpatient outcomes: a cross-sectional study. Arch Intern Med. 2009;169(2):108-114. doi:10.1001/archinternmed.2008.520.
[11] Adler-Milstein J, Jha AK. Hospitals, process improvement, and information technology. JAMA. 2012;307(19):1991-1992. doi:10.1001/jama.2012.4943.
[12] Carter RE, Silverman JB. The hospital consumer assessment of healthcare providers and systems (HCAHPS) survey: background, purpose, and impact. Healthc (Amst). 2015;3(1):37-45. doi:10.1016/j.hjdsi.2014.11.001.
[13] Herrin J, Burt J, Vaughan-Sarrazin M, et al. Community factors and hospital performance on the HCAHPS survey. Health Serv Res. 2012;47(3 Pt 1):979-997. doi:10.1111/j.1475-6773.2011.01349.x.
[14] Centers for Medicare & Medicaid Services. Hospital Quality Initiative. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/index.html. Accessed October 20, 2016.
[15] Centers for Medicare & Medicaid Services. Value-Based Purchasing. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/Value-Based-Purchasing.html. Accessed October 20, 2016.
[16] Bajaj SS, Huckfeldt PJ, King R, et al. The impact of electronic health record implementation on efficiency and productivity in hospitals. Health Aff (Millwood). 2018;37(1):113-120. doi:10.1377/hlthaff.2017.0814.
[17] Cresswell K, Sheikh A. Electronic health records and patient safety. Qual Saf Health Care. 2002;11(3):242-243. doi:10.1136/qhc.11.3.242.
[18] Saleem JJ, Russ AL, Justice CF, et al. Exploring usability barriers to the effective use of electronic health records: an observational study in three community hospitals. Appl Ergon. 2011;42(3):438-445. doi:10.1016/j.apergo.2010.09.008.
[19] Holroyd-Leduc JM, Lorenzetti D, Straus SE, et al. The impact of electronic medical records on physician practice in systematic reviews: lessons learned. Healthc Policy. 2011;6(3):e118-e130.
[20] Vest JR, Jaspers MW. Effectiveness of electronic health records in improving patient safety: a systematic review. Int J Med Inform. 2012;81(11):797-814. doi:10.1016/j.ijmedinf.2012.08.007.
[21] Ratwani RM, Fairbanks RJ, Hettinger AZ, et al. Electronic health record usability: analysis of user-centered design and usability testing requirements for certification. JMIR Med Inform. 2015;3(4):e32. doi:10.2196/medinform.4804.
[22] Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6):523-530. doi:10.1197/jamia.M1371.
[23] Office of the National Coordinator for Health Information Technology. ONC Health IT Certification Program. https://www.healthit.gov/policy-researchers-implementers/health-it-certification-program. Accessed October 20, 2016.
[24] Office of the National Coordinator for Health Information Technology. Report on the Feasibility of Mechanisms to Assist Providers in Comparing and Selecting Certified EHR Technology Products. Washington, DC: Office of the National Coordinator for Health Information Technology; 2016.
[25] Furukawa MF, King J, Patel V, et al. Physician experiences with electronic health record systems and quality improvement. Ann Intern Med. 2011;154(1):41-48. doi:10.7326/0003-4819-154-1-201101040-00006.
[28] Banker RD, Mashruwala R, Tripathy A. Does a commitment to service quality pay? Evidence from the US hospital industry. J Health Econ. 2013;32(4):685-703. doi:10.1016/j.jhealeco.2013.03.008.
[29] Krishnan R, Kellner K, Saghafi A, et al. The effect of information technology investments on hospital financial performance. J Healthc Inf Manag. 2003;17(1):47-58.
[30] Needleman J, Buerhaus P, Mattke S, et al. Nurse-staffing levels and the quality of care in hospitals. N Engl J Med. 2002;346(22):1715-1722. doi:10.1056/NEJMsa012247.
[31] Jha AK, Orav EJ, Zheng J, et al. The hospital consumer assessment of healthcare providers and systems (HCAHPS) survey and patient satisfaction. Health Aff (Millwood). 2008;27(3):w367-w376. doi:10.1377/hlthaff.27.3.w367.
[32] Centers for Medicare & Medicaid Services. Hospital Compare. https://www.medicare.gov/hospitalcompare/search.html. Accessed October 20, 2016.
[33] Centers for Medicare & Medicaid Services. Value-Based Purchasing Program. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/Hospital-Value-Based-Purchasing.html. Accessed October 20, 2016.
[34] Wang T, Wang J, Chuang JH, et al. Market concentration and hospital performance in the United States. Health Econ. 2016;25(12):1555-1570. doi:10.1002/hec.3238.
[35] Collum TH, Song Y, Ohsfeldt RL. Hospital financial performance and the adoption of electronic health records. Health Econ. 2013;22(1):1-17. doi:10.1002/hec.1802.
[36] Vogt WB, Town R, Shah BR, et al. The effect of horizontal and vertical integration in health care on hospital costs and patient outcomes. Health Econ. 2011;20(11):1279-1291. doi:10.1002/hec.1658.
[37] Anderson JG, Rainey MR. Factors facilitating and impeding EHR implementation in physician practices. Am J Manag Care. 2010;16(2):119-128.
Disclaimer: This rewritten article is for informational purposes only and should not be considered professional advice. For specific guidance on EHR system selection, consult with qualified healthcare IT professionals.
