Abstract
Background and objectives The US Centers for Medicare and Medicaid Services (CMS) End Stage Renal Disease Prospective Payment System and Quality Incentive Program requires that dialysis centers meet predefined criteria for quality of patient care to ensure future funding. The CMS selected the Consumer Assessment of Healthcare Providers and Systems In-Center Hemodialysis (CAHPS-ICH) survey for the assessment of patient experience of care. This analysis evaluated the psychometric properties of the CAHPS-ICH survey in a sample of hemodialysis patients.
Design, setting, participants, & measurements Data were drawn from the Adelphi CKD Disease Specific Program (a retrospective, cross-sectional survey of nephrologists and patients). Selected United States–based nephrologists treating patients receiving hemodialysis completed patient record forms and provided information on their dialysis center. Patients (n=404) completed the CAHPS-ICH survey (comprising 58 questions) providing six scores for the assessment of patient experience of care. CAHPS-ICH item-scale convergence, discrimination, and reliability were evaluated for multi-item scales. Floor and ceiling effects were estimated for all six scores. Patient (demographics, dialysis history, vascular access method) and facility characteristics (size, ratio of patients-to-physicians, nurses, and technicians) associated with the CAHPS-ICH scores were also evaluated.
Results Item-scale correlations and internal consistency reliability estimates provided support for the nephrologists’ communication (range, 0.16–0.71; α=0.81) and quality of care (range, 0.16–0.76; α=0.90) composites. However, the patient information composite had low internal consistency reliability (α=0.55). Provider-to-patient ratios (range, 2.37 for facilities with >36 patients per physician to 2.8 for those with <8 patients per physician) and time spent in the waiting room (3.44 for >15 minutes of waiting time to 3.75 for 5 to <10 minutes) were characteristics most consistently related to patients’ perceptions of dialysis care.
Conclusions CAHPS-ICH is a potentially valuable and informative tool for the evaluation of patients’ experiences with dialysis care. Additional studies are needed to estimate clinically meaningful differences between care providers.
Introduction
As standards in healthcare administration and delivery shift to value-based purchasing of health care services, evidence-based measures of performance are required to drive high standards and improve the quality of care. The outcomes of such measures are increasingly used to determine future reimbursement payments. They also provide information to help care providers and clinicians identify specific areas in which care experiences could be improved.
The US Centers for Medicare and Medicaid Services (CMS) recently implemented an End Stage Renal Disease Prospective Payment System (ESRD PPS) for Medicare reimbursement of outpatient dialysis-related services along with a Quality Incentive Program (QIP) to monitor and ensure dialysis centers maintain quality patient care (1). There are currently two types of QIP metrics: clinical measures and reporting measures. The clinical measures have specific outcome standards that dialysis centers must meet, whereas the reporting measures require only collection of specific data. Failure to achieve these standards results in reductions to dialysis payments for future payment years.
The CMS selected the Consumer Assessment of Healthcare Providers and Systems (CAHPS) In-Center Hemodialysis (CAHPS-ICH) survey for assessing patient experience of care (2). The CAHPS-ICH development team created the survey by conducting a literature review, examining 13 existing surveys, convening a technical expert panel, soliciting public comment, conducting patient focus groups, and performing 71 cognitive interviews with draft items. The CAHPS-ICH survey was field tested with 1454 respondents (46% response rate) using a mixed-mode data collection effort (mail with phone follow-up); however, the results have not yet been published. Patient experience of care was included as a QIP reporting metric for 2014 (1). Although the CMS makes this recommendation, it also notes that the cost burden may be quite high based on the volume of patients treated. The CMS estimates a $10 million annual burden to the system (3).
This study evaluates the psychometric properties of the CAHPS-ICH survey in a sample of hemodialysis patients.
Materials and Methods
Data Source
Data were drawn from the Adelphi CKD Disease Specific Program (DSP), a cross-sectional survey of real-world clinical practice conducted in the United States in the first quarter of 2012 (4). The objective of the DSP is to provide a holistic understanding of the management of patients with CKD, including their treatment environment.
The DSP is a patient record-based survey of nephrologists in the United States and their patients with CKD. Nephrologists, identified from public lists of healthcare professionals, who manage at least 20 patients with CKD per month on dialysis were eligible to participate in the research and were drawn from different types of centers across the United States. Nephrologists were asked to provide information on their dialysis center and complete a patient record form for their next 10 CKD dialysis patients seen. Information captured included patient demographics, history and monitoring, symptoms, concomitant conditions, treatments, management, and hospitalization details.
The same patients were invited to fill out a patient self-completed questionnaire that included consultation history, symptoms, and the CAHPS-ICH (Supplemental Material).
The research was performed in full accordance with the US Health Insurance Portability and Accountability Act of 1996 (5). Each patient provided informed consent for the deidentified and aggregated reporting of research findings. All data were fully deidentified and aggregated before analysis. Full details of the methodology were previously published (4).
CAHPS-ICH Scoring System
The CAHPS-ICH survey has 58 items (questions), including 3 that ask for a global rating of the patients’ kidney doctor, the dialysis center staff, and the dialysis center itself. Patients are asked to rate each using a 0–10 scale that is subsequently collapsed to a 1–3 scale, with a score of 1 comprising responses of 0–6 on the original scale, a score of 2 comprising responses 7 and 8, and a score of 3 comprising responses 9 and 10. Three multi-item scales assess the following: nephrologists’ communication and caring (NCC scale; 6 questions with a final score between 1 and 4), quality of dialysis center care and operations (QoC scale; 17 questions with a final score between 1 and 4), and providing information to patients (PI scale; 9 questions with a final score between 0 and 1) (6). All scale scores were calculated according to the CAHPS-ICH scoring methodology (6).
Statistical Analyses
The three multi-item scales were evaluated using several different psychometric techniques (7). In step 1, item convergence (or the correlation of each item with the scale score), correcting for item overlap, was estimated to ensure that each item was assigned to the appropriate scale. A cut-off of ≥0.4 was used to indicate a large enough correlation (8).
In step 2, item discrimination was evaluated by comparing correlations of an item (correcting for item overlap) with its hypothesized scale, against its correlations with the other two scales, to ensure each item in a scale does not correlate as highly or more highly with other scales.
In step 3, the internal consistency reliability of each scale was also evaluated to provide a single summary measure of the extent to which the items had acceptable reliability. A threshold of ≥0.7 was used to indicate an acceptable level of reliability (7).
In step 4, floor and ceiling effects were estimated to determine whether the scales and global rating items distinguished adequately between respondents. All other factors being equal, measures are better when there are few patients scoring at the top (ceiling effect) and bottom (floor effect) of the scales.
Finally, in step 5, three separate confirmatory factor analysis (CFA) models were estimated to evaluate the overall fit of the hypothesized three scales to the data. Separate models were estimated to help identify any potential issues or problems with the multi-item scales. For each CFA model, two goodness-of-fit statistics were reported: the comparative fit index (CFI; 0–1) and the root mean square error of approximation (RMSEA; 0–1), in which a higher CFI and a lower RMSEA indicate better fit of the hypothesized model to the observed data. CFI values ≥0.95 and RMSEA values ≤0.06 are desirable (9).
In addition, to assess the extent to which patients within the same clinic share similar patient experiences and how well the CAHPS scales and global rating items distinguish between clinics, intraclass correlation coefficients (ICCs) and reliabilities at the clinic level were estimated using one-way ANOVA models.
Finally, F tests were performed to assess whether significant differences exist between the mean global ratings and composite scales across subgroups of patients defined by, for example, demographic characteristics and disease status. A Bonferroni adjusted P value of <0.0005 was used as the threshold for statistical significance to account for the multiple tests being performed.
Results
Demographics
Physicians and patients at 76 dialysis centers across all regions of the United States were included in this study. A total of 819 patients receiving dialysis were recruited. Of these, 404 patients were eligible for analysis based on having received hemodialysis for >3 months and having completed the CAHPS-ICH survey. Table 1 provides an overview of the characteristics of the recruited sample (n=819), and those included (n=404) and not included (n=415) in the analysis. Chi-squared tests and Fisher’s exact tests were performed and there were no statistically significant differences between the two groups. The mean (± SD) age of eligible patients was 57±14.7 years and the median time on dialysis was 48 months (interquartile range, 24–28 months).
Characteristics of the sample comparing patients included and excluded from the analysis
Psychometric Properties of the CAHPS-ICH
NCC Scale.
Five of the six NCC items correlated strongly with the scale score, and all items had a stronger correlation with the NCC scale than with the other two scales (Table 2). Internal consistency reliability (coefficient α) was 0.81. The CFA model demonstrated a good fit (CFI, 0.96; RMSEA, 0.10), although the factor loading (the correlation of the item to the hypothesized composite) for item Q9 (physician fully informed) was only 0.17 (Table 3). Factor loadings for the other five items in the composite were higher, ranging from 0.68 to 0.83.
Item-scale correlations and internal consistency reliability of the CAHPS-ICH composites
Confirmatory factor analysis of the hypothesized three-composite structure
QoC Scale.
Fourteen of the 17 QoC items correlated strongly with the scale score, with most items correlating more highly with QoC than the other two scales (Table 2). The coefficient α for the 17-item scale was 0.90. The CFA model did not quite demonstrate the desired fit (CFI, 0.92; RMSEA, 0.08). Factor loadings for items Q16 (dialysis staff privacy of information) and Q26 (dialysis staff dietary information), were only 0.19 and 0.17, respectively (Table 3), with the loadings for the other 15 items ranging from 0.42 to 0.82, thus indicating a stronger correlation with the hypothesized scale.
Floor and Ceiling Effects
Floor effects (or lowest possible scores) were minimal, ranging from 0% to 8%, whereas ceiling effects (or highest possible scores) varied from 15% (QoC composite) to 75% (kidney doctor global rating). This suggests that the ICH measures may be limited in their ability to distinguish between patients experiencing the most positive levels of care.
Reliability at the Hemodialysis Center
The dialysis center item was the most reliable of the three global rating items and three multi-item scales (ICC, 0.31), indicating that within dialysis center scores are more consistent for this item than the other CAHPS-ICH scores. All other global ratings and composites had ICCs <0.2. The number of patient responses required for reliable results (defined as a reliability of 0.70) varied from 5 (dialysis center global rating) to 16 (NCC composite).
Associations of Patient and Dialysis Center Characteristics with CAHPS-ICH Scores
A higher ratio of patients per nurse was associated with lower PI scores (ranging from 0.84 for facilities with >12 patients per nurse to 0.91 for facilities with <5 patients per nurse; P=0.0001; Table 4), whereas a higher ratio of patients to physician was associated with worse global ratings of both the dialysis center (ranging from 2.37 for facilities with >36 patients per physician to 2.80 for facilities with <8 patients per physician; P=0.0001) and the dialysis center staff (ranging from 2.44 for facilities with >36 patients per physician to 2.78 for facilities with <8 patients per physician; P=0.0001). Longer waiting room times were associated with lower scores on the NCC composite (ranging from 3.68 for facilities with waiting times >15 minutes to 3.97 for facilities with waiting times of 5 to <10 minutes; P=0.0001) and the QoC composite (ranging from 3.44 for facilities with waiting times >15 minutes to 3.75 for facilities with waiting times of 5 to <10 minutes; P<0.001). This suggests that a more positive experience of care is demonstrated by patients treated in dialysis centers that are appropriately staffed.
Relationship between CAHPS-ICH scores and global rating items and key dialysis center characteristics
Black/African-American patients reported worse global ratings of the dialysis center than white/Caucasian and Hispanic/Latino patients (ranging from 2.40 for African Americans versus 2.72 for whites/Caucasians and 2.68 for Hispanics/Latinos; P=0.0001 overall; Table 5). Finally, patients receiving anemia education reported more positive experiences regarding PI (ranging from 0.89 for educated patients versus 0.78 for patients not educated; P=0.0001).
Relationship between CAHPS-ICH scores and global rating items and key patient characteristics
Discussion
Our analysis demonstrates good item convergence and item discrimination, and satisfactory internal consistency reliability for the CAHPS-ICH NCC and QoC composites. The PI composite had less than desirable internal consistency reliability (α<0.7) and the data did not fit the CFA model well. The ICCs at the center level ranged from 0.13 to 0.31 and the estimates of the number of responses required to obtain a reliability of 0.70 were relatively low, ranging from 5 to 16, which suggests that reliable information can be obtained with relatively few patients. Using a Bonferroni adjusted P value of <0.0005, we identified some patient and dialysis center characteristics that were associated with the CAHPS-ICH scales. These included the number of patients per nurse, which differentiated scores on the PI composite (P=0.0001), and the ratio of patients per physician, for which a higher ratio was associated with lower mean scores on the dialysis center and the dialysis center staff global ratings (P=0.0001 for both).
These results are consistent with recent patient experiences of care in United States hospitals (10,11). Jha and colleagues examined data from the CAHPS Hospital Survey (HCAHPS), which captures the performance of hospitals in the United States in terms of patient experience (10). Hospitals with higher ratios of nurses to patient-days performed better than those with the lowest ratios, with 70% and 64% of patients, respectively, indicating that they “would definitely recommend” the hospital (P<0.001). In 2009, Kutney-Lee and colleagues examined the relationship between nursing and HCAHPS scores across 430 United States hospitals (11). They also found that nurse-to-patient staffing ratios were significantly associated with multiple HCAHPS domains, including whether patients were likely to recommend a hospital.
CAHPS surveys are used in several therapeutic areas to inform patient experiences with care in a variety of settings such as ESRD (12), women’s health (13), nursing home residents (14), family member experiences of nursing home care (15), and dental care (16). An evaluation of the CAHPS Medicare survey for patients with ESRD conducted in 2009–2010 was recently reported (12). The cohort included 3794 patients with ESRD drawn from a database of 823,564 Medicare beneficiaries. The survey evaluated access to care, physician communication, customer service, and access to prescription drugs and drug information as well as ratings for overall care, personal physician, specialist physician, and the prescription drug plan. Overall, the care experience for patients with ESRD was at least as positive as that reported by those without ESRD. Within the ESRD cohort, patients who were black or had less education reported worse experiences than other patients with ESRD. A similar pattern was observed in this analysis of the CAHPS-ICH, with black/African-American patients reporting worse experiences than white/Caucasian and Hispanic/Latino patients in terms of the dialysis center. Bagchi and colleagues recently explored this apparent disparity through a series of focus groups with African Americans, Asian Indians, Latinos, and whites (17). They found that racial/ethnic groups were more likely to regard cultural competency and the provision of a holistic approach to care as important to healthcare quality.
Patient experience of care reflects the value of the interpersonal aspects of their health care experience and complements health-related quality of life evaluations in assessing care from the patient’s perspective (18). Information derived from direct evaluation of patient experience of care can be used to identify areas for improvement and support changes in care provision with the aim of improving the overall quality of care for patients. The recent inclusion of patient experience of care evaluation as a component of the ESRD PPS for Medicare reimbursement of outpatient dialysis-related services means that individual dialysis centers need to attempt to administer the survey to their patients and report that they did so in order to meet one of the three reporting measures that together comprise 10% of facility’s total QIP score (19). The CAHPS-ICH is a key element of this quality-led evaluation of care provision and this analysis supports the utility of this tool in ensuring that dialysis centers strive for and achieve delivery of high-quality care through the use of public funds (20,21). Utilization of the CAHPS-ICH should support comparisons between dialysis centers in terms of the quality of care they offer and sharing of best practices between centers. However, some issues such as the ethnic/racial distribution of the patient population may have an effect on between-center comparisons, as previously discussed.
The survey was conducted in a limited number of centers. To assess the generalizability of the results, we compared the patient cohort included in the current analysis with the US Renal Data System 2011 dialysis population and found the populations to be similar in terms of age and sex (22). When considering the results presented here, the limitations of the survey approach should be considered. We were unable to compare patients who did and did not complete the survey in terms of relevant variables that may have affected their experience of care and the likelihood of their completing the survey.
The CAHPS-ICH provides a valuable and informative tool for the evaluation of patient-perceived quality of care. This analysis suggests that at the center level, staff-to-patient ratios and the information provided to patients may be of particular relevance to patients in determining their perception of the quality of care they receive. However, measurement properties are specific to the setting in which they were assessed, so it will be important to repeat the analysis in other settings in order to test the replicability and thus the generalizability of the findings. Furthermore, although this analysis has focused on the composite scores, individual item results could also be used to assess aspects of care and may serve as markers for specific quality improvement endeavors. Additional tools will be needed to evaluate other aspects of quality of care, such as health-related quality of life, with clinical elements such as anemia education and management and vascular access, which form a significant part of the dialysis treatment routine. Additional psychometric evaluation would be useful before using CAHPS-ICH as a CMS QIP metric. Equally, further work to establish minimally important cut-off scores and changes or differences in scores that are clinically important may also be required. The CAHPS-ICH has the potential to assess the quality of service in renal dialysis units in the United States, yet more research is warranted to elicit the clinically meaningful difference between dialysis centers.
Disclosures
R.W., G.T.-S., and J.P. are employed by Adelphi Real World Ltd. C.J.P. is employed by Amgen, Inc. R.D.H. is a consultant to Amgen, Inc. and the Critical Path Institute. M.G. was employed by Amgen, Inc. at the time this study was initiated and conducted.
Acknowledgments
Medical writing assistance was provided by Tracey Lonergan of Adelphi Real World Ltd., and this support was funded by Amgen, Inc. Adelphi Real World Ltd. sponsored and conducted the research. After the research was completed, Amgen, Inc. subscribed for access to the data analyses. Amgen, Inc. participated in the data analysis and interpretation, and review and approval of the manuscript. All authors were involved in the analysis of the data, development, and revision of the manuscript, and decision to submit the manuscript for publication.
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
This article contains supplemental material online at http://cjasn.asnjournals.org/lookup/suppl/doi:10.2215/CJN.10121013/-/DCSupplemental.
See related editorial, “Incentives for Caution: The In-Center Hemodialysis Consumer Assessment of Healthcare Providers and Systems Survey and Experience of Care,” on pages 1005–1006.
- Received October 3, 2013.
- Accepted February 18, 2014.
- Copyright © 2014 by the American Society of Nephrology
References
In this issue
