Factors Affecting Results of HEDIS/CAHPS® 2.0H Health Plan Member Satisfaction Surveys and Resulting Accreditation Scores Technical Information to Assist Consumers, Purchasers, and Researchers in Interpreting Results Download our free Power Point Presentation of this Report Here (Microsoft PowerPoint Required) In the media and in health plan marketing materials, consumers are often presented with claims and statistics based on surveys of plan members. Such survey results are useless for comparison of health plans unless each plan's survey is done by an independent survey organization using the same survey instrument and the same survey procedures. The only member survey data that meet these criteria and are currently available on a substantial number of plans are from the Consumer Assessment of Health Plans Survey (CAHPS) conducted under the procedures dictated by the National Committee for Quality Assurance (NCQA). The CAHPS survey was developed with funding and leadership from the U.S. Agency for Healthcare Research and Quality (AHRQ). The NCQA-dictated procedures for the CAHPS survey call for the following sequence of four mailings: questionnaire with cover letter, reminder postcard, second questionnaire with cover letter to nonrespondents, second reminder postcard. For nonrespondents to these mailed attempts, either a third questionnaire mailing is done or at least three attempts are made to complete the survey by phone. NCQA also allows members to be offered the option of completing the survey on the Internet. The initial sample size for each plan is typically over 950 and response rates are generally in the 45 percent range, yielding about 400 to 450 responses. NCQA requires each plan to have its survey carried out by an independent survey research firm. The plan can choose the survey firm, but the survey firm has to be trained and certified by NCQA, and NCQA monitors the survey process. Each plan supplies its survey firm its entire list of eligible members and the survey firm draws a random sample of these members to survey. The list of eligible members is checked by an independent audit firm to assure that it is complete. The surveyed members score the plans using several different scales - Zero to 10 "Big problem," "Small problem," or "Not a problem" "Never," "Sometimes," "Usually," or "Always" When you see comparisons of plans, you need to check that each plan is being compared based on the same score cutoff. For example, if the question is scored on a zero to 10 scale and one plan's results are reported as the percent rating it 8, 9, or 10, you need to be sure that the results for any plan to which it is compared are not being reported as the percent rating it 9 or 10, or the percent rating it 7, 8, 9, or 10. Despite the relatively high rigor and uniformity of the NCQA-supervised CAHPS survey, several cautions need to be considered when using the member survey data: If you just look at the overall ratings of each plan, you might not spot serious problems. One can imagine a plan having restrictive policies that routinely deny members life-saving but expensive therapy while at the same time having an excellent newsletter and a special arrangement to get members a big price break on memberships at local health and fitness clubs. Since only a few members each year need expensive life-saving treatment, their negative ratings of the plan would not have much effect on the plan's overall percent of favorable ratings, while the favorable ratings from the many members who might enjoy the newsletter or the fitness club discount  would have given the plan a big lift in its overall percentage of favorable ratings. There may be plan-to-plan differences in the characteristics of the survey respondents, and some plans might have been rated by respondents who are more or less difficult to please than other plans' respondents. In extensive analysis of survey data from hundreds of plans, we have found that differences in age, education level, and health status account for significant differences in plan scores. NCQA does not dictate that scores be adjusted for these or other respondent characteristics. Fortunately, the effects of adjustment, while significant, would not be large. Very few plan scores would change by more than two percentage points if the scores were adjusted. Among plans that would be in the top quarter of all plans based on actual scores, only about 10 percent would cease to be in the top quarter if the scores were adjusted. The members who respond to the survey might have different opinions and experiences with the plan than nonrespondents. Our review of the data indicates that younger members and men, especially men under age 35, are less likely to respond than older members and women. Young males also tend to give somewhat lower ratings than older members of either gender. Fortunately, with roughly half the sampled members responding for most plans, the respondents do at least represent a substantial portion of the members. And there is some evidence that scores would not have changed much if an even larger portion of members had responded. We have done special studies in which an extra survey mailing was done, increasing the number of respondents by 10 to 15 percent, and these extra respondents on average rated their plans only two to three percentage points lower than earlier respondents. Differences in plan scores might be explained by chance alone. A plan might have simply had the bad luck that the persons who like it least happened to respond. For a plan rated by 400 members, you can be 95 percent confident that its score would not have been more than five percentage points higher or lower than reported if the sample of respondents had simply been larger. Some differences in plan scores result from differences in the mode - mail versus telephone - by which the respondents are surveyed. Some plans reach a relatively large portion of their respondents by mail; others get relatively few by mail and more through telephone followup. (Many plans don't have phone numbers for many of their members, so it is difficult for the surveyors to reach members by phone.) We have done analyses indicating that telephone respondents rate plans differently from the way mail respondents rate them. The overall rating of plans tends to be about seven percentage points higher for telephone respondents than for mail respondents. We even have data on members who responded twice to the same survey, having been reached by phone before the Postal Service delivered their returned mail questionnaire. On the overall rating of the plan, the ratings given by these mail-and-phone respondents were about twice as likely to be higher by phone than mail as they were to be higher by mail than by phone. Fortunately, since few plans get large portions of their responses by phone, the effects of mode differences on plan-to-plan differences in scores is rarely more than two or three percentage points. Some scores might have been affected by use of the Internet. A substantial number of plans use an "enhanced" survey procedure in which members are offered the option to complete the survey over the Internet. Special test data we have analyzed indicate that within the same plan, members who are offered the option to rate the plan via the Internet and choose to do so tend to rate it nearly 15 percentage points lower than members who are given the mail and phone survey options. Fortunately, most plans that offer surveyed members the Internet survey option actually get few responses via the Internet, so the effect of those responses on plan scores was probably less than one percentage point. Scores might have been affected by the language of response. NCQA allows surveyors the option of sending both an English-language and a Spanish-language questionnaire to each surveyed member. Some plans with substantial percentages of members who have expressed a Spanish-language preference take this approach, but others do not. Our analysis indicates that members of the same plan rating the plan on the Spanish-language questionnaire tended to give ratings about 20 percentage points higher than members using the English-language version. This effect is not likely to explain many differences in plan scores, however, because very few plans adopt the two-questionnaire approach, and even in those plans, few members use the Spanish-language questionnaire. Scores might have been affected by a plan's decision on the specific product line to survey. Some plans survey only their basic HMO members or only their POS members. Others survey members from both lines of business, HMO and POS. Among members of the same plan, POS members are about two to three percentage points less likely than basic HMO members to give high ratings to the overall plan. The plan variant that the surveyed members rated might not be the same plan variant you are considering. If the member ratings you see are for a basic HMO and you are considering a POS option or an HMO without a gatekeeper requirement also offered by the same organization, the ratings won't be strictly applicable to your situation. The plan's performance in your specific geographic area may not be the same as in other areas from which some or most of the surveyed members were drawn. The cumulative effects of all these considerations might have a substantial impact on the extent to which the information presented to you allows you to draw sharp distinctions among plans.