Factors Affecting Results of HEDIS/CAHPS® 2.0H Health Plan Member Satisfaction Surveys and Resulting Accreditation Scores
Technical Information to Assist Consumers, Purchasers, and Researchers in Interpreting Results
Download our free Power Point Presentation of this Report Here (Microsoft PowerPoint Required)
In the media and in health plan marketing materials, consumers are often
presented with claims and statistics based on surveys of plan members. Such survey results
are useless for comparison of health plans unless each plan's survey is done by an
independent survey organization using the same survey instrument and the same survey
procedures. The only member survey data that meet these criteria and are currently
available on a substantial number of plans are from the Consumer Assessment of Health
Plans Survey (CAHPS) conducted under the procedures dictated by the National Committee for
Quality Assurance (NCQA). The CAHPS survey was developed with funding and leadership from
the U.S. Agency for Healthcare Research and Quality (AHRQ).
The NCQA-dictated procedures for the CAHPS survey call for the following sequence of four mailings: questionnaire with cover letter, reminder postcard, second questionnaire with
cover letter to nonrespondents, second reminder postcard. For nonrespondents to these
mailed attempts, either a third questionnaire mailing is done or at least three attempts
are made to complete the survey by phone. NCQA also allows members to be offered the
option of completing the survey on the Internet. The initial sample size for each plan is
typically over 950 and response rates are generally in the 45 percent range, yielding
about 400 to 450 responses.
NCQA requires each plan to have its survey carried out by an independent survey
research firm. The plan can choose the survey firm, but the survey firm has to be trained
and certified by NCQA, and NCQA monitors the survey process. Each plan supplies its survey
firm its entire list of eligible members and the survey firm draws a random sample of
these members to survey. The list of eligible members is checked by an independent audit
firm to assure that it is complete.
The surveyed members score the plans using several different scales -
- Zero to 10
- "Big problem," "Small problem," or "Not a problem"
- "Never," "Sometimes," "Usually," or "Always"
When you see comparisons of plans, you need to check that each plan is being compared
based on the same score cutoff. For example, if the question is scored on a zero to 10
scale and one plan's results are reported as the percent rating it 8, 9, or 10, you need
to be sure that the results for any plan to which it is compared are not being reported as
the percent rating it 9 or 10, or the percent rating it 7, 8, 9, or 10.
Despite the relatively high rigor and uniformity of the NCQA-supervised CAHPS survey,
several cautions need to be considered when using the member survey data:
- If you just look at the overall ratings of each plan, you might not spot serious
problems. One can imagine a plan having restrictive policies that routinely deny members
life-saving but expensive therapy while at the same time having an excellent newsletter
and a special arrangement to get members a big price break on memberships at local health
and fitness clubs. Since only a few members each year need expensive life-saving
treatment, their negative ratings of the plan would not have much effect on the plan's
overall percent of favorable ratings, while the favorable ratings from the many members
who might enjoy the newsletter or the fitness club discount would have given the
plan a big lift in its overall percentage of favorable ratings.
- There may be plan-to-plan differences in the characteristics of the survey respondents,
and some plans might have been rated by respondents who are more or less difficult to
please than other plans' respondents. In extensive analysis of survey data from hundreds
of plans, we have found that differences in age, education level, and health status
account for significant differences in plan scores. NCQA does not dictate that scores be
adjusted for these or other respondent characteristics. Fortunately, the effects of
adjustment, while significant, would not be large. Very few plan scores would change by
more than two percentage points if the scores were adjusted. Among plans that would be in
the top quarter of all plans based on actual scores, only about 10 percent would cease to
be in the top quarter if the scores were adjusted.
- The members who respond to the survey might have different opinions and experiences with
the plan than nonrespondents. Our review of the data indicates that younger members and
men, especially men under age 35, are less likely to respond than older members and women.
Young males also tend to give somewhat lower ratings than older members of either gender.
Fortunately, with roughly half the sampled members responding for most plans, the
respondents do at least represent a substantial portion of the members. And there is some
evidence that scores would not have changed much if an even larger portion of members had
responded. We have done special studies in which an extra survey mailing was done,
increasing the number of respondents by 10 to 15 percent, and these extra respondents on
average rated their plans only two to three percentage points lower than earlier
- Differences in plan scores might be explained by chance alone. A plan might have simply
had the bad luck that the persons who like it least happened to respond. For a plan rated
by 400 members, you can be 95 percent confident that its score would not have been more
than five percentage points higher or lower than reported if the sample of respondents had
simply been larger.
- Some differences in plan scores result from differences in the mode - mail versus
telephone - by which the respondents are surveyed. Some plans reach a relatively large
portion of their respondents by mail; others get relatively few by mail and more through
telephone followup. (Many plans don't have phone numbers for many of their members, so it
is difficult for the surveyors to reach members by phone.) We have done analyses
indicating that telephone respondents rate plans differently from the way mail respondents
rate them. The overall rating of plans tends to be about seven percentage points higher
for telephone respondents than for mail respondents. We even have data on members who
responded twice to the same survey, having been reached by phone before the Postal Service
delivered their returned mail questionnaire. On the overall rating of the plan, the
ratings given by these mail-and-phone respondents were about twice as likely to be higher
by phone than mail as they were to be higher by mail than by phone. Fortunately, since few
plans get large portions of their responses by phone, the effects of mode differences on
plan-to-plan differences in scores is rarely more than two or three percentage points.
- Some scores might have been affected by use of the Internet. A substantial number of
plans use an "enhanced" survey procedure in which members are offered the option
to complete the survey over the Internet. Special test data we have analyzed indicate that
within the same plan, members who are offered the option to rate the plan via the Internet
and choose to do so tend to rate it nearly 15 percentage points lower than members who are
given the mail and phone survey options. Fortunately, most plans that offer surveyed
members the Internet survey option actually get few responses via the Internet, so the
effect of those responses on plan scores was probably less than one percentage point.
- Scores might have been affected by the language of response. NCQA allows surveyors the
option of sending both an English-language and a Spanish-language questionnaire to each
surveyed member. Some plans with substantial percentages of members who have expressed a
Spanish-language preference take this approach, but others do not. Our analysis indicates
that members of the same plan rating the plan on the Spanish-language questionnaire tended
to give ratings about 20 percentage points higher than members using the English-language
version. This effect is not likely to explain many differences in plan scores, however,
because very few plans adopt the two-questionnaire approach, and even in those plans, few
members use the Spanish-language questionnaire.
- Scores might have been affected by a plan's decision on the specific product line to
survey. Some plans survey only their basic HMO members or only their POS members. Others
survey members from both lines of business, HMO and POS. Among members of the same plan,
POS members are about two to three percentage points less likely than basic HMO members to
give high ratings to the overall plan.
- The plan variant that the surveyed members rated might not be the same plan variant you
are considering. If the member ratings you see are for a basic HMO and you are considering
a POS option or an HMO without a gatekeeper requirement also offered by the same
organization, the ratings won't be strictly applicable to your situation.
- The plan's performance in your specific geographic area may not be the same as in other
areas from which some or most of the surveyed members were drawn.
The cumulative effects of all these considerations might have a substantial impact on
the extent to which the information presented to you allows you to draw sharp distinctions