How We Gather (and How to Interpret) Our Data
We surveyed area Checkbook and Consumer Reports subscribers for their feedback on services they have used. We also invited a random sampling of other area consumers to participate.
On our surveys, we ask consumers to rate their experiences with companies they had most recently used on several aspects of service. Our ratings tables show the percent of each company’s surveyed customers who provided a rating of “superior” (as opposed to “inferior” or “adequate”) for questions on our survey. We sometimes also report the percent of surveyed customers who rated each company “adequate” or “superior” (as opposed to “inferior”) for the “overall quality” question on our survey.
We have included on our ratings tables all of the companies for which we received at least 10 ratings on our customer surveys. If a company is not listed on our ratings tables, it simply means we did not receive at least 10 ratings for it.
Since many companies were rated by rather small numbers of raters, small differences between two firms in the percentage of raters who gave a particular rating (say, “superior”) should be ignored. The table below gives a rough guide to minimum differences you should look for in deciding on one company over another.
When using these survey data, remember that the questions are to some degree subjective and that the differences among companies might be explained by differences in the personalities, backgrounds, critical standards, and other characteristics of the raters or by biases these raters might have.
Our ratings tables show the number of complaints filed against each company with government offices of consumer affairs in Fairfax, Howard, and Montgomery counties; the Consumer Protection Division of the Maryland Attorney General’s Office; and the Virginia Office of Consumer Affairs. These complaint counts are for a two-year period.
Our ratings tables also often show counts of complaints filed with the Better Business Bureau (BBB) for a three-year period.
For the counts of complaints reported for local government consumer offices, we have attempted to include only complaints that relate to auto repair work (as opposed to complaints about bodywork, new or used car sales, etc.). Although the BBB categorizes complaints it receives into broad categories, these categories don’t separate out complaints relating to auto repair work. For all of the shops on our ratings tables, we have included those complaints that fell into the category of problems with product/service, which accounted for the most complaints filed against independent auto repair shops. We did not count complaints that were categorized as relating to advertising/sales, billing/collection, delivery, or guarantee/warranty issues.
Where we were able to, we have also reported on our ratings tables a complaint rate for each company, calculated by dividing the number of complaints by our measure of the number of full-time-equivalent mechanics who work for each shop. These complaint rates are intended as a rough way to take into account volume of work and the fact that companies that do more work are exposed to a greater risk of incurring complaints.
When using the complaint information, keep in mind that complaints are not always justified; sometimes the customer is unreasonable. Also, be aware that some companies may be at greater risk than others of incurring complaints because of the specific types of business they do. And remember that the measure of business volume we use in calculating complaint rates (our estimate of the number of full-time-equivalent mechanics) is at best a very rough indicator.
We always recommend that you look for substantial differences in complaint counts and rates. We also advise giving little weight to complaint counts if the total count against a company is less than three or four.
Price Comparison Scores
For each shop, this score is intended to suggest the price a customer might expect to pay for repairs that would cost $100 at the “average” shop repairing the same type of car. The price comparison score is based on prices quoted to Checkbook’s undercover shoppers for up to three specific repair tasks. For each shop, a make or several makes of cars were selected from among the makes the shop regularly services. All prices quoted were for a make of car the shop repairs and the shop was compared to other shops with price quotes on the same make and repair. Shops that do not regularly repair any of the makes were not scored. Note that some dealers were scored on a kind of car they repair but do not sell. For instance, Lincoln dealers were scored on the basis of their charges for repairing Fords.
To compute our price comparison scores, we calculated an average price for each job or item for all the companies that quoted on that job or item. Next we compared each company’s price to the average. One company might come in at 120 percent of the multi-company average for a particular job, and another company might come in at 90 percent. We took each company’s percentage score on each job or item, standardized it, and assigned a weight to each job or item, based on our judgment. We then averaged the standardized, weighted percentage scores to find how the company compared to other companies overall. Finally, we multiplied this overall percentage score by a flat dollar amount, say, $100.
The price comparison score, then, is intended to indicate the relative prices we found for the companies, adjusted to the base of this flat dollar amount. These scores are imperfect for various reasons: for instance, the jobs or items checked may not be representative; the weighting of various jobs or items in the scores may not accurately reflect typical expenditure patterns; and the number of jobs or items is small.
Cars Repaired, Warranty Work, and Labor Rates
Information reported on makes of cars repaired in and out of warranty and labor rates came directly from the shops’ representatives. We mailed each shop a survey. For shops that did not respond to our mailed survey, our researchers called each shop to complete a survey over the phone and then followed up with a mailed verification form.
Timeliness of the Data
All of the data must be interpreted in view of timeliness. Our customer survey data are from surveys conducted from January 2012 to September 2021. Survey respondents were asked to report on experiences in the preceding year. Our data on complaints for the Better Business Bureau are for a three-year period dating back from a date in September or October 2020. Our data on complaints for the government offices of consumer affairs are for a two-year period between June 1, 2018 and May 31, 2020. Our price data were collected from February to September 2020. The data from our surveys to ask about facts such as makes of cars repaired and hourly labor rates were collected from August to November 2020.
Our tables include companies for which we collected 10 or more ratings on our customer survey during the customer survey period mentioned above, but we do not report data for periods prior to companies’ changes of name and ownership. As a result, some large dealerships and shops are not listed. If only name or ownership changed, we do report the data. Changes subsequent to the dates listed above may not be taken into account.
We give checkmarks to companies that score highest on a scoring system that we devise for each service field. Our scoring systems weight the various data in our tables and text based on our subjective judgment of their importance. Since the scores are based entirely on information presented, you can apply your own subjective judgments, and decide whether you prefer companies we have not given checkmarks. Where we do not have important data on a company, we cannot give our checkmark.
|A rough guide for deciding whether the difference between two percentages is important||If one firm had this number of ratings:||And a second had this number of ratings:||Do not give much importance to the difference between percentages unless the difference is at least this many percentage points:|
|Assuming the average of the two firms' percentages is 50 percent||10
|Assuming the average of the two firms' percentages is 80 percent||10