How We Gather (and How to Interpret) Our Data
Our tables rating individual companies will be more valuable to you if you know how the data were gathered and how they should be interpreted.
Customer survey scores reported on our ratings tables are from our surveys of area consumers—primarily Consumer Reports and CHECKBOOK subscribers—conducted via email and regular mail.
Since many companies were rated by rather small numbers of raters, small differences between two companies in the percentage of raters who gave a particular rating (say, “superior”) should be ignored. The table below gives a rough guide to minimum differences you should look for in deciding on one company over another.
When using these survey data, remember that the questions are to some degree subjective and that the differences among companies might be explained by differences in the personalities, backgrounds, critical standards, and other characteristics of the raters or by biases these raters might have.
Our ratings tables report the number of complaints filed against individual companies with local Better Business Bureaus (BBB) for a three-year period.
There are limitations to the complaint data. One problem is that some complaints may be unjustified, simply filed by cranks. Another problem is that we didn’t have a measure of business volume and therefore couldn’t control for differences in companies’ exposure to complaints.
We always recommend that you look for substantial differences in complaint counts and rates. We also advise giving little weight to complaint counts if the total count against a company is less than three or four.
Survey of Companies
To gather much of the other information on our ratings tables, we surveyed the companies. In general, our researchers surveyed companies by phone (sometimes without revealing their affiliation with CHECKBOOK), but in some cases, data were collected by mail or from companies’ websites, or phone responses were confirmed by mail follow-up.
Price Index Scores
To compute our price index scores, we calculated an average price for each job or item for all the companies that quoted on that job or item. Next we compared each company’s price to the average. One company might come in at 120 percent of the multi-company average for a particular job, and another company might come in at 90 percent. We took each company’s percentage score on each job or item, standardized it, and assigned a weight to each job or item, based on our judgment. We then averaged the standardized, weighted percentage scores to find how the company compared to other companies overall. Finally, we multiplied this overall percentage score by a flat dollar amount, say, $100.
The price index score, then, is intended to indicate the relative prices we found for the companies, adjusted to the base of this flat dollar amount. These index scores are imperfect for various reasons: for instance, the jobs or items checked may not be representative; the weighting of various jobs or items in the index may not accurately reflect typical expenditure patterns; and the number of jobs or items is small.
Timeliness of the Data
All of the data must be interpreted in view of timeliness. Our customer survey data are from surveys conducted from January 2005 to October 2018. Survey respondents were asked to report on experiences in the preceding year. Our data on complaints for the BBBs are for a three-year period dating back from a date between June 4 and August 3, 2012. The data from our surveys of companies were collected from April to August 2012. Our price data were collected from January to April 2012.
For the most part, our tables include companies for which we collected 10 or more ratings on our customer survey during the customer survey period mentioned above, but we do not report data for periods prior to companies’ changes of name and ownership. As a result, some large companies are not listed at all. If only name or ownership changed, we do report the data. Changes subsequent to the dates listed above may not be taken into account.
We give checkmarks to companies that score highest on a scoring system that we devise for each service field. Our scoring systems take into account the various data in our tables based on our subjective judgment of their importance. Since the scores are based entirely on information presented, you can apply your own subjective judgments, and decide whether you prefer companies we have not given checkmarks. Where we do not have important data on a company, we cannot give our checkmark.
|A rough guide for deciding whether the difference between two percentages is important||If one firm had this number of ratings:||And a second had this number of ratings:||Do not give much importance to the difference between percentages unless the difference is at least this many percentage points:|
|Assuming the average of the two firms' percentages is 50 percent||10
|Assuming the average of the two firms' percentages is 80 percent||10