Hooking you on the promise of quality

Like most new build homebuyers, you will probably have seen HBF Star Rating logos displayed by housebuilders on flags around their developments and generally anywhere prospective customers might see them and draw the conclusion that lots of stars = lots of quality and happy customers.

So what are Star Ratings, exactly?

Every year, the HBF (a political lobby group that represents ~80% by volume of the UK housebuilding industry) and the NHBC (the UK’s biggest provider of new home warranties) run a customer satisfaction survey for new build buyers. The survey is sent to owners around 8 weeks after they move in, and results for the previous year are published in or around March of the following year.

For their Star Ratings, HBF works out the percentage of ‘yes’ answers to one of their survey questions, and this determines the number of HBF Stars each participating builder gets ‘awarded’: 80%+ gets you 4 stars. 90%+ gets you 5. The percentages that get you less than 4 stars are irrelevant because no-one gets less than 4 stars.

The question HBF asks homebuyers in order to generate its Star Ratings is:

“Would you recommend your builder to a friend?”
[answer: yes or no].

That is not a direct question about construction quality or after sales customer satisfaction, so there is no direct correlation between a ‘yes’ answer and the housebuilder’s general build quality or customer satisfaction levels, but that is what the Stars are used to promote. Not only that, but by asking the question only 8 short weeks into ownership, buyers are unlikely to have formed a clear understanding of overall construction quality or to assess whether the customer service team is any good at fixing things.

So, not really that representative of construction quality or customer satisfaction then.

But at least the statistics themselves are reliable, right?

Wonky Statistics?

HBF publishes its survey results and awards its members with their Star Ratings annually, and have done so each year since 2005.

Every year, HBF almost invariably promotes the message that customer satisfaction is continuing to improve year on year and has never been higher. Only, to compare something year on year, would you not have to measure the same thing in the same way each year, and include all the returned surveys in your analysis?

We reviewed HBF’s annual reports for 2005 and 2012 and found that HBF’s stated methodology for producing key statistic change significantly over time, with no explanation for how this may have affected the claimed results. Perhaps some of the claimed improvements over time reflect changing methodologies rather than improvements in construction quality or customer satisfaction?

We also found that in any given year, somewhere between 4.9% and 23.8% of surveys were apparently ‘disregarded’. HBF does not say why. Perhaps some homeowners gave the ‘wrong’ answer?

No outside organisation appears to be auditing these survey results, which are used not only to market to new buyers but also to keep politicians at bay with assurances that there is no need for regulatory intervention within the industry (something many buyers and representative groups appear to disagree with).