The Financial Conduct Authority (FCA) said staff at the firm had focused on the "quantity not quality" of sales to the detriment of customers, many of whom were vulnerable older people.
HomeServe, which insures more than 2 million people in Britain against burst pipes, broken gas boilers and electrical problems, has been punished for "serious, systemic and long-running failings, extending across many key aspects of its business". The fine is the largest ever for mis-selling to retail customers in Britain, beating a £28m penalty imposed on Lloyds Banking Group last December.
The regulator found that between January 2005 and October 2011 HomeServe had mis-sold insurance policies and failed to investigate complaints adequately.
During that period, pay to sales teams was structured in a way that gave staff incentives to increase the volume of products sold, irrespective of customers' need for the products. In addition, inadequate IT systems meant some customers were overcharged or paid for duplicate cover.
For three years between 2008 and 2011 the pay structure meant that people handling complaints were paid according to how many they closed, whatever the outcome for customers, the regulator said.
Meanwhile, the board and senior management of the company were also found to be "insufficiently engaged" with compliance matters.
The FCA said it considered the failings to be particularly serious given that a significant proportion of HomeServe customers were of retirement age and therefore more vulnerable.
On top of the fine, HomeServe has already compensated customers to the tune of almost £13m, but is expected to pay out about £4m more. It has been in touch with many customers caught up in the scandal, and said everyone will have been contacted by the end of March.
Tracey McDermott, the FCA's director of enforcement and financial crime, said this was a serious case. "HomeServe is another example of a firm that has acted without proper regard for its customers over a long period of time," she said. "HomeServe promises to provide customers with peace of mind when things go wrong. In fact, the firm's culture, controls and remuneration structures meant that staff were focused on quantity, not quality, and there were customers that paid the price for that."
The firm had already warned shareholders that a large fine was imminent. The actual fine is lower than the £35m it was prepared for, and also lower than it would have been had HomeServe refused to settle at an early stage.
HomeServe's chief executive, Richard Harpin, said: "We sincerely regret that some customers have been affected by these issues.
"We have transformed the business, rebuilding and strengthening the management team, retraining staff and restructuring systems and controls. We have worked very hard over the last two years to put customers back at the heart of our business and we are committed to offering valuable products with a high quality of service."
In late 2011, HomeServe shocked investors and customers when it suspended its entire sales workforce amid fears they had been mis-selling its products. The West Midlands-based firm said it had suspended its sales operation and was retraining 500 call-centre staff voluntarily.
The Guardian reported at the time that the move was prompted by amember of HomeServe's customer services team contacting the FCA's predecessor, the Financial Services Authority, to warn that customers who complained to the company after receiving poor service may have simply been ignored.
An internal document revealed the firm was reviewing 48,000 complaints from the previous winter, when weeks of sub-zero temperatures in many parts of the country led to a huge number of call-outs.
There were claims at the time that the company had been unable to cope with the level of calls, leaving some people without heating for weeks.
In April 2012, HomeServe was fined £750,000 by Ofcom for making excessive numbers of silent cold calls.