case study - insurance quote

Problem

Classic car insurance sales temporarily plummeted due to COVID-19 economic panic.

Like many other companies, early 2020 kicked Hagerty's butt. Springtime economic panic due to COVID-19 temporarily reduced classic car insurance sales by over 30% compared to 2019. Hagerty's 2020 goals included exceeding 2019 insurance sales by 14%. For a time it looked as though this wasn't going to happen.

A screenshot of early 2020 web-based insurance sales over time. In mid-March 2020, Hagerty experienced an abnormally steep decline in insurance sales (which looked worse than reality because of an anomalous surge in sales around the beginning of March).

Early attempts to make up sales were fruitless.

Folks stepped up from all directions forming desperate task forces proposing solutions that sounded great but were not viable. Early spaghetti-to-the-wall attempts were costly for the company and we weren't learning much. There were no concrete user-centered problems identified that Hagerty tried to fix.

Delicious, desperate, mindless grabs for sales.

Sales came back on their own over the next few weeks, but the damage was done.

By the middle of Spring 2020, economic panic impacting insurance sales subsided. But sales growth goals for quarter 2 were already a wash. Now the company was asking, can we regain the sales we were expecting? Can we make up for the money we spent? (Staff: can we still get our full bonuses?). After sales growth rates went more-or-less back to expected, people started asking, "Well, what do users really need, anyway? How do we find that out?" <Skies opening>

A screenshot of web insurance sales over time, depicting a big hole in sales in March and April 2020. By the end of April, sales rates went back to expected, on their own.

Can meeting user needs make up for a loss in sales?

I was asked to put some of my other projects on hold to handle a high priority business problem - help identify ways to increase sales through a "UX lens." What were users experiencing when they applied for insurance? How could we influence them (without being dishonest) to buy more insurance? Were there pain points in applying for insurance? What did prospective insurance buyers need that we weren't providing?

Approach

My roles: designer, researcher, optimizer

I was responsible for planning and executing qualitative and quantitative research activities, and then providing high-impact actionable recommendations to product teams. I also worked with 1. product owners to prioritize jobs, 2. developers to confirm feasibility of recommendations and develop AB tests, 3. designers to help problem solve issues and make some spec'd visual designs, 4. information analysts to help query the data warehouse and review my data analyses, and 5. partnered with another user researcher to plan and execute research.

Usability tests and analytics helped me uncover salient user pain points in the online insurance quote and in the pay-per-click insurance flows.

One of the first things I did was conduct qualitative usability and 5-second tests of the online quote and application, and collect call center data about users calling in about the insurance quote. I moderated 4 sessions of users attempting to complete tasks in the online quote. The other researcher and I surfaced several dozen usability pain points while reviewing notes and recordings. I dug through analytics, partnering with another information analyst to triangulate usability insights with quantitative data about users' form inputs and where they were most often exiting the quote. Looking at referral traffic and keyword data helped me identify potential problems with the paid search landing page for insurance. Running some more usability tests, 5-second tests, desirability tests, and interviewing stakeholders revealed credibility issues and a wonky paid-search-traffic strategy that siphoned current clients from account management to marketing brochureware.

"It looks like a page that's trying way too hard to sell me something."
- user

A data table of quote result exits for the first half of 2020. Of note: roughly 50% of users do not proceed past quote results into an application. Usability tests indicated the quote result page felt amatuerish and salesy to users, and hard to skim and scan.

Analytics showed roughly 50% of users do not proceed past quote results into an application. Usability tests indicated the quote result page felt amateurish and salesy to users, and hard to skim and scan.

A quote fallout chart for the first half of 2020. Of note: 35% of users bail as soon as they start a quote, and 30% of users who make it to the demographics page bail when they're asked to provide personal and contact information. 5-second tests revealed users didn't know the quote start page was the beginning of a quote form. Usability tests indicated users are tired of filling out forms once they get to the demographics page.

Analytics showed 35% of users bail as soon as they start a quote, and 30% of users who make it to the demographics page bail when they're asked to provide personal and contact information. 5-second tests revealed users didn't know the quote start page was the beginning of a quote form. Usability tests indicated users are tired of filling out forms once they get to the demographics page.

A data summary that shows organic traffic converted 186.86% better than paid traffic in the first half of 2020. The e-commerce team pays for a lot of irrelevant traffic to visit the quote and they direct account-management bound users to the quote instead of where they need to go.

Analytics showed organic traffic converted 186.86% better than paid traffic in the first half of 2020. The e-commerce team pays for a lot of irrelevant traffic to visit the quote and they direct account-management bound users to the quote instead of where they need to go.

A screenshot of the old pay-per-click page. People who searched 'Hagerty login' on Google were directed here. But there's nowhere obvious for them to go! Very bad for retention.

A screenshot of the old paid search landing page. People who searched 'Hagerty login' on Google were led here by our paid search ads. But there's nowhere obvious for current customers to log in!

I reviewed my findings with product owners, developers, designers, the marketing team, and other stakeholders

After identifying high-impact issues, I helped set priorities by reviewing findings with several teams and calculating conservative potential impact to revenue for several usability issues. I found 4 quote and paid search related opportunities that predictably could gain Hagerty $1.2 million in additional revenue over 1 year and $15.5 million over 5 years (accounting for our average customer retention, seasonality, and average insurance premium rates), all other things being equal.

Iterating and testing design changes

Once priorities were set, I helped designers, developers, and product managers understand the usability problems critical to fix. The designers and I crafted solutions, and I reviewed them and tested with target users via usability tests, 5-second tests, desirability tests and AB tests, depending on the research questions. We followed a rough process of design > test > iterate > test > development and QA > Release > Optimize for 4 main opportunity areas, including starting the quote, entering personal information, reviewing quote results, and lastly, the pay-per-click marketing page. AB tests (for 2 solutions) took 2 months each (due to relatively low visitor traffic and the need to power sample sizes for AB tests to mean anything).

The AB tests had to be done 1 at a time so that 1. they didn't interfere with each other, 2. confounding variables were minimized, and 3. we could gather adequate sample sizes for each without prolonging product changes. If we had AB tested all 4 solutions at once, nothing would have been permanently released until Apr. 2021. The way we did it made sure we had consecutive things to release to the public. (While AB tests ran, we all worked on other projects, too.)

Impact

Improved relevance for users

I recommended concrete changes for improved usefulness and more credibility. I helped product teams and designers empathize with our users. And I helped test and iterate design changes that made things easier for users to do and find. While we didn't finish all of the changes for this project until 2021, we more than made up for 2020's loss of sales revenue.

My work also enabled improved experiences for existing customers who entered the site on the pay-per-click page. 123%+ more account-bound users are able to get to their accounts or pay their bills now.

$3.12 million more for Hagerty

Uncovering user needs and recommending concrete actionable improvements to insurance products helped generate about 10% more new online insurance business (impact to retention won't be knowable for some time). Since the start of the project we've made $3.12 million more than we would have (and plausibly $14.26 million in additional revenue over the next 2 years, all else being equal).

8.66% more paid traffic quote completions and 123.18% more online bill payments
Users are smart. They plainly saw the paid search landing page from Hagerty's paid search ads was full of hyped, subjective language. During usability tests and desirability tests, e.g., one user said, "It looks like a page that's trying way too hard to sell me something." Another said, "I couldn't find anything about why I should get it. It was all just stuff saying how awesome Hagerty is." Many others said that "everyone" claims to save you money - how were we any different? They didn't see anything to concretely answer their questions.

Poor credibility wasn't the only problem. Since Hagerty was bidding for search terms like 'Hagerty login', we were preventing current clients who needed to pay bills and manage their policies from doing so. (it turned out that at least 16% of the pay-per-click traffic was trying to do something account related). The paid search landing page, e.g., did not have a login link. These folks then most often resorted to calling to pay their bills.

We were able to appeal to users' genuine questions instead of providing dubious or empty claims. E.g., instead of hyping how awesome we were, we appealed to concrete reasons our insurance coverage is good for classic cars. And, I appealed to the marketing team to stop paying for 'Hagerty login' keyword searches. Compromises had to be made on some of the copy to appease Brand team members and make sure Hagerty's "twinkle" was still present, even though the data suggested users are anything but impressed with the "twinkle".

B increased insurance quote completions from paid traffic by 8.66% with 99% Confidence. And B increased paid traffic pay bill completions by 123.18%, almost eliminating call center calls for paying bills for visitors of the paid landing page altogether.
A v B pay-per-click page. There was nothing objective about A's copy except for the statement that we're 36% cheaper than standard insurers, which is just false in most contexts. B reduces self-promotion and subjectivity in favor of appealing to users' real-life questions about the product.
I helped reduce clutter on the quote results page, and front-loaded more useful information so users could more easily find what they were looking for. This increased overall online insurance sales by 12.0%.
The paid search landing page wasn't the only place we prioritized marketing content over information users needed. The quote result page was also too full of marketing content. One user said, "This marketing is super dated." Another said, "The 'learn more' [popups] don't have anything useful." When we updated the quote results page, we made sure to reduce marketing content. Unfortunately, we were unable to introduce new features to edit coverages and add-ons directly on the page (development resources were scarce). What we were able to do was remove a lot of irrelevant content, re-order quote information to prioritize policy coverage information over quote minutia like quote numbers and the users' email address, and update some of the styling to match the insurance content webpages on the site. Even though we weren't able to do everything we wanted to do, it was still a vast improvement. During the AB test of the quote results page, we increased sales by 12.0%, with 99% Confidence.
This was the initial mobile view of the A version of the quote results page.
The initial view of B quote results (screenshot of prototype).
I helped users better understand where they were in the quote process.
There was nothing on the quote start page to indicate to users they were about to start a quote. This was especially troublesome for users who entered the quote directly from a Google search result. Once they landed on the quote start page, they'd get confused and leave. So we made a few changes to make it more obvious this was the start of an insurance quote.
The mobile view of the old version of the quote start page before we made any changes. Notice there is nothing on here that would indicate to a user they are about to start an insurance quote.
The "new" view of the updated B quote start page. We added a header to the page, 'Let's start your quote,' changed the button copy from 'Next' to 'Start my quote,' and changed the progress bar label from 'Location' to 'Start.' Usability tests demonstrated users were much better able to know where they were and what they were doing on this page.
I helped users get through their weariness while filling out the quote.
By the time users get to the demographics page, they've filled out dozens of form inputs and start to lose steam. Users needed a little reminder they were nearly done getting a quote. So we updated the header on the demographics page from "Demographics" to "Your quote is only one click away. We just need these last details." This change was usability tested, but not AB tested. Users verbally stated their relief. "Oh thank god."

We didn't have developer bandwidth to AB test this, so we released and monitored analytics. Its release coincided with a 23% increase in quote completions.
The header on this image used to say, "Demographics." Now it says, "Your quote is one click away. We just need these last details."