Iterative Usability Testing of a Conversational AI Chatbot for Auto Insurance Quotes
One of the largest US auto insurers was building a conversational AI chatbot to guide shoppers through a quote. Marketade, a UX research firm, ran 3 rounds of iterative usability testing with 18 prospective customers, from an early prototype to the working build before launch. Each follow-up round tested the fixes from the one before, so the team knew what worked and what still needed attention.

Last updated: November 18, 2026
Round 1: Foundational Study
The first round of usability testing used a clickable prototype of the chatbot, before the team had built it.
Goals
The company, one of the largest auto insurers in the US, sells policies directly to consumers, and many shoppers start a quote on their phones. Its digital team was designing a conversational AI chatbot for the mobile quote flow. The chatbot asks the quote questions in a chat, has its own name and personality, and can answer questions along the way.
The chatbot research was one cycle of a continuous UX research program that Marketade ran with the company’s digital experience team. Each month, the program took on a new product question and carried it from kickoff to prioritized findings.
The chatbot was still a prototype. Before building it, the team wanted evidence that it would give shoppers a better experience than the existing quote flow and make them more confident and comfortable along the way.
The research set out to:
- Learn whether people understand what the chatbot is for and where the chat confuses them
- Hear how people react to the chatbot’s personality
- Check whether people know what happens after the chat ends
Approach
We ran the study as moderated usability testing, the UX research method of watching people use a product while they think aloud. The core was 6 one-on-one sessions with prospective customers on a mobile prototype of the chatbot, plus 1 pilot session. The work followed 5 steps: discovery, recruiting, research, analysis and report.
1. Discovery
Discovery started with a kickoff meeting with the product team. We confirmed what the chatbot was for and what comes before and after it: shoppers arrive from the homepage, enter a zip code and say why they are shopping before the chat begins. We also agreed on who to recruit. From there we wrote a test plan with the goals, 4 research questions and a session script, and walked the product team through it before the sessions.
2. Recruiting prospective insurance customers
We recruited people who might shop with the company but weren’t customers yet. Participants had to meet these criteria:
- Not a current customer of the company
- A mix of people who had recently gotten an auto insurance quote and people who hadn’t
- Likely to shop for insurance on a phone
- A mix of iPhone and Android users
We sourced candidates through an online research panel and screened each one by phone, checking their fit and that they would be comfortable talking through their thoughts. The 6 participants ranged in age from 26 to 52, and their quote experience varied widely. Some had compared prices on quote comparison sites, one had only ever gotten quotes in person or by phone, and one had no policy at the time.
3. Research: think-aloud usability testing on mobile
Each usability session opened with a short interview about the participant’s current insurer and how they last got a quote. Participants then used the chatbot prototype on a phone, imagining they had come to the company’s site for a real quote. They read everything aloud and talked through what they expected at each step.
We stopped at 2 points to probe: the first choice in the chat, and the list of coverages, where we asked how they would see or change their coverage. Afterward, participants described in their own words what they had been interacting with and, if it were a person, what its personality was like. We also asked about other virtual assistants they use, how long the process felt, what information they missed and what they expected to happen after buying.
Testing a chatbot as a prototype was the hard part. A clickable prototype can only simulate a conversation, so participants may never stumble on some of what the chatbot can do. We planned a moment in each session to see whether participants could work out how to ask the chatbot a question, and a pilot session beforehand let us refine the script. Any team testing an AI feature early can do the same: script the moments where you want to see whether people discover what the AI can do, rather than waiting for it to happen.
4. Analysis
We analyzed the sessions through collaborative analysis, a structured way for a research team to sort observations and agree on what matters most. During each session, observers wrote each thing they noticed on its own sticky note. We sorted the notes by participant and theme, then grouped related notes and titled each group with the problem it described.
Dot voting ranked the resulting 10 findings. The top 3 went to solution brainstorming in small groups, where each group summed up the problem and proposed a fix. We closed by checking the findings against the original goals and research questions.
5. Report
The summary report covered the study’s goals, the participants, the method and each step of the analysis. It listed the 10 findings in priority order, the agreed direction for the top 3 and next steps with an owner for each, and it linked to the session recordings.
Outputs
By the end of the study, the team had:
- 10 findings ranked by priority, so the team could focus on the 3 that mattered most
- An agreed design direction for each of the top 3 findings
- 3 next steps with owners, from new mockups to rewritten transition copy and a follow-up round of usability testing
- Answers to each of the 4 research questions
- Recordings of every session
The usability testing answered the team’s main question. Participants moved through the quote flow successfully, but the chat itself didn’t register: they didn’t sense they were in a conversation, and they didn’t pick up how to get help or change an answer. Three examples show the kind of insight the study produced.
People didn’t notice the conversation. Participants treated the chatbot like a step-by-step form. They answered each question but didn’t realize they could ask the assistant for help. The team chose to make the chat’s input area more visible and draw attention to it, so people would know they could type a question.
The persona went unseen. The chatbot had a name and a designed personality, but participants often didn’t notice the assistant at all. A persona only helps if people register it. The team explored ways to make the assistant more visible during the chat, such as an onboarding step and color.
No clear way to go back. In a linear chat, participants didn’t know how to go back and change an earlier answer. It was the top-voted finding. The team designed ways to draw attention to the edit option, such as an onboarding overlay or an animation, and planned to test them.
Each pattern applies to AI features in general: people need to notice the AI, understand what it can do and know how to correct course.
Rounds 2 and 3: Follow-Up Testing Before Launch
The insurer’s team built the round 1 findings into the chatbot. As launch approached, it asked Marketade for 2 more rounds of usability testing. Both rounds were remote, moderated sessions on the working build, with 6 participants each on phones and desktop browsers. Each round tested the fixes from the round before.
Round 2: Testing the working build
Goals. Round 2 was the first test of the chatbot as a complete, working experience. The team wanted to know whether the changes made after round 1 had helped, catch any show-stoppers before release and line up issues to fix after launch. The round kept round 1’s research questions so the results could be compared.
Approach. We recruited people who weren’t customers, some of whom had recently gotten a quote from another insurer. Between them, participants used iPhone, Android, Mac and Windows across 4 browsers. In 30-minute remote sessions, each one got a quote as they normally would while thinking aloud, then answered the same follow-up questions as in round 1.
Outcomes. Round 2 produced 7 findings and 7 recommendations. Four of the findings apply to AI features broadly:
- The conversation felt slow and long. Participants found the assistant slower to respond than they wanted, and most said it asked too many questions. One said she would have left before finishing. We recommended faster responses and fewer questions.
- Some questions felt too personal for a bot. Several participants balked at giving details like their job title and Social Security number for a quick quote from a chatbot. We recommended dropping the questions that weren’t essential.
- The personality read differently to different people. Some described the assistant as straightforward, thorough and no-nonsense. Others heard it as bubbly and light.
- People expected to fix mistakes by talking to the assistant. Most participants still didn’t know how to change an earlier answer. They guessed they might type the correction into the chat, tried the menu, or said they would call or start over. We recommended a clear cue on each answer, such as a small pencil icon.
Other findings covered comparing quotes and the detail people wanted before buying. Most participants said they would still call to confirm before purchasing.
Round 3: Confirming fixes and testing the purchase flow
Goals. Round 3 checked 2 fixes made after round 2: faster responses from the assistant and a more visible search field in a long list of options. It also added the purchase flow at the end of the quote, with a new research question on whether people understand how to buy a policy.
Approach. We ran another 6 remote sessions on the working build, across iPhone, Android, Mac and Windows in 4 browsers. Participants got a quote and continued through the purchase.
Outcomes. Round 3 produced 5 recommendations, and it confirmed what round 2’s fixes had achieved:
- The fixes worked. All 6 participants found the quote quick and responsive, and said it met or exceeded their expectations. Once the search field had a subtle highlight, most participants found it.
- Reactions to the personality were mostly positive. Participants described the assistant as straightforward, direct and encouraging, though several also found it robotic.
- Editing was still a problem. Most participants missed the edit option again. The one who found it liked being able to jump back to an answer without scrolling through the whole chat. We recommended testing an edit option that highlights when people scroll back up.
- Price changes needed an explanation. In the purchase flow, participants were surprised when the price changed after they had chosen a plan, and they wanted to see totals and fees. One pointed out that a person could simply have explained the change.
What 3 rounds showed
Iterative testing let the team see which fixes worked and which problems needed another approach. Response speed and the search field were solved within one round. Editing earlier answers came up in all 3 rounds, a sign that people in a chat need a way to correct course that they can’t miss.
About the Project
- Industry: Auto insurance
- Platform: Chatbot on mobile and desktop web, from prototype to working build
- Audience: Prospective customers shopping for auto insurance
- Methods: Iterative moderated usability testing and collaborative analysis
- Stakeholder teams: Digital strategy, mobile, UX and development
- Company size: More than 25,000 employees
More Case Studies
How Podium Used Mixed-Methods Research to Guide AI Product Strategy
Marketade conducted mixed-methods discovery research to inform the product strategy of an AI software company serving 100,000+ local businesses.
Helping an AI Digital Health Team Move Toward Product/Market Fit with Mixed Methods Research
Marketade conducted strategic research for a digital health team using a 150-person survey, 12 interviews, and a persona workshop to explore product-market fit.