A real estate company adopts a culture of research and testing, and within months achieves a 180% increase in leads.
Challenge: Renter Leads Down 60%
The multi-state apartment finder company called us in a panic. They had overhauled their site a few months back, with a new design, new content, and a new property search flow.
Some metrics were up, but the most important ones were down. Their #1 KPI, renter leads, had declined rapidly over the last few months, and was down 60% over prior year.
Solution, Part 1: Customer Interviews, Personas, and Usability Testing
One of the first things we did was conduct 1:1 interviews with the company’s leasing agents and customer support staff. We quickly gained an understanding of their renter customers: their behaviors, pain points, questions and concerns.
We prototyped 2 primary renter personas — along with 2 sub-personas — that we would refine over time as we talked to actual customers. Early on, we saw that new and improved content was a big opportunity with the site, so we had our UX writer Nora Fiore lead the customer research and persona creation.
Next we started usability testing the site with people who were in the market for a new apartment. The stakeholder team observing the sessions included the real estate firm’s marketing director and CEO, along with their design and development team.
Over the course of multiple rounds of testing, the research team identified over 50 problems with the site and web app. For many of these, we turned to Google Analytics to validate and size the problem.
We collaborated to identify initial solutions for most of these problems. To help prioritize the work, we gave each solution an “Opportunity Score” of 1 to 10, which combined separate sub-scores for Impact and Effort. We ended up with five “9”s and fifteen “8”s. Of these, we identified a number of solutions worth A/B testing.
Solution, Part 2: 7 A/B Tests Launched in 8 Weeks
The A/B testing ideas were the ones the team was most excited about. They included the most ambitious changes, as well as some that we hoped would be quick wins.
Over the years we’ve learned a surefire way to kill energy and momentum around a research-and-testing program: saddle already-busy developers with all the solution implementation work.
So we used Optimizely to implement all of the tests, allowing us to bypass the developers. We launched the tests in waves of 1 to 2 at a time every couple weeks. By the end of 8 weeks, we’d launched 7 tests. 3 of those were purely content changes; the other 4 included content and layout/design changes, including some that affected multiple pages.
We tracked all experiments in a shared spreadsheet that included short names for each design variation, links to screenshots, and a hypothesis for each experiment.
A 2nd thing we’ve learned over the years: even though it’s exciting to launch A/B tests and get quantitative results, you can’t drop the qualitative user research. As you’re launching the experiments, you must circle back to moderated UX testing to observe users interacting with the new designs. Qualitative research might show you why a new design is failing. It might suggest ways that a winning new design could be tweaked to perform that much better. Or it might open your eyes to a completely new solution.
Result: 180% Lift in Renter Leads
Of those first 7 experiments, 4 saw no significant change one way or the other. The other 3 each achieved renter lead conversion rate increases of over 25%.
The team was encouraged by these results. As the experiments passed surpassed 90% or 95% confidence levels, we passed the variations onto developers to implement as the new control. But as we looked more closely at the numbers and annualized the increases, we realized we’d still be well below last year’s lead numbers even with all the improvements in place.
That’s when the return to qualitative research saved us.
It was in one of the follow-up usability testing sessions that something struck the marketing director. “Renters seem to understand what we’re offering better now,” she said to the team. “And they’re more engaged. More of them are jumping into the apartment search functionality and having a better experience there. But it’s still not a great experience. And we know from analytics that most people do a few searches and then bail.”
“Our online inventory can’t compete with Apartments.com. But once we get someone connected with a leasing agent, they have a great experience and usually find an apartment. What if we tried removing the apartment search option completely, and just relied on all that great new content we’ve created to get them excited about working with one of our agents?”
It seemed like a crazy idea. And the rest of the team was skeptical.
The group’s assumption to that point was that the only path to renters becoming leads is after they do some property searches on the site. These days, how could you be a web-powered apartment finder company and not offer the ability to look for apartments?
If the team hadn’t just watched a bunch of users have a sub-par experience searching for an apartment on the site, the idea would have gone nowhere. But as the marketing director reminded us of what we’d just watched, the idea started to seem less crazy. And discussion shifted to “How much work would this take to test?” and “Could we do this in Optimizely?”
Property search functionality and calls-to-action were integrated throughout the site, including in the navigation, so it seemed at first like a daunting task. As it turned out, it took less than a day to launch a new, search-less version of the site within Optimizely.
It took less than a week to gain 95% confidence that the new design was a big winner. The percentage of site visitors to convert into a renter lead shot up from 1.5% to 4.3% — a 180% improvement — putting the team on track to erase their lost lead volume and then beat past performance.
We emailed the results to the team. The CEO responded enthusiastically and publicly thanked his marketing director: “It was a brilliant idea!”