We recently interviewed John Nicholson about his key performance indicator (KPI) work with AARP.
Marketade: A while back you led a KPI discovery exercise with AARP. What part of AARP was this for?
John: They had recently launched their Life Reimagined site and mass media campaign. They asked us to help them identify KPIs for that business unit for the upcoming calendar year.
What prompted them to reach out?
They were having trouble reaching consensus internally and wanted a more objective outsider to take a look at their business and help them identify KPIs that better withstand scrutiny from their board.
Got it. So how did you start the process?
The first thing we did was get buy-in on using a collaborative process. We knew that if we went off and did our own research — even if it involved stakeholder interviews — and came back to them to present our recommended KPIs, the process wouldn’t work well. One, we’d have trouble getting buy-in and consensus if they weren’t part of the process, And more importantly, because we wouldn’t be tapping into their expertise as well, we’d be more likely to pick the wrong KPIs.
Was it tough to get buy-in on a collaborative process?
Not in this case. Once we explained the process, they were excited about it and they didn’t have much trouble getting all the right people in the room for 2 sessions spread over a couple weeks. This group was eager to have a say.
Tell me about the first session.
It was a 2-hour session. We had about 10 people there, mostly from the Life Reimagined unit and one or two from central corporate offices that oversaw strategy. From LR, we had their chief marketing officer and a few other marketing people. We had their main research and analytics person. We gathered around a table in a room that had the right supplies: lots of wall space, sticky notes of different colors, markers, etc. I started off by giving an overview of the process, which was a combination of Avinash Kaushik’s KPI measurement model process and Jared Spool and UIE’s affinity mapping process (aka KJ method). Avinash’s approach provides the overall structure and goals (we use a lighter weight version than what he uses), but it doesn’t tell you how to reach consensus with a diverse group of opinionated stakeholders. When we tried that in the past using traditional brainstorming and discussion techniques, it failed. So we borrowed a page from our user research playbook, where after doing things like usability testing, we’d seen how effective affinity mapping was at reaching consensus and prioritizing the most important findings. This was one of the first times we applied affinity mapping to a KPI exercise.
Great, so walk me through how that combination worked.
First, we used affinity mapping to identify core business objectives for LR, answering the question “why do you exist?”. We reached consensus on 3 pretty quickly. Then we used affinity mapping to identify 1 or more KPIs for each business objective, answering the question “how will we measure progress against this objective?”. This part took a while because we had a huge volume of potential measures on sticky notes. As we often see, people have a hard time remembering the “Key” aspect of KPIs, and so we had a lot of metrics like visits and time on site that lacked outcome focus. But affinity mapping worked its magic, and because we’d stressed “outcomes” early and often, the group was able to reach tentative consensus on a solid list of KPIs.
A lot of metrics people have probably never used affinity mapping. Talk about the broad steps you used to reach consensus.
Right. So the steps in this case were: 1) brainstorm individually and add each idea you have (for a business objective or a KPI) to a sticky note; then everyone adds their stickies to the wall at once; no names attached; 2) group the notes into categories — so for instance, if there are a bunch of objectives related to revenue, you group them together; 3) name the groups on separate sticky notes above the groups; this part is always hard, but is critical for getting people to process all the notes and groups); 4) vote on the top groups; 5) order the top groups based on votes and identify the top 3.
Earlier you mentioned the challenge of keeping the group focused on outcomes. Were there any challenges with this group?
Keeping them from talking. Nearly everything I described earlier was done in silence. A big part of the reason affinity mapping works is that we’re avoiding a shouting match where loud voices and senior voices win. We were able to do it, but it took a lot of reminding, including reminding some senior-level people. If we hadn’t done this a lot before and realized how important this part is, we wouldn’t have been as disciplined and it would have devolved into the typical frustrating meeting. And there’s no way we would have reached consensus within 90 minutes.
OK so that was session #1. What happened in session #2, the next week?
In session #2 we recapped where we’d arrived on business objectives and KPIs, and gave people an opportunity to raise questions and concerns. Large ones we discussed on the spot, small ones we identified a plan to investigate after. Then we covered *how to track and count the KPIs*. Which data source or analytics tool? Was it set up to track this behavior already? If not, could it be? And finally we covered *initial KPI targets*. Since we didn’t have good baselines in place for many of the KPIs, we created a plan to quickly set up tracking and develop baselines so our targets would be better informed.
Anything you’d do differently based on how it went?
Since running this, we’ve discovered and used Google’s Design Sprint. It has a lot of overlap with Jared’s affinity mapping approach. One way it differs is by having a distinct role for the Decider. While affinity mapping is wonderful at elevating all voices in the room and leads to much better results than a typical meeting, it is sometimes too democratic. In reality organizations have hierarchy and it’s better, we’ve found recently, to strike a balance and give more authority to the ultimate decision-maker. It also helps to move things along during the session when the group is divided.