How to Design Research That Keeps Paying Off

In research you encounter many people who are entirely forgettable. And then there's the people who your team is still talking about weeks or even months after fieldwork has wrapped.

It makes logical sense that you'll find very average, unremarkable people in the pursuit of generalizable insight. But in my experience, it's the remarkable people that are still being discussed later, as teams make strategic choices, that end up having the most commercial impact.

Everyone knows that some people are more convivial, engaged, or outgoing than others. That's just human nature.

But what is it about memorable respondents that is fundamentally different, and how can we systematically design research so that we're increasing our odds of finding these people?

Because ultimately, research is only as useful as its ability to provide us with the fuel that powers the critical decisions we need to make. And those memorable respondents are rocket fuel.

Quality begins with motivation

Someone who is motivated by curiosity or personal interest will always give you more than someone who is just there for the money.

There's a paper in the international journal of market research comparing the participation motives of professional and non-professional respondents when it comes to survey completion. As you might expect, all people have both intrinsic (eg. curiosity, helping others, topic interest) and extrinsic (eg. financial incentives, obligation) motivations for participating in research. Both have a role to play and are ever-present.

But, when the authors did a forced-choice comparison of what gets panel-derived respondents and real people into a survey, they discovered their primary reasons for participating are vastly different.

Money is a precondition for the majority of panel people, but a top priority for none of the groups of non-panel people they looked at. The TL;DR of the paper is that while professional respondents are driven by a mix of intrinsic and extrinsic motivators, real people are driven primarily by intrinsic motivators.

Other research has shown that intrinsic motivation and quality are linked. Intrinsic motivation is what predicts the quality of what people contribute or produce, while extrinsic motivators mostly predict quantity.

In the research world, this is why we tie payment to output, like survey completes. That's what drives volume. We deliberately use a reward structure that weakens the link between the study and intrinsic motivation, and therefore quality as well. It's a conscious trade-off made in service of getting the survey completes you need in a reliable time frame.

This trade-off is why we so often recommend mixed method approaches. First, hand-pick intrinsically-motivated people who generate deeper insights in qual. Then, let surveys do what they're best at: demonstrating how that insight generalizes.

Finding great people to do the first part can be done haphazardly, or it can be done with intention and care. Here's how we think about it in our work…

"Creative and articulate" is not enough

Every recruiter or research vendor does some form of articulation or creativity screen as part of how they vet the people who show up in your research. But in my experience, this isn't enough. Most research firms are preoccupied with ensuring that talkative and palatable people end up in front of clients – but that's as deep as it seems to go.

When you go about recruitment this way, it becomes about how someone performs in the format of research. This advantages practiced respondents – people who are pleasant to observe – rather than being the kind of people your team is still quoting three months later.

This is bad for everyone involved, because as it turns out, those memorable respondents are also more likely to make commercially useful contributions to research. Another paper identified that people with what they called "an emergent nature" are able to develop product concepts that mainstream consumers find significantly more appealing and useful compared to those developed by others – including lead users.

Ultimately, what sets them apart is how they process. Not their level of creativity, nor their strength of opinion. It's that they can dual-process. They experience things intuitively, as most people do when you put stimulus in front of them, but they can also reflect on that experience rationally. They can stand outside their reaction, and reason about why, where it leads, and what that means in the context at hand.

As a bit of an ironic aside, this reflection is something focus group moderators are generally trained to coach respondents out of (eg. "I want your opinion, not your thoughts on other peoples' opinions" or, "Don't overthink it, I just want your gut reaction").

Anyhow – this dual processing runs on a mix of underlying traits, including reflectiveness, creativity, openness to experience, and strong visual imagination. It's about more than someone being articulate and creative in the most general sense.

Ultimately though, this emergent nature is something you can screen for. There is a peer-reviewed and validated scale for it, which means finding those commercially impactful "star" respondents can be a deliberate design decision rather than a lucky roll of the dice, as it is with big panels and research firms.

Finding the right people

We look for both of these things in our work, going out of our way to find people who are either intrinsically motivated, or emergent in how they process. Ideally both.

The first practical consequence is where you look, because you're far less likely to find intrinsically motivated people sitting inside a panel, self-serve or otherwise. You can identify and recruit people from communities, clubs, comment sections, and referrals just as quickly as you can from a panel – there is no difference in speed to field. We've done 50-person FMCG recruits, our way, in a week's time.

Someone who has left a detailed comment on reddit about mobile network reliability is going to be far more informative than an average Jane/Joe if I'm trying to understand the consumer mental model for that concept. The same goes for every type of FMCG innovation I've come across in my journeys. Even the most general and mundane topics can be intrinsically-motivating for people if you approach things in the right way.

The second practical consequence is what you listen for once you find people. Motivation and emergent processing are easily discernible inside of a fifteen-minute conversation, which we have with every single qual respondent in all of our engagements.

A question as simple as, "tell me how these products fit into your life" provides a big enough opening to talk, and get to the heart of whether or not someone is intrinsically motivated by the topic at hand.

In our experience, people who exhibit emergent processing are also usually very nerdy about something, so you can also get a good sense of how they think by asking them questions about the things they spend their energy, money and time on.

The typical articulation and creativity screen, which is almost always asked through the lens of the category being studied, with a frame very close to the objective of the research, is functionally useless. It's not at all a reliable predictor of how useful someone's contributions will be in a research project.

Putting it into practice

In the relatively short history of Faster Horses, the one thing that has consistently surprised, even me, has been how consistent the insight we get from hand-picked respondents in qual is with what we see in follow-on quantitative research. Our initial insights are almost always supported by what we then see if our survey work.

Concern about bias is the natural worry that tends to emerge when I speak with people about how we design research. But let's be clear: There is no neutral, "unbiased" alternative we're foregoing here in service of our approach. Traditional panel recruits are designed for availability and incentive-responsiveness. That's it. There is nothing about them that makes them somehow more representative of real people, or capable of leading you toward generalizable insight, than if you really lean in to the recruitment process and treat it like a design exercise.

Regardless of the approach taken, you're never choosing between a selected group and an unselected one. You only choose what you're selecting for.

You can select for signal intensity. You can select for the political safety that comes from choosing a big consultancy, or research firm. But you can't select for generalizability. Not in qualitative research.

If your goal is to achieve meaningful differentiation, break new ground, or make choices about your organization's future, choose signal intensity first. Seek generalizability second. That's how you end up with people on your team repeating what someone said months after your research finishes, and the numbers to back it all up.