A Focus on Marketing Research
When businesses want to find answers to questions in marketing, whom do they ask? Do they set up experiments to test their ideas, pitting the approach they think is most effective against alternatives? Do they survey consumers on a large scale? Do they go to experts who have questioned and requestioned their theories? Surprisingly, the answer is no. Most often, businesses rely on small “focus groups” to answer big questions. They rely on the intuition of about 10-12 lay people with no relevant training who ultimately have no idea what they’re talking about.
I wonder how can this be a useful strategy? Why ask those who are lacking any kind of proficiency when, by definition, experts are more knowledgeable on the topic and have experience that could actually be beneficial? And even if experts are more narrowly focused, and tunneled vision, how can this be better than carrying out their own research?
Research in psychology and behavioral economics has shown time after time that people have bad intuitions. We are very good at explaining our behavior (sometimes shocking and irrational), and to do so we create neatly packaged stories – stories that may be amusing or provocative, but often have little to do with the real causes of our behaviors. Our actions are often guided by the inner primitive parts of our brain – parts that we can’t consciously access — and because of that we don’t always know why we behave in the ways we do; still, we can compensate for this lack of information by writing our own versions. Our highly sophisticated prefrontal cortex (only recently developed, by evolutionary standards) takes the reigns and paints a perfect picture to explain what we don’t know. Why did you buy that brand of fabric softener? Of course, because you love the way it makes your clothes smell like a springtime breeze when you pull them out of the warm dryer.
So, why do businesses go to our imagination when we know it’s just a cover for what’s really going on? Indeed, why do businesses go to the imaginations of a group of people to find real answers? I suspect that the story here is linked to another one of our irrationalities: As human beings, we have an insatiable need for a story. We love a vivid picture, a penetrating example, an anecdote that will stay in our memories. Nothing beats the feeling of knowledge we get from a personal story because stories make us feel connected – they help us relate. Just one example of customer satisfaction has a stronger emotional impact than a statistic telling us that 87% of customers prefer product A over product B. A single example feels real, where numbers are cold and sterile. Although statistics about how a large group of people actually behave can tell us so much more than the intuitions of a focus group, the allure of a story is irresistible. Our inherent bias to prefer the story compels us to believe in the worth of small numbers, even when we know we shouldn’t.
This “focus group bias” is not just a waste of money it is also most likely a waste of resources when products are designed according to the “information” gathered from these focus groups. We need to find a way to base our judgments and decisions on real facts and data even if it seems lifeless on its own. Maybe we should try and supplement the numbers with a story to quench our thirst for an anecdote, but what we can’t do is forget about the facts in favor of fairy tales. In the end, the truth lies in empirical research.
–
My latest book, The Upside of Irrationality, explores some positive and some negative ways that irrationality plays out in our lives.
Great, I am very happy that you deal with this topic. Harvard Professor Gerald Zaltman wrote a nice book on marketing in 2003 “How Customers Think” where he directly attacked the use of focus groups.
With his 6 Marketing Fallacies he made a start at breaking down some long-held -and obviously flawed- habits of marketers.
Too bad Zaltman was mainly concerned in pushing his own patented marketing research method ZMET.
Until now I don’t see much change in the use of focus groups. It seems to me that marketing managers are indeed less interested in the truth and are satisfied with anecdotal data that helps persuades their bosses.
Dan
isn’t the problem not with the focus group or the use of focus group but with what the marketers do with the results. When we know nothing about a product or customer preferences, the focus groups give us some hypotheses (some extreme and may not reflect the rest of population) that we can go test with a wider population?
-rags
-rags
Having been in so many ‘focus groups’ at several different stops on my career path, my biggest problem with these is the “group think” that can occur. It’s often difficult to have someone speak up who disagrees with where the group opinions seem to be going. (I, though often strongly opinionated, will frequently observe and process my differences internally rather than with the group. When I do dissent, one of two things frequently happen. First, there’s almost always complete silence when someone disagrees. Second, there’s usually several people who come up afterwards, thanking me for the different opinion that they shared but didn’t want to bring up.)
How this transfers out of any kind of strategic planning into consumer preference, I’m not sure. I do know that any product I’m passionate about isn’t based on any statistical analyses presented to me… pretty though the charts might be.
A friend had a high level post at a world-renowned ad. agency for many years. She told me that the focus group organisers always asked the subjects if they had liked the music on the TV ad that they had just watched. They almost invariably said yes . . .
Even when there wasn’t any . . .
It became kind of a standing joke at the agency.
This is changing, at least in the web world. Companies are using A/B testing on their sites, and building systems to automate testing so the internal cost of testing is very low.
http://en.wikipedia.org/wiki/A/B_testing
Love the new blog, but your rss feed – what we use to keep track and read every post from your blog – is currently only giving me the last articles from 2009 (“2008 was a good year…” etc)
Just thought you’d like to know before the 40-something thousand followers who’ve been redirected here by your old site’s post subscribe and don’t get new updates. Hope this can be fixed soon, and glad to hear Arming the Donkeys is back!
The new RRS feed is: feed://danariely.com/feed/
Isn’t the big question about a focus group one of sampling? Is it representative of the target population? But what a focus group “can do that surveys can’t is allow you to see how people interpret things. The lesson I draw from the “How did you like the music?” anecdote is not that focus groups and the people in them are stupid but that questions, even simple ones, don’t always mean what we think they mean. (And, Dan, “reign” doesn’t mean “rein.”)
Sorry Dan, you’re way off the mark here.
First of all, qualitative research is one of many tools to help provide direction for marketers. With limited research dollars it is not ideal to base decisions on groups alone, but if the research is conducted well, it’s certainly better than the gut feelings of an individual (usually a business owner or senior executive) – which will always be more biased than consumers.
While I’ve seen many of my clients misuse and abuse focus groups, they are still only one piece of a larger effort to gather insights, nearly always dominated by quantitative work.
Second, the notion that anyone is more “expert” than people who use a product is absurd.
While I could go on and on about this…I think the most critical point here is that real, in the moment, high-touch human contact is essential. It is about imagination and emotion. Brands are always about something “bigger” – like the “springtime fresh feeling you talk about.” Who makes decisions based on facts? No one.
How else could Sarah Palin be taken seriously by so many people? It’s not that they’re stupid (though many are!), but that she taps into strong feelings of anger, fear and discontent.
Whatever those feelings are, that’s the job of a great brand (like it or not) – to tap into deep seeded emotions. Observing those emotions come out in the moment is priceless – not just what people say, but how they say it.
Done well, focus groups can be epiphanies.
Hi Jeff,
Done well — and in addition to other data — maybe they could be useful.
But most people use them as the main source of data and assume that they are very informative…
This is what I worry about
best
Dan
Hi Dan
Given your comment above, it seems you are more worried with the interpretation of groups, than the groups themselves.
Marketing research might be conducted for a multitude of reasons – for some of these reasons groups will be a valid exercise, but for others they won’t.
Qualitative research is by its nature illustrative and exploratory. It can be a first stage to prompt further thoughts by “experts” or create hypotheses to be more rigorously tested.
I think it is quite arrogant to assume “experts” should be more trusted. “Experts” may have certain skills or experiences but these are likely very different to the majority of the population – only by speaking to the “lay people” can we reconcile the theory with the practice
Best,
Simon
Hi Dan,
I hope you’re doing well. A couple of notes here…
I can appreciate the set up to emphasize the importance of examining the utility of focus groups, but I think your argument starts off a little strong:
“Do they set up experiments to test their ideas, pitting the approach they think is most effective against alternatives? Do they survey consumers on a large scale? Do they go to experts who have questioned and requestioned their theories? Surprisingly, the answer is no.”
This just isn’t the case. Large sample studies with experimental design are commonplace in marketing research. In fact, we’re engaged in these types of studies everyday. And advanced firms have even begun to implement some of the best coming out of the behavioral sciences (implicit association testing, deliberate decoy effect testing, emotional inputs into choice algorithms). Granted there is a lot of work to be done in transitioning the rest of the industry to more advanced experimental design, but many firms are already practicing these methods.
I couldn’t agree more with your emphasis that:
“We are very good at explaining our behavior (sometimes shocking and irrational), and to do so we create neatly packaged stories – stories that may be amusing or provocative, but often have little to do with the real causes of our behaviors.”
I am constantly beating this same drum (www.sentientinsight.com/happiness-is-a-warm-face).
But this is also where I think the conversation on focus groups needs to go to the next level. It’s not that focus groups are inherently bad, but rather it is a question of how the information is gathered based on what you are seeking to understand. Explicitly asking consumers to self-report ‘why’ they like a product (whether in a focus group or in a quantitative survey) is no longer appropriate given what we know from the behavioral sciences. But using indirect measures and projective techniques with a group of consumers as an exploratory phase in multi-stage research design, to generate hypotheses for testing, is actually good practice. It’s akin to a brainstorming session and allows for the discovery of potential drivers of behavior that we may not have thought of otherwise. The problem isn’t with asking consumers questions, the issue is with how you ask (or gather) and what you are trying to achieve with the information.
I think your analysis on why traditional focus groups are so compelling to marketers is spot on. I can’t tell you how many times I’ve seen an executive’s face light up when some quote from a consumer confirms something s/he was thinking. A single customer quote can be more compelling than a 1,000 points of data. As you say, this is the essential challenge for the industry – how do we create greater reliance on sound empirical data? Our solution to date, has been to use qualitative research as an exploratory hypothesis generating phase, complete an experimental testing phase to gain our conclusions, and then utilize quotes, videos and imagery from consumers to reinforce the empirical data.
To take this to a meta-level of analysis, as behavioral scientists, how can we make the importance of this argument more compelling to the rest of the industry? Perhaps the answer is in stories. What if we created a repository of examples of how new product launches based solely on the use of focus groups asking explicit questions have failed? (I know of a great one for an HIV testing kit that people said they would buy when it came on the market…that was an ultimate failure). And what if we had similar stories of the unbridled success of products that were borne from sound experimental research design? If people are going to make decisions based on stories, let’s give them stories (grounded in empirical knowledge).
I met you recently and asked you about this very topic. At the time you said, “I suspect focus groups are more confirmatory than enlightening.” This has stuck with me and I am still pondering it. What I have seen often with focus groups is that people observing them anchor on one or two moments and then the “example becomes the rule.” It is very hard to get observers to consider all the information gathered in a group or groups. These anchors can serve to justify previously held beliefs and solidify ideas that existed before the data was collected. As someone employed as a market researcher, I want to be sure we use it to learn new things so we can make better decisions. I often worry that we use market research to justify decisions we’ve already made. If you have ideas how to prevent this anchoring effect, I would love to hear them!
“I often worry that we use market research to justify decisions we’ve already made.”
That is inherently human behaviour and not sure this could ever go because, well, we’re irrational.
Like someone mentioned above, I think technology used with concepts like Crowd Sourcing and Creative Problem Solving is the future of how we honestly and transparently source customer feedback.
So there are ways but the issue here is whether or not those CEO’s who want their own way want to actually listen to the people and what they want.
I agree “focus group study” is for sampling, which isn’t sufficient for product decision.
And if we’re talking about software product especially a new one I believe the three elements of Core Functionas, Usefulness and Ease of Use, and Target Customers must mingle, play together well and speedy iterations…
Like in most things in life this is not a binary answer. qualitative market research, i believe, can be a very powerful tool if the research objective and the technique is right. i do believe you cannot research and reveal the next need or create innovation but you can research and get a fair sense of the type of emotional chords some of the marketing elements are striking. I am personally not a marketeer and researcher but my wife is – hence this is a slightly aware view not a biased view. The key issue in research is that it loses effectiveness when its self serving… a lot of the stuff that you have written in your PI book like anchoring etc obviously vitiates the response – however the trick is to get the nuances through surrogates. I havent done or heard of controlled research to prove the same but some of the techniques i have heard from my wife seems to make sense and should lead to decent conclusions – but as i said of the more straightforward issues (e.g comparison of advertising campaigns; packaging connotations) and not of the NEXT product.
http://www.guernicamag.com/interviews/1782/illouz_6_1_10/
I don’t do email, and have no reason to believe that either party is even remotely interested in what I might have to say; nonetheless, I’m compelled to do some match-making.
Dan, meet Eva.
Hi Dan,
A colleague of mine recommended your book, Predictably Irrational, after we began speaking about a Gen Y loyalty in wine purchasing study that we have begun work on. I had been reading Levitt and Dubner’s new tome, and after citing a couple of facts, she mentioned that I might be interested in your book. I’m currently reading Ben Goldacre’s ‘Bad Science’, and it’s from my recollections of that book that I fear some of the content being written here by your community.
I have to start by pointing out that I am a wine marketing academic who lives in Dijon, France. I came into academia after 20 years in the wine and hospitality sector. The attraction to academia for me was that practitioners tend to ignore marketing science; and for their part, academics spend too much time sulking about how practitioners ignore science. So, I wanted to start by changing that.
This discussion is about focus groups, and I couldn’t agree with you more. At least one of the world’s largest retailers (I am not permitted to say, I’m sorry), does not allow the use of focus group results as a support for why a new product should be listed in store for exactly the reasons mentioned above. Baddeley (2006) discussed the sense of reciprocation that the [focus group attendee] feels when receiving a gift of an honorarium for attendance at the focus group. Accordingly, just like the problem that quant scientists suffer with respondents answering questions just because they feel a need to supply an answer, our focus group attendees feel that they need to support the product presented to them simply because they’re a part of the focus group.
In my opinion- any good researcher SHOULD KNOW THIS. Consequently, any focus group being conducted to ascertain the generalisability of the level of acceptance of a product is both an example of poor science and a waste of time. As mentioned above, a focus group should be used to generate hypotheses from an informed sample of respondents.
In 1999, for my honours thesis, back in Australia, we investigated the wine trade’s perceptions of alternative closures to natural cork, in order to find out how screwcaps and synthetics could look for ways to combat the virtual market monopoly of the natural cork. We hand-picked members of the wine trade with the most knowledge so as to ascertain the range of considerations that might require attention in an effort to encourage product adoption. The results fit beautifully as a series of considerations for testing in line with product adoption theory, going back to Perry’s work on the curve, and fitting in nicely with Urban’s modern contributions on consumer advocacy.
Focus groups worked brilliantly for this purpose, but should never be used to ascertain any sense of applicability to a market in general. For generalisability, you need a quantitative method.
Dan, it’s clear that your page stimulates both interesting and valuable debate among the marketing community. Long may it continue.
Damien Wilson, PhD
Director, MSc Wine Business
Burgundy School of Business
Dijon, France.
I worry that this post represents a rather lopsided view of Market Research which actually employs a vast range of qualitative and quantitative techniques to help clients add consumer perspective and data to their marketing strategies. Whilst I agree that focus groups are beset with a multitude of inherent flaws they can also provide rich context and insight for clients.
I’ve worked in market research for the last eight years, having previously worked on academic research studies, and believe the best approach uses multiple sources of data – qualitative and quantitative – and devises a tailored solution for the client’s needs.
Your post makes two assumptions: that focus groups are the only source of data used in market research and that they are populated with just anyone. Both of which are rarely the case in market research. We think carefully about what information and data clients already have and who they should talk to in order to further explore their issues.
Most recently I have been working with a client on the challenge of prompting behaviour change across a host of transports modes and situations. I have read your book and blog for information on human behaviour. One of the key challenges we face is unpicking the distinction between what people say and what they do – whether talking about road safety decisions or switching to a more sustainable mode of transport. We constantly look at our research findings against the backdrop of behavioural economics and academic thinking to contextualise our findings.
We use observation sessions, ethnography, in-depth interviews, quantitative surveys and experimental design alongside focus groups to help clients answer marketing challenges. We currently have a department that evaluates the effectiveness of experiential marketing using a control cell and experimental design.
However, focus groups, and other qualitative research add a valid consumer voice to the process of decision making. We have conducted a host of studies on teenagers and I would argue that a focus group with teenagers whilst not without its own flaws is more robust and representative of their mindset than the assumptions of the marketing team, and I think our clients would agree!
In my business and in economics (thanks to Nassim Taleb in The Black Swan) you don’t get sales and clients by looking the facts in the eyes. In fact congress is looking to fund a stimulus bill for builders to build more houses when there are 19 million empty houses already in the US with only 2 million of those on the market.
We are not rational and I work in the most irrational of all humans.
http://qedrealestate.wordpress.com/2010/06/07/goodbye-v-shaped-recovery/
Placing too much faith in focus groups is indeed a good example of one of the most common types of fallacious thinking in use by Americans today: proof by anecdote. You see this on the Oprah show, you see it on what passes for “news” programs, and yes, you may even see it in market research.
A person or small group of people are interviewed about a personal experience. (“I’ve been eating blueberries for breakfast for three months now and my sex life is so much better!”) Often no other data is presented, or there may be some sloppy and/or unexplained references to scientific research. Yet, it is obvious that we are expected to draw broad conclusions from this “data”. The tragedy is, most will.
Just another argument for sound basic instruction in applied science and statistics (real life stuff), for every schoolkid.
Ben Saurez writes that he conducted tests with a couple dozen or so people, and a dozen or so products.
Each person would get $100 (this was in the 80s when that was Real Money).
They could “buy” whatever they wanted. And keep it.
What people bought, that’s would Suarez would build for direct marketing.
Usually worked, and well.
Dan -
this is a very controversial idea that you lay out here, and it has been admirably responded to by both sides of the focus group fence ( so to speak!). I agree with Kat and think it needs to be recognized that focus groups are rarely the ONLY avenue that a company is acquiring information about its product. Different kinds of market research provide different shades of information, depending on the what you are looking for. Ideally researchers are gaining holistic research that recognizes and responds to the sensory, emotional, economic, and social desires of the people buying, the consumers. And there are all sorts of exciting ways that this is being accomplished in a more integrated way in ADDITION to focus groups, as they provide a certain shade of information about how people respond to a product as a group. But most often companies are on a tight budget and holistic research is not gained because of the financial restraints.
InsightsNow, a holistic research company has a webinar series “WebinarsNow” that covers this topic in detail. Their November 2010 Webinar “Overcoming Budget Challenges – Research Programs that Deliver More with Less” is particularly relevant to this topic. You can register or just look around at http://www.insightsnow.com/resources .
Hope this helps!