Experiments? Not!
A few days ago we wrote the nice people at Whole Foods about some ideas for doing experiments together. These were going to be experiments on taste perception, willingness to pay, and on how we get people to eat healthier and enjoy more fruits and vegetables.
Here is the official response we got back:
Thanks so much for considering Whole Foods Market for this opportunity…. While the concept is quite interesting and the subject matter is aligned with our stores, we will unfortunately be passing on this opportunity at this time.
Thanks so much for considering us and please don’t hesitate in reaching out to us in the future.
Best regards,
Michael
What annoys me is the combination of ”aligned with our stores” and “no.” And of course it is not just Whole Foods that is in this boat. Companies in general are willing to spend lots of money on consultants, they are willing to spend lots of money on gambling that their intuitions are correct, and sometimes they even pend money on focus groups. But, when it comes to testing things empirically, the typical answer is “interesting, but not for us.”
I suspect that the reason for the reluctance to engage in experiments is that no one (for example Michael from Whole Foods) wants to take any risks. Everyone just wants to do their job. Because of that companies continue to behave in the same way without taking any new interesting directions…. very sad.
Sorry about this post, but I had to vent somehow.
The Upside of Irrationality, explores some positive and some negative ways that irrationality plays out in our lives.
Even far away here in Germany, I can hear the sadness from your post.
Keep up your very good work, please!
You fail when you stop trying. Go to the next company, and the next… and until you find the one that will do it. At the end — their answer was not unexpected…
Best, S
Yup, I think you’re on the spot. I was just reading how big firms tend to buy equipment, hardware and software that come from the biggest name brands, but aren’t necessarily optimal for the firm. It’s all about accountability.
Perhaps reaching out directly to the CEOs / owners of large privately held companies stands better chances of getting a “yes”.
Perhaps if you offer them confidentiality they will be more receptive. If I were Whole Foods I would be concerned that news of my experimenting on my customers to see how much they would pay, perceive food quality, and to manipulate them even for a perceived “good”, like eating veggies, might damage my brand.
I agree with Clyde. With the addition that the company probably would like to supress some possible outcomes, eg Whole Foods veggies tasting about the same as Walmart veggies.
If you want the cooperation of a big company I think you need to prove that you both can provide them something that either can be used as good PR and that you’re not going to cause them any harm if the results turn out to be undesirable.
You just have to find the right companies. Hewlett-Packard, for instance, has an experimental economist on payroll.
The real problem for them was the “aligned”. Whole Foods are really not nearly as “green” and not about healthy food as they appear. Several exposes about them and their ilk show that.
Brings to mind testing of notions to verify them… Prof. Ariely, test the notion that they are about healthy food. You will find material to sew doubt in your mind about that notion.
This is probably what Michael was afraid of, more negative exposure that they are just another unecological, unsustainable, un-green “box grocer”.
C-L
RE: “aligned with our stores” but “no”
Methinks the problem of alignment might not be alignment with corporate goals but with individual security. For the marketing folks, or the senior corporate leaders, the prospect of empirical research might seem like a big no-win risk for them: it might indicate that they have been doing the wrong things, which won’t look good on them, and even if it shows they have been doing right things, they will feel like they have lost credit for it. Consultants can cause the same level of anxiety, but they usually have enough screening and control over consultants that they get the answers they want to hear.
My guess is that a proposal like this, like many sales situations, requires finding an internal champion to wage the political battles behind the scenes and craft a perception of it that will protect everyones sense of security.
Gord you hit it out of the park. I concur.
Now you have a new research path.
Interesting since John Mackey, CEO of Whole Foods, said it pretty well earlier this year: “The effectiveness of teams is tremendously enhanced when they are fully empowered to do their work and to fulfill the organization’s mission and values. … Empowerment unleashes creativity and innovation and rapidly accelerates the evolution of the organization. Empowered organizations have tremendous competitive advantage because they have tapped into levels of energy and commitment which their competitors usually have difficulty matching.” Given this philosophy you’d think the organization would actually be more open.
My “intuitive” perception is that most companies, especially in documents like their annual reports, express words like these to foster a culture of innovation and measured risk, and the people writing these words might actually sincerely believe them. However, risks or innovations that don’t succeed can be punished severely in the corporate environment, and at a very personal level for those who proposed or were closely associated with them.
I guess it’s a matter of not drilling your good intentions down to a sufficiently detailed level.
A friend recently sent me this charming quote from a 10-K of a US public company:
“…we believe when financial compensation is tied to performance we motivate the right behaviour for our shareholders”.”
Gotta laugh ’til you cry.
Of course, if Whole Foods were truly as cool and “in the know” as they’d like us to think they are, they would be aware of your work Dan, and would not have hesitated to participate.
Try Trader Joe’s.
Trader Joe’s – maybe. Certainly good and mainstream but with the sometimes organic focus.
I’m wondering if local cooperative grocers might be willing – and more authentic and curious about the issue. Many are organized through the National Cooperative Grocers Association. If you want an introduction I’d be pleased to provide one.
I think that this is what I will try next.
Disappointing but hardly surprising. We face similar answers when offering to implement new ways of working that improve productivity and employee satisfaction.
Organizations wish to innovate and stand out of the pack but are un-willing to try new ideas and concepts. They jump on the bandwagon only when the concept has proven to be highly effective but at that point, it no longer gives them a competitive advantage.
Dan,
A lot of it is “Don’t bother me with the facts. My mind is made up.”
Another is “Facts may make me change…and I do not want to change.” “I’m lazy.”
And “I’m doin’ okay. Why bother.”
Could be fear. The dentist and pain thing maybe scared them that they’ll find out something that their customers will not like.
Anyway, keep up your spirits. In Southern California, maybe Trader Joe’s would be interested in your project. They have many n”nutrition-aware” customers.
Gord is right on. Companies don’t take risks, people do. What is Michael’s incentive to potentially look foolish or even lose his job? You’ll have much better luck reaching out to people who will benefit from taking on the risk….maybe the folks over at 37signals? They are into experimentation and probably have a lot of data to work with. Good luck.
Helen is right. Trader Joe’s is very much in tune with the customer and might be willing to jump in. Because they are so successful, perhaps they have already been using your experiments! (given they seldom offer more then a couple of choices!)
Whole foods and others like them are part of a mold designed for corporations and what works for them, not for the customer. Keeping the customers addicted to HFCS, salt, fats and sugar are what they are concerned about, not how to get people back to eating healthfully.
If they cared about the customer, junk food wouldn’t be 50 to 80% of the ‘food’ on their shelves.
Dan,
I’m an economist with professional experience built on “organizational change”. You are not alone! Some days I wonder why I bother, but when the “tipping point” is reached and change occurs it is really incredible.
I’m a big fan of your work! Keep it up!
It seems to me like they left you with the perfect opening for you to follow up. Stay in touch, approach them again, and maybe you can close the deal.
I wonder if they are concerned that you will write about the experiment on your blog or in a book.
You might try Central Market….they are the same sort of store but up-scale. Less emphasis on all natural and more on the gourmet. Privately owned, Headquarters in San Antonio TX. It’s the kind of place I take out of state visitors to. And even though they protest all the way to the store…they always walk out with a big bag of “stuff”. The CEO is Steven Butt, I’m told.
I know the feeling of frustration, but it is unfair to assume too much about their motivations – there could be many reasons they said no. Who knows, maybe Brian Wansink approached them first and they said yes to him…
In any case, try Wegman’s!
Well, most of all, I appreciated the high level of creativity that went into that rejection letter. Positively breathtaking.
Some good comments, but I think there’s more to the idea that everyone ‘just wants to do their job’.
Did the message get to someone who can actually make a decision, or be an executive sponsor? Likely not unless you had an inside channel.
What thoughts might a ‘manager’ have about an unfamiliar technique that they’ve never done before?
What if I feel dumb in a meeting to discuss this with my boss?
What if I can’t get him/her to feel like its their idea?
What if this draws resources from ‘real’ tasks for an ‘ivory-tower’ project?
What if I recommend this and it doesn’t pay off in an explicit way?
I want to be innovative but only in the context of what we regularly do.
I can’t put this on a spreadsheet!
Usually people will prefer familiar mediocrity to unfamiliar possibility of failure. Businesses abhor uncertainty, even if their certainty is artificial.
As the other David pointed out, this is a potentially fruitful research area.
And yes, try Trader Joe’s.
Dan – I am a big fan of your work, but as several of the comments above reflect, I don’t think the problem is that they don’t want facts. They do want facts. They just want to be able to control who knows about them and what is done with them. Grocery stores like Whole Foods (and Trader Joe’s) and Krogers and any large chain base their placement of items on extensive empirical research on how to maximize purchases. Similarly, many of the manufacturers and sellers of the items to be sold engage in research to identify optimal selling conditions, shelf placement, etc. It is by no means merely intuitive. It is just that they would prefer that the general public not be aware of how structured that environment is and to what ends. Moreover, the sellers of some products (candy for example) clearly don’t want information such as you propose to gather, least of all in the hands of some one who might report the outcomes to general (that is not just academic) readers in bestselling books! I will be interested to see if you get any takers.
I read this post and smiled sadly to myself.
As a User Experience expert, I always try to convince companies to base their decisions upon research of real needs and behaviors of their target audience, not just hunches…
Two hours after reading this post I found myself in a meeting with the marketing team of a leading academic institute.
The conversation sort of went like this:
Me: “What are the main motivations for people who are looking into going to college?”
The team: “We don’t know”
Me: “So let’s find out…”
The team: “Not now, let’s just guess and spend money based on that”.
And those are the representatives of the academy…
Dan, is it really that surprising? I think you said it yourself in an earlier post (or interview). Companies are driven to optimize and to be efficient. Coming up with new ideas requires experimentation and exploration, however. It is something you do in a startup and start to lose once the company is running. I think what companies need are little bursts of risky behaviour in units that are set up like startups within the company.
That’s too bad. You might try a smaller store. A big company already has their ideas and ways of doing things. It’s just like you say in your latest book, they fall in love with their ideas and so of course they don’t want to end that love affair. A smaller, newer store might not be as set in it’s ways and might be more open to doing something that will increase their business and costumer satisfaction.
You might think about buttering them up a little bit first also. Give them a few compliments, tell them how their ideas are backed up by your research and how their doing so many things right. Nassim Taleb talks about how at first he would tell economist that their statistical models were all wrong and they were crap. They obviously got angry at him. The next time he did a presentation he explained first the places it worked well and then the few places not to use it. After that he was a hit. Sometimes all you have to do to get what you want is pretend the other person is smarter than they are.
Dan, there are very few Toyotas in this world. My observation: Businesses are willing to spend large amounts of money to verify their intuitions. The last thing in the world an egomaniac CEO wants to hear is they’ve been doing things wrong. And more problematic: Had you actually reached the CEO, not some flunky in Corporate Comm, he might have said yes. Companies fail when their decision makers become too far removed from contact with the real world and day-to-day ops, making decisions in a relative vacuum. Try a smaller store, maybe Sprouts or Sunflower Market.
And keep trying, just don’t try something nasty on me in the Sunflower Market!
SS
Hey Dan,
Keep up the good work. You will find some company willing to hear you out & try. Unfortunately, most of Corporate America runs on the concept of “perception management” and “CYOA”.
Perception management because they need to keep thier jobs & their managers happy.
CYOA because they know they lack “courage of convinction of doing the right thing”
Both these feed well with each other.
That is OK Dan! It is frustrating to see that people have a hard time taking risks to find truth. This reminds me of the work of L.Kohlberg, J. Loevinger, R. Kegan, and other developmentalists, pointing to adult stages of development from a cognitive, moral, and emotional standpoint and how certain people get “stuck” on a particular stage and can’t move above it (unless they want to, or situations force them to).
I think, deep down, they know that they aren’t offering any better products than Walmart and their customers are just paying for the feeling. If it is proven, it’s the end of them. The emperor has no clothes!
That reminds me of an e-mail I got last week from a large charitable organization that said we could no longer legally say we were giving a portion of our t-shirt sales from October to their charity, and if we wanted to make a donation we had to enter into a written agreement that included a MINIMUM donation. These are the kinds of things that leave a little hollow spot in your heart. But you move on – and we chose a different charity who, of course, was MORE than happy to accept any donations!
Don’t you also love the “don’t hesitate in reaching out to us in the future” part? If you go to a restaurant and they don’t have any of the choices you would like, would you go back, don’t think so. Then why put all this fluffy bs in an email when they don’t mean it. Silly *real* world.
well, there’s the downside of irrationality! .. from where I’m standing what you are offering is like a harsh medicine that the society needs in order to heal. It’s difficult for those in charge to see it this way, and it’s understandable. But without the medicine, the society is doomed to chronic pain. Please persist. You’re one of the most inspiring scientists I know.
I’ll stop shopping there in solidarity! Well, actually, I stopped shopping there for other reasons — opposition to health care reform, questionable labor practices — a while ago. This is just another instance where their “alignment” seems questionable.
Sounds like a super-important avenue of research. Maybe some of the smaller organic foods stores in your area would be more interested in hosting the experiments?
I agree with you. Companies dont mind when it comes to spending big bucks on advertisements and signing up celebrities for an exhorbitant price to promote their market and increase their brand value. But when asked to experiment a bit by offering discounts or by the other way, they see you as if you stepped on their toes. Sad indeed.
One thing that strikes me that has not been mentioned yet. The power of the crowds and customers. Dan, you saw examples of this at PopTech I believe (yes I was there too).
Why not use the fans of Whole Foods to help you get in? A digital call out to Whole Foods. “Help Dan in his quest to find a way to get America to eat healthier.” Instead of shades of gray, make it 100% transparant, set up a site on the web were WF fans can comment on the data that you will reveal. This way, the chain can hopefully be pushed into it by the crowd, and also see a end result that is beneficial in yet another way.
Perhaps a long shot. Just had to suggest it.
What tool or process were you contemplating for this effort? I am familiar with preference analysis tools and have an investment in one that has successfully been sold by Accenture and others to the likes of Lenovo, Samsung, United Healthcare, Intel, and others. I would be happy to discuss this tool and the process of getting buy in from potential users. It is a decidedly longer and more personal process than merely proposing an experiment. You need to provide an economic reason for pursuing the project. I am sure you did this but, I might be able to help you with concrete examples of success.
Dan:
Empirical data risks changing management’s preconceived notions for which they already spent huge marketing $$$$. This response doesn’t surprise me one bit.
Would Wall Street Banks be interested in a like-kind analysis? Probably not but what’s the downside to ask.
Dan,
What wasn’t clear to me from your posting was what problem you were trying to solve for Michael at Whole Foods. That is, a problem that Michael might agree is there and worth fixing. I’d be surprised if you didn’t have that in your pitch and I’m a bit curious as to what angle you took in this regard. BTW, a huge fan of your work.
/Mike
Good point. Is Dan a good salesperson to the uninitiated?
This is about your survey: I didn’t know a more relevant place to comment on it. After you told me that people generally desire to live in greater income equality countries, I couldn’t ditch the idea and it permeated my responses. Thus the survey is, from that perspective, hopelessly biased and useless. I assume you knew that, therefore, what was it really about? Just to test participants’ math levels? ????
Thanks for the very interesting comments and I agree that there are lots of other incentives involved here.
I will keep on trying and if we ever get a partner I will let you know.
As mentioend when we spoke, if you want to do online testing we can help. We have lots of ‘rational’ clients (or at least we have made them so by now
Try the New Seasons Market chain, headquartered in Portland, OR. http://www.newseasonsmarket.com.
Considering that we in the US have a huge surplus of do-gooders and lawyers, I am not surprised about the response.
If you switch to something like fair trade coffee I can help!
Our company spent >200k to find out x, at the same time i was analyzing the actual consumer data (every sale for 18 months) and concluded the opposite. 5k data points vs 475k. Which did they run with? Not mine, why? They paid 200k for the analysis, if they didn’t use it they would be wasting money!
If I could find a good program I would love to design programs that teach corporate types about fallacies, irrational thinking, and experimentation. Unfortunately they pay me too much money to be able to go full time phd.
It occurs to me that in avoiding whatever risks he perceived, Michael ran straight into another one – there’s now a public story about him declining to pursue a research opportunity that could increase shareholder value.
He’s probably not too happy that you published his reply. If I were a Whole Foods shareholder I might have some questions now about what kind of research Whole Foods is basing its business on.
Dan,
As a conversion optimization testing company (www.widerfunnel.com) your challenge rings incredibly true and familiar.
In our experience, certain types of business managers are very receptive to using statistically valid tests rather than opinion and guesswork for decision making — and others just are not. Period.
The distinction has less to do with the industry they work in and more to do with their personal background and motivation. This is especially true for managers in marketing roles.
In general, we see three factors at play:
1. If they got into marketing because they wanted to express their creativity (but needed a paycheck so writing and painting were out as career options) then for them measurement is threatening. On the other hand, if they see business as a discipline and a profession, measurement is the optimal way to make decisions.
2. If management gets rewarded for performance, then measurement is in the company’s DNA. This is especially true if the CFO demands accountability from marketing (rather than avoid the topic altogether)
3. If the individual manager sees him/herself as a “champion” within their organization, they will want to demonstrate thought leadership and will embrace rational and logic measurements options to improve the business.
In other words, “Michael” is (ahem) “predictably irrational” – you just have to figure out his motivators.
Anyone who won’t work with Ariely is clearly “irrational”.
Aside from that, my experience in corporate america is there are very few companies with personnel empowered to actually be innovative, creative or experimental. They all speak to it, but when presented with action, high percentages duck and keep their heads low. Don’t rock the boat. Don’t risk it. I don’t want my head cut off. There is also a size issue. With big companies, it’s often difficult to know who really calls the shots because decision making is so dispersed or handled by committee.
In the end, I think it is a leadership and culture issue, and culture comes from leadership.
The question is whether they are rationally maximizing their interest, or irrationally escaping testing
Your frustration is both entirely justified, and pretty interesting. Feel free to continue posting such rants in the future.
A few more thoughts on this subject: http://allyourcode.wordpress.com/2010/11/07/should-companies-experiment/
Catharsis is good.
It might even generate some testable hypotheses about how corporations align their interests with the world.
In the first case, of course, they generally do nothing until the corporate entity perceives an opportunity for engagement.
The blog post itself was interesting, because the absence of relevant information, notably the information presented in the original request, the channel it was submitted to at WF, and he role of “Micheal” role in the scheme of things, allows this event to be considered as a Rorschach test, on several levels.
At the Dan to WF level: viewing WF from the outside, where should the request go? what information should it contain? what context should the information be presented within? what commitment should it require?
At the WF level: Who is this guy? What does he want? How much do we need to process this information? Does it fit within an existing organizational template? If not, how high up the chain of command does it need to go? Does it require us to commit resources? Does it require us to disclose customer information? Etc.
At the WF to Dan level: did anyone actually read the proposal? Did anyone with authority to decide consider the proposal? Did they understand it? What reason(s) led to denial? Are there valid business reasons that support the decision process? Do our customers want us to allow WF to use them in “experiments”? Do they know we collect information about them? Will this experiment make that process salient? Etc.
It would be interesting to call Michael and see whether he’s willing to shed light on the actual vs. the imagined process that was followed, ie. the guy you originally sent the request to forwarded me your email on Friday afternoon, I had no idea who you were or why you wanted to do this, but noticed you need customer info that we don’t provide to outside parties as a matter of policy, so I sent back our standard “alignment” letter.
PS. The alignment language reminds me of people driving down the Garden State Parkway: we just want to remain in alignment, listen to the radio, and not be bothered by any action requiring active thought.
Organizations fear change more than anything else: change threatens the status quo, by definition, and those in power under the status quo will fight change tooth and nail because they fear that a change in status quo will mean a loss in power.
Funny how the market for consulting firms full of Ivy League suits has never been better when any business would be better off with a room full of social scientists doing research on incentives, behavioral studies, and armed with experimental data. It’s probably because businesses are run by suits that they have such a “suit bias”. Somebody should do a study on that.
It is rather difficult to comment without seeing your “pitch” to Whole Foods. But I can understand how their executives would be reluctant to take on a project which may not contribute to their profitability; I assume that you cannot guarantee results from your research.
Was the research intended to be performed openly? Whole Foods might also have been reluctant to be seen experimenting on their customers for financial gain.
Consultants, and others, get their work by having relationships of trust with the people hiring them. You have no connections to the people, you don’t know the intricacies of their business, and they don’t know they can trust you.
It does go down to not making risky decisions. If you make the decision less risky, then they’ll be more likely to take it. Get to know their people. Then try again.
I just tried your weekly survey but with the example of net worth, decided it wasn’t worth my time. In your first example, if you take out a mortgage for $100,000 you should have an asset worth at least that, leaving your net worth either neutral or slightly positive. You can’t ignore the value of the asset purchased. If I had $50k in net worth before the house transaction, I’d have about the same after.
Dan,
This blog post (well, the discussion thread, really) inspired this post: “Would Don Draper Run Conversion Optimization Tests?” here http://bit.ly/aOpxev as comScore seems to think the offline world does a lot of testing. Ha!
hi Dan,
I agree with Ginnie Bivona. Central Market is Whole Foods’ biggest competitor in Austin, TX, Whole Foods’ HQ and hone of the flagship WF store. It would be a kick in their pants.
hi Dan,
I agree with Ginnie Bivona. Central Market (http://www.centralmarket.com/Stores/Austin-Central.aspx) is Whole Foods’ biggest competitor in Austin, TX, Whole Foods’ HQ and home of the flagship WF store. It would be a kick in their pants.
You may be mixing up two issues. First, you are correct that many companies do not want to do research. Often that is because it is an expense, the results may or may not help guide business decisions, and sometimes because they don’t want to learn that they made a mistake. (I once had a clinet tell me that decison making was “so much easier when I didn’t have to consider what the customer wanted.”)
But in this case, I suspect the reason for balking may have been a little different: by partnering with you Whole Foods would lose control of the dissemination of the results. Thus they would run the risk of results being reported that might be embarrassing to them or to their customers or vendors. Very few businesses are willing to lose control of data that may be harmful to their business, regardless of how unlikely that might be.
Venting is good… I got one of those rejection letters from Whole Foods too. Basically it said: thanks for the samples and the laughs but we’re afraid your product titles are too risque for our stores. Something like that, I don’t keep rejection letters but this one did try to let me down gently
Big companies don’t like to take risks — go for a small one and get press coverage.
What’s amazing to me is the difference between e-commerce sites who feel that NOT testing would be risky and brick and mortar stores who somehow feel that testing is a high risk activity of some sort.
But… are they worried about testing or are they worried about having those tests go public?
At finalmile, experimentation is built in to our delivery model and we work with the usual suspects. Many clients tell us its easier to roll out full scale than experiment. We have stuck to our stand and only work if experimenting is part of the project. However, we realized that mostly its only us who are interested in the experiment, most clients take little interest. Status quo bias in full effect.
I wonder whether it is because they are risk averse. Because they are the same set of marketers who are willing to subject their advertisements to expensive pre-tests and run the risk of failure. Think its probably more of status quo issues and the inability to think of experiments, which often measures results in terms of sales and action rather than opinion and preferences.
Dan,
I know that you can’t type because of that accident but if you can I’d be interested in your comment on this.
In a pub in Woking they did an experiment I think on a certain day every week.
On that day or evening the price of drinks was determined by demand. The more pints that were ordered the higher the price went up and the less consumed the lower the price.
I like it because it’s the opposite of the Happy Hour concept.
I don’t know the outcome and I’m trying to get details as I’ve forgotten the name of the pub.
Actually, it sounds like they just don’t need your ‘service’. Why would a Fortune 500 company need to hire a consultant to do testing? They’ve got loads of internal teams that do all of that.
Now how about we discuss the psychology behind why you had so much trouble receiving a ‘no, thank you’?