Good Decisions. Bad Outcomes.
If you practice kicking a soccer ball with your eyes closed, it takes only a few tries to become quite good at predicting where the ball will end up. But when “random noise” is added to the situation — a dog chases the ball, a stiff breeze blows through, a neighbor passes by and kicks the ball — the results become quite unpredictable.
If you had to evaluate the kicker’s performance, would you punish him for not predicting that Fluffy would run off with the ball? Would you switch kickers in an attempt to find someone better able to predict Fluffy’s involvement?
That would be absurd. And yet it’s exactly how we reward and punish managers. Managers attempt to make sense of the environment and predict what will result from their decisions.
The problem is that there’s plenty of random noise in competitive strategic decisions. Predicting where the ball will go is equivalent to deciding whether to open a chain of seafood restaurants on the Gulf Coast. The dog running off with the ball is the BP oil spill. When the board reviews the manager’s performance, they’ll focus on the failed restaurants. The stock is down. The chain lost money. Since the manager’s compensation is tied to results, he’ll incur financial penalties. To save face and appear to be taking action, the board may even fire him—thus giving up on someone who may be a good manager but had bad luck.
The oil spill example is an extreme case. In the real world, the random noise is often more subtle and varied—a hundred little things rather than one big thing. But the effect is the same. Rewarding and penalizing leaders based on outcomes overestimates how much variance people actually control. (This works both ways: Just as good managers can suffer from bad outcomes not of their own making, bad managers can be rewarded for good outcomes that occur in spite of their ineptitude.) In fact, the more unpredictable an environment becomes, the more an outcomes-based approach ends up rewarding or penalizing noise.
In the last year I’ve asked many board members how much of a company’s stock value they think should be attributed to the CEO’s strength, and the answer is surprising. They estimate that you’ll get about 10% more stock value, on average, from a good CEO than from a mediocre one. Implicit in that estimate is the understanding that many outcomes are outside a leader’s control.
We can’t entirely avoid outcome-based decisions. Still, we can reduce our reliance on stochastic outcomes. Here are four ways companies can create more sound reward systems.
1. Change the mindset. Publicly recognize that rewarding outcomes is a bad idea, particularly for companies that deal in complex and unpredictable environments.
2. Document crucial assumptions. Analyze a manager’s assumptions at the time when a decision takes place. If they are valid but circumstances change, don’t punish her, but don’t reward her either.
3. Create a standard for good decision making. Making sound assumptions and being explicit about them should be the basic condition for getting a reward. Good decisions are forward-looking, take available information into account, consider all available options, and do not create conflicts of interests.
4. Reward good decisions at the time they’re made. Reinforce smart habits by breaking the link between rewards and outcomes.
Our focus on outcomes is understandable. When a company loses money, people demand that heads roll, even if the changes are more about assuaging shareholders than sound management. Moreover, measuring outcomes is relatively easy to do; decision-based reward systems will be more complex. But as I’ve I said before, “It’s hard” is a terrible reason not to do something. Especially when that something can help reward and retain the people best able to help you grow your business.
The Upside of Irrationality, explores some positive and some negative ways that irrationality plays out in our lives.
I think the underlying assumption with this idea is that we know a good decision when we see it. Unfortunately managing is a classic example of the problem of unknown unknowns. If there was a standard set of information one needed to take in to make a good decision, anyone who knew how to read a checklist and use an excel spreadsheet could be a top manager. Any novel technique, style or idea by definition will not be validated.
The scary but true fact is that all most of us see is noise. The really good managers see true patterns in that noise, but the truth or fiction of those patterns doesn’t become clear until after the fact. In such an environment, it is actually statistically optimal to reward on outcomes. While each individual outcome is obviously driven by a lot of noise, on average those who succeed saw some pattern that their peers didn’t see.
I think your right that managing could benefit from more explicit statements of assumptions and methods, but that information would be most beneficial in the aggregate to make all managers across the board better by identifying chronically good and bad ideas faster.
Dan,
Thank for this post. It makes a lot of sense. Actually since performance rewards are tied to outcomes, most compromise on long term good decisions to take the low hanging fruit. Which in the long run causes a lot of problems, since no body has more than a quarter long plan.
Sonia
Dan:
The theory makes sense but Wall Street’s evaluation of a public company’s stock will not agree. Results and outcomes are the metric. Rewards and penalties follow accordingly. Until you change that dynamic, not much progress will follow.
I agree with you intuitively, but I wonder what we would find if we were to study this scientifically. Someone should investigate the impact of going public on companies’ long-term planning. Do companies tend begin to focus on “hitting quarterly targets” at the expense of long-term strategic projects?
Perhaps a simple experiment would be to see how people do on projects that require significant development before any “results” become visible when their progress is evaluated at different time intervals (e.g. daily, weekly, monthly, quarterly).
Video games offer a means to explore how people make trade-offs between between short and long term gains. Commercial games are probably not good for scientific purposes, because they are too complex, but a simplified version of an existing game might work well. The two games that come to mind are StarCraft and Civilization. Both require players to invest in different aspects of their army. Some choices have an immediate pay-off in terms of army strength (e.g. train a soldier), while others require more time to realize a “return on investment” (e.g. researching new technology).
You could introduce a new rule to the game that says “you cannot attack for the first 10 minutes”, and see what effect this has on the strategies that people choose. Are they more likely to choose slow to research technologies that produce more effective soldiers? I believe behaviors would be radically altered if players are not faced with early and frequent engagements that “test” their armies.
No one uses anything nearly as complex as starcraft of civ, but people do use some very simple games to get at this same idea of immediate rewards vs long term planing. The keyword to look up is “temporal discounting”
I cannot completely agree.
There was a nice joke about it:
- “Why have you shredded half of job applications for that position?”
- “I don’t employ unlucky people”
Some people are just more lucky (or have better intuition) and I think that counts. You do not want someone who seems that can solve the problem and then fail. You want someone who actually solves the problem.
I think you’ll be hard pressed to find someone in business who counts on luck.
Intuition is also problematic. How do you tell the difference between that and luck?
This also applies quite well to the current wave of education reform (outcome-based accountability for teachers). Teachers operate in complex and unpredictable environments (and in fact, in the “failing” schools, the students and environment are even more unpredictable (more turnover in lower income families), yet we insist on outcomes-based reward and accountability. Whereas in business, where outcomes-based evaluation often narrows the outcomes to those convenient to measure (such as stock price), in teaching the focus on outcomes-based accountability for teachers focuses on narrow standardized test scores.
When you are done translating the basic insights of the psychology of decision making into economics, I urge you to continue on to K-12 education policy, which could really use your help.
Dan,
How do *you* define *good managers”? The analysis should follow from there but you haven’t defined the term.
Thank you.
Raquel
Outcome-based evaluation is problematic to many jobs. I’ve visited this topic a few times in my own blog:
http://goo.gl/2yEP3
http://goo.gl/joG9X
To draw an example from my own profession (software engineering), it’s not a good idea to evaluate people based on how many features or products they launch. This leads to a meriad of problems:
1. premature launches
2. poor support after launch
3. rushing to develop code that mostly works, but is in bad shape, and difficult to extend and maintain.
This mentality also systematically fails to reward developing basic facilities that make everyone else more effective, because releasing such code does not correlate with product or feature launches. This type of work is often unglamorous, because people do not notice the large indirect impact it has on the whole company.
The fundamental problem with outcome-based evaluation is that it completely ignores the process that a worker uses leading up to the final outcome. In other words, it’s too shallow to give a true picture of someones performance. When it is possible to follow a good process, but have bad outcomes, we need to take a careful look at what the process a person takes (i.e. what he or she actually did) in order to decide whether they did a good job. As Dr. Ariely points out, many organizations do not take this route, because they find it less expedient.
DHGlZa I’m not easily impressed. . . but that’s impressing me!
Interesting ideas about outcome evaluation. I agree that there is an overly short term focus in many businesses because of the boards direct link to share price etc. The idea of capturing the decision basis at the time is also interesting. I suspect the key challenge will be uncovering the assumptions that have been used as many of these are likely to be sub-conscious and therefore difficult to elicit. Many of the assumptions are likely to be skewed in favour of the short term outcome clouding the issue further.
>> Reward good decisions at the time they’re made.
The problem is to distinguish good decisions from bad ones when they are made, without looking at the result. In fact, my questions is, what is a good decision if it doesn’t result in good output?
Tricky !!
Sometimes, the difference between a good decision and a good outcome is clear.
Let’s take Texas Holdem poker as an example. If you have pocket Aces, the best hand in the game. It’s a smart move to push people around and call a big bet before all the cards are shown. If you do that, you might still loose occasionally, but on average you will win.
When the noise is small and the signal is large, like simple games with only 52 possible cards, then good decisions become easy to find, even if good outcomes aren’t assured.
The problem I have with the post is that business seems like a situation where the noise is huge and the problem space has a lot more than 52 moving parts. This leads to a situation where it’s impossible to independently understand what a good decision is apart from the outcome without billions or trillions of observations.
Dan,
This is a good post but it needs one more thing. I believe that the time of the reward is crucial because if one opened chain restaurants in the gulf a year before the oil spill did well until the spill occurred should you reward the short term gain? Even if there were insurmountable losses following the spill? In my opinion there has to be a time period that allows your decision to develop so someone is not rewarded just for short term gain or fired for short term loss. The long run needs to be inside of the decision.
Dan, I don’t see what the problem with reasonable performance bonuses is. Sure, they will be distorted by noise, but over time that noise should balance out. The BP oil spill was preventable with well upgrades, from what I understand, and Fluffy wouldn’t have run off with the ball if the kicker had the foresight to lock her in the house. Sure, some stuff is beyond people’s control, but that doesn’t mean we should abandon performance incentives altogether.
Although graduate school is a lifetime away for me, I remember a course in decision making that didn’t classify decisions as good or bad but consistent or inconsistent with available information. Now if you could analyze each decision on that basis, it would be preferable to a system based solely on outcomes. However, that would be a difficult and tedious task perhaps with more cost than benefit. It would seem that a reward system based on outcomes adjusted for “bad luck” or the occurrence of unpredictable events like the BP oil spill or Hurricane Katrina, payable at the end of a business cycle might be a better system.
Dan,
I have been thinking about this framework of evaluating decisions. One could think of a poker game, where each decision to bet, call, raise has a certain expected value (eg. how often it wins and how much is in the pot). However, it should not be evaluated using the ACTUAL hands you bet into, but all the possible hands. (this is a typical linguistic framework for poker players). The idea is, even though a play cost you all your money, its not necessarily a bad decision based on all the possible hands your opponent might show up, him showing up with 10% of the hands that happens to beat you is just “noise” or “variance”.
Back on to the point, given 100% completely information, we should theoretically be able to evaluate decisions independently and calculate that “expected value” of each decision. However, no one has all the information but you could think individuals with more information of the real world are in general, capable of making better decisions. At the same time, two exact same decisions shouldn’t be credited equally. Joe could have done his research before opening up a shop that turns out to be successful, while John could have done exactly the same thing but only did it on a whim, so who is a better decision maker?
Another way to frame this, which is more realistic.
Represent each decision at time X as a distribution of outcomes which include failure and success. Every decision contains a risk of exposing the company to both bankruptcy and huge profits.
Notice:
1. Good decision makers have a mean that is more toward success than failure.
2. BIG POINT: Some distributions contain tails that are larger than the tails of other distributions. That is, there are some huge successes but huge failures. Do we want those (e.g., BP, subprime mortgages)? This is an index of risk that becomes irresponsible once the losses are sufficiently large.
I suggest that we reward executives not on ACTUAL outcomes, but on attaining the most efficient a priori bankruptcy/profit ratio. Too many executives are doing little more than putting all their chips on red and hoping their color comes up.
(Of course, the devil lies in the modeling.)
Better yet, reward them 2 years after their decisions are made based upon better parameters in the model (when riskiness is better evaluated).
In any case, we have to admit that executives are being paid to be oracles and we are somehow amazed when they cannot do it well.
“The thing is – it is very odd how we change our opinion on past decisions/events over a period of time… what seemed correct a few years back, changes to ‘obviously’ wrong as time passes by…..” Do read about it more in this link: http://bit.ly/nfvPlQ