Of Course They Are

This past week a story ran on Huffington Post that surprised a lot of readers and resulted almost immediately in a vast chorus of scorn and derision – as usual. On any given day you can expect to find stories of varying impartiality on every topic from international politics to local sports, some attempting to sound mature and some that are openly incendiary. But what made this story different was that the controversy was arising over food, and specifically about a listing of the 30 best barbeque restaurants in the United States. What made it so controversial is that not one of the 30 restaurants listed are from Tennessee, Louisiana, North Carolina, South Carolina, Mississippi, Alabama, Arkansas, Georgia, Florida, or Missouri, while states like New York, Vermont, Wisconsin and Ohio all had multiple entries…

One might reasonably ask how any such survey managed to bypass the entire region of the United States most associated with barbeque – but only if you had also failed to note the headline of the article, which states that these are the 30 best barbeque restaurants as identified by Open Table, the online restaurant reservation system. To their credit, the folks at Huffington Post do note that these are the top 30 barbeque restaurants that accept reservations on Open Table, and speculate that many of the great Southern barbeque joints don’t take Open Table reservations. Given that a lot of great barbeque is produced by hole-in-the-wall operations with a devoted local following, which therefore do not want or need Open Table, this is likely correct. However, I thought it was a good example of how bias gets into otherwise interesting data – and renders it utterly worthless…

Clearly, there is no practical way for a single reviewer to sample all of the restaurants in a given category in the United States in one lifetime; there’s just too much ground to cover. But unless the same reviewer (or group of reviewers) is doing all of the samples, there is no way to avoid having matters of personal, regional, national, ethnic, professional or other preferences from influencing the data. But in this case we are adding an additional problem in that all of these choices have been filtered by a factor that has nothing to do with how well a restaurant produces food, let alone how well it makes a specialty type of food relative to other providers. Even granting that having the technical ability, knowledge and willingness to use Open Table would allow you to make better barbeque – which seems unlikely, frankly – this sample is automatically excluding everyone else from the survey…

Now, this type of bias isn’t limited to business applications. You can see it in everything from people hiring employees because of pre-conceived ideas about gender, ethnicity, socioeconomic background, or appearance to former Vice President Dick Cheney leading the search for the best candidate for Vice President and discovering (no doubt much to his own surprise) that it was himself. You will sometimes see this referred to in the management literature as Confirmation Bias, which is the tendency to see whatever information is available as proof that your existing opinions are correct. In some cases this will result in bad decisions, when people use unrelated or even negative evidence to convince themselves that the choice they already wanted to make was the best one, while in other cases people will stop gathering information once they find enough to confirm whatever they already believe…

I don’t have any magical way of dealing with this issue – I’m as capable of confirmation bias as the next man. It is only by questioning our assumptions – not just at the start of the project, or at the end of the day, but continuously – that we have any chance of recognizing these errors, let alone avoiding them. But if you need a good example to work from, consider that you may confidently expect that the best restaurants in any category take Open Table reservations – if the only restaurants you are considering are from the Open Table database, that is…