“I think where I would hide my kids from shooters every time I am in public. No matter where. Not just movies or public events. I was in the grocery store last weekend with my four year old. I found myself scouting places I could hide my little boy.”
— Kevin Bloxom (of Louisiana) quoted after the San Bernardino attacks on the front page of the New York Times
This is part 1 and you can read the full series here. This piece was originally published on December 2016 in Priceonomics.
With alarming regularity, there seems to be another terrorist incident, mass shooting, or police shooting in the news – and it’s hard to know how scared we should be. Should we take personal precautionary measures? Should we be careful when we’re in public? How accurate is the news in covering what’s going on in the world? What military action should we take? How much of our taxes should we dedicate to preventing these incidents? Who should we vote for that will protect the most American lives? What other risks should we be aware of?
Even when fair-minded journalists report events, we’ll see how media (Facebook, Twitter, newspapers, 24 hour news networks) can negatively impact personal and societal decision-making – leading to suboptimal decisions that can negatively impact our quality of life. Over several posts, we’ll determine an objective way to measure distortion in media, highlight it in a number of areas, and then try to understand what is introducing the bias. We’ll also see the impact on politicians, journalists, and thoughtful citizens.
But before we get there, we’ll begin in 1986 when one of America’s great tragedies occurred.
Part 1: The Space Shuttle Challenger
On January 28, 1986, the Space Shuttle Challenger exploded 73 seconds after liftoff, killing seven crewmembers and traumatizing a nation (see a video of the launch; warning the contents are graphic). Millions of viewers (including many schoolchildren) watched the launch live partly because Christa McAuliffe, a social studies teacher who was to be the first civilian in space, was on board.1
Source: History.com
The cause of the disaster was traced to an O-ring that had failed due to the low temperature (31°F / -0.5°C) at launch time – a problem that several engineers noted, but that NASA management dismissed. NASA’s own pre-launch estimates were that there was a 1 in 100,000 chance of shuttle failure for any given launch – and poor statistical reasoning was a key reason the launch went through.
Seconds After Shuttle Explosion (bottom is Christa McAuliffe’s parents and sister)
Source: CBS News (top), NBC News (bottom)
In 1989, Siddhartha Dalal (my father), Edward Fowlkes , and Bruce Hoadley would write a paper (“Risk Analysis of the Space Shuttle: Pre-Challenger Prediction of Failure” ) analyzing the data available before launch to determine if the failure could have been predicted. Using standard statistical techniques, they determined that the evidence was overwhelming that launching at 31°F would lead to substantial risk of failure (~13% likelihood of O-ring failure at 31°F, compared to NASA’s general shuttle failure estimate of 0.001%, and a 1983 US Air Force study of failure probability at 3-6%).
Both the post-disaster presidential commission report and Risk Analysis of the Space Shuttle highlighted NASA’s misleading use of data that showed number of O-Ring failure incidents only vs. temperature.
Look at the graph below, and see if you can spot any correlation between temperature and failure rate. If you were the decision maker for launch and had this graph, would you have allowed the space shuttle to launch at 31°F?
Number of O-Ring incidents vs. Joint Temperature (failures ONLY)
Source: Report of the Presidential Commission on the Space Shuttle Challenger Accident, 6 June 1986, Volume 1, Page 145, (link) Color added.
NASA management used the data behind this first graph (among other pieces of information) to justify their view the night before launch that there was no temperature effect on O-ring performance. After all, it’s hard to find any consistent relationship between temperature and failure rate in the provided data.
Now look at a graph of the full data set (this time including successes, rather than just the failures). Do you now see any correlation between temperature and failure rate? Would you still allow the space shuttle to launch at 31°F?
Number of O-Ring incidents vs. Joint Temperature (failures AND successes)
Source: Report of the Presidential Commission on the Space Shuttle Challenger Accident, 6 June 1986, Volume 1, Page 145, (link) Color added.
Successful launches (those with no failure incidents) had not been listed in the first data set we saw, and if included would have led most to conclude there was a definite temperature effect. If the data behind this second graph had been available, it’s likely that the launch would have been postponed – saving seven American lives. While data and statistics might seem like a dry topic to some, this comparison highlights just how important having an unbiased set of data is – and how it can literally mean the difference between life and death.
We might visualize the difference between the two graphs as follows, with the data leading to inferences that impact the decision about whether the space shuttle should have launched:
| Dateset | Inference from Data | Decision | Result |
|---|---|---|---|
| Biased Data (actuality) | Low temperatures have little to no effect on O-ring failure rate | Allow space shuttle launch at low temperatures | Space Shuttle Challenger explosion leading to 7 deaths |
| All Data (ideal) | Low temperatures have a substantial effect on O-ring failure rate, especially as all launches below 65° had O-ring issues | Don’t allow space shuttle launch at low temperatures, due to the substantially elevated risk of failure | Unknown; likely, significantly lower risk of O-ring failure and subsequent explosion |
I drew several lessons from the Challenger disaster – most notably, looking at biased data that excludes critical information leads to ineffective decision-making. As humans, when we seek to make sense of a large number of data points, we construct a limited data set – we observe or include only a subset of all data points through a process known as sampling – which can lead to a biased data sample if this misrepresents the underlying data.
Further, “data” is more than numbers, but any source of information that can be used to inform decisions. It can come from a number of sources we don’t typically consider data: journals, newspaper articles, TV segments, web videos, social media feeds and posts, Reddit, or personal experiences to name a few. In the Challenger’s case, it was a negative selection bias for failures alone that led to a flawed graph, a subsequent flawed decision, and seven deaths. In other cases, such as academic publishing, it is a positive bias for successful experiments.
Nearly all forms of media (social media, journalism, science publishing, among others), sample a subset of data from the full underlying data set (everything going on in the world), and have a selection bias for content that is most engaging to its consumers (readers, viewers). We – the audience – favor certain topics over others, forcing the media to produce content on a subset of popular topics, to the detriment of other important stories, increasing audience engagement but distorting our view of the world and negatively impacting the decisions we then make.
To understand this, we’ll have to define “bias”, determine how to objectively identify it, and see how it affects our decisions. Our goal will be to tease out how media coverage is different from the full underlying data, seeing how as readers we might make the wrong inferences based on coverage alone. While we’ll note that media coverage could be better, we’ll especially argue that any American (including key societal decision makers like politicians) should make significant adjustments before using media coverage for making decisions.
First, we’ll look at media coverage for one particular topic that’s top of mind and widely covered – which we’ll do next time.
This is part 1 in a four part series. Read the full series here.
- See Space Safety Magazine for more details on the launch and subsequent investigation. ↩