Dark Data¶

The whole book is about this: data may be the oil of the new world, and while science insists on letting data speak, we must recognize that the people collecting and designing data, the machines used in the process, and the interpretation of results all shape what the data says. Reading data carefully is essential.

The author identifies 13 types of dark data:

1. Data we know is missing: Known unknowns — survey fields left blank by people who refused to answer.
2. Data we don't know is missing: Unknown unknowns — e.g., not knowing who might respond, so failing to notice that certain people never responded at all.
3. Selecting only part of the situation: e.g., surveying survival rates at a specific time while missing those who have already died.
4. Self-selection: e.g., patients choosing whether to be included in a study (dropping out mid-experiment).
5. Missing key variables: Because of dark data, the true correlation is invisible — e.g., "shorter skirts correlate with higher ice cream sales."
6. Changes over time: Results vary with the environment — different eras, different contexts.
7. Definition of data: Different interpretations lead to radically different conclusions.
8. Data summaries: Once data is aggregated, granular detail is lost.
9. Measurement error and uncertainty: Margin of error.
10. Feedback and gaming: When the data collection process is influenced by the data itself — e.g., stock price bubbles.
11. Information asymmetry: Insider information, etc.
12. Deliberately darkened data: Intentional data manipulation.
13. Fabricated and synthetic data: Potentially fraudulent constructed data.

programming¶