[Book Summary] Weapons of Math Destruction—Cathy O'Neal

Rating: ★★★★☆

(Written with aid of ChatGPT.)

Cathy O’Neil’s Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (2016) is a compelling critique of how algorithms—often perceived as objective and efficient—can perpetuate and amplify social inequalities. O’Neil, a mathematician and former hedge fund quant, introduces the term “Weapons of Math Destruction” (WMDs) to describe algorithms that are opaque, unregulated, and scalable, making them particularly harmful when applied to critical areas of life such as education, employment, finance, and criminal justice.

What Are WMDs?

O’Neil defines WMDs as algorithms that share three key characteristics:

1. Opacity: They operate as “black boxes,” making it difficult or impossible for individuals to understand how decisions are made or to challenge those decisions. Sounds like the graduate school application process or job hunting process, right?
2. Scale: These models are deployed across large populations, meaning that any embedded biases can affect vast numbers of people.
3. Harm: They often reinforce existing inequalities, disproportionately impacting marginalized and disadvantaged groups.

Real-World Examples

1. Education: Teacher evaluation systems, such as the IMPACT model in Washington D.C., which relied heavily on student test scores, led to the dismissal of competent teachers based on flawed metrics. Besides, the famous/notorious US News Grad School Ranking is probably something you are familiar with.
2. Employment: Automated hiring processes that filter candidates using personality tests or credit scores (Surprised? I certainly was.) can unfairly disadvantage applicants, often without their knowledge.
3. Finance: Credit scoring algorithms may penalize individuals for factors unrelated to their financial behavior, such as shopping at certain stores, leading to unjust consequences like higher insurance premiums. This maybe a reason why some people shop with cash instead of credit cards.
4. Criminal Justice: Predictive policing tools can create feedback loops that target minority neighborhoods, increasing surveillance and arrests in those areas, thereby perpetuating systemic biases.

A particularly striking example involves the use of credit scores in determining auto insurance premiums. O’Neil highlights a case where individuals with poor credit scores but clean driving records were charged significantly more than those with excellent credit but a history of drunk driving. This underscores how WMDs can produce counterintuitive and unjust outcomes, often without individuals being aware of the underlying reasons. 

By the way, I was charged around $2.4k a year for my auto insurance. However, my experian credit score was over 770, and my driving record has been clean in the United States. I guess the insurance companies don't need a reason to rip you off.

Though the author pointed out the WMD's power and impacts, the solutions she presented are in broad terms and lack detailed implementation strategies. For instance, while the idea of algorithmic audits is compelling, the book does not delve deeply into how such audits would be conducted or enforced. The section on solutions is weaker than the illustration of the problem.