by John Geracitano
In the book Weapons of Math Destruction, author Cathy O’Neil critically examines mathematical models and machine learning (ML) algorithms. Leveraging her expertise in mathematics and her experience in the financial industry, O’Neil unveils the troubling (mis)use of and growing reliance on algorithms. Using real-world examples, she highlights the adverse societal impacts of “Big Data.” Even models that originate with good intentions can devolve into pernicious feedback loops, exacerbating societal divides. These algorithms are used in various domains, such as hiring, insurance, and law enforcement, often perpetuating bias and inequality. O’Neil argues that these systems can cause significant harm to individuals and communities without proper oversight.
While the military might not heavily rely on these technologies to operate effectively, their prevalence spills over into our domain. Our most valuable resource–our people–can be influenced by external entities that the chain of command cannot control. Therefore, we must educate ourselves to understand how artificial intelligence and ML models operate and how they impact our soldiers’ daily lives.
O’Neil has credibility. A data scientist and founder of mathbabe.org, she earned a PhD in mathematics from Harvard University before working in a private sector hedge fund. Amidst working at various startups, she witnessed the crippling impact algorithms played in the 2008 housing crash, which motivated her to raise awareness and take strides to ignite change. Here are the core concepts I learned from O’Neil’s book.
The Nature of Models
O’Neil begins by elucidating the concept of models. Simply put, a model is a conceptual representation of a process. She uses the example of a smoke alarm, a primitive model that uses the presence of smoke to trigger an action. Modern models, however, are dynamic, incorporating numerous factors to determine outcomes. For instance, in baseball, dynamic modeling is used to shift players based on specific hitters’ statistics. This transparent and benign application of models in baseball, popularized by the 2002 Oakland Athletics, showcases how mathematical models can objectively enhance decision-making.
Another example of a well-constructed dynamic algorithm is the FICO credit score, which comprises five transparent elements: payments owed, length of credit history, payment history, credit mix, and new credit. This transparency allows individuals to understand and improve their scores. However, O’Neil argues that mathematical models have become increasingly opaque and harmful in recent decades.
Opaque Algorithms: Modern-day Phrenology Weapons of Math Destruction (WMDs)
O’Neil defines a weapon of math destruction (WMD) as an algorithm characterized by opacity and scale, causing significant damage. These algorithms manipulate, control, and intimidate people, often hiding behind complex mathematics that obscures their true structure. Not all WMDs are created with malintent, but they often use complex mathematics to obscure their true nature. Organizations hide behind the alleged objectivity of algorithms, offering little explanation or justification for any of their outputs due to their ambiguity.
The nineteenth-century phenomenon of phrenology is analogous to today’s pseudoscientific WMD implementation. In phrenology, individuals were diagnosed with conditions based on the measurement and location of bumps on their skull, a practice based on a small amount of empirical science and a large amount of creativity. Similarly, algorithms have become opinions codified in code, starting with subjectivity and little scientific basis. This is an acceptable beginning to the iterative development of an algorithm, but fine-tuning must occur along the way to prevent the negative consequences that often emerge.
O’Neil illustrates WMDs best through the U.S. News and World Report college ranking system, where the school’s reputation (a subjective factor) was the most heavily weighted element. When they initially developed their ranking system, U.S. News observed the top colleges, extracting variables that made them a great school (which automatically keeps them among the top). Abdulaziz University and TCU were two schools that gamed the system, injecting the proper attention and money into specific evaluation categories, causing their rankings to soar quickly. The result is a system that favors the privileged and furthers the educational divide.
Feedback Loops and Societal Impact
A recurring theme in O’Neil’s examples is WMDs’ creation of feedback loops. For example, for-profit colleges like the University of Phoenix and Corinthian College used predatory models to target specific populations. These included impoverished zip codes and veterans. They aimed to recruit the most students possible who could land government loans/funding. These “diploma mills” marketed a brighter future through the path of education, but their degrees were essentially worthless. These students then had loan debt they couldn’t pay off because they didn’t have the high-paying jobs they were misled to believe they’d obtain, furthering the social divide within already impoverished demographics and communities.
Other pernicious feedback loops discussed in depth in this book include:
- Job-hiring screening software.
- Auto-insurance algorithms that use irrelevant factors such as the FICO score to determine premium rates.
- A recidivism model that uses an LSI-R questionnaire favoring more educated respondents.
- The flawed mathematical models that fueled the housing crisis and market crash in 2008.
- Crime prediction software that increases arrests in high-crime areas, further decreasing the chance of success for those living in such communities.
- Peer-to-peer lending services.
Through these examples, O’Neal establishes substantial evidence that WMDs exist and have negative consequences if left unchecked.
Proposed Solutions and Ethical Considerations
While O’Neil paints a grim picture of the current state of WMDs, she offers solutions to harness Big Data for positive outcomes. She suggests that data scientists adopt a Hippocratic Oath akin to that in medicine, emphasizing transparency, accountability, and ethical responsibility in model development. Additionally, she advocates for using collected data to improve societal outcomes, such as reducing recidivism through targeted prison training programs.
O’Neil acknowledges that most mathematical models are not designed with malicious intent. For example, crime prediction software aims to reduce major crimes but inadvertently exacerbates socioeconomic divides through increased petty crime arrests. Despite its good intentions, the software’s negative second-order effects highlight the complexities of algorithmic impact.
Implications for Leaders
Overall, Weapons of Math Destruction is an enlightening book for the information age, simplifying complex concepts and revealing the often unseen impacts of algorithms on society. O’Neil’s unique perspective and contrarian viewpoints shed light on the urgent need for ethical considerations in Big Data and machine learning. Her work is a compelling ‘call to action’ for greater transparency and accountability in algorithmic decision-making.
As leaders, we must understand the external–and potentially invisible– forces that impact our people and organizations. Including a 15-minute information brief on AI and ML during the next administrative training day is worth it. This technology is all around us, affecting us in ways we’re unaware of. It might seem innocuous, but you never know how the knowledge of these models might alter significant decisions in someone’s life.
John Geracitano is a U.S. Army Signal officer currently serving as a doctoral student at the University of North Carolina at Chapel Hill. Formerly an Armor officer, John’s most recent positions include Deputy G6, Brigade S6, and Task Force Operations Officer.
