A new book from Nobel laureate Daniel Kahneman examines the flawed nature of human judgment and offers ways to reduce the noise that interferes with better decisions.
“Unwanted variability in judgment,” or ‘noise’ is problematic in any situation, and particularly so for legal professionals. Noise: A Flaw in Human Judgment, a new book by Daniel Kahneman (author of Thinking Fast and Slow), Olivier Sibony, and Cass R. Sunstein, illuminates plenty of examples across industries as they explain the concept. A New York Times article reviewing it, “For a Fairer World, It’s Necessary First to Cut Through the ‘Noise’ (May 2021) opens with a dramatic example cited in the book from the legal profession:
A study of 1.5 million cases found that when judges are passing down sentences on days following a loss by the local city’s football team, they tend to be tougher than on days following a win. The study was consistent with a steady stream of anecdotal reports beginning in the 1970s that showed sentencing decisions for the same crime varied dramatically — indeed scandalously — for individual judges and also depending on which judge drew a particular case.
Noise is explained in the context of bias, which is more consistent than noise. Decisions can be biased and noisy. But because bias without noise follows a consistent pattern, it’s easier to detect, as highlighted in the New York Times article:
As the authors explain in their introduction, a team of target shooters whose shots always fall to the right of the bull’s-eye is exhibiting a bias, as is a judge who always sentences Black people more harshly. That’s bad, but at least they are consistent, which means the biases can be identified and corrected.
While hard to detect, ‘noise audits’ can help determine how much noise exists in any organization. A process suggested by the authors was shared in a recent CNBC article (May 2021):
Kahneman describes a protocol for a noise audit that involves studying how a group of experts in a firm (he suggests a minimum of 12 participants) react to two or three realistic case studies on an individual basis. Each member would have to summarize the case and the judgment either numerically (in dollars, percentiles, or probabilities) or on some scale with at least five degrees (such as “very strong,” “strong,” “average,” “poor,” or “very poor”). They would not be allowed to communicate with each other.
The executives who assemble the case study are surveyed beforehand to assess how much they expect their experts to agree on any given case, and what level of disagreement would be acceptable.
If the results diverge significantly from expectations, then there is a noise problem.
Fighting Noise with ‘Decision Hygiene’
‘Decision hygiene’ can help but might be difficult to implement, depending on the type of noise. ‘Occasion noise’ is more easily addressed than ‘system noise’ — the two are distinguished in the New York Times article:
Some decision hygiene is relatively easy. “Occasion noise” — the problem of a judge handing out stiffer sentences depending on whether a favorite sports team won or lost or whether it’s before or after lunch (yes, studies have found that, too) — can, like bias, be recognized during a “noise audit” and presumably dealt with. “System noise,” in which insurance adjusters, doctors, project planners or business strategists assess the same facts with that unfortunate variability, requires a more energetic decision hygiene.
A few ways to practice decision hygiene, aimed more at organizations than individuals, are shared in the CNBC article:
- The goal of judgment is accuracy, not individual expression. Kahneman calls this “the first principle of decision hygiene.” Individual differences “lead different people to form different views of the same problem. This observation leads to a conclusion that will be as unpopular as it is inescapable: judgment is not the place to express your individuality.” Can we replace human judgment with rules or algorithms? Kahneman says while it may be undesirable to completely eliminate human judgment, using algorithms can improve judgments by making them less dependent on what he calls “the idiosyncrasies of one professional.”
- Resist premature intuition. Professionals make very quick decisions based on past experience, which is a key source of bias and noise. “Intuition need not be banned, but it should be informed, disciplined, and delayed,” Kahneman says.
- Obtain independent judgments from multiple judges, then consider aggregating those judgments. This is another well-studied bias: A group of, say, financial analysts who have an opinion on the direction of the stock market will often change their opinion after being exposed to a group expressing different opinions. Taking a large independent sample size, whether you are dealing with evaluating an X-ray, an engine problem, or the future price of stocks, will improve the precision of estimates.
Fighting Bias Simultaneously
With algorithmic bias an obvious related concern, the authors acknowledge that balance is key given other limits of decision hygiene, which the New York Times article summarized:
Thus, the authors cite the lawyer and author Philip Howard, who in books such as “The Death of Common Sense” has documented the dangers of bureaucracy, laws, rules and numerical ratings replacing human judgment in so many decisions.
Kahneman, Sibony and Sunstein also acknowledge the judicial backlash against federal sentencing guidelines passed in 1987 that were meant to reduce the massive inconsistencies. Many judges fervently believed that these federal guidelines — and even more stringent ones legislated in many states — sidelined them from making the human judgments they were put on the bench to make. That continuing backlash, and the fact that prosecutors and judges learned to game the new rules, has been a key force behind recent criminal justice reform efforts.
The authors’ general argument, however, is that there is now so much noise that a major hygiene effort is in order across multiple disciplines. In too many arenas, they maintain persuasively, we’ve allowed too much noise at too high a cost.
The trick is finding the right balance, not looking for perfect fairness or accuracy, which will always be illusory.
Kahneman elaborates on the backlash against the federal sentencing guidelines in interview with Behavioral Scientist (May 2021):
We do not want to eliminate intuition from the process, and by intuition, I really mean that subjective sense that it’s a judgment that you are making. What we’re trying to do is to delay intuition and process information prior to exercising your intuition. If we could convince judges to do that—to engage in a disciplined thought process before they form an intuition or global judgment—that would be a big improvement.
This is really essential. Clearly, if you prevent people, do not allow them to feel that they are doing an intellectual job, a job that occupied their mind, they will resist you. They will sabotage it, and this is definitely not the way to go.
With racism as one of the most urgent problems resulting from both bias and noise, a few related resources follow:
- JUNE 17, 2021: Upstander Advocacy in the Legal Profession (Free Event from Massachusetts SJC Standing Committee on Lawyer Well-Being and collaborating bar organizations)
- How to Be an Effective Antiracism Ally in the Legal Profession (LCL MA Well-Being Blog, 2020)
- The Role of and Need for Implicit Bias Training (ABA Law Practice Today, 2021)
- How Fast Thinking Slows Lawyers Down (Mass LOMAP Blog, 2018)
- Companies Need to Think Bigger Than Diversity Training (Harvard Business Review, 2020)
- How to Bust Bias at Work (WorkLife with Adam Grant, 2021)
- The Path to Ending Systemic Racism in the US (TED2020)
- How Racial Bias Works — and How to Disrupt It (TED2020)
- How to Be an Antiracist: What Lawyers & Law Students Need to Know (LCL MA Blog, 2020)
