(x)slates

Goodhart’s Law

Goodhart’s Law states that: “When a measure becomes a target, it ceases to be a good measure.”

The first comment in this Lesswrong’s article about the law is indicative of this phenomenon in industry:

A good example from my history of doing this is when I worked for an ISP and persuaded them to eliminate “cases closed” as a performance measurement for customer service and tech support people because it was causing email-based cases to be closed without any actual investigation. People would email back and create a new case, and then a rep would get credit for closing that one without investigation either. The replacement metric was one I derived via the Theory of Constraints, inspired by Goldratt’s “throughput-dollar-days” measurement. The replacement metric was “customer-satisfaction-waiting-hours” — a measurement of collective work-in-progress inventory at the team level, and a measurement of priority at the ticket level.

KPIs are the biggest culprit. Unsophisticated reasoning and copycat behavior lead many to believe that anything that can be measured should be a KPI and that each KPI should have a goal. If you’re careless, KPIs can become a massive time suck and lead to endless debates about which should be included or discarded. Albeit catastrophic numbers, they rarely nudge anyone to alter their behaviors, making them effectively useless.

Data Skeptic has an informative video explaining this phenomenon.