Automating a Decision Nobody Had Actually Made

The Global Signal

On August 1, 2012, Knight Capital Group, then one of the largest market makers in US equities, deployed new trading software intended to route orders under a new exchange program. The deployment reused an old, unused flag in the company's order-routing system, a flag that had previously triggered a defunct testing function called "Power Peg." Knight Capital's technology staff updated seven of the eight production servers running the relevant code with the new software; the eighth server retained the old code, which, when the new flag was sent, activated the dormant Power Peg function and began generating a rapid, unintended stream of live orders in 154 stocks. In approximately 45 minutes before the issue was identified and trading was halted, Knight Capital incurred losses of approximately $440 million, an amount that exceeded the firm's own capital and forced it into an emergency recapitalization that ultimately led to its acquisition by Getco later that year, forming what became KCG Holdings. This sequence is documented in detail in the US Securities and Exchange Commission's October 2013 administrative order, "In the Matter of Knight Capital Americas LLC" (SEC Release No. 70694, October 16, 2013).

No one at Knight Capital had explicitly decided, at any point in the deployment process, what should happen if the new code reached some servers and not others, or what the automated system should do if it encountered the old, dormant flag. The decision the incident actually turned on, how the system should behave under a specific, foreseeable deployment inconsistency, had never been made by a person. It had been left, by default, to whatever the leftover code happened to do.

The Hidden Signal

Automating a process converts a series of decisions that used to require a person's judgment into a fixed set of rules that will execute exactly as written, including in situations no one explicitly considered when the rules were written. Consider a hypothetical scenario, smaller than Knight Capital's case but illustrative of the same mechanism: a retailer automates its pricing-exception approval process, so that discount requests below a certain threshold are approved automatically without human review. No one explicitly decides what the system should do if a request is submitted twice in rapid succession due to a customer double-clicking a button, so the automated system, having never been asked that question, approves both, silently doubling a discount that was only supposed to apply once. The automation did not remove the decision about how to handle a duplicate request; it simply meant no person ever made that decision before the system started acting on it.

Why the Visible Metric Misleads

An automated system passing its test cases and functioning correctly under normal conditions is often treated as evidence that the underlying operating model has been thought through, when passing normal-case tests says nothing about whether anyone has explicitly decided what the system should do under the specific, foreseeable edge conditions that eventually occur in practice, a partial deployment, a duplicate request, a sensor disagreement. Knight Capital's new trading code worked correctly on the seven servers it was properly deployed to; the entire loss originated from the one condition, a partial rollout leaving old code active on an eighth server, that no one had explicitly decided how the system should handle.

Implication for Operators

Any organization automating a workflow that previously required human judgment should assume that every foreseeable edge condition the system will encounter represents a decision that either a person makes explicitly before deployment, or the code makes implicitly and silently once it is live. The practical shift is treating "what should happen here" as a required, documented question for every foreseeable edge case, not an acceptable gap to discover after deployment.