Another lesson is that smart professionals might give an…
““Another lesson is that smart professionals might give an instruction to a program based on a sensible-seeming and normally sound assumption (e.g. that trading volume is a good measure of market liquidity), and that this can produce catastrophic results when the program continues to act on the instruction with iron-clad logical consistency even in the unanticipated situation where the assumption turns out to be invalid. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. This is a theme that we will encounter again.””
About This Quote
This interpretation was drafted with AI assistance. It is one reading of the quote, not the author's own explanation.
Algorithms follow instructions literally, so faulty assumptions can cause disastrous outcomes when conditions change unexpectedly.
In simple terms: Programs act on assumptions without understanding context.
Key Takeaway
Validate assumptions before automating.
Themes
technology
risk
automation
assumptions
Mood
cautious
analytical
Type
warning
philosophical
When to use this quote
- financial trading
- AI deployment
- software development
- risk assessment
Key Concepts
logic
robustness
system design
Questions to Reflect On
- How can we detect hidden assumptions?
- What safeguards prevent rigid logic from causing harm?
A Different Perspective
Assumptions may be hidden and hard to test.