Real strength Quote by Joel Spolsky
““In any case, even if a usability test resolves a dispute, it doesn't do it in any kind of a statistically valid way. Unless you test thousands of people from all walks of life under all kinds of conditions, something that not even Microsoft can afford to do, you are not actually getting statistically meaningful results. Remember, the real strength of usability tests is in finding truffles—finding the broken bits so you can fix them. Actually looking at the results as if they were statistics is just not justified.””
About This Quote
Source Blog: Joel on Software, 2005
Usability tests reveal specific problems but lack statistical generalizability without massive samples.
In simple terms: Tests find bugs, not broad stats.
Key Takeaway
Use tests to spot issues, not prove hypotheses.
Themes
usability
testing
statistics
product design
Mood
analytical
cautious
Type
cautionary
informative
When to use this quote
- software development
- website redesign
- app testing
Key Concepts
human‑centered design
sample size
validity
Questions to Reflect On
- When is a small test sufficient?
- How to supplement with other data?
A Different Perspective
Statistical claims may mislead decisions.
![“In the real world of research, conventional tests of [statistical] significance seem almost worthless.” quote by Fischer Black](https://sms4smile.com/quotes/img-quotes/3a4/fischer-black-60662_conventional_in-the-real-world-of-research-convention_163947.jpg)
![“It is not a simple matter to differentiate unsuccessful from successful experiments. . . .[Most] work that is finally successful is the result of a series of unsuccessful tests in which difficulties are gradually eliminated.” quote by Anonymous](https://sms4smile.com/quotes/img-quotes/66e/anonymous-1_difficulty_it-is-not-a-simple-matter-to-differentia_1365505.jpg)
