Understanding and Overcoming Our Industry’s Knowledge Failures

Photo by K. Mitch Hodge on Unsplash


Software engineering is an industry notable for its strongly-held beliefs. Sometimes, we are also noted for our inability to reflect on those beliefs and question whether there is a factual basis underlying them.

While our beliefs usually serve us well, there are times when they don’t, especially when we believe something lacking in truthfulness. In such cases, we are likely to be led astray by our misguided notions, making us no different than our fellow humans, as best captured in the following quote:

Why our predictions continually fail and how to improve our results with learning-based approaches

Which way to go?
Which way to go?
Photo by Jon Tyson on Unsplash

Key Takeaways

  • The software industry has a dismal track record when it comes to predicting and planning in the face of uncertainty.
  • There are significant biases preventing us from learning, including cognitive biases and compensation structures.
  • Statistical approaches to predictions can be successful if we expend the effort to create learning-based models such as Monte Carlo simulations.
  • Highly uncertain environments are best exploited using the iterative learning models inherent to Agile methods.
  • Extremely uncertain, non-deterministic environments are best exploited by the incremental learning model of hypothesis testing (Hypothesis-Driven Development) and learning to accept the discomfort associated with uncertainty.

The Best Laid Plans…

“Prediction is very difficult…

J. Meadows

J. Meadows is a technologist with decades of experience in software development, management, and numerical analysis.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store