Face it. Software development teams suck at estimating when their projects are going to be completed. For anyone involved in software development; be it engineers, product managers, and customers this scene is all too familiar:
"How's the project coming? Will it be done in mid-April like you said?""Well, we're actually way behind. April doesn't seem doable, but that was always just an estimate.""How is that possible? You estimated up front that it'd only take you 3 months!?"
These are awful conversations to be a part of as all parties share feelings of frustration, disappointment, and often times anger. In fact, many legal battles have begun over a software team being incapable of meeting estimated deadlines (I've seen this first hand and they are nasty). The fact that this scenario is so common begs the question: "Why do software engineering teams miss their estimated deadlines so often?"
Related: Fast, Good, or Cheap. Pick Three?
I put this question to the community of Certified ScrumMasters and here are their responses:
More than a product of synergy between stakeholders and developers, software development is creative engineering.
Building software is different than erecting a skyscraper. There are dozens upon dozens of technologies one has to consider and integrate with during a large development project. Also there are many ways to 'skin a cat' and, depending on your performance criteria, one developer could engineer a solution and have it look completely different than had his peer developed it. It's this level of creativity and elegance which separates the best developers from the rest but this also makes it hard for us to gauge exactly how long it will take to develop.
I've heard some development teams take their estimates, double them, and then add some buffer time on top of that. While that may sound safe in theory, the reality is if you give a developer extra time, he will likely use it all and more. Developers see that as extra time to try and engineer the most elegant solution. Remember that we have to find the balance between creativity and time. There is no such thing as the perfect solution.
Related: Why IT Projects Fail: Avoid These Common Reasons
Asking for an accurate estimation from a developer is like asking an artist how long it will take him to complete a work of art. He's not sure, but he can tell you how long it took to complete similar pieces in the past.
If IT groups continually erase their past by regularly rebuilding their IT teams during a project, is it any wonder why they have a hard time with accurate estimates?
This is is a recipe for estimation disaster by agile development teams since a huge contributor of our estimations are based on team velocity. If the components of the team are in a constant state of flux, no meaningful estimation is possible. Patrick Henry once said: "I have but one lamp by which my feet are guided; and that is the lamp of experience. I know of no way of judging the future but by the past." If we don't have the experience of a consistent team, we have a very dim lamp lighting our way.
It is the nature of writing software that makes accurate estimations difficult. Only very well known processes can be accurately estimated.
Developing software involves balancing dozens of technologies, many of them unknown at the onset. This is best visualized with a graph of the Cone of Uncertainty.
This diagram clearly identifies that the largest amount of unknown factors exist at the start of the project and only as time progresses can you begin to understand what would be considered a reasonable estimate. The reasons for this are numerous; the team feels more comfortable with each other over time, the components which they'll be interfacing with have been researched, engineers are more familiar with the code-set as they contribute, and more. With this graph in mind, it's obvious that any estimations made in the beginning of a project are not 'promises to be written in blood.'
"Release new business value (as defined by 'people are paying money for it') every two weeks and people will stop harassing you about the accuracy of your estimates." - Malcolm Anderson at Pragmatic Agility
This is the beauty of agile software development. Businesses configured to engage with Scrum or Kanban teams are more concerned with the rapid delivery of features on a regular basis as opposed to waiting for the whole project to be dropped in their lap months later. More times than not in the latter scenario the project looks different than what it was intended to and the market has already moved on. By releasing and iterating regularly, your customers benefit right away and use their learnings to shape the software to best suit their quickly-shifting needs.
As you can see there are many reasons that software development teams suck at providing accurate estimations, but there is also strong support for businesses to focus less on the accuracy of the estimate and more on the benefits it can provide its customers each iteration. Remember, an estimation is only supposed to yield a rough value. It is unfair, even to the definition of the word, to expect a high precision outcome. As Swisher's Law states: A software project's timeline is immediately off-track as soon as it is provided (credit to Business.com's CTO Robert Swisher).
ScrumMaster Pro-Tip (courtesy of Malcolm Anderson): Because the meaning of the word "estimate" has been changed to mean: "promise written in blood" ask your stakeholders to promise you that they will never use the word again, and instead replace it with "speculation."
So next time you are looking for an estimate from a software development team, remember you may get a more accurate answer if you asked a geologist how much a rock weighs? And that's okay.