Agile vs. Estimates
Once of the most common issues I hear when it comes to Agile adoption is the "lack of estimates." I'm not sure where this notion comes from precisely. Looking at the manifesto and principles, there doesn't seem to be mention about estimation directly, but I suppose it could be construed from this one line of the manifesto:
Customer Collaboration over Contract Negotiation
The terms contract and negotiation have fairly negative connotations in my mind, but perhaps the business folks are more comfortable with them. After all, the work of software is to perform a service or operation in some way which saves or makes an organization money.
Any time the business spends money, this is why. The customer doesn't care about PHP or HTML5 or NoSQL; your system could run on Unicorn blood and Kraken oil so long as it solves their problem. You deliver a product or service which solves a problem and they are paying you to help them solve it. Contracts are the formalization of what it means to solve their problem with the understanding of what it will cost to solve that problem. Negotiation is the process of establishing a contract, so that all parties have a common understanding.
When adopting the Agile mindset, does that mean you eliminate contracts and negotiations? Hardly. Remember the post-script, it isn't a black-or-white, it is a matter of the degree:
That is, while there is value in the items on the right, we value the items on the left more.
We can't reasonably eschew contracts or negotiations, but we can change the form in which they exist. In fact, most Agile methodologies have contracts, only with different names. What else would you call working agreements, story cards, and definitions-of-done? Heck, even the notion of regular iterations are a form of contract - it states that the team will deliver working software every x days.
How are Estimates Created?
The customer would obviously prefer if devs gave a precise dollar amount and a completion date for a given scope of work. Of course that supposes that the scope of work is precisely defined, which despite the customers' best efforts is quite difficult.
You know an organization which is actually good at estimating projects? NASA has a pretty good reputation, and their missions are large, ambitious, complicated, and (admittedly) expensive with tons of uncertainty. I don't think many organizations, however, are willing to go to their lengths to ensure stewardship of taxpayer money. Running a Monte Carlo simulation analysis, while actually covered by the PMI literature, isn't something I've see in practice. That is to say, typical business customers care about accurate estimates, but are not actually willing to devote the resources to make that happen.
Put simply, the more effort you spend estimating, the higher the accuracy. Crazy, right?
Instead what I normally see is estimates pulled essentially from thin air, with a veneer of authenticity provided by historical averages, with 20% added by senior developers as a CYA measure, which then gets another 20% from seasoned (jaded?) project managers. And then the project gets blown by 50% on time and budget anyway. Nobody is happy, but we keep doing this over and over again, as if this form of self-flagellation will eventually purge us of our sins.
sigh
Truths about Estimates
Agile doesn't mean we skip estimates. Â Instead, we strike a different deal than the one above. There's still contracts, there's still negotiations, but they happen differently.
It means we acknowledge that we're not going spend enough time doing estimation up-front on a large project to be very accurate about it. That's partially because there isn't much value delivered in efforts spent on estimation. That's partially because the time taken to estimate delays responding to the business environment. The thing to understand is that we're relying on human understanding and intuition to build these estimates, which means there are psychological factors to keep in mind.
Smaller estimates are more accurate than larger ones
The fact is, it is easier to reason about a small task than a larger one. I can't imagine much argument on this, but there is always a tendency to want every project to be some grand pursuit. Real Artists Ship, and so the focus on delivering value and completing the entire process every iteration means keeping things rooted. This is why there is such an emphasis on planning poker or other story-level estimation - that's the smallest unit of work we manage at the team level. Even on a larger feature-level, the game is the same, though we understand that the accuracy will be lower. The (sometimes modified) Fibonacci sequence helps with this; larger values are more spread out and help convey this ambiguity.
Relative estimates are more accurate than absolute ones
In general, people can gauge differences more easily than they can judge absolutely. Quickly: which is taller, the Statue of Liberty or the Washington Monument? I'll wait....
If you follow the links, you'll see that one is nearly twice the size of the other. And yet I had trouble coming up with the answer even though I've seen both of these structures in person. I would imagine this is true for most people. In the mind, both of these are considered "large" and "significant structures", and tend to be grouped together despite their differences. How could we be so unsure about answering the question given such a large disparity in height? This is because you've never seen them close together enough to compare them. From the height of buildings to the distances between cities to the color you want to paint the dining room, it is hard to be absolute about value, but reasonably easy to see differences when the items are side-by-side. This is the power of relative estimation. We may not know exactly how long something will take, but we can more easily say that one thing will take (roughly) the same/more/less effort than another.
Nearer estimates are more accurate than distant ones
How long will it take to prepare dinner tonight? How about 3 days from now? Next Friday? Labor Day? You may know that tonight the plan is for spaghetti, but 3 days from now may not be planned at all. Or if it is planned, you probably haven't thought much about the time it will take to prepare it, only that you know you'll need to. The longer you have to look into the future, the less you know about the context in which things take place. The further out a task is, the less that can be understood about the domain, and so the estimate grows uncertain.
More, smaller estimates
To wrap things up, a more Agile approach would involve smaller, more comparable, and more timely units of work to estimate. In this way, the accuracy improves on the estimates, and the business can more easily make decisions about where to invest their resources. As with most things Agile, this requires more collaboration between developers and customers to shorten the feedback loop so that these estimates continue to get better and provide more value for everyone in the process.