Project Estimation

Estimating projects is notoriously difficult, and the larger the project the harder to estimate. But even small pieces of work for a single person are easy to underestimate. When you make an estimate base it on actual elapsed times of similar projects, always try to overestimate the time, and reduce the scope before promising more than you can deliver.

Everyone knows that construction jobs are typically going to take longer and cost more than quoted, from home renovations to major construction projects. But even though everyone knows this it still happens. Part of this may be due to misaligned incentives; you may be more likely to choose the builder with the lower and faster quote, so there is an incentive to under-quote even though it damages their reputation. But another part of it is the planning fallacy, where estimates are based on likely best case scenarios; no rainfall, all materials arrive on time and no mistakes that need to be fixed occur during construction. However in a long project some of these are certain to occur, which is why the time and cost is higher than quoted, rather than using an informed average of previous projects.

A similar thing happens in software and analysis projects. When planning you think through all the things that need to be done, make rough estimates of how long they would take (which are typically best case), and add them to get a total estimate. This “inside view” will be optimistic because there will be other work that needs to be done concurrently, coordination issues, technical and data issues even assuming that the scope will not change. This is a big problem because if you give this estimate you’ll inevitably disappoint stakeholders that are expecting your quote to be accurate, and lose their trust.

Kanhneman discusses this problem in Thinking Fast and Slow Chapter 23, The Outside View. The best solution is to look at how long similar tasks have taken in the past as a reliable indicator. Keep track of how long (in elapsed time) it actually has taken you to deliver similar solutions, and ask others involved in similar projects how long it has taken them. He also suggests adjusting the estimate slightly if you know the conditions are more or less favourable than typical, but I’d be tempted to keep these estimates small.

This works even better if you can have small frequent deliverables. This is a useful technique because people rarely know exactly what they want from analytics and software projects, but if you have something to show them they can evaluate it and give feedback. Having smaller deliverables also means you’ll more quickly build a database of time estimates for similar projects, so you can estimate more reliably.

When you give an estimated delivery date you’re going to be held accountable. Being late is bad, especially because there are often knock-on effects for downstream processes that will be delayed. So when estimating you shouldn’t be aiming for the median time, but somewhere near the 80th percentile. This might feel disingenuous but you’ll build up a reputation of someone who can be relied on to deliver.

Giving larger estimates can cause pushbacks. Sometimes there’s an existing deadline, or they have prior expectations of how long it should take. I recommend staying strong on your estimates and educating your stakeholders that this is how long it will really take. If the estimates don’t meet their needs you can negotiate on scope and resourcing. Spending a lot of time understanding the business problem places you better to reliably deliver the most value.

If you continue to be told your deadlines are too long and they’re not willing to negotiate on scope or resources it shows a lack of trust. The project is unlikely to be successful unless you can get past this, and it may be best not to accept the project if possible.

Being able to give reliable estimates builds trust and helps your stakeholders plan. The only way to do this is to track how much time it really takes to do things, and aim for the upper end when estimating. This is easier to do by building smaller immediately useful deliverables. When you build a reputation as someone who can reliably deliver what they promise you’ll build better relationships and do better work.