How to calculate how long a large project will be delayed? How likely it will be completed at all?
I'm talking Really Big Projects.
NASA plans to have astronauts skinny-dipping in Shackleton Crater by 2030. Norway wants to build the first ever commercial ship tunnel, done by 2030.
That's the level I'm talking about.
And also, I'm not talking about cost overruns or resource requirements, or anything like that.
I just mean, really big, really public, engineering projects -- typically chock full of political complications -- get announced in the news, like ...
"NASA to build Super-Duper-Big Telescope on Olympus Mons, for $8B, by 2034"
-- and everyone knows it's actually going to cost 6-16x that amount, and be ready to use in the late-2040s ... assuming it gets built at all.
I want something vaguely formulaic to calculate how much and how long it will actually take NASA to build that telescope.
Back before he became universally hated, it was kind of a meme-level joke that Elon Musk was already living on Mars time, so all of his project-completion estimates had to be multiplied by 1.88 (to convert from Martian back to Earth years). The funny thing, as a rough guideline ... that was actually a pretty reliable formula.
I want something like that formula, but given a bit more thought and research, maybe some optional variables to take into account the project circumstances.
There are plenty of studies (I've been looking) that seek to identify and calculate the causes of large-project delays and cost-overruns, and how to minimize/avoid them. But I'm not trying to rein in the delays and overruns ... I just want to have a way of semi-realistically calculating, right at the start, just exactly how badly people are underestimating the timeline for building their Big Shiny New Thing.
The only way to do this is to understand the project enough to be able to mentally divide it into segments, estimate how long each segment will take, figure out what can be done in parallel and what has to be done in series, then do the math. Obviously, that's a lot of work. It also requires more than just a passing familiarity with X project. A few methods are to see how long similar projects have taken in the past and to make consistent, small updates to your estimates as you take in new information.
The book Superforecasting isn't about this specifically, but is about how to refine your estimates and guesses to make them as accurate as possible.
One of my professors in college was heavily into Fred Brooks' The Mythical Man Month. Its lessons, unfortunately, have still not been learned by contemporary business culture. Namely, "throw more people at it" will make the project take longer, not speed it up, as you dramatically increase the time spent on onboarding and coordination, usually sidelining the most knowledgeable people.
My favorite quip about this...
What is IBM's definition of a Man Year?
720 guys trying to knock it out before lunch.
I have a problem on a technical level with this becsuse the idea is that you do eventually see gains.
I agree in spirit because, especially in engineering, management vastly underestimates how long onboarding can take. And that's the real problem. Garbage in, garbage out. If yohr planning misses such big parts of the equation, it will be thrown off for deadlines.
That and the fact that US careers being so fleeting these days doesn't help. Between politics, learned behavior, and lack of labor protections, it can be hard to build up knowledge when people are leaving/laid off/burned out every 1-2 years. If this happens to key oersonell, then you effectively need to re-plan the entire project. A project being redesigned every 5 years surprisingly struggles to performs an 8 year plan. Significantly.
Now a plan as lofty as space travel? We'll have "leaders" spend an entire career dragging their feet.
As someone potentially moving in a more project oriented role, thank you for that recommendation.
I was thinking of something more like a statistical analysis.
Eg, looking at 100 historic NASA projects, come up with an average delay, weighted by things like initial time and cost estimates, maybe political party in charge at start and/or completion, # of dependent precursot projects, etc.
Things that aren't project-specific dependencies.
Call it the Drake Equation of Megaengineering...
And cool book recommendation. It's already on my short-list.
I think to do this, you would get wildly different results based on what sample set you use, and who is to know which would be accurate?
In your example of a telescope on Olympus mons, do you look at all mega scale engineering projects? Do you just look at NASA projects? Do you look at just manned NASA projects? Do you look at just manned NASA projects with a large scale engineering element?
All of those will give you wildly different results, and it's hard to say which ones would be most accurate without digging deep into the implementations and figuring out exactly what they're doing and how they compare to past efforts. At that point you're no longer doing statistical analysis, you're just doing project management, and if we could figure out how to accurately do project management, we wouldn't have the cost and time overruns the first place.
You may well be right. I kind of suspect that, if anyone actually did a wide-ranging analysis – in NASA's case at least – the political climate surrounding a project during its lifetime would be a much better predictor of its success, than the project's actual complexity.
But that's just my hunch.
Does anyone know if there are any studies that have looked at things like this? Definitely not just/explicitly NASA, but really, any wide-ranging analysis that looks for common non-project-specific factors?
I think there are only two non-insane ways to approach this problem. (The problem of trying to be better than NASA at predicting how long a NASA project will take.)
The first is to give up entirely. Recognizing that to accurately forecast a project requires understanding the components in detail and that doing so is very difficult for a non-NASA employee, we can just throw in the towel. That's not insane.
The other is to throw modeling out the window entirely - even NASA's modeling - and say "look, problems like this are so intractable that even the experts at NASA who think they know what they're doing actually don't. For that reason, I'm going to take their estimate, multiply it by an arbitrary 50%, and call it good."
If you wanted more precision, instead of the arbitrary 50% you could use roughly how long project overruns tend to be. That average number is imprecise, but easy to calculate (if you're willing to put some work into the research), while trying to guess the specific timeline for any one project will never be right.
The difference between this and what you suggest is the weights. I suggest that determining those weights is impossible (and crazy) and will make you crazy. Instead, just determine "how long is the average project delayed?" and use that, no additional inputs advised.
I respect your desire to apply complicated statistical modeling to this, but there's just no way that'll be the best answer. Big projects are simply too different, too diverse, too multifarious to model accurately like that. We humans - especially those of us who are science-minded - fall so easily into the trap of trying to model things. This has sometimes been called 'mathiness,' or the veneer of sophistication that comes with numbers. It's an easy trap to fall into - but we have to avoid it. When we can model something properly, it's an invaluable tool. But when we can't, we should avoid modeling entirely.
c.f. Merton and Scholes, who created the first mathematical formula defining the price of a financial derivative and then used that formula to create a hedge fund... that went bankrupt and required a $3B bailout.
I strongly, strongly recommend the book "How Big Things Get Done" by Bent Flyvbjerg and Dan Gardner.
You need a book to answer this question in a satisfying way. This is that book.
So far no-one has mentioned the key word to a number of the world's largest industries: Logistics.
Construction is all about logistics. Trade is all about logistics. Production/manufacture is all about logistics. Technology development is all about logistics.
How do you get people to spend as little time as possible waiting and as much time as possible doing the things needed to be done in the right order to finish. How do we wait as little as possible? The waiting is what costs money.
Logistics is the art of planning. It's the entrepreneurial organization of resources to get the most out of as little as possible to maximize whatever you're trying to do.
When and how will we know the price and completion date of the ship tunnel?
Someone will make a contractually binding bid for how much they're willing to accept as payment for doing the job under a particular set of constraints.
Sure, there may be delays. The company awarded the job may go bankrupt, but that's relatively rare. If the public institutions buying write good procurement contracts, things generally get delivered on time and on cost.
The issues here are extremely often one of two things:
For guessing at when new technologies will mature, how much they'll cost and so on, all we've got are previous estimates and previous experiences developing previous technologies.
Those previous technologies are necessarily different to the new undeveloped technologies.
Another huge factor in play here are the systemic forces that lead to known underbudgeting to get a political project rolling, knowing that once money is spent, more money will follow.
Getting the ball rolling on building a project is what matters. That's a completely different game, and it's a huge business game with a lost of vested interests.
No other politician will stop a half-constructed bridge. If you start a bunch of things you can't afford to do all at once, you're forcing the hand of future politicians so they have to spend their limited time and agency on your project rather than whatever project they see as most important when they're in the deciding-chair.
This same dynamic exists for CEOs and companies, within families, within marriages and so on.
So, for the first section, in my mind ... Logistics means Gantt chart. The more accurate the chart, the closer the project is to staying on budget and on track.
The last section, yeah, that's a huge black box for me, and IDK if it's even possible to model the political aspect of large, public projects.
The other political aspect you didn't touch is the whole "districting" aspect ... If the project brings jobs to the politician's district ... I mean, at that point, completing the project at all, let alone on time/at cost, may well be considered a negative outcome.
Just finished this. Great recommendation, thanks.
I would think that the biggest determining factor would actually be its estimated completion year. But it also depends what organizational structure led to the formation of that estimate. An estimate that was made at NASA with the NASA waterfall method for making Gantt charts is going to be a lot different than an estimate from SpaceX.
Generally speaking, if an estimate is only a few years away it means people have a pretty good idea how long it will take and that it is a smaller project (or that they being very ambitious or they know it will take longer but the short time-span is more exciting and they'll be able to string along investors so it doesn't matter that their estimate is inaccurate). If an estimate is far into the future then it means that it is a big project or that there are more unknowns. I would guess that the error bars are going to look a lot bigger for projects which take longer.
There is an element of Earnestness, an element of Scope (more interconnected parts or more diversity of occupational fields), and an element of Novelty that you would need to take into account while evaluating a project's estimate.
(where a bigger Duration and Novelty also increases the error bars; I don't think Project Size really changes the error bars unless a project is badly managed--but a bigger Size / larger Scope make it more likely that there is more waste as different components just sit in warehouses while being blocked by other parts etc--however, an earnest estimate will account for the bulk of this)
If something takes longer to complete that also means it is more likely to be cancelled--especially as earlier political priorities and social inertia change. Also, the project may have originally been seen as an important scientific endeavor but the longer the project the more likely it is that we'll find satisfactory scientific answers through a smaller, cheaper experiment (or from a different branch of science altogether).
Years ago I developed a model for a decision aid system but never got around to publishing it. The long and short is that I used a combination of BERD data, generalized versions of the federal Technology Readiness Level, generalized version of the Manufacturing Readiness Level, and some other user inputs to estimate:
So you could put in high level information about a project, including what technologies and NAICS sectors it depends on, and it will estimate cost, timeline, and suggest how to achieve it through public policy.
Unfortunately, this is limited to US projects, and despite being designed for non wonks, it still requires some detailed knowledge about the goals of the project because it is intended to help improve project planning and allocation. Fundamentally, it takes tools used to assess a specific technology or manufacturing capability and abstracts it to the planning phase using economic data and some historical statistical analysis to try and improve decision making upstream of acquisition.
I think a good starting place works be to ask how they came up with these original estimates. How much were they driven by politicians and budget? How much were scientists and engineers involved? How much of the project is made up of unknown unknowns?