I will show in this article:
- It's wishful thinking to think you can estimate your development process. Or it is not, in fact, a development process (like creating a new simple page on CMS engine is rather professional use of ready-made software).
- Tight deadlines might be a way to failure. They often lead to estimate project's end-time constantly to next or overnext week, which actually delays getting it ready.
- Loose deadlines are not a solution.
- Agile processes are hard to follow, but well-reasoned.
In software development in general it is very hard to estimate times exactly - this fact might seem counter-intuitive, because in many areas it's simple to do time estimation by calculating units to be built, average time spent for one unit before and the percentage of units done from total number of units. There can be several kinds of units and some additional procedures, but usually it's still possible.
In software development, it's usually not possible to estimate time. It's possible to say roughly, based on experience and some facts, how big a project is - is it more like a month or like a year. Anyway, there are no hard facts supporting even such kind of prognosis - you can't say number of units needed. Number of code lines is always more or less taken from an air, number of packages is more like a problem of dividing those code lines into categories than strong estimate about how many of them there will be.
Thus it's natural conclusion and a good practice used in iterative and incremental programming to not estimate too much. At least programmers usually can understand the fact that times are unestimatable - especially experienced programmers, who don't take those numbers taken from air too seriously not fall into numerous wishful thinking traps.
This fact is not so obvious for customer. Customer is usually a business with limited resources, competitors and strong pressure from all directions. Yet worse - customer has long tradition of setting clear deadlines and creating timely project. Customers have also standard methods of calculating cost and benefits, pros and cons. Software is a detail of some more general product or effort of customer. Having some part of bigger picture need some totally different method of risk calculation, planning and estimates is obviously very uncomfortable for them, often simply unacceptable. For clients, iterative planning with it's uncertainties is totally different paradigm - they try to fit it into their general business method and this cuts away most important parts of it.
This, now, leads to serious problems of many kinds. Usually, some general estimate of time is given and client supposes that whole functionality is ready for that time. This estimation, in practice, is rarely the worst-case scenario - it's sadly often the best-case scenario, especially in case of less experienced managers. People under pressure tend to do what pressure at that moment needs, not what would be reasonable or produce correct result in long term, because it's simple to use wishful thinking when handling long-term problems. As there is no middle way - you cant make deadlines partially fixed so easily. For software developers it's also clear that when they set some deadline they can guarantee, some competitor will agree with shorter one - and clients usually wait a little bit more if deadline has passed, thus it's even better strategy. It might even be said that for many, solution to this problem is simply choose such deadline that they would have the smallest risks in general - risk to loose the project and risk to fail. Nontrivial projects still fail often.
Inside company and for those managers themselves this kind of decision making strategy is often not clearly worded - it's stabilized by free market method, which simply chooses those managers, who set deadlines with smallest risks. Those managers will write books explaining their reasoning and that reasoning will spread as giving best proven results. This is one line of defense, but it still raises problems inside a company, which form another set of problems raised by such timings. It's also to be noticed that even if such setting of deadlines is actually not so much agile methods - agile methods do not support setting deadlines with known resources and functionality -, it's still practiced by many, who are using agile methods. Agile methods are an effective tool for managing a lot inside a company, but they alone do not provide proven and satisfactory client relations unless for companies, who are working for very professional and experienced customers. Also, in fact, if you tell to client that this application could possibly be ready in five weeks, client will read that it's possible to get it ready in five weeks - so, they will mark to their project plan that it's five weeks plus some percentage of buffer time. They do not accept that you haven't yet written those code lines, which all take some research and many unpleasureful surprises will come on the way. Many managers seek for their perfect programmers like some alchemist might seek tree of life or formulae of success - in programming world, there are no formulas of success. If you get so much experience with some tool to code it with guaranteed time, you will compile your code into library and have no need to code it at all, then there will soon be some similar GNU library or at least something similar per each competitor. Real work is and stays research work - and well known quote from science goes that "if we knew what we are doing, it wasn't science". If we knew what we are doing, it's also not programming - and it's really hard to accept this fact in world of many people wanting guarantees and safety.
This other problem is defined as follows: finishing a project is made up from a series of clearly defined steps. Those include planning, implementing, testing, releasing. Their actual consistence is more complex than this simple linear series of few steps. Now, when you move the deadline to some time after a week, a programmer will try to fit everything into this week. Fearing to lose job or just not wanting to argue and prove, programmer uses wishful thinking to follow everything as if this deadline was possible. Some programming effort will be done in first, say, three days. This includes fixing a few obvious bugs and adding some very buggy and low-quality piece of code for each missing function. After that, a day or two is spent to test this code, which is buggy by nature. After testing, some try will be there to handover this product to client unless it's impossible to ignore the fact that it's not ready (like the case of having simply nothing on screen). If there are some bugs, like application crashing after first 10 seconds, it will be given to client saying that there are a few bugs. So the week is spent to produce some code with so low quality that searching bugs from it might take more than the code itself took, test this obviously unready thing as a whole and try to communicate to client that it's almost ready. Sadly, there will be weeks and weeks of such tactics, possibly wasting some 50% or 75% of time to absolute nonsense. It also makes some need to have many hours of unnecessary discussion, searching mistakes and blaming everyone and everything - in case of more experienced manager, there is often no direct blaming, but there is also no clear solution to the problem, actually the real problem is simply dismissed. In practice, if it's clear that a project is far from ready when deadline is there, it would be safe to at least double the time estimation - anyway, it often seems reasonable to managers to add time week by week. It does not seem to contain any big problem to demand things sooner than possible - but it's trap, demanding things to take less time than they actually take will mean that they will take more than they would have taken otherwise.
I have worked for companies, which simply do not set deadlines. This, anyway, is only possible to large companies having enough resources to take some clearly defined risks, having research units, which do the projects with very high risk (called experimental); having several contracts with different companies to create the same required functionality with different tools or methods etc. All of those are impossible for small company - small company simply will not face the fact that they are risking with their existence and playing some kind of dice. Small companies take the numbers on paper and change them until they seem to make sense from some angle and do not mean a bankruptcy. Otherwise it would be hard to keep some level of moral - or in worse cases, to keep workers from already searching for their next job. Anyway, several reasons like ambition, life needs and underestimation of necessary effort lead companies into taking such projects. Projects without a risk are often boring, small and dead-end projects. One manager told me that not setting deadlines is good for two reasons - if you set it too small, you gain nothing; if you set it too large, programmer will slow down. If you are good enough to understand, when time is used efficiently and when it's not, you are able to lead without time estimates given to programmer.
Now, there is another solution - once again, an official part of agile methods. It's to tell client that you really don't know what's ready for given time and that you need a priority list and create software based on that. You can closely cooperate with a client. Not surprisingly, even that works best with client with some experiences in software development. For beginner customer, it's hard to accept - they want their results and they want to know, they don't want some company, who does not know how to get results and does some research and learning in process. They want a product. They can buy multimillion-dollar worth of effort from shop as Windows, they can get Linux for free in a few hours - and now, someone is going to ask such sum from some simple widget totally lost into richness of their desktop? And says that this will be a research effort? Clients won't want to create a priority lists or live with uncertainties. Anyway, if you can make it clear that a priority list is needed, you will face another problem - it's actually really hard to create a good priority list. I have personally worked on software for quite a while and still learning, how to prioritize tasks. Most people have no way to prioritize tasks in their life. Another problem with that - you as a software developer can actually imagine new features, customer often does it very vaguely, they have to see and touch things to say how important they are and are they correctly done at all. So, in most cases those decisions are actually mostly done by programming company. That's some of the places where the "good communication skills" needed by many CV-s come into play. But first you have to know, what you want to communicate.
As a conclusion - it's hard to get around the current accepted methodologies, but it's also hard to get into them or make them clear to customers. As each day makes some new things clear about the software you are developing, you must make some new decisions each day - to continue, to stop, to re-evaluate or reprioritize. To tell client that some estimate was wrong as soon as it gets clear, which must be tactically well thought-through. To make some painful decisions and observations - some of those mistakes might be avoided by facing the truth, facing the incompleteness of this truth and doing the best in uncertainties. Some can be avoided by choosing simpler projects and going in smaller steps - but it is also well-known fact that big results are usually achieved by taking big risks. In total sum, it produces a few really successful companies and when looking one-by-one, it produces a lot of failures. Hopefully to learn from them.
No comments:
Post a Comment