Monday, January 4, 2010

If people lived a 1000 years

Sometimes I think what it would be if people lived some 1000 years. Most people, I mean, not just the one who found some tree of life. I know some negative sides of this coin, but there are positive ones, which are strong. Lets say they don't get children each year etc.

I have thought about the following:
  • Number of friends and other people in your life.
  • Number of places you know well.
  • Number of fields of interest, knowledge.
  • Value of experience.
  • Music and things you own.
And, one by one...

Number of friends

Counting firends family etc. Who is socially inactive, will go through lifetime with closest family members and few friends - I have one close relative, 28 years old, who is like that. Who is socially really active, will find one or two friends each year and meet them all often.

You can skip this part: Young people see their friends each month. Older people see their friends several times per year. Oldest people have, in case they are still active (lets take my grandmother) thousands of friends - people they know well -, who all have their children, children of children etc. Oldest people see their friends, many of them, few times per year and talk about the biggest life events, mostly, and most important things they have found out meanwhile. They still have enough memories together to be close, their children get to know each other and they have many hidden connections through that. Oldest people may know people many generations back in time and they understand, thus, many more relations between people. They also have had time to know tens of thousands of people - when total music maniacs I know might know some few thousand musicians at age 20 or 25 (when they have been aware of this world maybe from age 10), they also know few hundreds or possibly few thousands of people in their life (family, friends, friends of friends, people from schools and workplaces) and a little bit they know some people who sell food at shops, people who sell kayaks and people who repair their electric devices. They also know some well-known people - presidents, many politicians and such, which might add some more thousands. They know actors (at least few, but when you start thinking, you can recall new and new ones without an end even if you don't watch much TV, just think about movies and people in those). This is possible, that people at my age know some 10-20 thousand other people; they have met and talked with several thousands they remember. I can recall some people from food shops and supermarkets I have seen over years, someone I used to talk at some school (who I have never met afterwards, just some small talk) etc. I'm certain that total number is several thousands. After more 5 times as much time, there could be ten or twenty thousands - especially if I kept the full health and activity. And I'm not the kind of person who hangs around at parties all time - I do it few months per year, other time I'm alone and programming or being with my wife alone.

So, after 1000 years (this makes maybe 900 years of active time instead of 60) you will get 15 times more close friends and those less close friends - many of those get closer over years (with a person you see once per year you have met already 500 times after 500 years instead of 20 - that's like being together over one full year) ...this means that instead of knowing 100 people well, 100 people twice less well and 1000 people somewhat after 10 years, you might know, after 1000 years, 10 000 people well, 10 000 people twice less well and 100 000 people somewhat. From movies, television etc. you will know million if you now know ten thousands!

This is not just quantity. You will know relations between people and entities. When you see some totally different and new kind of person once per year, you know them 900 after 1000 years, you have broad circle of understanding.

Number of places

Ok, that does not need explanation. You know, how much you travel. When you meet a new place and live there a few months, you start traveling there more often, your world grows bigger. If you, for now, know 2 cities well and this is 30th year of your life - after thousand years, you might know some 40 cities as well. But that's possible you start travelling more often over years. If you know 10 cities well, you will know 400. If you travel to one place for one week twice per year and live some two months in some place in each five years - ok, calculate yourself.

Thus, you will know a lot of different places and relations between them. Earth becomes more smaller. You will know, after 1000 years, many places similarly well as you know your home city right now. That's a lot.

Number of fields of interest

People, who don't learn fast and much, still learn over time - they read some wikipedia articles each month, new needs will appear in their life and they read newspapers or watch TV. Everyone learns.

I had, for age 10, some interest in physics, math, logic and other fields. For age 20 I knew a lot about programming. I still develop myself in fields not my choice - but at the end of my life, I will probably know only few fields very well. Anyway, those fields I know in degree of 10% of professional knowledge (and there are many of those) I would know 40% when I'm 70 years old. I would have professional knowledge about 4 or 5 fields at that point in case I manage my time effectively - like programming (number of programming fields) and several related fields, which help to make my programming job more practical and effective - you know that, programming has no big value in case the program is not written being fully conscious about some real-world problems this program is supposed to solve.

After 1000 years, I would be professional (at todays standards) in some 40 or 50 fields (and that's a lot!) given that I know 4 or 5 at year 70, also for those I know some 40 or 50% professinality at year 70 (there are tens of that) I would have learned maybe 100 at professional level or just additional 90 fields in 40-50% professionality level. In each case, there would be people being professional in several hundreds fields and normal people would be professional, as much as they are now in few fields, in tens of fields. Also, what I didn't even consider - in time, you will find connections between different fields. I mean, if you know both physics and chemistry well, they have a lot in common - and actually learning one is also somewhat learning another. Learning about latin language will make you better in several other languages and learning some general math helps in finances and architecture. Thus, actually, instead of knowing so many fields, you would know just so much more about the world.

Value of experience

We make mistakes. It makes a few long years to figure out some of them and find the ultimate solutions. As we get rid of small mistakes, we get bigger challenges, responsibility in bigger things and thus do bigger mistakes. This is a normal course of life - how much responsibility one can carry. How well one can finish the task. How fast one will understand the new topic (this is a skill and it develops slowly). How much one can take feelings, matters of things and finances, normal course of life events, different things into account when making some decision. How often one will fail with things they start.

Someone, who is been here some 30 or 40 years, will always do mistakes - they can fail with project, loose job, go into conflict with wife, buy some expensive thing, which will not give what it was meant to give, create some unsuccessful company, hire someone who will fail in something, whatever. Many of those are serious things. If you are some 70 years old, you usually fail less often. If you was 1000 - you would possibly carry some things out, all the time, with really good results. Think that you will have also mentors, teachers and authors of books you read with same experience level! You have people, who have learnt five years and then simplified their knowledge so much that they could say in five sentences something, which would have taken some other 100 years to learn for you. And, still, you are also consious about some millions of details behind things in life, you can make up your own conclusions and recognize such truths when you see them. You have thousand of people to ask, you know their interests.

Music and things you own

You have listened some number of songs, some number of complex rythyms etc. You would have listened hundreds times more. You have been in hundreds of shops, chosen things you like - you would have been in many times more shops, your home would be much more better (and even if it burnt down, you would at least know, what to buy and where, who is able to provide you that). You would know, what is a quality - something, what even money can't give you.


Thus, the life would be different. Not only longer, but different.

Software development problems with time estimations

After some 10 years of commercial programming, I would like to share some observations about how some common practices in time management can fail.

I will show in this article:
  • It's wishful thinking to think you can estimate your development process. Or it is not, in fact, a development process (like creating a new simple page on CMS engine is rather professional use of ready-made software).
  • Tight deadlines might be a way to failure. They often lead to estimate project's end-time constantly to next or overnext week, which actually delays getting it ready.
  • Loose deadlines are not a solution.
  • Agile processes are hard to follow, but well-reasoned.

In software development in general it is very hard to estimate times exactly - this fact might seem counter-intuitive, because in many areas it's simple to do time estimation by calculating units to be built, average time spent for one unit before and the percentage of units done from total number of units. There can be several kinds of units and some additional procedures, but usually it's still possible.

In software development, it's usually not possible to estimate time. It's possible to say roughly, based on experience and some facts, how big a project is - is it more like a month or like a year. Anyway, there are no hard facts supporting even such kind of prognosis - you can't say number of units needed. Number of code lines is always more or less taken from an air, number of packages is more like a problem of dividing those code lines into categories than strong estimate about how many of them there will be.

Thus it's natural conclusion and a good practice used in iterative and incremental programming to not estimate too much. At least programmers usually can understand the fact that times are unestimatable - especially experienced programmers, who don't take those numbers taken from air too seriously not fall into numerous wishful thinking traps.

This fact is not so obvious for customer. Customer is usually a business with limited resources, competitors and strong pressure from all directions. Yet worse - customer has long tradition of setting clear deadlines and creating timely project. Customers have also standard methods of calculating cost and benefits, pros and cons. Software is a detail of some more general product or effort of customer. Having some part of bigger picture need some totally different method of risk calculation, planning and estimates is obviously very uncomfortable for them, often simply unacceptable. For clients, iterative planning with it's uncertainties is totally different paradigm - they try to fit it into their general business method and this cuts away most important parts of it.

This, now, leads to serious problems of many kinds. Usually, some general estimate of time is given and client supposes that whole functionality is ready for that time. This estimation, in practice, is rarely the worst-case scenario - it's sadly often the best-case scenario, especially in case of less experienced managers. People under pressure tend to do what pressure at that moment needs, not what would be reasonable or produce correct result in long term, because it's simple to use wishful thinking when handling long-term problems. As there is no middle way - you cant make deadlines partially fixed so easily. For software developers it's also clear that when they set some deadline they can guarantee, some competitor will agree with shorter one - and clients usually wait a little bit more if deadline has passed, thus it's even better strategy. It might even be said that for many, solution to this problem is simply choose such deadline that they would have the smallest risks in general - risk to loose the project and risk to fail. Nontrivial projects still fail often.

Inside company and for those managers themselves this kind of decision making strategy is often not clearly worded - it's stabilized by free market method, which simply chooses those managers, who set deadlines with smallest risks. Those managers will write books explaining their reasoning and that reasoning will spread as giving best proven results. This is one line of defense, but it still raises problems inside a company, which form another set of problems raised by such timings. It's also to be noticed that even if such setting of deadlines is actually not so much agile methods - agile methods do not support setting deadlines with known resources and functionality -, it's still practiced by many, who are using agile methods. Agile methods are an effective tool for managing a lot inside a company, but they alone do not provide proven and satisfactory client relations unless for companies, who are working for very professional and experienced customers. Also, in fact, if you tell to client that this application could possibly be ready in five weeks, client will read that it's possible to get it ready in five weeks - so, they will mark to their project plan that it's five weeks plus some percentage of buffer time. They do not accept that you haven't yet written those code lines, which all take some research and many unpleasureful surprises will come on the way. Many managers seek for their perfect programmers like some alchemist might seek tree of life or formulae of success - in programming world, there are no formulas of success. If you get so much experience with some tool to code it with guaranteed time, you will compile your code into library and have no need to code it at all, then there will soon be some similar GNU library or at least something similar per each competitor. Real work is and stays research work - and well known quote from science goes that "if we knew what we are doing, it wasn't science". If we knew what we are doing, it's also not programming - and it's really hard to accept this fact in world of many people wanting guarantees and safety.

This other problem is defined as follows: finishing a project is made up from a series of clearly defined steps. Those include planning, implementing, testing, releasing. Their actual consistence is more complex than this simple linear series of few steps. Now, when you move the deadline to some time after a week, a programmer will try to fit everything into this week. Fearing to lose job or just not wanting to argue and prove, programmer uses wishful thinking to follow everything as if this deadline was possible. Some programming effort will be done in first, say, three days. This includes fixing a few obvious bugs and adding some very buggy and low-quality piece of code for each missing function. After that, a day or two is spent to test this code, which is buggy by nature. After testing, some try will be there to handover this product to client unless it's impossible to ignore the fact that it's not ready (like the case of having simply nothing on screen). If there are some bugs, like application crashing after first 10 seconds, it will be given to client saying that there are a few bugs. So the week is spent to produce some code with so low quality that searching bugs from it might take more than the code itself took, test this obviously unready thing as a whole and try to communicate to client that it's almost ready. Sadly, there will be weeks and weeks of such tactics, possibly wasting some 50% or 75% of time to absolute nonsense. It also makes some need to have many hours of unnecessary discussion, searching mistakes and blaming everyone and everything - in case of more experienced manager, there is often no direct blaming, but there is also no clear solution to the problem, actually the real problem is simply dismissed. In practice, if it's clear that a project is far from ready when deadline is there, it would be safe to at least double the time estimation - anyway, it often seems reasonable to managers to add time week by week. It does not seem to contain any big problem to demand things sooner than possible - but it's trap, demanding things to take less time than they actually take will mean that they will take more than they would have taken otherwise.

I have worked for companies, which simply do not set deadlines. This, anyway, is only possible to large companies having enough resources to take some clearly defined risks, having research units, which do the projects with very high risk (called experimental); having several contracts with different companies to create the same required functionality with different tools or methods etc. All of those are impossible for small company - small company simply will not face the fact that they are risking with their existence and playing some kind of dice. Small companies take the numbers on paper and change them until they seem to make sense from some angle and do not mean a bankruptcy. Otherwise it would be hard to keep some level of moral - or in worse cases, to keep workers from already searching for their next job. Anyway, several reasons like ambition, life needs and underestimation of necessary effort lead companies into taking such projects. Projects without a risk are often boring, small and dead-end projects. One manager told me that not setting deadlines is good for two reasons - if you set it too small, you gain nothing; if you set it too large, programmer will slow down. If you are good enough to understand, when time is used efficiently and when it's not, you are able to lead without time estimates given to programmer.

Now, there is another solution - once again, an official part of agile methods. It's to tell client that you really don't know what's ready for given time and that you need a priority list and create software based on that. You can closely cooperate with a client. Not surprisingly, even that works best with client with some experiences in software development. For beginner customer, it's hard to accept - they want their results and they want to know, they don't want some company, who does not know how to get results and does some research and learning in process. They want a product. They can buy multimillion-dollar worth of effort from shop as Windows, they can get Linux for free in a few hours - and now, someone is going to ask such sum from some simple widget totally lost into richness of their desktop? And says that this will be a research effort? Clients won't want to create a priority lists or live with uncertainties. Anyway, if you can make it clear that a priority list is needed, you will face another problem - it's actually really hard to create a good priority list. I have personally worked on software for quite a while and still learning, how to prioritize tasks. Most people have no way to prioritize tasks in their life. Another problem with that - you as a software developer can actually imagine new features, customer often does it very vaguely, they have to see and touch things to say how important they are and are they correctly done at all. So, in most cases those decisions are actually mostly done by programming company. That's some of the places where the "good communication skills" needed by many CV-s come into play. But first you have to know, what you want to communicate.

As a conclusion - it's hard to get around the current accepted methodologies, but it's also hard to get into them or make them clear to customers. As each day makes some new things clear about the software you are developing, you must make some new decisions each day - to continue, to stop, to re-evaluate or reprioritize. To tell client that some estimate was wrong as soon as it gets clear, which must be tactically well thought-through. To make some painful decisions and observations - some of those mistakes might be avoided by facing the truth, facing the incompleteness of this truth and doing the best in uncertainties. Some can be avoided by choosing simpler projects and going in smaller steps - but it is also well-known fact that big results are usually achieved by taking big risks. In total sum, it produces a few really successful companies and when looking one-by-one, it produces a lot of failures. Hopefully to learn from them.