I’ve worked in software development for 20 years. Through those years, in the context of dozens of different projects in different industries, I have witnessed the evolution of software process organization from Waterfall, to RUP until nowadays, agile methodologies.
What is so challenging and difficult about software development processes that sparks so many discussions and even requires people and organizations who make a living by simply advising on the matter?
Well, my answer is – quality of the people. Second thing is – size of the team and the third thing – leadership by example.
I will start with an example of a project I once worked on back in 2004. Five guys (me included), were assigned with a task to develop a new worldwide part ordering system for a client in the automotive sector. Existing database (mainframe) was there. The requirements were there. We had no process at all. Our lead architect was developing with us, the analyst was sitting at a table next to us. The manager would come from time to time with a demand for the next delivery, we would all listen to him at that moment and return to work after that. Everything on that project went perfectly smooth and in time and budget. How would we call this? A success story that was agile even before the term was even used? Could be.
But an even better answer is that this was a project where only people with enough common sense and ownership were involved. And there were not too many of us. The guy who knew everything about the domain really cared about the business and was coding with us.
The following is an example of another extreme. A project with almost 100 developers, many managers, testers, analysts – all paid from a huge government budget. In the end, it did not deliver anything useful and huge amounts of money were spent. Many experienced process organizing experts were paid to come and help. It simply did not work. Ownership was not encouraged; common sense was lost long ago. The size of the team compromised both ownership and common sense in a very short period.
Why was the team so big, and why did we not move forward? The answers to these questions are specific each time something like this happens, but the rule is to never get in a situation where you need so many people working under the same deadline – unless the type of work you are doing can be done by a well-written script – there are products these days that generate source code for that type of work.
Common sense. Ownership. Size of the team. Lead by example. Agile concepts are nothing more but a derivative of these few terms, written by people who were lucky enough to spend time in such environments.
Why are we unable to ‘stabilize ‘software development in a sense that ‘throwing’ enough people at it will always solve the problem in an amount of time that is proportional to the team size increase? To make a comparison with another industry, let’s say that I am about to build a huge building and the architecture is already worked out in detail. My thinking is that having 100 construction workers will probably be 40 times faster than building it with 2 of them. I would lose some efficiency in the overhead of organizing all those people, assigning them to their next task in an optimal way – but it surely would go way faster. In software development, that is not the case – things could in fact even get worse – and the question is why.
Developing software means – making something new, something yet not developed. Otherwise you probably wouldn’t even get the assignment to build it. Or, if someone else has developed something similar before – this time it needs to be rebuilt, slightly differently and with entirely new technologies, in a different context, and for a different client. In other words, each time a software endeavor is about to start, it requires some ice breaking. Ice breaking is not done with hundreds of ships that know how to sail (knows “how to code”), it is done by a limited number of ships that are skilled in ice breaking. Only after, and not before, the sea is open for sailing, you can put hundreds of other ships, with a simple purpose of traveling from point A to point B. But, by the time the sea is open for sailing, you realize that in fact there is no need for hundreds of ships at all. We realize that we were in fact mislead by someone telling us that having our sea cleared out of ice will need years and hundreds of ships to accomplish.
Actual coding – sailing in the open sea with some ice blocks floating around – is the easy and fun part of any software project. Good ships (developers) are needed to navigate and avoid those remainders of the floating ice. To tackle them by recognizing and challenging badly written requirements, by writing well designed and readable code, by making a test for every change of the code in order to avoid hitting any ice ever again.
Heading towards the exact same destination is the next challenge, a challenge that we can overcome only with perfect communication, that is, a team small enough to think and talk alike.
Good software development is about having a small team of people with willingness (ownership) and capability to understand what the business actually needs (create and implement well written user stories). It is about making sure that the chosen technologies will always work in cohesion (continuous integration); that the whole thing will actually deploy on the required platform and that the delivered solution will effectively be maintainable in the long run (use test driven development).
