When Alan Turing wrote his book “On Computable Numbers, with an Application to the Entscheidungsproblem” in 1936, he laid the grounds for computer science. The first “universal machines” were developed at Bletchley Park to decipher the Germans’ encryptions during the Second World War.
Between the 40’s and the 70’s computer science became more of a scientific instrument than a well-established business technology. It wasn’t until IBM and Apple introduced the PC and Macintosh computers that computer science began to spread outside the scientific institutions, making their way to the largest companies. It was at this time that software development started to take off and we saw the birth of large companies.
Up until the 70s, programs often were quite simple and were operated only by those who created them. But as systems became larger, it also became more difficult to develop and organize software development.
In 1970, a director at the Lockheed Software Technology Center, Dr. Winton W. Royce, published a paper entitled “Managing the Development of Large Software Systems: Concepts and Techniques”. Dr. Royce presented a more structured method for organizing software development. This technique was inspired by the manner in which fields like Civil Engineering and Manufacturing organized their development.
The basic idea is that everything is done in sequential phases. This means that you need to understand everything in a specific phase before you can start doing the next phase. If you change your mind in a later phase it will cost you and be hard to finish the project in time. First you need to understand all requirements, and then you need to do all the design (big design up front) and so on.
Each phase was handled by specialized groups like business analysts (for defining the requirements), system analysts (for designing the programs), programmers (for developing applications), testers (for testing applications) and deployment personnel (for overseeing operations). These groups communicated mostly in writing, and handed over work from group to group.
Managing software development with the Waterfall Model (I discuss this model later in this section) is to investigate what the system is supposed to do, make plans so that it does what it is supposed to do, and to stick to that plan. This model, however, had its setbacks: first, people learned a lot from the first system requirements until they went into production and were used by users. This made it difficult to take advantage of what was learned in the various processes.
Second, it often took a long time between the requirement phase and the user feedback phase. If you didn’t figure out what the users wanted or the users themselves didn’t know what they wanted, that meant more time and money had to be spent to change or adapt the system to users’ needs.
In defense of Royce, it would be fair to say that he actually did warn that these things could happen and he therefore proposed an iterative way of work. But no one adopted this part of his model. That’s how it came to be called the Waterfall.
When the US Department of Defense needed a software development process, they looked at Royce’s paper and they adopted a part of it (unfortunately they adopted the worst part) and named it DOD-STD-2167 (Department of Defense Standard 2167).
When the NATO later needed a model they thought that if it was the best model the US military could find, then it ought to be adopted. And from there, more and more people adopted the theories of the Waterfall. Even if the US Department of Defense changed the standard in 1995, it remained the basis of what the academic world is teaching to this day.
The rise of plan-driven methodologies
In the 80th and early 90th a myriad of new methodologies where invented that where focusing on design. These gained popularity in the same speed that object-oriented programming languages like C++, ADA and Smalltalk gained practitioners.
Naturally there were design methods before this time, even object-oriented ones but the popularity of C++ created the need for a new approach. Most design methodologies before this were data driven and/or functional in nature. When programming in an object-oriented language they were found to be not adequate.
Methodologies that became popular where Rumbaugh OMT, Booch, Coad-Yourdan, OOSE from Jacobsen and Shlaer-Mellon to name a few. All were quite good in certain areas, but seem to not cover the whole design process. Each methodology had its own type of notation and often only concentrated on a sequence of even in a system.
Because of this it was hard to use only one tool; developers adapted their favorite method and added other tools into their hybrid design methodology, splintering the industry even more. The so-called “Method Wars” arise and people where arguing endlessly about the pros and cons of their adapted methodology.
But in the mid-90th three design creators Jim Rumbaugh, Grady Booch and Ivar Jacobson joined forces at a company specializing in design tools called Rational. They became known as the famous “three amigos”. They declared the “Method Wars” over and soon came out with a first release of the Unified Modeling Language.
The RUP process and other methodologies from this time were based on a plan-driven but iterative assumption. The critics was mostly based on that these methodologies was to document focused. You would still need to understand the whole problem before you started the next step everything should be documented and this created a very big overhead that didn’t create a business value. A large complex problem where documented and explain rather good but less complex and more simple problem directly became as big to administrate.
The rise of agile methodologies
During the late 90th people started to react against the plan-driven models. Many people were very frustrated on the demands presented that developers became more agile to business demands, adapting better to knowledge gained during the project. Plan-driven methodologies were considered bureaucratic, slow, demanding, and inconsistent with the way software developers actually perform effective work.
New ways of work like XP, Scrum, DSDM, ASD, Crystal, FDD and Pragmatic Programming where developed as an alternative to documentation driven, heavyweight software development process. These new methodologies, however, had something in common: they focus on a construction and planning plan in the beginning phase and stay with that plan. Many of the Agile methodologies were inspired by “New Product Development Game”, an article written by Hirotaka Takeuchi and Ikujiro Nonaka and published in the Harvard Business Review in 1986. This article is often used as a reference and could be considered the birth of agile methodologies.
In February 2001 a group of different methodology developers meet in a ski resort in Utah, to talk, ski, and relax and to try to find common ground of what they were trying to accomplish. The output of this weekend became known as “The agile software development manifesto”.
Another movement that has gained substantial support within organization of software development in the 21th century is the Lean Software Development theories with Mary and Tom Poppendieck as the main figures. Lean is based on the system thinking theory, which sees an organization as a system. The system shall fulfill a clear customer-focused purpose in a so productive way as possible. They mean that your purpose is probably not to develop software. Your organizations customers probably want their demand satisfied or a problem fixed. If the customer could solve their problems without software, they would be delighted. They way Leans work is that it analyzes the system that the software shall be used and also how to produce that in a so productive way as possible, that focus on adaptive behavior, knowledge acquisition and knowledge workers.
To summarize the history of organizing software development, I will use the words of agile guru Martin Fowlers: “from nothing, to monumental, to Agile”.