Organizations often try to adopt a software development methodology, in hopes of reducing the risk of cost and schedule overruns in their software development project. Veterans of previous methodology adoption attempts roll their eyes or actively resist, while less experienced team members drool in anticipation of some order and predictability. In this post I describe how software development methodology adoption efforts can fail, based on my own experiences.
This is a personal perspective, so it doesn’t go back all the way to the early days of computer science. It starts when I started paying attention, which was in the 1990s. Nevertheless, I think it’s pretty accurate with respect to what was going on in the 1990s through today (late 2015).
Here goes.
Once upon a time, in the final years of the 20th century, there were some experienced software folks who had been through some successful projects and some failed projects, who had developed a sense for what to do and what not to do in certain situations.
In an effort to share their knowledge, and to cash in on their experience, they each wrote a Software Methodology Book, and then noticed that there were like-minded folks writing similar Software Methodology Books, so they joined forces and wrote a Unified Software Methodology Book. These books were intended for people who were sort of like them, but less experienced. They were full of advice like “do these things, but of course customize all this to fit your own project”.
They then constructed elaborate Computer Aided Software Engineering tools, which attempted to embody the structure of their customizable software development methodology while making it possible to generate code from written specs, and hyperlink from everything to everything else. Ideas like “requirements traceability” from a line of code back to a specification that explained why it was there (and from a spec to the test that verified it was implemented) were the goals of this effort.
This software was astonishingly expensive, buggy, and very hard to use, but some people bought it and attempted to adopt it, because this seemed very much like a silver bullet to those who had never read the essay, “No Silver Bullet“.
Worse, the people who read their Software Methodology Books were not all like them. Many of these were people who lamented the failure of their own software projects, who had read all manner of books from the past 100 years describing how to manage an industrial production line, but had not had much success running software projects using the model of a boss telling low-skilled minions what to do.
A brief digression: An intrinsic part of the hierarchical mindset of an early industrial-era business is the belief that the folks at the top of the org chart are the genius inventors who do all of the complicated thinking. As you go down the org chart, you pass through the well-read project organizers, and eventually reach the interchangeable, low-skilled workers who repetitively follow a cookbook in order to crank out whatever widgets the geniuses came up with. This was a pretty successful structure a century ago, and made the industrialized world a vastly better place.
Unfortunately, this model was so popular that its assumptions have become ingrained in the culture of industrialized societies. These include the assumption that in a company that is not successful, the thing to do is to look for ideas outside the company (since the ideas can only come from the boffins at the top, and the company’s own boffins are coming up short). Somewhere out there is a bold innovator, taking risks and blazing a new trail, which people in the failing company could then follow, saying “Look at what that person is doing! Let’s copy them!” It’s only slightly politically risky to suggest that an organization stop failing, if one can point to an example of a successful project which documented how it succeeded. (It’s far riskier in an environment like this to suggest an original idea, since the downside is that the failure is blamed on the person who suggested it.)
These industrially-minded managers regarded Smart Person Books as if they were cookbooks, and sought professional training to better memorize and adhere to the latest methodology. They still hoped that they could hire inexpensive workers and command them to follow the methodology book by the letter (managed by the recently-certified methodology police), and get just as good results as the original team of expensive, experienced software folks got when blazing a new trail. When in doubt, they would continue to ignore their own team’s judgement and perspective, and force everyone to stick to the guidelines in The Book – guidelines which had been elevated to Rule status.
The imposition of rigid bureaucracy upon a low-skilled team turned out not to be a successful strategy in software development. As workers gained experience, read some books themselves, and honed their technical and management skills, they found that the environment they were working in was carefully designed to stifle their creativity. They left these stifling companies, seeking work at other places that weren’t so stifling. As a result, these bureaucratic organizations didn’t learn, but a large body of software developers matured, rebelling against the rigid methodologies which had previously been forced upon them.
They were happy to find a new, like-minded set of people writing about a lighter-weight Software Methodology, first in a Wiki (the very first Wiki ever, incidentally), and then in books. These authors talked about trusting smart workers to identify the right thing to do and to do it, to tailor their ongoing practices based on the experience that they gained from working on that specific product, and above all, to focus their time on making the product with a minimum of wasted effort. The creative genius was embedded within the workers, and the workers were expected to be introspective about how to work more effectively and to be flexible enough to work in an ever-changing way.
Unfortunately, when the rest of the world saw a new Software Methodology Book, a process took place that was similar to that which had happened the last time. Certifications were created; methodology police were re-educated; teams were again made to follow processes that made no sense to them because the Book said so and their managers figured that the Book knew better than their workers. Some methodology police, who read the book but weren’t able to make it work for them, said that the methodology itself was to blame, resulting in one of the new methodology’s authors writing an essay called “We Tried Baseball and It Didn’t Work“. (This essay actually describes how the failure to fully adopt the author’s methodology as specified can lead to a failure to reap its intended benefits, which is the exact opposite of the larger point I’m trying to make here, but I interpret it as being a cautionary tale, advising software professionals to avoid adopting the practices of any methodology when you don’t understand the reason for the practice.)
Again, the wiser among software developers read all of the relevant new books, and treated them as mere recommendations, “rolling their own” processes on their own projects, using their own sense of what was working and what wasn’t, and being willing to try some practices for a little while, even if they seemed like they might turn out to be a waste of time. Again, they moved on when they found they had outgrown organizations that wanted to cargo-cult the contents of Software Methodology Books and force them to do things that were obviously stupid, or things that didn’t fit with the way these people preferred to work.
Then, one day, when the cargo-culters were well and truly disillusioned with the poor results of rigidly imposing a bureaucratic maze they had created based on the various Extreme Programming methodology books, they found a new Software Methodology Book about something called Scrum. They quickly set about signing up for certification classes, and building a new rigid bureaucracy to impose their interpretation of Scrum upon their workers.
The moral of this story is that you should read all the books, but roll your own process, and never stop modifying it to work better over time in your own circumstances. Someone else’s experiences led them to come up with some good ideas that worked on their team and their project, but may or may not work for your team on your project. It’s best to have a large bag of tricks that you can draw from when you have a problem to solve, while remaining brave enough to try things that (as far as you know) nobody else is doing. There is no silver bullet, but your best bet is to build a skilled team, and empower them to make good decisions.
tl;dr: Do not adopt software development methodologies; instead, loot them for good ideas.