More Thoughts on Scrum

I've been doing a little more thinking about Scrum, and have had a little success in getting it kick-started at work. Our first sprint, with a team of seven excellent developers, starts on Monday.

Here's some more thinking on the subject. The basic theory behind scrum is based on traditional process control theory. At first glance, it appears to be simply "management by to-do list". But there really is method behind the madness. Most existing software development processes assume that software development is a defined process--similar to that of building a car on an assembly line. If you've been doing software for a while, I'm sure you realize that is absolutely not the case.

Scrum is different. It acknowledges that software development is really an empirical--unpredictable--process. The methods used to control an empirical process are vastly different from those used to control a defined process. For a defined process, you can follow a recipe or series of steps to produce the correct output. For an empirical process, you closely examine the inputs prior to performing a process and the outputs once the process is complete. You put controls in place to react and change course when things go haywire.

So it sounds ideally suited to certain software development activities. But I'm trying to apply it to production support--maintenance of a system that is already in production. As with any support organization, our reacts to constantly changing priorities: we scramble to deal with the latest "emergency." Maintenance releases occur weekly on a regular basis, and sometimes as frequently as once a day. To complicate things even further, 80 percent of our support team resides in India, while the other 20 percent is located here in the U.S.

Scrum is really intended more for longer development periods, prioritized task lists that don't change, and development teams that have the luxury of being co-located and being left alone. There has been some talk about evolving Scrum to deal with concurrent development periods, and even weekly maintenance releases. And that will be the subject of my next blog.

Processes

I've been thinking about software development processes recently. I've actually been interested in them for quite a while. While at Andersen Consulting, I worked on an applied research process where we applied some of the latest object-oriented techniques on projects for a defense contractor. Those processes later became the object track for Andersen's Method/1.

Since then, I've applied both Extreme Programming and the Rational Unified Process to various projects. I've also worked on a very large-scale project where we followed a "homegrown" process influenced by the company's product development processes. This process is very much a waterfall process. As I said, the project is large, and in my experience, the larger the project, the harder it is to apply newer, less rigid processes.

But like any waterfall processes, it has its weaknesses. Because requirements must be specified up front, they often arrive late in the cycle and are rarely as complete as we'd like them to be. Because requirements are late, we struggle to complete designs in time for a thorough review. Because we rush through design reviews, we struggle with defects after the code is written. Although we test pretty thoroughly, the system is complex and we've introduced a lot of "hidden" defects into production. The burden of bringing those production defects under control is what's currently on my mind.

We have a team of about 30 people working on production support. The team does a great job of responding to problems quickly, but it's a thankless job. We're keeping our heads above water, but just barely. Our fix rate is just slightly larger than our incoming defect rate.

So I've decided to give Scrum a try. Scrum is a process, not a methodology. Based on process control techniques, it is almost more a philosophy than a process. With Scrum, software is developed by relatively small teams in 30-day sprints. The team controls what they develop, how they develop, and when they develop. This isn't the ideal scenario for Scrum. I'm not talking about a product development activity, I'm talking about software maintenance. I'll also have an up-hill battle to get our leadership team to accept this "untried" process.

I'll post updates as we make progress.

Jeff