Scrum: process that makes empirical sense

Well, enough of the irascible old programmer rants. Today I'm going to turn the eye back to
the technical.

Some managers just *love* processes. They jump on the next new one like a dog on the last steak in a butcher shop. I've always been rather critical of this. Its always seemed to me that most software processes are either ( a ) more interested in the process then the result or ( b) are thinly veiled efforts to couch the inventors personal preferences and prejudices in something that sounds scientific.

To date, most of the attempts at software "process" have come out of the legacy of the assembly line given us by Henry Ford. This works pretty well for producing craft, but does not work well for producing art.

When I say craft v. art what do I mean? A craft is a well known application of a well known skill. The hallmark of a good craftsman is consistency. He or she can do the same job over and over again and produce exactly the same results. Its a case where, given known inputs, in this case the quality of material and tools and the skill of the craftsman, one can exactly know the resulting output. There is actually an entire highly rigorous and mathematical discipline called Process Control. Process control engineers call this kind of process a defined process. Most highly advocated software processes to date have attempted to use defined process control

In an art, however, every output is unique. The artist approaches new ideas and new problems with every job they take on. their tools can vary widely as can their techniques and their results. very often all the inputs are not known at the start of the process. Process control engineers call such a process an empirical process and there is a wide body of research on empirical process control.

Myself, I believe that good software engineering is more like an art then a craft. It is, by its nature, an empirical process and that requires empirical process control. Empirical process control is experimental and iterative. It does not start out by saying "this is the right way to get this done" but provides a structured and measurable framework in which to discover the right way over time.

I learned about process control myself by reading up on a new software development process, Scrum. The thing about Scrum is that, although it has a scientific base, as working engineer when you look at it, it just makes sense in a way no other process I've ever tried to use has.

Basically, Scrum says "we don't know all the details." Product requirements can and do change. new software always involves discovering new ways of doing things. No where is this more obvious then the game industry, where every product must be different from what came before and the ultimate goal of "fun" is not really even definable in an engineering apriori manner. On the one hand, a software process must be flexible enough to handle changing inputs. This means that the software development team must be able to handle these changes and factor them in. On the other hand, it must be structured enough so the engineering team does not thrash and can get real work done. This is what Scrum attempts to do.

Scrum has value for both the engineers and the management, but in order to have this value, it needs to be implemented completely. If one side or the other tries to "improve" it, they are likely to unintentionally remove the value for the other side.

For management Scrum has two very attractive results. First, it allows the management to review and modify the project requirements as the project progresses. Secondly, it provides monthly check points of demonstrable functionality. This gives non technical managers the comfort of seeing how the work is progressing. An equally or more important advantage is that it allows the engineers to work in a highly productive manner. This is an advantage though that most management won't understand until they see it in retrospect.

For software engineers, its recognizes the realities of their work. Work is divided up into 30 day "sprints" to the next demonstrable point. During that period no outsider is allowed to change their goals. (There is an exception, but I'll get to that.) It also provides them immediate feedback to management on a daily basis of anything that is getting in their way.

It allows the team to "self-organize" into a structure and style that works best for them to get the job done. This is very important. Artists by nature all have different styles. No externally imposed structure is going to fit well. Give a motivated and dedicated team the space to do it, however, and they will figure out themselves how to organize their work most effectively.

Scrum is a low-load process. It recognizes that process and management is the job of management, not your engineers. A good process should leave your engineers mostly free to do uninterrupted engineering. After all, that is what you hired them to do. It expects that everyone on the team is a mature adult who wants to do the right thing (I have seldom run into anything else in the field) and leaves them free to do it.

Scrum puts a lot of responsibility on the shoulders of the engineering team. This can be frightening at first. But it also provides protections so the team is free to wield that responsibility in the way they need to to get the job done. It has always been my experience that engineers are diligent, hardworking and responsible people by nature. Management should trust their engineers. After all, they hired them. If the engineering team is non-functional, then that is indicative of a management lack and they should get help in straightening their hiring out.

Importantly, Scrum recognizes that not all progress is linear. Finding out a job is harder then you thought is still progress towards the goal. Finding out an approach won't work is also progress. Although the team has a functional and demonstrable goal to hit, they are free to make modifications to the plan to get there, modify the definition of the details of the goal, or even to fail to get thereat all. Even a failure is progress as you have learned what *not* to do.

I mentioned there is a way for management to change priorities mid sprint. It is possible, it is not advisable. The action management has to take to do this is to cancel a sprint. If a sprint is canceled
then the team is no longer responsible to show any progress resulting from the work to date on that sprint. A team can also cancel a sprint early if they find that even modified, the sprint goal is unattainable. This puts them immediately back into the planning phase of a new sprint. Canceling a sprint is embarrassing. No one likes to do it, which is good because it is costly. But on a rare occasion you do need to cancel to avoid wasting the entire sprint period on a goal that is either unnecessary or unattainable.

As I said at the beginning, I am not a process hound. In general I've always found that they diverge seriously from the realities of engineering in their assumptions and expectations. I also don't believe that Scrum is a substitute for good high level planning. We are building a system that needs to be highly scalable. Scalability doesn't just happen. In the large it must be designed in to your architecture. But where most failed projects fail is not in their design, but in the incredibly complex details of their implementations. (What management sometimes calls 'execution.") And here, i think Scrum will be very valuable to us.

I mentioned that its an iterative process. It assumes nothing starts out perfect, even the process itseklf. This is my first Scrum project as a Scrum leader, and only the second time I've run into Scrum at all. We will all be learning how to do this, even me.

But I have high hopes. It makes sense to me.