I have repeatedly heard that "Scrum succeeds largely because the people doing the work define how to do the work" (see From The Top by Ken Schwaber for his full text). However, I do not think that is even a third of why Scrum works - and be clear, I am certainly not questioning that Scrum works - I just want to get at the why it works.
Before I go into that, let me say a few words about why I am interested in the why. I have been doing some form of Agile development for 20 years. The first 10 years or so, my approach was somewhat informal, although I remember one project that had complete automated testing (for my piece anyway) back in 1984. In the last 7 years, as a trainer/coach/mentor in Lean and Agile methods, I have had the privilege of seeing dozens of teams migrate to Lean-Agile methods. As an educator (I guess my word for "change-agent") I have been intrigued with what helps to shift people's behavior (not just their understanding).
Shifting behavior requires more than simply knowing what to do and how to do it. It also requires more than having done it a few times. It requires a confidence that what they will do will make a difference when things get tough because, when things get tough, people tend to go back to their previous (bad) habits. They begin to fear that they will not succeed with the new approach, so they revert to what they have been doing for years, even if they've had short-term success with better methods.
I joke in my classes that it's a good thing that we didn't have our attitude about learning as babies that we have now. If we did, when we were trying to learn how to walk we'd get up to try to walk, fall down, get up again to try, fall down again and then say to ourselves - "To heck with this, I'm crawling, it's faster."
Shifting to effective behavior requires both a doing and an understanding of why the doing works, both practice and intellect. I think eXtreme Programming faced a bit of resistance in spite of its obvious validity because when it first came out little was done to explain why it worked (no question that it does). Scrum is having great success because it is a great process (yes, I think Scrum is a process, but that'll be another blog). The problem with misplacing why Scrum works is that you have little guidance to follow if you try to use it and non-standard issues come up. Perhaps more importantly, it gets hard to convince management (who definitely needs an understanding of why) if you tell them the wrong answer (Manager: "You mean I should just let my team figure it out? The same guys who can't deliver software on time? Yeah, right!"). Yes, I am being a bit facetious and unfair here, but you get what I mean...I hope.
Why does Scrum Work?
OK, so why does Scrum work? Remember, I am not challenging that it does, just asking why. I would say Scrum works for the following reasons:
A Scenario: Structure may be more important than autonomy
Note, when considering the reasons I believe Scrum works, self-motivated teams are not even on the list here. I would consider all of the items above to be more important.
Imagine two different teams:
With a decent PM telling the second team what to do, the will perform much better than the first team. Why? Because the first team's constraints make it impossible to implement the real reasons (available customer, working one project, co-location) that actually has the productivity be high. True, the second team would do even better if they can make a lot of decisions for themselves. But, in these cases, it is the constraints on the first team (something over which they have no control) that keeps them from doing as well as the second.
Be careful that I am not arguing against autonomy. Nor am I arguing against self-organizing teams. My experience is that most self-organizing teams that understand Scrum will want to do the things I have listed as "reasons". But I am trying to be clear that it is those effective actions that is causing things, not the self-organizing.
You can say that my case is unfair. I would say my case is realistic. The structure within which teams find themselves often has a lot more to do with what they can accomplish than the freedoms they are given within that structure. This should come as no surprise. Edwards Deming, whose philosophy and teachings are the foundation for Toyota's Lean methods, incorporate the notion of systems being the source of most errors.
OK, enough theory, what's my evidence for this?
As I mentioned, over the last 7 years I have seen dozens of teams go agile. Many times (especially when the teams were small), the team could define what their work was. More often, especially in larger companies, they weren't given this ability. However, several of the items I gave above as reasons were put in place (by management decree) and these teams still succeeded.
I've also been involved in assessments of companies where little autonomy was given yet there was a 4-1 ratio in effectiveness between some of the teams. The main differences in the teams were the effective teams:
Now, it may be that a team that can define its work will pick these things. I buy that. I would say that a team that defines its work will be effective because it is choosing the right things to do. But it is effective because it is doing the right things - not necessarily that they chose them.
This is an important distinction if we are going to understand why things work. It's also an important distinction if we have managers who want an effective team but don't quite trust them to define their own process. It also helps us when we are in organizations where the people doing the work are afraid to make decisions on their own. In many organizations teams don't quite believe that they can make decisions on how to run things. While this is an important issue and eventually needs to be addressed, it is not necessarily where you need to start.
This is actually a very good thing. I have heard many Scrum coaches complain about the need for trust to get started doing agile. I would say that is not true. While trust helps a lot, just doing iterative development, focus on integrating the team, having QA define test cases concurrent with getting requirements, co-locating teams, .... (see reasons above) will make a lot of progress. Trust can come from success as well as success coming from trust.
What I am saying is that while self-organizing teams who are knowledgeable about how to do effective software development will tend to become effective, the practices they adopt have a lot to do with their effectiveness. Knowledgeable, self-organizing teams select practices that support them and avoid practices that don't. Their process becomes their baseline of effort and they change it when it doesn't work. They refine their process (although they may not think of it as a process). This is all good. But it assumes they live in a structure where they can select the practices that will work best. Therefore, it is important is to understand what works and why. It may requirement more than the team to get these practices in place. It also may be that teams don't know what to do or that they don't believe they can act on their own. Of course, a self-organizing team that gets management support to do the right thing is the best situation.