Challenging why (not if) Scrum works

May 24, 2007 — Posted by Al Shalloway

I have repeatedly heard that "Scrum succeeds largely because the people doing the work define how to do the work" (see From The Top by Ken Schwaber for his full text). However, I do not think that is even a third of why Scrum works - and be clear, I am certainly not questioning that Scrum works - I just want to get at the why it works.

Before I go into that, let me say a few words about why I am interested in the why. I have been doing some form of Agile development for 20 years. The first 10 years or so, my approach was somewhat informal, although I remember one project that had complete automated testing (for my piece anyway) back in 1984. In the last 7 years, as a trainer/coach/mentor in Lean and Agile methods, I have had the privilege of seeing dozens of teams migrate to Lean-Agile methods. As an educator (I guess my word for "change-agent") I have been intrigued with what helps to shift people's behavior (not just their understanding).

Shifting behavior requires more than simply knowing what to do and how to do it. It also requires more than having done it a few times. It requires a confidence that what they will do will make a difference when things get tough because, when things get tough, people tend to go back to their previous (bad) habits. They begin to fear that they will not succeed with the new approach, so they revert to what they have been doing for years, even if they've had short-term success with better methods.

I joke in my classes that it's a good thing that we didn't have our attitude about learning as babies that we have now. If we did, when we were trying to learn how to walk we'd get up to try to walk, fall down, get up again to try, fall down again and then say to ourselves - "To heck with this, I'm crawling, it's faster."

Shifting to effective behavior requires both a doing and an understanding of why the doing works, both practice and intellect. I think eXtreme Programming faced a bit of resistance in spite of its obvious validity because when it first came out little was done to explain why it worked (no question that it does). Scrum is having great success because it is a great process (yes, I think Scrum is a process, but that'll be another blog). The problem with misplacing why Scrum works is that you have little guidance to follow if you try to use it and non-standard issues come up. Perhaps more importantly, it gets hard to convince management (who definitely needs an understanding of why) if you tell them the wrong answer (Manager: "You mean I should just let my team figure it out? The same guys who can't deliver software on time? Yeah, right!"). Yes, I am being a bit facetious and unfair here, but you get what I mean...I hope.

Why does Scrum Work?

OK, so why does Scrum work? Remember, I am not challenging that it does, just asking why. I would say Scrum works for the following reasons:

  • The iterative nature of Scrum development (including the re-assessment of priorities between iterations)
  • The workcell nature of the Scrum team (everyone who needs to work together is together)
  • The fact that many Scrum teams are given the chance to co-locate
  • Most Scrum teams work on one project at a time
  • Many old, cumbersome checkpoints/forms/status reports that used to be required are abandoned because the team is doing something new
  • A focus on defining acceptance tests concurrent with requirements
  • The availability of a product champion (product owner to Scrum enthusiasts)

A Scenario: Structure may be more important than autonomy

Note, when considering the reasons I believe Scrum works, self-motivated teams are not even on the list here. I would consider all of the items above to be more important.

Imagine two different teams:

  1. A team that has to work on 5 projects without the availability of a customer and is not co-located.
  2. A team that is somewhat told what to do but has a customer available whenever needed, works on one project at a time and is co-located.

With a decent PM telling the second team what to do, the will perform much better than the first team. Why? Because the first team's constraints make it impossible to implement the real reasons (available customer, working one project, co-location) that actually has the productivity be high. True, the second team would do even better if they can make a lot of decisions for themselves. But, in these cases, it is the constraints on the first team (something over which they have no control) that keeps them from doing as well as the second.

Be careful that I am not arguing against autonomy. Nor am I arguing against self-organizing teams. My experience is that most self-organizing teams that understand Scrum will want to do the things I have listed as "reasons". But I am trying to be clear that it is those effective actions that is causing things, not the self-organizing.

You can say that my case is unfair. I would say my case is realistic. The structure within which teams find themselves often has a lot more to do with what they can accomplish than the freedoms they are given within that structure. This should come as no surprise. Edwards Deming, whose philosophy and teachings are the foundation for Toyota's Lean methods, incorporate the notion of systems being the source of most errors.

The Evidence

OK, enough theory, what's my evidence for this?

As I mentioned, over the last 7 years I have seen dozens of teams go agile. Many times (especially when the teams were small), the team could define what their work was. More often, especially in larger companies, they weren't given this ability. However, several of the items I gave above as reasons were put in place (by management decree) and these teams still succeeded.

I've also been involved in assessments of companies where little autonomy was given yet there was a 4-1 ratio in effectiveness between some of the teams. The main differences in the teams were the effective teams:

  • were organized as workcells
  • were co-located
  • worked on one project at a time
  • had available customers

Now, it may be that a team that can define its work will pick these things. I buy that. I would say that a team that defines its work will be effective because it is choosing the right things to do. But it is effective because it is doing the right things - not necessarily that they chose them.

This is an important distinction if we are going to understand why things work. It's also an important distinction if we have managers who want an effective team but don't quite trust them to define their own process. It also helps us when we are in organizations where the people doing the work are afraid to make decisions on their own. In many organizations teams don't quite believe that they can make decisions on how to run things. While this is an important issue and eventually needs to be addressed, it is not necessarily where you need to start.

This is actually a very good thing. I have heard many Scrum coaches complain about the need for trust to get started doing agile. I would say that is not true. While trust helps a lot, just doing iterative development, focus on integrating the team, having QA define test cases concurrent with getting requirements, co-locating teams, .... (see reasons above) will make a lot of progress. Trust can come from success as well as success coming from trust.


What I am saying is that while self-organizing teams who are knowledgeable about how to do effective software development will tend to become effective, the practices they adopt have a lot to do with their effectiveness. Knowledgeable, self-organizing teams select practices that support them and avoid practices that don't. Their process becomes their baseline of effort and they change it when it doesn't work. They refine their process (although they may not think of it as a process). This is all good. But it assumes they live in a structure where they can select the practices that will work best. Therefore, it is important is to understand what works and why. It may requirement more than the team to get these practices in place. It also may be that teams don't know what to do or that they don't believe they can act on their own. Of course, a self-organizing team that gets management support to do the right thing is the best situation.

Subscribe to our blog Net Objectives Thoughts Blog

Share this:

About the author | Al Shalloway

Al Shalloway is the founder and CEO of Net Objectives. With 45 years of experience, Al is an industry thought leader in Lean, Kanban, product portfolio management, Scrum and agile design. He helps companies transition to Lean and Agile methods enterprise-wide as well teaches courses in these areas.


Hi Alan,

Long time listener, first time caller. Love keeping up with your thoughts on the software industry. Keep up the great work.

A major difficulty that I have with Scrum and with Agile literature in general is that it gives too casual a treatment of the fundamental 'why it works' points that you discuss in this article. A key concept that I take from your article is 'focus'. I would qualify this concept as the need for business focus in software teams. Here's an anecdote from my experience.

Imagine a team that sits together, works on the same thing together, works on one project at a time, has available customers, delivers working software every month, but has a 'task completion' culture. In this team the technical aspects (tasks) of many different features are worked on every sprint, but the features themselves are not completed to a useable state: the feature implementations span sprints. The customer is engaged to show that progress is being made on all of their important features. Where had my team (ouch) gone off the rails?

From an outside perspective, and with the benefit of experience, I guess it is obvious that our business focus was missing. We weren't targeting business outcomes and consistently delivering value to our customer. Developing a really efficient database for 6 months does not add value for the customer. They cannot play with the database. The database on its own does not make them any money. We were delivering technical tasks instead of business value.

So for me, the practices underlying Scrum are fantastic enablers for focus, but remember to question what you are focusing on. Think about why you are coming to work every day and you may get more mileage from your process!

Just to clarify, I think Scrum is great. I think the intent is absolutely there for it to focus the team on business outcomes. I don't blame Scrum for the fact that we were not doing business in the most efficient way. If Mr. Schwaber had looked at our project I can almost guarantee that he would have either said:
1. You need to build working functionality that provides business value to the customer every sprint. And if we still didn't listen:
2. You suck and that makes me sad.


Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Guy Beaver
Business and Strategy Development, Executive Management, Management, Operations, DevOps, Planning/Estimation, Change Management, Lean Implementation, Transitioning to Agile, Lean-Agile, Lean, SAFe, Kanban, Scrum
Israel Gat
Business and Strategy Development, DevOps, Lean Implementation, Agile, Lean, Kanban, Scrum
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, SAFe, Kanban
Ken Pugh
Agile Design and Patterns, Software Design, Design Patterns, C++, C#, Java, Technical Writing, TDD, ATDD, Certifications, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Marc Danziger
Business and Strategy Development, Change Management, Team Agility, Online Communities, Promotional Initiatives, Sales and Marketing Collateral
Max Guernsey
Analysis and Design Methods, Planning/Estimation, Database Agility, Design Patterns, TDD, TDD Databases, ATDD, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile
Steve Thomas
Business and Strategy Development, Change Management, Lean Implementation, Team Agility, Transitioning to Agile
Tom Grant
Business and Strategy Development, Executive Management, Management, DevOps, Analyst, Analysis and Design Methods, Planning/Estimation, Innovation Games, Lean Implementation, Agile, Lean-Agile, Lean, Kanban