Why Scrum Works and How This Tells Us When It Won’t

August 11, 2012 — Posted by Al Shalloway

Abstract

We hear that Scrum works because it allows teams to self-organize.  I believe, however, that Scrum works because it mandates a method that manages queues implicitly. Self-organization may get you there, and will certainly be useful in sustaining improvements achieved with Scrum’s implicit queue management, but it is not the real reason Scrum works. Knowing why Scrum works at the team level provides insights into why it doesn’t work well, without a lean context anyway, at the enterprise level. 

Queue management at the team level is different than queue management at the organization level.  Understanding the need and value of Lean flow is essential to make the leap from team to enterprise.   Understanding how Scrum works at the team can help you see how to extend your thinking to make it work at the enterprise – or to jump to a Lean method more suited to enterprise wide product management. If you don’t see how simple Scrum methods will work across the organization, I’d suggest trusting your instincts and learning more before starting.

Background Story

This section tells how I got here. It is not essential to get to the bottom-line of the blog (feel free to just jump there).  But, if you are interested in learning how I got here, please read on.

I’ve been introspective about development since about 1984.  Before then, I was interested in how to improve – learning new methods, etc. But I can specifically remember the insights that had me start asking why things worked, not just what I could do to do better.  For a while, this was a personal  endeavor.  But when design patterns came out, I started asking publicly why they worked.  I challenged the notion they had to be discovered.  I was interested in what people were thinking that had them write code that later was considered to be good enough to be a pattern.  In other words, I wasn’t as interested in learning what the good designs of others were, I wanted to know what they were thinking so that I could think that way as well. This eventually culminated in my book Design Patterns Explained: A New Perspective on Object-Oriented Design.  This book is still relevant and one of the few that deals with the true issue of what patterns are.  For those who want a shorter treatise, I recommend my article Can Patterns Be Harmful or my more recent blog Going Beyond Design Patterns.

When XP came out I continued my questioning – why did it work?  BTW: I knew that it did, I just wondered why.  I questioned why we needed to do all twelve practices to get value when they all seemed individually useful.  I knew they worked because I had used them all individually at some point in my career with great value.  I thought XP was brilliant for both putting them together and for giving us a framework for using them.  For whatever reason, neither of these continued probes got me any points with thought leaders.  I greatly appreciated Rebecca Wirfs-Brock’s understanding one day when she told me I was just a model-builder but it made people uncomfortable.

When Scrum came out, we (Net Objectives) started teaching it. For the first few years of the 21st century we were one of the biggest Scrum training organizations around.  And, of course, I started asking “why did Scrum work?”  After a few years of this I finally wrote a blog - Challenging why (not if) Scrum works - on this.  I had clearly come to a different conclusion than the party-line.  Again, my questioning was not appreciated and led to the first time I was thrown off the Scrum Development users’ group (the second was when I had the audacity to challenge if Scrum-of-Scrums can work in dysfunctional organizations).

I believe challenging doctrine and questioning why something works or is something that should even be done is both useful and the epitome of Agile.  I find it both ironic and unfortunate that it is not the least bit appreciated by some Agile leaders.

The Bottom Line

In early 2007, I wrote a blog Challenging why (not if) Scrum works that described our realization that cross-functional teams doing waterfall out-performed siloed teams doing waterfall by 3-to-1.  Others have told me it can be as high as 10-to-1.  I had been suspecting for quite some time before this that it wasn’t self-organization that was making Scrum work but that it was the cross-functional teams making things work better. Cross-functional teams result in shorter queues, fewer hand-offs and significantly less time from making a mistake until it is discovered.  This literally lowers the amount of work that needs to be done (see Real Tenets of Lean: Avoid Creating Waste By Eliminating Delays on our Lightning Webinars page). 

Of course, self-organizing teams are important and result in sustainability of improvement.  I wasn’t (and am still not) questioning the value of self-organization.  Rather I am saying that knowing how to do things the right way is often more useful than having to figure it out for yourself – and that self-organization doesn’t always result in this understanding.

So why do cross-functional teams (and therefore Scrum) work? I would suggest it is because having cross-functional teams implicitly manages queues.  By having shorter queues, errors are detected quickly. Scrum mandates this with cross-functional teams, but never explains why they work.  Kanban provides a deep understanding of lean-flow. This, coupled with explicitly discussing work-flow policies provides the basis for continual improvement. 

It is ironic that explicit policies are a great boon to self-organization but have been vilified by many in the Scrum community. Without a theory on why a group’s implicit workflow works, self-improvement often flounders.  We see this in many Scrum teams – they have a great start only to languish later.  This happens because self-organization requires principles to organize around once the easy gains are achieved. 

Scrum can achieve great success at the team level because implicit knowledge is sufficient and is easily shared across team members.  Unfortunately, implicit knowledge at the team level is difficult to share across teams.  This impedes creating a common vision (or at least does not help create one).  Different teams attempt to accomplish different things and this causes thrashing to take place when they attempt to integrate their work.  Scrum-of-Scrums attempts to bridge this, but since there is not a shared approach across the teams, it often does not help align the teams. In any event, a higher view is required, one which the individual teams cannot provide. This is similar to the way separate fire-fighting teams may not ever see the big picture of fighting the forest fire.  Someone well away from the front must be in charge and give objectives to the teams.  This is the same reason that teams participating in the Scrum-of-Scrums may never be able to see the bigger picture required to achieve the high-level result desired.

Besides all of this lies the fact that queue and work-flow management at the team level is considerably different than at the enterprise level.  In a team, it is easy to see when individual (local) optimizations that hurt team (global) results occur.  Time-boxing is extremely useful for this.  Developers may have coded all the stories, but if they haven’t been tested at the end of the sprint they’ve not completed their global challenge. Many companies see this on projects requiring multiple teams – all of the teams have completed their sprint’s stories but they don’t have completed functional code across the teams.  Scrum will tell us of this impediment, but doesn’t help us see how to fix the situation.

It is often said that that’s not the point of Scrum – telling us how to fix impediments.  The purpose of Scrum is to reveal them in 30 days.  Personally, I’d rather use a method that 1) predicts the impediments that will happen, 2) exposes them when they do, and 3) tells us how to fix them. Kanban and an understanding of Lean-flow do just this.   I don’t know why re-inventing the wheel seems to be an attitude so prevalent in the Agile world.

Challenges at the enterprise level are much more complicated (even complex) than at the team level.  They do not reveal their secrets as easily as team challenges. Some science (i.e., lean-flow) is necessary.

Using Lean-Flow to Avoid Challenges Met By Organizations Using Scrum

After several years of success at the team level using Scrum, we started running into a series of challenges we could not solve with the insights/methods Scrum provided.  These included:

  • Handling the cross-team technical and business dependencies of multiple teams
  • Creating enterprise wide architectures
  • Managing the workload of people who are required by several teams (this occurs due to any number of reasons but includes folks who have needed subject matter expertise, critical skills or rare knowledge of legacy code).
  • Coordinating multiple stakeholders with multiple teams
  • Aligning groups with disparate motivations

We’ve found insights from Lean to be invaluable in helping solve these.  Our webinar series, Lean-Agile at Scale and at the Team: The Value Stream Series describes how we’ve used Lean and Kanban to help solve these.

Does This Mean My Investment In Scrum Has Been Wasted

In a word – no.  In two words - absolutely not. Scrum provides many insights that help get teams started down the Agile path. This does not, btw, imply that it is always the best place to start.  But what has been learned is definitely of value. It also doesn’t mean you must abandon Scrum and go to something else.  But you likely do want to extend your use of Scrum and possibly change your mindset about Agile as well = see my blog Mindsets: Waterfall, 1st & 2nd Generation Agile or our Lean-Agile at Scale and at the Team: The Value Stream Series.

The key is to realize that Scrum does not provide the insights required to help manage enterprise queues.  These are much more complicated because of the organizational dynamics involved (see People Are Complex, Software Development Isn’t ).

What To Do Next

I highly recommend reading our Lean-Agile Roadmap and/or following our Lean-Agile at Scale and at the Team: The Value Stream Series.  While many in the Agile space have been bemoaning how difficult scaling Agile is, many others, using Lean and Kanban, have been discovering how to do it. If you are beginning to think about starting an Agile adoption, I would also suggest reading one or both of the following:

In addition, I’d especially suggest that you trust your instincts.  If you don’t see that a method will solve your problems you should investigate more methods before undertaking a transition.  Of course, I’d be happy to help you get a better understanding. BTW: If you happen to be at the Agile conference, please come by and see me.

Alan Shalloway
CEO, Net Objectives

Subscribe to our blog Net Objectives Thoughts Blog

Share this:

About the author | Al Shalloway

Al Shalloway is the founder and CEO of Net Objectives. With over 40 years of experience, Alan is an industry thought leader in Lean, Kanban, product portfolio management, SAFe, Scrum and agile design.


Comments

It really really really puts things in perspective and connects the dots... has been a long time since I read such an insightful post...

May I make a suggestion: use your post in the Reactor14 conference to lead a session named something like "Toward a Unifying Theory of Software Methods." I think it has the potential to get us beyond the "my method is better than yours" crap that has plagued the community(ies) over the past N year.

Again, very many thanks - yours are precious insights!

Best,

Israel

Israel:Thanks for the kind words.  I've been a bit negligent in following comments and just read this.  Great idea re reactor14

Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Guy Beaver
Business and Strategy Development, Executive Management, Management, Operations, DevOps, Planning/Estimation, Change Management, Lean Implementation, Transitioning to Agile, Lean-Agile, Lean, SAFe, Kanban, Scrum
Israel Gat
Business and Strategy Development, DevOps, Lean Implementation, Agile, Lean, Kanban, Scrum
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, SAFe, Kanban
Ken Pugh
Agile Design and Patterns, Software Design, Design Patterns, C++, C#, Java, Technical Writing, TDD, ATDD, Certifications, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Marc Danziger
Business and Strategy Development, Change Management, Team Agility, Online Communities, Promotional Initiatives, Sales and Marketing Collateral
Max Guernsey
Analysis and Design Methods, Planning/Estimation, Database Agility, Design Patterns, TDD, TDD Databases, ATDD, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile
Steve Thomas
Business and Strategy Development, Change Management, Lean Implementation, Team Agility, Transitioning to Agile
Tom Grant
Business and Strategy Development, Executive Management, Management, DevOps, Analyst, Analysis and Design Methods, Planning/Estimation, Innovation Games, Lean Implementation, Agile, Lean-Agile, Lean, Kanban