The 5 whys of Lean as an answer to the But of Scrum

October 25, 2009 — Posted by Al Shalloway

In this blog I discuss the need to get to the root cause of why so many teams are not having success with Scrum.  Merely saying management is not removing impediments or labeling it "Scrum but" does not give much indications as to what or where the problems are. The question is "why does this happen and what can we do about it?" I will suggest one of Lean's problem solving tools - "5-whys" - may assist Scrum teams in moving forward.

In an Agile Collab interview, Ken Schwaber acknowledged that only 1 in 4 Scrum implementations achieve the success desired by the companies trying it. He attributes this to the organizations accommodating the impediments the team faces instead of solving them. This number has been generally accepted by the Scrum community as accurate (as discussed at Scrum gatherings and on user groups). What surprises me isn't the low rate of success but rather the general lack of interest into why it occurs. I've looked at both sides of this (see Challenging Why (not if) Scrum Works and Challenging Why (not if) Scrum Fails). My own belief is that one has to take a scientific approach and do an inquiry into why things work or don't work. The intent of both blogs was to spark a conversation about how to improve Scrum.  Unfortunately, neither blog achieved this.

In the past, the generally accepted reason simply seemed to be people were not disciplined, educated or motivated enough to do Scrum properly.  More recently, the belief that developers don't properly understand what they need to do has resulted in much discussion about having Certified Scrum Developer courses to solve this problems.  But I am afraid both of these are superficial answers.

The issue of teams doing a pseudo Scrum by cherry picking their practices has reached epidemic proportions. One hears of this so much that there has become a name for it "Scrum but." That is, "we do Scrum but for this".  However, why do people do this?  This is an important question.   In some situations this is probably a good thing.  After all, Scrum is not a prescriptive methodology but rather a lightweight framework. One would suppose that would mean that there aren't any absolutes in Scrum, but I've heard several CSTs say if you aren't doing daily stand-ups or retrospections then you aren't doing Scrum. The reticence of many Scrum CSTs to present a model of why and how Scrum works (many claim such a thought process is actually counter-productive) only exacerbates this situation. In other words, if one doesn't have a set of principles to work from, one is left with little understanding of why practices work or not.

So what's supposed to be in Scrum and what's not supposed to be in Scrum and who is to decide? If one believes that practices are context dependent and that teams need to be self-organizing then one would expect the teams to decide what they should be doing. Tailoring Scrum should therefore be left to the team and an outsider shouldn't pass judgment on the team's efforts. However, since teams don't understand Scrum well at the beginning of their adoption of it the advice is to require teams to follow a starting set of rules and to only tailor them after you understand Scrum. But why should we believe the starting set works in all situations?

Ironically, many Scrum thought leaders believe software is so complex there is no causality you can understand. In other words, you need to just solve the problems as they show up. In other words, after you've tried things and learned them, then you should do things in a way that works for your situation. While this makes sense, it seems to have led to the current situation that if it works, we'll call it "Scrum" but if it doesn't, well, we'll call that "Scrum but."

The dilemma of Scrum becomes evident. While we don't want to be prescriptive, when you learn Scrum there are rules you need to follow.  Then, as you get better, you can tailor it to your needs.  The unexamined assumption is that you can always do this.  I do not think it is as simple as this. I believe we need a real analysis of why teams accommodate their problems or why teams don't follow certain Scrum practices. In other words, let's get to the root cause of Scrum not working or teams doing "Scrum but."  To do this, let's use a Lean tool called "5-whys."

5-Whys

When something goes wrong, we often jump to conclusions about what caused it. Even when correct, we often stop there – instead of asking ourselves, "well, what caused that?" In other words, we don't get to the root cause.  Thus, even if we fix this problem, if the root cause is still present, it'll likely cause another problem.  A popular Lean tool to get to root cause is called "5-whys."  It basically is an a technique where one keeps asking "why something happened" until the root cause of the original incident which set off the query is uncovered. This may take anywhere from 2-9 levels of questioning – so "5" should not be taken as the exact correct number of "whys" to ask.

Let's use 5-whys to determine root cause for accommodating impediments and "Scrum but".

The Root Cause of Accommodating Impediments

One of the common impediments teams face in corporate-wide Scrum implementations is that team members are often assigned to work on several projects. It is easy to dismiss this on the lack of commitment of management. If it's available for them to solve it, there is a kind of implication (and insult) that they're not up to it – either in intelligence or in commitment. But the truth is somewhat different when one looks at the issue.  The question "why don't they solve the impediments?" and "what do they need to know to remove the impediments the team is facing?" are questions that should be asked, but rarely are.

Teams often run into impediments whose cause are outside the team. The teams therefore can't solve them. But why doesn't management? For example, many Scrum teams are composed of members who work on other things besides their main project. This causes thrashing an delays.  

Why does this happen? Because management believes that one of their jobs is to make sure that everyone has productive work to do.

Why? Because management tends to look at people's productivity in making management decisions.

Why? Because productivity is what they have been taught and used.

Why haven't they been taught more useful agile methods? Shouldn't this change in Agile? Not necessarily.

Why? Because many Scrum thought leaders believe you shouldn't teach much, just create a framework.

BTW: See Learning To Manage What Matters – Not Always Intuitive for more on the counter-productiveness of managing productivity.

Implicit Assumptions That May Not Be Correct

Two (unexamined) tenets of Scrum are impediments that slow the team down will be recognized as impediments and that the team is capable of removing them. Besides my earlier statement that impeded teams may not have the capacity to remove the impediment I will go further and state that teams may not even recognize all of their impediments as just that. Many things that slow a team down are often just viewed as the way it is. There is no notion that it is an impediment. For example, I don't consider the fact that I have to take planes when I want to travel long distances an impediment – it's just the way it is. Now in this case, I think my perception of reality is correct (but it would be cool if one day I woke up and found out I could fly on my own!).

The Whys of "Scrum But"

Let's take another example of applying 5-whys to a common "Scrum but" - retrospections.

Why is the team not doing retrospections?
Because they don't see the value of the retrospections.

Why don't they see the value of the retrospections?
Because they think there is more value in doing their work.

Why do they think there is more value in doing their work?
Because the retrospections haven't really done anything for them.

Why haven't the retrospections done anything for them?
Because they often don't get agreement on what to do.

Why don't they get agreement on what to do?
Because they think software development is about intelligent people using their judgment – that is, there are no real rules to use or a scientific method to apply.

Why do they think this?
Because software is so complex and no one has presented them with a theory to explain it.

BTW: This is one reason I talk a lot about the theory of flow underneath Lean-thinking. I have seen that it can provide a thought process that helps unite the team.

The Dilemma (Faulty thinking?) of Scrum

Scrum is in a dilemma here. On the one hand, Scrum is based on being a light framework that doesn't give a model to work from. On the other, without such a model many Scrum practitioners don't realize the importance of the practices or how to manifest the affect of the practices in their own context. It seems that the reaction has been to insist that certain practices be done (at least at the start) – which is ironic because Scrum is not intended to be prescriptive. If there isn't a model, then how can one tell if one is doing Scrum? The common thinking is that you're doing Scrum if you are removing your impediments, but you're not doing Scrum if you are accommodating them. This, of course, assumes that Scrum can always work - a dangerous assumption, in my mind. It also makes it very difficult for new teams to know how well they are doing.

Getting Beyond "Scrum But"

To get beyond "Scrum But" one needs to be able to understand when the practices of Scrum work and when they don't.  I believe this requires both an understanding of why Scrum works and a scientific approach in how to apply these rules.  Unfortunately, I have not seen the Scrum community embrace this approach.  See the following blogs for more:

Author: 

Share this:

About the author | Al Shalloway

Al Shalloway is the founder and CEO of Net Objectives. With over 40 years of experience, Alan is an industry thought leader in Lean, Kanban, product portfolio management, SAFe, Scrum and agile design.


Comments

Hi Alan,

I've seen you make this argument several times. It sounds as though you have a beef with the SA in general and Ken Schwaber in particular. The problem I find with your critique is that it lacks precision and makes broad generalisations that are obviously not true.

  • Firstly *Agile IS NOT Scrum* You seem to compound the two, and make the assertion that Scrum is flawed hence Agile is flawed too.
  • *Scrum is not Ken Schwaber or the SA* Scrum has a history that starts out with Hirotaka Takeuchi and Ikujiro Nonaka. Other early pioneers include Jeff Sutherland and Mike Beedle. Bundling the work of all these individuals into the views and opinions of 'ken Schwaber" is niave at best and disingenuous.

This brings me to the most important point. No external Agent can enforce "Agile Adoption" on a company or organisation. Change is an internal process and must come from within. Organisations may hire Scrum Consultants or Lean Consultants, but it is not the Consultant that will adopt Lean or Agile values and principles it will be the organisation if they choose to. At best the consultant can hold up a mirror, perhaps depart some knowledge, but it will only be through the commitment and hard work of the organisation itself that knowledge will be translated into understanding and understanding into action and change, eventually leading to skill.

Scrum takes the approach of "just do something" and use empirical feedback so that organisational problems and impediments become visible. For an organisation motivated to change, then this mechanism can be sufficient, and they will ask why? Five times perhaps seven times, but they will ask the questions themselves, and not rely on the consultant, and seek answers and adapt as needed.

In other scenarios, the Organisation may want answers to questions before they start, in which case they may be interested in the "science" behind Scrum. Well the science exists. It stems from social complexity and it explains why they must act to create change, and why rules for change can't be external to the organisation and imposed from the outside.

And this is the problem I find with the alternative vision you are providing. A consultant can turn up with his bag of tricks, but they are his bag of tricks. He may even have some success convincing management that these tricks are the elixir of life. The problem is though, that any model the consultant provides will be wrong. Why? because the model will leave out lots of important details needed for success, and any one of those details may have a tremendous impact since software development organisations are complex systems, and complex systems are non-linear, and it is impossible to predict which process details and which interactions in any given context are indeed essential. (The science behind Scrum explains this).

So the management impose the Consultants flawed model onto an organisation, without the understanding or the skill to fill in the gaps for themselves. Where does that leave us? Well it leaves us where management science fads have left us for the last 40 years. Passing fads imposed by management, and suffered by the workers and a bunch of Management Consultants plying their wares :)

If we agree that all models are flawed, then what we as change agents can best hope for is to provide organisations with the tools to effect change for themselves. Scrum is one such tool. The scientific knowledge you speak of is another tool. But at the end of the day these are just tools. In the end the responsibility for action lies with the organisations themselves.

2 years from now, when Lean and Kanban is being challenged by the latest shiny new Management fad, and are being labeled failures, when in fact it is the adopters who are failing, then you will understand exactly what I mean. At that time you may look at your past criticisms of Scrum in a different light.

Paul.

Paul:This issue is an important one, so yes, I have discussed related issues before, but mostly on user groups.  I do not believe I have ever mentioned 5-whys to help "Scrum but" in a blog before.  I don’t have a beef with the Scrum Alliance in general or Ken in particular.  I have a problem with a generalization in the industry that Scrum can do it all when it clearly can not.  I also have a beef with the fact that few people seem to be inquiring into why this is so. And, maybe I have a beef with the fact that whenever I engage in an inquiry as to why things work or don't work I am labeled as having a beef. ;)  I think this is a very important question.

And yes, Scrum is not Agile, but only one kind of Agile. However, I do think the industry confuses this issue, and this is one other thing I am trying to clear up as well – there are other, in my opinion, better (when going beyond a single team) Agile methods than Scrum.  I only referred to Agile methods in the imaginary conversation I thought someone doing Scrum would have. Sorry if that was confusing

I totally agree that no external agent can enforce anything. I have never said that, nor implied it. Or at least, did not mean to.  We seem to keep getting into this type of discussion and I keep agreeing with you that this is true and I am a little bewildered as to why you keep bringing up something we all agree with. Good consultants coach and open up opportunities that their clients don’t see.  At Net Objectives we have a history of showing people new ways to look at things that result in them making better decisions.  It is imperative that consultants/coaches do not make an organization dependent upon them – so part of their coaching/consulting must be to enable the clients to see new things for themselves.

I do want to respond to your comment about Lean and Kanban.  I have been doing Lean Software development for 5 years now based on my experience with Lean/Deming that goes back to 1984.  It is not about doing something as much as it is about an approach and being something.  Lean has a long history of several decades.  It is way too old to be called a fad.  I agree that it is the adopters that are failing.  However, if you have a flawed tool (which I believe Scrum is) that tends to have adopters fail because it doesn’t give them the right information, then I believe the tool bears some responsibility for the failure.  Lean does not guarantee any success.  But it has a much better set of tools.  It has a much richer history.  It presents a better paradigm for success.  And, perhaps, more importantly, it continues to evolve and grow to meet ever-changing needs.

Alan Shalloway, CEO Net Objectives

Hi Alan,

Just wanted to respond to one specific point. The term "Lean" was coined I believe in around 1995, and is relatively new, so I fail to see how you can claim that "Lean has a long history of several decades". I assume that you mean "post war Japanese industrialism" has a long history. I agree that in Japan that their industrial mindset has a long history, and has been very successful, and has gone way beyond being a fad.

Several have tried to transplant the very same mindset and success to the West, and over the years we have seen several labels applied to it like Just-in-time, Quality Circles, Six Sigma, Total Management, Total Quality Control, Total Quality Management, Lean,... just to name a few. I've personally witnessed several of these Management Sciences come and go over the years as passing fads. Now this doesn't mean that the ideas behind these labels were flawed, it merely means that the label became popular, there was a phase of wide spread adoption, then the popularity passed as the adopters became dissatisfied, and moved on to something new.

Lean is just a label. Interestingly the Japanese do not use the "label" Lean to describe what they do. In the west however, Lean has been advocated in fields as disperse as accounting and health care. Are the benefits of "Lean" proven in these non-industrial fields? In fact is the benefits of "Lean" proven outside Japan and Japanese industrialism?

If you check the literature you will see that these questions are still open to debate. One could argue that the serial relabeling of "post war Japanese industrial principles and practices" is a result of these practices loosing credibility in the West and passing as a fad, followed by the need to relabel, rehash and re-market the same ideas as "something new" so as to fuel yet another new fad phase.

Is Lean/Kanban the latest in this series of Japanese inspired fads? I think it is too early to say. Complexity Science however offers us a way out. By admitting that any given formula for organisational success cannot be readily transplanted and reproduced in a new foreign context, we place responsibility where it truly lies, with the organisations themselves.

Regards,

Paul.

I'm pressed for time today, so a quick response.  I am not certain the original coining of the term "lean." I know it was used in "The Machine that Changed the World" copyright 1991.  But I think Lean was used before then.   Don Reinertsen got to it by looking at nuclear submarine systems and combined that with seeing the affect of time on business to come up with his ideas on flow.  Taiichi Ohno of Toyota got to it from Deming and adding Just In Time (JIT) and autonomation.  I have done a few talks about Lean not being based on toyota anymore but bsed on the science of flow (lean-thinking) the management that opens up (lean-management) and continuous process improvement (we sometimes call this lean-knowledge stewardship). Our book's last chapter (Lean Agile Software Development: Achieving Enterprise Agility ) deals with this to some extent.

Theories of flow and batch size go back almost a hundred years to the nascent telephone industry.  Deming and Shewart not quite so far, but before the 50s.  So, yes, Lean has a long tradition on which it builds.

It only took about 3 years before I found the limits of Scrum.  I have been using lean for 5 years since that point.  Every year Lean becomes more manifest in its ability to help - not less.  So, yes, there may be something after lean - who would doubt that.  Given my track record of embracing new ideas and technology (object-orientation, automated testing, XP, design patterns, agile, scrum, lean) I will probably embrace that as well.  Actually, my own experience is that lean in software has changed more over the last 2 years because of the kanban community than Scrum has done in its entire history.  So I am not sure we'll see another big thing as much as an evolution of what is happening.

All of this conversation is academic, however.  What I know is that my compatriots and I have helped transform large IT and product organizations (4000+ being the biggest one) using techniques that Scrum doesn't afford. That's where the real rubber meets the road.  We used to be very big in the Scrum community.  We moved away from that when we found something better. We still use Scrum in the cases we find it applies.  But that is typically always small orgs that typically go the route of CSM training. When someone shows me how to kick off 75-100 people at once with great success using generic agile or specific Scrum I'll pay a little more attention.  BTW: I think Jim Highsmith may be able to do this (not with Scrum, however) - I am currently reading his latest book and am hopefully optimistic.

Alan Shalloway, CEO Net Objectives

So all the "Lean" adoption failures of the 80's and 90's and all the Agile adoption failures of the last decade now have a savior...

All they need to do is call NetObjectives :)

That's quite some claim. Lets hope that the word doesn't get out too quickly. You could end up stampeded in the rush :)

Paul.

I don't claim we are unique.  Nor does using us guarantee success.  One has to be committed.  I do claim that the methods we use can work when applied with commitment.  I also claim they are better than your standard get a Scrum pilot going and scale it up.

There is no silver bullet.  There is no silver company.  But we do good work with good methods, and I will stand by that.

Alan Shalloway, CEO Net Objectives

Free Email Updates!

Sign up for free email updates from
Net Objectives Thoughts Blog

Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean-Agile, Kanban, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, Kanban
Ken Pugh
Software Design, Design Patterns, Technical Writing, TDD, ATDD, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile