The Need for Leadership in Scrum - Updated

December 15, 2009 — Posted by Al Shalloway

(This updates an entry originally posted April 21, 2007.)

From 2000 to 2004, I trained and coached teams in Scrum in a manner that was fairly consistent with what the Scrum Alliance said Scrum was. Beginning in 2004, I began to introduce Lean thinking into the mix. At first, I was mostly adding Lean insights into Scrum practices but over time, I began to let Lean thinking motivate and lead what I taught. It became clear to me that the ideas and attitudes of many of the thought leaders behind Scum were at odds with Lean thinking. Today, as I look at what iterative, time-boxed approach to software product develop should be, I can no longer label it as "Scrum." I don't do Scrum with Lean. I do something else - something I consider to be much more effective

This blog describes one of the biggest differences I have with many Scrum thought leaders: the proper role of management in Agile teams. Too many Scrum trainers/coaches/practitioners continue to dismiss the need for management in organizations using Scrum.

Distrusting management 

I was recently part of a series of email discussions that included many experienced Scrum practitioners. I was surprised by the number of posts describing leadership as either manipulative in nature or just a plain old bad thing. Then I noticed a post by Ken Schwaber that seems to underscore an attitude that management typically involves itself only in bad ways (see here for the full text):

The catch is that Scrum, like all Agile methodologies, depends on self-managing teams—the exact opposite of most top-down directives. While management may understand that Scrum succeeds largely because the people doing the work define how to do the work, resisting the urge to intervene can be very hard, particularly when someone has stuck her neck out to make the top-down implementation happen. Managers who have initiated the Scrum implementation often feel responsible for its success and revert to command-and-control tactics.

Many Scrum practitioners seem to have a fundamental belief that self-managing teams and direction from the top are opposed to each other. Now, I don't doubt that there are times that managers interfere in team practices but that does not mean that management is inherently bad. Yes, direction in the form of micro-management is bad but direction in the form of vision and matching people to their abilities is not! Many Scrum practitioners seems to have an inherent distrust of management to do the right thing and some practitioners even teach that the team must be protected from management. I fundamentally disagree. This attitude is a serious impediment to Scrum succeeding at the enterprise level.

Of course, we don't want the direction of the team to be changed at a whim. Why not educate management why such change is bad? Why not make the team's process transparent to management - as Lean/Kanban suggests - so that they can see clearly the impact of such change and can balance this impact with other economic forces? Contrast this healthy transparency with the "Chicken and Pigs" metaphor that is basic to Scrum: We can’t have Chickens (management) even have the option to talk for fear they will derail our team. Why? Because deep down we don’t trust management... and yet, we expect them to trust us to self-manage. This approach obscures the team's process from management and creates unhealthy isolation between the team and its management.

Unfortunately, this attitude is pervasive across the Agile community. Given the heavy focus on over-bearing process that has gripped this industry for years, this may not be surprising. However, it is time to include leadership and management in the transition to agile methods. We must see how we can marry respect for people (and hence the team) along with process.

This requires leadership and proper management.

The need for leadership and management 

Let me define what I mean by leadership and management. Marcus Buckingham, in his excellent book, The One Thing You Should Know, does a great job describing these two distinct yet complimentary roles.

  • Leadership is the "ability to create a solid vision of a better future for those people he/she is leading. " A leader must have a compelling desire to move towards that vision.
  • Management is the "ability to match people’s tasks with their skills. "

A good manager is like a skilled Chess Master. Unlike checkers, where all the pieces are the same (until one gets kinged), in chess different pieces have different strengths. A good chess player knows how to use these strengths. A good manager knows how to maximize the strengths of his staff. Leadership and management are not opposed to each other – they just have different functions.

Ironically, the need for proper leadership and proper management is illustrated in another excerpt in the same article by Ken Schwaber that I alluded to before:

Command and control has two negative effects on a Scrum team. First, developers get in the habit of waiting to do anything until they are told what to do. As a result, they often are afraid to take the initiative and are used to blaming management for stopping them from doing so. Second, management often feels as if it is not doing its job if it doesn't come up with answers and instead waits for developers to figure out solutions on their own, including letting them make mistakes.

In practice, teams require leadership. Without it, it is very likely they will continue moving forward in the way they have in the past. Many Scrum consultants say this is the job of the Scrum Master. But even here, the community is split equally between the Scrum Master’s role to remove impediments from the team and the Scrum Master’s role to lead the team. Depending upon which CSM course I am listening to or which Scrum Gathering I am going to I hear different things.

In any event, even if the team can provide its own leadership, it is still left to work within the existing structure of the organization in which they find themselves. This may not be a good thing.

Keeping Buckingham’s views in mind, the role of leadership and management then is to create the vision for the Scrum teams to be effective and to staff the teams with the appropriate resources. But just creating the vision is not enough. Leadership and management must help create the structure within which the team works to assist the team. This is something Scrum teams cannot do on their own. Ignoring this issue often causes great frustration for the team.  This leads Scrum teams to feel they must protect themselves from the organization when what they need to do is help the organization see how to better support the team.

Lean thinking helps 

If Scrum can’t help us here, what can? This is where Lean thinking comes in. Lean-Thinking tells us that there are rules we must pay attention to when developing software. These mostly center around achieving flow by eliminating waste and maintaining visibility so we can control things. This philosophy is based on systematic thinking. Insisting that a team's system be visible opens the door for managers/leaders to coach the team when they see something is wrong.  They don't need to (and, in fact, shouldn't) jump in with solutions but can provide guidance and coaching as needed. Of course, this attitude assumes that there is a science underneath software development - something many of the Scrumologists who don't believe in management involvement don't believe exists either.  These two actually go together - if there is no science, then just let the developers figure it out themselves. 

Ironically, much of the inspiration for Scrum came out of an article by Takeuchi and Nonaka which was based on their study of Lean methods. Lean’s underlying foundation is based on a combination of:

  • Respecting people
  • Continuous process improvement
  • Avoiding waste by eliminating impediments to flow

Lean doesn’t set up a “management vs team” mentality. Instead, it embraces management and team. Management to assist the team in building the best process possible.

At some point, I hope all Scrum/Agile practitioners also embrace management, not as a problem to overcome, but as a valuable resource to enlist. I suggest that by learning more about Lean principles that Scrum practices can be improved. Without embracing leadership and management, we will have great difficulties in getting Scrum to be successful in the enterprise.

Let us help you 

If you are interested in how Net Objectives can help you extend and support your Scrum practices with Lean principles, please send me an email at alshall @ netobjectives.com (leave out the spaces) and I will be happy to discuss it with you or forward you on to the correct person in our organization. You can also get more information from our website by registering here.

Subscribe to our blog Net Objectives Thoughts Blog

Share this:

About the author | Al Shalloway

Al Shalloway is the founder and CEO of Net Objectives. With 45 years of experience, Al is an industry thought leader in Lean, Kanban, product portfolio management, Scrum and agile design. He helps companies transition to Lean and Agile methods enterprise-wide as well teaches courses in these areas.


Comments

Hi Alan,

I don't want to defend Scrum, but I believe that there are a number of misunderstandings here that should be addressed. Perhaps you've been hanging out with the wrong Scrum crowd :), but there is a fundamental message behind Scrum and other Agile approaches that is indeed different from Lean and needs some explaining. Let's look at the ontology of work in its entirety:

http://nooperation.typepad.com/.a/6a00e54ff8b9c188340120a5bc34d8970b-pi

The axis in the diagram describe order and Un-order, and rules and heuristics. I won't go into detail explaining what these are, but from common daily experience we all know that certain types of work are more amenable to fixed rules, whilst others rely more on judgement (heuristics). We also know that certain jobs can be made very mechanical and predictable, whilst others are somewhat Un-ordered and unpredictable. Depending on which quadrant your work fits in to will determine the most appropriate management and leadership style.

Lets choose some areas of work away from software. First example, a call centre: Whilst in a call centre the arrival of calls cannot be precisely predicted (although I'm sure statistical probabilities are applied to good effect), the processing of the call can be made very predictable. The operator can be given a script to follow to resolve calls. If the caller asks a question that isn't on the script then the operator can be given a standard response and can forward the caller to another queue. This is a well defined and ordered process. Its mechanical in nature and we've all experienced the feeling of being herded around like cattle and bounced from on queue to the next when calling a call centre.

Another example: The movie industry. If I'm the CEO of Time Warner and I want to make a film, I need to hire/nurture the best directors and producers I can find and then release them to make great movies. The key word here is "release". I can't expect to tell them what to do and expect great work. I need to relax the levers of control and let them do what they do best. If I'm a director on set, I'd expect my actors to know the script, but I would also expect them to stray away from the script, ad-libbing and changing the dialogue as they see fit. I would see it as part of their creative license. New dialogue "that worked" would become part of the movie and would make the final cut.

These two very different types of work call for different kinds of management style and different kinds of leadership. The call centre is ordered and rules based. The film studio is Un-ordered and relies on heuristics.

Most companies treat software development like the call centre. Here you need rules, and well defined scripts. You also need to be able to predict how long each call will take and have an idea of the wait time. Here leadership is provided by the automated call answering system, which encapsulates rules that were determined in advance.

Agile approaches suggest that we should treat software development more like a film studio. There is still some control, the CEO can still pull the plug on a movie that is running hopelessly over budget and isn't showing much promise, but the director and the actors are released to do their thing, and leadership comes from different people at different times (director, writer, actor , etc..) in a self-organising and organic way. It just wouldn't work otherwise.

Now in either case the distinction isn't black or white. At times the call centre operative will be called upon to think on her feet when a clever caller throws a question at her that isn't in the script :) Likewise the film studio probably has some hard and fast rules like no talking by the film crew whilst filming and perhaps a fixed schedule for lunch , etc. In all jobs there are aspects of the work that falls in each quadrant. Each job tends to have a dominant quadrant, however.

What Agile is saying is that software development is mostly Un-ordered and based on heuristics. Most organisations tend to assume order and rules, the realm of Process Engineering. System Thinking assumes a mix of the two, with order and heuristics.

What I believe many Scrum practitioners point out, is that traditional management practice tends to have a cognitive bias towards the call centre approach, command and control. As far as that is true, then Ken Schwaber is absolutely right. System thinking says that in an ordered context, that a person in a position of authority can provide useful leadership. As far as that is true, you are absolutely right :)

So it depends on your context and where you think your work fits best in the matrix. I tend to believe that software development is complex and best understood through the lens of social complexity...

This is a view that I believe is shared by the majority of the Agile community.

Paul.

Paul:

When you say "Most companies treat software development like the call centre" this is the paradigm of micro-management and should be avoided.  When you talk about traditional agile, that becomes the management of herding cats and should also be avoided.  Lean provides a possibility of of coaching and managing people while enabling them to figure out what to do.  You theories are nice - but I prefer the evidence those in the Lean/Kanban communities have achieved. Look at the Lean Kanban conference in Miami and the Lean UK Conference in London for the proceedings which have documented this.   The standard Agile/Scrum approach at an enterprise level would be laughable if it weren't for the low productivity that ensues and that people's careers are being hurt. I would suggest you look at either David Mann's Creating a Lean Culture or Peter Scholte's The Leader's Handbook to learn more about how lean management helps teams be more creative by providing them the proper organizational structure from within which to work.

While lean can be used to manage a predictable process, no one (I repeat no one)  in the Lean Software community  believes software development is a linear process. All believe that it is chaotic and done by people which comprise a complex adaptive system.  By incorporating feedback, understanding some principles on which the work is based and providing proper leadership, lean can work at levels way beyond the scale that Scrum can.So, if we continue this conversation, please bring up examples you've seen of how Lean Software proponents have been espousing incorrect management.  

BTW: I should add that not all of Scrum follows the approach that I am describing - but it seems most does.  There is a split in the Scrum community but unless you pay attention it is hard to notice. I like the approach Jeff Sutherland takes.  If you read his material about management you do not hear disparagement.  He does not talk about banning management from the daily standups, btw.  He also talked about Scrum being done at different levels but was trying to make a distinction between team Scrum and more enterprise friendly methods.  But Ken Schwaber nixed that, saying "it's all Scrum" and added to the confusion of where and how Scrum applies.  The irony is Scrum claims a need to apply different methods to different contexts but then does the same thing everywhere ("inspect and adapt" while denying rules of software development exist).  Unfortunately I think you are right that I've been hanging out with the wrong crowd - that is, I am mostly familiar and referring to mainstream Scrum.

Alan Shalloway, CEO Net Objectives

Hi Alan,

That's just it. The theories I refer to aren't just theories, they are backed by practical experience too. My experience, Googles experience and the experiences of others...

There is more then one way. More then just one lens.

I'm glad that you have differentiated between the different perspectives even in the Scrum community. I for one would never say that management shouldn't attend a Scrum. I would say that management should be comfortable listening at a Scrum meeting and perhaps the Scrum meeting itself isn't the best time to make their voices heard.

Lean thinking is just one lens. There are others lenses. My goal was to give an accurate account of the Agile lens and where I believe it is most applicable. Thanks for the opportunity.

Paul.

I have never said there is one only one way or only one lens.  I have said that there are some that are always useful.  It is actually many in the Scrum community that say that. 

Please show me evidence of your theories in the software world.  I actually agree with the theories as theories but disagree with how they have been applied to software. I suggest my views are very consistent with a proper application of chaos and complex adaptive systems - even more so than most of the agile community.  Structures can help CAS.  You require feedback to manage non-linear systems.  Again, point to their application in the software world where people were guided by them. 

In any event, I believe your continued aspersions that lean is a micromanagement, prescriptive process represents a different understanding of lean than the lean thought leaders in the software area.

Alan Shalloway, CEO Net Objectives

The link to Ken Schwaber's article is invalid. Does someone have another link?

Apparently the Scrum Alliance has removed Ken's article from their site.  Anybody else know where  a copy of "From the top" by Ken is?

Alan Shalloway, CEO Net Objectives

Hi Alan,

Do you have a working link to the article you reference?

Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Guy Beaver
Business and Strategy Development, Executive Management, Management, Operations, DevOps, Planning/Estimation, Change Management, Lean Implementation, Transitioning to Agile, Lean-Agile, Lean, SAFe, Kanban, Scrum
Israel Gat
Business and Strategy Development, DevOps, Lean Implementation, Agile, Lean, Kanban, Scrum
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, SAFe, Kanban
Ken Pugh
Agile Design and Patterns, Software Design, Design Patterns, C++, C#, Java, Technical Writing, TDD, ATDD, Certifications, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Marc Danziger
Business and Strategy Development, Change Management, Team Agility, Online Communities, Promotional Initiatives, Sales and Marketing Collateral
Max Guernsey
Analysis and Design Methods, Planning/Estimation, Database Agility, Design Patterns, TDD, TDD Databases, ATDD, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile
Steve Thomas
Business and Strategy Development, Change Management, Lean Implementation, Team Agility, Transitioning to Agile
Tom Grant
Business and Strategy Development, Executive Management, Management, DevOps, Analyst, Analysis and Design Methods, Planning/Estimation, Innovation Games, Lean Implementation, Agile, Lean-Agile, Lean, Kanban