Complexity, Causality and Why Keeping It Simple May Keep You Stupid

January 12, 2013 — Posted by Al Shalloway

Many in the Scrum community take an attitude of starting with as simple a framework as possible. While I agree having too much process can be bad, having too little can be just as damaging.  As Einstein once said – "Make things as simple as possible but not simpler."  Making things simpler than they should be makes things simplistic.

I believe Scrum is often presented in a simplistic manner - one that has negative consequences to those using it for the first time.  This opinion is based on the patterns of challenge that are readily apparent to many adopting Scrum.  These are evidenced every time I go to a conference (about 8 times a year).  These patterns of challenge are at both the team and the organizational level. Please be clear, I am not attacking Scrum, I am suggesting it needs to be extended as well as used where it is well suited.  To understand the need for both of these we need to have a basic understanding of complexity.

In my earlier blog – Scaling Scrum – What Works – I mentioned organizations are typically at one of three different levels of complexity:

  • Low complexity is when there is a dedicated stakeholder, projects are independent of each other and there are well defined project objectives and priorities.
  • Medium complexity is where there are multiple-team projects, multiple stakeholders exist, there are enterprise release impacts of the projects and teams are given business requirements requiring further discovery. 
  • High complexity organizations have projects implemented by multiple development groups, have new technology being used, have external dependencies, and have multiple stakeholders with competing priorities.

The more complex a situation you are in, the more difficult the challenge of scaling.  In The New New Product Development Game (the article which inspired Scrum), the method described was intended for low complexity situations. Not surprisingly, Scrum is very well suited for these - although I still believe an understanding of Lean-flow is invaluable. Unfortunately, there is a lot of evidence (as well as theory) that Scrum is not well designed for medium and high complexity situations.  Scrum-of-Scrums (SoS) is the common method to extend Scrum into these more complex endeavors. Unfortunately, Scrum of Scrums has proven to be ineffective in leading a bottom-up, team-of-teams approach to scaling.  See my upcoming webinar on what will and won’t work to scale Agile –The Three Ways to Scale Agile and One That Doesn’t Work So Well, January 31. The reason is that scaling Agile takes an holistic view, something difficult to achieve from a bottom-up, or team-of-teams approach.  Lean-flow provides the holistic view required.

Given the evidence that Lean-flow assists Scrum teams and that Scrum-of-Scrums is not an effective method to lead an organization to enterprise agility, why has this attitude persisted?  I believe some of it is remembering the pain of too much process.  But I think much is also due to the misunderstanding of complexity by many in the Agile community (see a 3+ year old blog - Types of Processes - by Don Reinertsen where I lay this out).  Complex systems are those where the relationship between cause and effect can only be perceived in retrospect. One often hears that “software development is complex” with an implication that means all we can do is “inspect and adapt.” But even complex systems have patterns (e..g, weather).  And not all of software development is complex – see People Are Complex Software Development Isn’t.  In fact, much of the practices of software development are very predictable.  Here are a few:

  • The more software developers are spread apart by time (e.g., off-shoring) the more difficulty will occur in communication with a corresponding amount of wasted effort
  • The more delays in the workflow taking place, the more waste will be created. This is especially true for delays between:
    • getting requirements and using them
    • getting requirements and validating them
    • coding and testing
    • testing and integration
  • The more work items your staff works beyond their capacity the greater the aforementioned delays will be
  • The larger the size of the work entering the input queue the greater the aforementioned delays will be

The bottom line is that delays cause waste and it is very clear what causes most of these delays.  Not arming your teams with this information is like training someone how to drive but neglecting to tell them which side of the road to drive on.

In other words, cause and effect is very clear in many cases even if it is at a macro, not micro level.  All of the above items are fully explained by the theories of Lean-Product Development Flow (see Don Reinertsen’s Principles of Product Development Flow: Second Generation Lean Product Development).  My point is that given these relationships, that exist in all situations, why would we not want all teams to know about them?  And, why would we not want to include practices that assist us with these principles? And by principles, I mean principles that for software developers are like the laws of physics - they always apply - for example, certain types of delay cause waste.  We need to understand this.

I hear that much of the reason for starting with a somewhat empty framework is we should be taking an attitude of KISS -  Keep It Simple Stupid.  What people forget is the KISS principle relates to products. Applying it to a development framework or method is another thing altogether.  Steve Jobs once said “Simple can be harder than complex.  You have to work hard to get your thinking clean to make it simple.  But it’s worth it in the end because once you get there, you can move mountains.”  In other words, simplicity should be a result, not a method.  

The result of this simplistic approach are patterns of challenge we see in virtually all but either the simplest situations or where folks are incorporating the principles of Lean into their Scrum practices:

  • Many stories are incomplete at the end of a sprint
  • Testing is not being completed at the end of a sprint
  • Inability to efficiently handle the interruptions that occur during the sprint
  • Estimation takes too long
  • Lack of predictability of business value delivered
  • Difficulties with distributed teams
  • Inability to create the requisite cross-functional teams
  • Starting too many things too soon
  • Too much time taken to do estimation
  • Lack of predictability of business value delivered
  • Inability to scale
  • Significant thrashing when multiple teams attempt to integrate their work
  • Inability to achieve alignment of stakeholders on what to build next
  • The list goes on

The tragedy is that virtually all of these challenges can be vastly diminished when lean-product development flow is attended to. In fact, these challenges are so predictable that we’ve developed a Scrum tune-up workshop to counter them. If you are having any of these challenges, you might want to watch our free webinar coming up next week – Enhancing and Extending Scrum with Lean where I discuss how Lean can help out at the team level.

At Net Objectives we’ve been doing Agile, in one form or another (Scrum, Lean, Kanban, …) for over 14 years).  As our methods have evolved,  we appreciate more and more the advantage to understanding as much as we can about the software development process.  True, there is no one size fits all in the form of practices, but there are core principles that shape software development.  Not attending to them is akin to building bridges without a firm grasp of gravity. We believe in arming yourself with the knowledge gathered over the last few decades.

The Bottom Line

Scrum is a very good team process in many situations.  However, in all but the simplest situation, understanding the principles of lean-product flow will help Scrum teams tremendously.  Trying to use Scrum to scale Agile with Scrum-of-Scrums to coordinate teams is, however, almost always a mistake.  Scaling requires the holistic view Lean flow provides.  Coming from a bottom up, coordinating teams is very difficult in complex situations.

Al Shalloway
CEO, Net Objectives

BTW: If you are having challenges such as I’ve described, please drop me a line, we’d love to help.

Summary of links in this blog:

Postscript

After writing this it occurred to me readers may like to read Why Scrum Works and How This Tells Us When It Won't.

Subscribe to our blog Net Objectives Thoughts Blog

Share this:

About the author | Al Shalloway

Al Shalloway is the founder and CEO of Net Objectives. With 45 years of experience, Al is an industry thought leader in Lean, Kanban, product portfolio management, Scrum and agile design. He helps companies transition to Lean and Agile methods enterprise-wide as well teaches courses in these areas.



        

Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Guy Beaver
Business and Strategy Development, Executive Management, Management, Operations, DevOps, Planning/Estimation, Change Management, Lean Implementation, Transitioning to Agile, Lean-Agile, Lean, SAFe, Kanban, Scrum
Israel Gat
Business and Strategy Development, DevOps, Lean Implementation, Agile, Lean, Kanban, Scrum
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, SAFe, Kanban
Ken Pugh
Agile Design and Patterns, Software Design, Design Patterns, C++, C#, Java, Technical Writing, TDD, ATDD, Certifications, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Marc Danziger
Business and Strategy Development, Change Management, Team Agility, Online Communities, Promotional Initiatives, Sales and Marketing Collateral
Max Guernsey
Analysis and Design Methods, Planning/Estimation, Database Agility, Design Patterns, TDD, TDD Databases, ATDD, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile
Steve Thomas
Business and Strategy Development, Change Management, Lean Implementation, Team Agility, Transitioning to Agile
Tom Grant
Business and Strategy Development, Executive Management, Management, DevOps, Analyst, Analysis and Design Methods, Planning/Estimation, Innovation Games, Lean Implementation, Agile, Lean-Agile, Lean, Kanban