Complexity is Relative

April 3, 2016 — Posted by Israel Gat

As software professionals, our fairly automatic response to complexity is to try to reduce it. Time and again many of us experienced the perils of overly complicated software. For example, in his 2013 Ph.D. dissertation[1], Dan Sturtevant analyzes eight consecutive releases of an application by a successful software firm, concluding that “files with high McCabe scores[2] are expected to have 2.1 times as many bug fixes as files with low McCabe scores.”

The emphasis on value in modern Lean-Agile thinking poses an interesting question with respect to how we should view complexity. Needing to fix more bugs due to a higher level of code complexity has, no doubt, additional cost associated with it. But, suppose the more complex code generated ten times higher value than that generated by a less complex version of the code? One could easily argue that it is worth fixing 2.1 as many bugs if the value generated by the corresponding code increased tenfold. In other words, value trumps complexity in this case.

Examining complexity relative to value enables us to go beyond the code per se to consider fundamental business realities underneath the code. Consider, for example, the following two scenarios:

1.     A startup has been highly successful introducing one line of products to market. Following its success with the first product line, the startup is struggling with managing the second line of products they launched a year after the first one. Assume the startup decided to accept additional complexity and invest in it by bringing aboard a capable brand manager and asking him/her to oversee the extension of the product management software to address the needs of two product lines. The investment in bringing aboard the brand manager, and in the software he/she oversees, might easily generate increasing returns to scale through having two successful lines of products (rather than just one) in the market. In other words, the incremental investment in hiring a brand manager and rewriting parts of the product management software to suit the extended needs has paid off in spite of the fact that it inevitably increased both organizational complexity and code complexity.

2.     A $500M software company is in disarray as a result of growing too fast. The company’s CEO decides to rein in the chaos by building a new PMO organizational entity that will be responsible for planning, budgeting and governing at the corporate portfolio level. For variety of reasons, the new PMO organization does not manage to develop adequate software to bridge the gaps between the corporate portfolio level and the individual project team level. Misalignment between the two levels grows exponentially as the project teams establish new realities on the ground every two weeks. For all practical purposes, the new PMO organization is reduced to a passive reporting function. The investment in forming the PMO function, hiring and training programs managers for it and developing software for their particular needs, only yielded diminishing returns to scale. In other words, in this case the investment in incremental complexity (in the form of the PMO organization and the software its uses) has not paid off.

My recommendation to the reader who struggles to put complexity in perspective is to view it in a relative manner, i.e. examine the extra value in light of the incremental investment in complexity. In certain cases, increasing complexity is actually worthwhile. In other cases, it might not as the returns on so doing are not sufficient.

I would like to conclude this blog post by pointing out a fascinating linkage between modern software and ancient history. in his classic book The Collapse of Complex Societies, Joseph Tainter avoids considering organizational complexity as ‘good’ or ‘bad.’ Rather, he views investments in additional complexity as something that a society should do when the returns to scale on the investment are increasing. Conversely, a society should stop investing (if it can) in incremental complexity when the returns on the investment are diminishing. If you ask me, his view of society and the roles complexity plays within it fit perfectly with the most recent research in software development as an organizational and cultural phenomenon.


[1] See System Design and the Cost of Architectural Complexity, http://dspace.mit.edu/handle/1721.1/79551.

[2] Dan chose to use the term McCabe Score in his Ph.D. dissertation instead of the conceptually similar term Cyclomatic Complexity.

 

Subscribe to our blog Net Objectives Thoughts Blog

Share this:

About the author | Israel Gat

Dr. Gat’s executive career spans top technology companies, he has led the development of products enabling companies to move toward next-generation system management technology. Dr. Gat is also well versed in growing smaller companies and has held advisory and venture capital positions for companies in new, high-growth markets. He was presented with an Innovator of the Year Award from Application Development Trends in 2006. Dr. Gat focuses his consulting and writing on technical debt, large-scale implementations of lean software methods and agile business service management (“devops”).



        

Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Guy Beaver
Business and Strategy Development, Executive Management, Management, Operations, DevOps, Planning/Estimation, Change Management, Lean Implementation, Transitioning to Agile, Lean-Agile, Lean, SAFe, Kanban, Scrum
Israel Gat
Business and Strategy Development, DevOps, Lean Implementation, Agile, Lean, Kanban, Scrum
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, SAFe, Kanban
Ken Pugh
Agile Design and Patterns, Software Design, Design Patterns, C++, C#, Java, Technical Writing, TDD, ATDD, Certifications, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Marc Danziger
Business and Strategy Development, Change Management, Team Agility, Online Communities, Promotional Initiatives, Sales and Marketing Collateral
Max Guernsey
Analysis and Design Methods, Planning/Estimation, Database Agility, Design Patterns, TDD, TDD Databases, ATDD, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile
Steve Thomas
Business and Strategy Development, Change Management, Lean Implementation, Team Agility, Transitioning to Agile
Tom Grant
Business and Strategy Development, Executive Management, Management, DevOps, Analyst, Analysis and Design Methods, Planning/Estimation, Innovation Games, Lean Implementation, Agile, Lean-Agile, Lean, Kanban