The Secret to Implementing Continuous Integration - CHEESE!

February 1, 2017 — Posted by Steve Thomas

Last time, I wrote about how Continuous Integration is like the Broccoli of the Agile world… everyone knows that it is good for them, but few really consume it.  

The hanging question from that blog was, "How can we convince organizations to actually eat their Broccoli and implement real Continuous Integration?"  

A friend of mine tells me that the secret to getting people to eat their broccoli is "Cheese - Lots of Cheese". So, what is the "Cheese" that will make people want to consume their CI Broccoli?  

I believe that the first key is to appeal to the right people in the organization. As we discussed last time, virtually everyone in the organization has reasons to oppose implementing CI. The linchpin is to convince the organizational leadership. If they decide to implement CI, they can usually bring the rest of the organization to overcome their individual reasons for opposing CI. The leadership team are the people able to authorize the expenditures in equipment, test environments, tools and staff to implement effective CI. The "cheese" that these executives are interested in is MONEY - Return On their Investment.  

In trying to get organizations to implement CI, I have tried several of the following arguments with executive leaders and managers. You may have tried some of these or others. I suspect that like me, you found these qualitative arguments to be ineffective in persuading the organization to really implement Continuous Integration:

  • It's required. CI is a core Agile practice which you need to implement to become Agile! You can't say that you are really using Agile methods without CI!
  • Everyone does it. CI is implemented by most advanced companies using Agile methods. And every company that is highly successful with Agile methods has CI as a cornerstone.
  • It's good for us. Implementing CI will lead to "good things" like faster development cycles, faster testing, faster feedback, better quality, etc.

All of the arguments above are like adding margarine to your broccoli: it enhances and highlights the flavor of the broccoli rather than covering it up. However, most people are looking for something to cover the flavor of broccoli. The leader that is already fully persuaded about the benefits of CI can use these to push the organization to implement CI. However, the clear majority of leaders aren't persuaded by this margarine and they want the cheese of money. We can give that to them with a clear business case built on cost / benefit analysis.  

In this blog, we are going to focus on understanding and quantifying the benefits. I have found that most of the time, organizations are able to identify their costs for implementing CI. Fortunately, it is not difficult to calculate the economic benefits of implementing CI. Below we show four quantifiable economic benefits of implementing effective CI.  

As we describe each of the economic benefits, we will show a concrete example of exactly how these savings would be reflected for a sample organization called ACME. ACME has a development organization composed of 12 agile teams (100 staff spread across locations in US and India) using SAFe(r) 4.0. The organization currently releases their software quarterly. Their average loaded headcount cost is $70K/person.  

Implementing an effective CI system will provide clear economic benefits in the following areas:

  • Hardening costs - Organizations that don't have effective CI, invariably need a "hardening sprint" or period. This is a period of time after the organization has completed the primary development and they are getting ready to release the product. During this hardening time, the development and test teams are almost fully consumed in the activities of testing, debugging and fixing the integration issues that arise. If the organization has effective CI in place, then the hardening time and effort would be eliminated. Therefore, all the hardening efforts represent potential cost savings for CI.
    • ACME - their hardening efforts represents 2.5 weeks for each release, or 250 staff weeks for a cost of $336K per release and $1.34M for the year.
  • Bug fix costs - Without CI, it is common that the bugs or issues that the organization finds as they integrate different features, components and systems together will be found after the story or feature is "completed". Many will be found during the "hardening" period described above, but some will also be found in the intervening sprints. Again, this effort would be eliminated with an effective CI practice.
    • ACME - has found that in each Program Increment that their teams spend about 5% of their effort on fixing issues that are either in the released product or in "completed" stories which have not been released yet. They believe that 75% of this effort would be eliminated with effective CI. For ACME, the potential savings of effective CI in reducing Bug Fix costs is $50K per PI or $200K annually.
  • Cost of Delay - Lack of effective CI is the primary reason that most organizations don't release updates to their products more often. As Don Reinertsen has repeatedly told us, "if you only quantify one thing, quantify the Cost of Delay". Using Cost of Delay any organization can calculate the benefit of releasing their product updates when they are ready, rather than holding them for the next batched release.
    • ACME - is releasing features that they have projected will benefit the organization by $10M in the current year. In aggregate, the features have a cost of delay of $800K per month. For the ACME organization, features are held on average 6 weeks after they development complete before they are released. Across all the features for the year, this means that they could realize an additional $1.1M if they release their features promptly when they are completed.
  • Cost of poor quality - Without effective CI, much of the manual testing that the organization does is repetitive of earlier test runs. The organization has relatively little ability to test the product in new ways including exploratory testing, stress testing, security testing, etc. The result is that the product delivered to the customer is of substantially lower quality than if they had effective CI and focused their testing efforts on expanding the testing regime.
    • ACME - In addition to the ACME development organization, ACME has support and operations groups to support their products. These groups have 15 people and they expect that they could save 10% of their costs when the quality improves. These potential savings will not materialize until significantly after effective CI is established. However, at that time, they expect to realize a savings of $110K or more annually.  

While there are other benefits of implementing effective CI, using these 4 simple factors will help your organization analyze the quantitative benefits. As you see from the above examples, the biggest financial benefits will come from reducing / eliminating the hardening period and by leveraging CI to release feature as soon as they are completed. In the example above, the ACME organization could realize a potential $2.75M each year or >25% of their annual budget. That is a lot of "cheese" to spread on your CI Broccoli and should make the job of convincing the organization to consume their "CI Broccoli"… a piece of cake.

Once your leadership team sees the enormous potential savings that they can get from CI, they will naturally want to explore deeper the risks and costs to taking this journey. In my next blog, I'll discuss the costs and risks of implementing Continuous Integration and its big brother Continuous Delivery.

Subscribe to our blog Net Objectives Thoughts Blog

Share this:

About the author | Steve Thomas

Steve Thomas is a certified ScrumMaster/Product Owner, Agile and Lean Coach experienced with facilitating Agile transformations in large organizations. He is experienced in all phases of the software development life-cycle.


Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Guy Beaver
Business and Strategy Development, Executive Management, Management, Operations, DevOps, Planning/Estimation, Change Management, Lean Implementation, Transitioning to Agile, Lean-Agile, Lean, SAFe, Kanban, Scrum
Israel Gat
Business and Strategy Development, DevOps, Lean Implementation, Agile, Lean, Kanban, Scrum
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, SAFe, Kanban
Ken Pugh
Agile Design and Patterns, Software Design, Design Patterns, C++, C#, Java, Technical Writing, TDD, ATDD, Certifications, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Marc Danziger
Business and Strategy Development, Change Management, Team Agility, Online Communities, Promotional Initiatives, Sales and Marketing Collateral
Max Guernsey
Analysis and Design Methods, Planning/Estimation, Database Agility, Design Patterns, TDD, TDD Databases, ATDD, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile
Steve Thomas
Business and Strategy Development, Change Management, Lean Implementation, Team Agility, Transitioning to Agile
Tom Grant
Business and Strategy Development, Executive Management, Management, DevOps, Analyst, Analysis and Design Methods, Planning/Estimation, Innovation Games, Lean Implementation, Agile, Lean-Agile, Lean, Kanban