Cost of Implementing Continuous Integration

February 21, 2017 — Posted by Steve Thomas

Someone in your organization is telling you, "We really need to implement Continuous Integration." They have probably told you many reasons how and why your development organization will be much better, faster, higher quality, efficient, etc. once you have Continuous Integration in place. They are absolutely right - I also am convinced that if you implement Continuous Integration you will gain fantastic benefits. See my previous post, The Secret to Implementing Continuous Integration - CHEESE! However, any solid business analysis requires that you identify the costs and risks of the choice as well as the benefits.

Generally, I find that most organizations can readily figure out the costs that they will have to implement Continuous Integration.  Below are the major costs that I have found are typically needed to implement CI:

  1. Build Automation - Your development and build team needs to fully automate the product builds. This is usually a small investment and in many cases, your builds are mostly automated already.
  2. Build Environments - First, you will be building your product much more often, so you may need additional build servers and environments to support the increase in build activity. 
  3. Test Automation - A fundamental ingredient of Continuous Integration is having a suite of automated validation tests which run on every successful build to validate them or give feedback to the developers.  Automating this set of tests to validate the sanity of each build is one of the largest efforts needed. As you are implementing these tests, your organization will likely continue to validate builds using your current (likely manual) methods.  So, you will need to simultaneously support the automation effort and the existing build test and validation procedures.  Eventually, the test automation activity should become a primary testing focus of your development and test organization displacing much of the manual validation testing that you are currently doing. 
  4. Automation Tools & Training - For an organization moving to CI, the question of which automation tools to select and use frequently looms large.  Of course, selecting a good Automation tool can make the automation tasks easier.  The costs to factor in here include the tool purchase costs, runtime licensing costs, training costs and the benefits that the tool can provide to your automation team. 
  5. Test Environments - In order for CI to provide prompt feedback to developers, the validation tests must be run in a timely manner.  In many organizations, this means running the automated test suite on multiple test environments in parallel.  As the number of automated validated tests increase, the number of test environments will also increase.  In addition, many organizations choose to run multiple layers of automated tests which provide different types of feedback and run in different time-frames.  Each of these test layers require their own test environment(s) as well as test suites to provide the designed feedback.  Some examples of these test layers include: Validation tests to provide fast feedback to developers, Complete Regression to run a comprehensive set of tests and provide thorough validation, Integration testing to test the product with other connected components, Security testing, Performance testing etc. 
  6. Simulators - Because virtually no system stands alone, testing the system in Continuous Integration means relying on other connected components, systems or products.  In order to keep the tests from failing every time there is an issue with one of these connected components, many organizations create simulators that will stub in and respond for these connected components during some of the tests.  The organization will need to create and maintain these stub-in components or simulators.
  7. Emulators - In some cases (e.g. involving hardware components, expensive equipment or third-party components), the organization will need to create or purchase emulators which will emulate many of the functions of the external, connected components.  These emulators enable the organization to run more tests which require the full functionality of the connected system and to run them more often, which shortens the feedback cycle for the development team.

There are several common pitfalls that organizations that are implementing CI fall in.  You will want to be aware of these and plan to avoid these common traps. 

  1. Tools Fixation - spending too much time, effort and money to select the right automation tools.  Many smart companies start with free and open source tools and begin to learn automation with these tools before making a major investment in other proprietary tools.
  2. Automation Perfection - believing that the automated validation suite needs to be very robust (cover all the current manual validation steps) before you trust the CI process.  This leads to a longer overlap time where you are both running existing validation methods and automating tests for the new CI environment.
  3. Automating manual validation tests - believing that all the current manual validation tests should be automated.  Manual tests generally have longer test scenarios that test a variety of functions leaving the tester to identify the failing part.  Automated tests are better designed as small focused tests, where only one thing can go wrong.  For automated tests, the test failure should indicate exactly what component or transaction has failed in the system.
  4. Tester Skillset - When you have Continuous Integration, the majority of your testing time, effort and skill will be spent in automating more and more tests.  As a result the majority of your staff involved in testing must be skilled in test automation.  In many cases, this means that you will need to retrain your testers and you may need to replace some of them.  Believing that you can continue with a separate test automation team or focused automation experts means that your test automation will lag.  And you have spent most of the costs for implementing CI, but are receiving only paltry benefits.
  5. Static Development Methodology - not changing your development practices and methods to take advantage of CI is the most expensive pitfall of all.  In this case you have invested all the costs but are achieving a small fraction of the benefits.  Developers need to learn a different style of development.  The guiding principles for developers who are leveraging CI are:
    1. No branching - all development is done on the main trunk.
    2. Always releasable - developing in such a way that the code is always releasable.  This means a different approach to changing interfaces and altering database schemas.
    3. Test automation - automating tests associated with code changes is a core part of the development job.

My previous article laid out the financial benefits of implementing Continuous Integration in your organization.  Here I have presented the major costs that you will incur in that implementation.  It is important to recognize that both the financial benefits and the costs are incremental.  That is you can decide which tests to automate and include in your CI environment.  Likewise, you can select which simulators and emulators to build or buy.  Each of those choices will be its own business decision based on the costs and the expected benefits.

Our hope is that you are now armed with information and ready to make informed business decisions about implementing Continuous Integration.  I am confident that as long as you avoid the pitfalls most of the time, you will find fantastic benefits to moving ahead with Continuous Integration.  Please write and let me know your story.

Subscribe to our blog Net Objectives Thoughts Blog

Share this:

About the author | Steve Thomas

Steve Thomas is a certified ScrumMaster/Product Owner, Agile and Lean Coach experienced with facilitating Agile transformations in large organizations. He is experienced in all phases of the software development life-cycle.


Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Guy Beaver
Business and Strategy Development, Executive Management, Management, Operations, DevOps, Planning/Estimation, Change Management, Lean Implementation, Transitioning to Agile, Lean-Agile, Lean, SAFe, Kanban, Scrum
Israel Gat
Business and Strategy Development, DevOps, Lean Implementation, Agile, Lean, Kanban, Scrum
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, SAFe, Kanban
Ken Pugh
Agile Design and Patterns, Software Design, Design Patterns, C++, C#, Java, Technical Writing, TDD, ATDD, Certifications, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Marc Danziger
Business and Strategy Development, Change Management, Team Agility, Online Communities, Promotional Initiatives, Sales and Marketing Collateral
Max Guernsey
Analysis and Design Methods, Planning/Estimation, Database Agility, Design Patterns, TDD, TDD Databases, ATDD, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile
Steve Thomas
Business and Strategy Development, Change Management, Lean Implementation, Team Agility, Transitioning to Agile
Tom Grant
Business and Strategy Development, Executive Management, Management, DevOps, Analyst, Analysis and Design Methods, Planning/Estimation, Innovation Games, Lean Implementation, Agile, Lean-Agile, Lean, Kanban