Greek Tragedy in Continuous Integration

February 24, 2017 — Posted by Steve Thomas

A business leader approached my colleague and complained, "I invested $50 million last year in new testing environments for Continuous Integration - It was a waste, we are hardly getting any benefit from it." How is it that an organization could establish a solid beginning for CI, investing in the environments, tooling and test automation and yet get little benefit. Sadly, we've seen this movie play out so many times that we know exactly what didn't happen in the organization.

Tragedy

The saddest part of this story is that there are many development and operations managers that would gladly give their left arms to have the executive support of a $50M investment in CI infrastructure. It is practically criminal that this investment was wasted. Even worse, this executive will tell his peers about their results and the likelihood that other executives will make the investments needed to realize CI, CD and DevOps will get even worse.

When I hear of situations like this one, I know exactly what has happened … I just don't know the specifics. I know that someone in the organization was able to effectively persuade the executives to invest in the environments, tools, build engineers and basic test automation to implement the building blocks of CI. I also know that almost certainly the major players in the development groups signed on and said that they would appreciate the new CI environment and they would use it and support it. Finally, I am sure that despite their assurances that the development community wasn't prepared to make the cultural and development practice changes needed to really support and embrace CI. It is also highly likely that the test organization was not prepared to retool their team to focus predominantly on test automation.

What turns this sad story into a Greek Tragedy is that the group that will benefit most from effective Continuous Integration is the development group. The is exactly the group that is mostly likely causing this Epic Failure of CI adoption.

In their book Leading the Transformation: Applying Agile and DevOps Principles at Scale, Gary Gruver and Tommy Mouser remind us that the most important and critically limited resource any organization has is its ability to absorb and embrace change. They point out that this is the critical resource needed for effectively implementing Continuous Integration, Continuous Deployment and DevOps at scale.

Development

Let's start with the development organization. The practice and methodologic changes needed in the development group are so pervasive that they require a shift in the culture of the development teams. Changing the culture means that managers need to take on this challenge so that they can impact all of the little elements that influence and are part of the culture.

The development group must change particular development practices and methods, in order to leverage the benefits of CI. They have to learn a different style of development based on the guiding principles:

  1. Frequent check-in - This means checking in code changes daily or even more often. As a developer, I have to change my development approach from big design for the feature to designing my changes and implementing them in small atomic batches. This is not the way most of us were taught to program.
  2. Always releasable - Developing my features in such a way that the code is always releasable. In order to do this, I can't simple change the interface to a method or class, instead I need to create a new version of the class. Developing this way means I have to take a different approach to changing interfaces and altering database schemas.
  3. No branching - As a developer, I am no longer working on a branch that isolates my changes for days or weeks or months at a time. As stated above, I use a different approach to making all of my software changes small and atomic so that they can be applied to the main trunk without breaking the existing functionality.
  4. Test automation - The development job is not done until there are new automated tests added to the test suites that will confirm that the new functionality is working. These tests protect the functionality from being accidentally broken by some future change.

At first the development group will likely moan and complain that this approach to development (particularly the requirements to make code always releasable and making test automation part of the development job) involves more work by developers and is less efficient. They will argue that the straightforward design-code-test pattern for developing features (or systems) is much more efficient. On the surface, they are absolutely correct. However, when they experience the whole life-cycle guided by these principles, then they will realize that this new approach is actually much faster and more efficient overall. This new style of development paired with Continuous Integration will make the entire cycle of getting changes to the product fully integrated much more efficient by eliminating the endless rework cycles.

Testing

In a similar manner the culture of the "test group" needs to shift. First of all the line between development and test will erode and fade. That line is significantly blurred when we require that test automation be a part of every piece of development. This means that test automation is not a separate activity or an after-thought. Rather the test automation is built in to the entire process. The best example of this is Acceptance Test Driven Development.

While there is still a place for manual and exploratory testing of the system, the primary focus of the people skilled in testing shifts to test automation. The tester's role shifts from designing and executing comprehensive test scenarios to designing and implementing small automated tests that are:

  1. Atomic - they test one item or aspect of the system. Passing or failing tells the whole development group whether a specific functionality is working or not.
  2. Fully automated - they are automated in setup, execution, checking the results and finally automated in communicating the test status - pass, fail, not completed to another automated script.

Conclusion

This situation was a tragedy. Sadly we have seen this tragedy play out several times. However, the bigger problem that we see is that organizations are not willing to move towards Continuous Integration. I addressed why people are reluctant in my blog "Why Continuous Integration is like Broccoli" and I addressed how to help your executives recognize the tremendous benefits they can get from CI in "The Secret to Implementing Continuous Integration - CHEESE!"

Please do your part in making the best possible use of the limited budgets that we often face to implement Continuous Integration. You can do this, by making sure that your organization understands and faces the cultural shifts needed for CI. Please write and let me know your story for implementing Continuous Integration.

Subscribe to our blog Net Objectives Thoughts Blog

Share this:

About the author | Steve Thomas

Steve Thomas is a certified ScrumMaster/Product Owner, Agile and Lean Coach experienced with facilitating Agile transformations in large organizations. He is experienced in all phases of the software development life-cycle.



        

Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Guy Beaver
Business and Strategy Development, Executive Management, Management, Operations, DevOps, Planning/Estimation, Change Management, Lean Implementation, Transitioning to Agile, Lean-Agile, Lean, SAFe, Kanban, Scrum
Israel Gat
Business and Strategy Development, DevOps, Lean Implementation, Agile, Lean, Kanban, Scrum
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, SAFe, Kanban
Ken Pugh
Agile Design and Patterns, Software Design, Design Patterns, C++, C#, Java, Technical Writing, TDD, ATDD, Certifications, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Marc Danziger
Business and Strategy Development, Change Management, Team Agility, Online Communities, Promotional Initiatives, Sales and Marketing Collateral
Max Guernsey
Analysis and Design Methods, Planning/Estimation, Database Agility, Design Patterns, TDD, TDD Databases, ATDD, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile
Steve Thomas
Business and Strategy Development, Change Management, Lean Implementation, Team Agility, Transitioning to Agile
Tom Grant
Business and Strategy Development, Executive Management, Management, DevOps, Analyst, Analysis and Design Methods, Planning/Estimation, Innovation Games, Lean Implementation, Agile, Lean-Agile, Lean, Kanban