A business leader approached my colleague and complained, "I invested $50 million last year in new testing environments for Continuous Integration - It was a waste, we are hardly getting any benefit from it." How is it that an organization could establish a solid beginning for CI, investing in the environments, tooling and test automation and yet get little benefit. Sadly, we've seen this movie play out so many times that we know exactly what didn't happen in the organization.
The saddest part of this story is that there are many development and operations managers that would gladly give their left arms to have the executive support of a $50M investment in CI infrastructure. It is practically criminal that this investment was wasted. Even worse, this executive will tell his peers about their results and the likelihood that other executives will make the investments needed to realize CI, CD and DevOps will get even worse.
When I hear of situations like this one, I know exactly what has happened … I just don't know the specifics. I know that someone in the organization was able to effectively persuade the executives to invest in the environments, tools, build engineers and basic test automation to implement the building blocks of CI. I also know that almost certainly the major players in the development groups signed on and said that they would appreciate the new CI environment and they would use it and support it. Finally, I am sure that despite their assurances that the development community wasn't prepared to make the cultural and development practice changes needed to really support and embrace CI. It is also highly likely that the test organization was not prepared to retool their team to focus predominantly on test automation.
What turns this sad story into a Greek Tragedy is that the group that will benefit most from effective Continuous Integration is the development group. The is exactly the group that is mostly likely causing this Epic Failure of CI adoption.
In their book Leading the Transformation: Applying Agile and DevOps Principles at Scale, Gary Gruver and Tommy Mouser remind us that the most important and critically limited resource any organization has is its ability to absorb and embrace change. They point out that this is the critical resource needed for effectively implementing Continuous Integration, Continuous Deployment and DevOps at scale.
Let's start with the development organization. The practice and methodologic changes needed in the development group are so pervasive that they require a shift in the culture of the development teams. Changing the culture means that managers need to take on this challenge so that they can impact all of the little elements that influence and are part of the culture.
The development group must change particular development practices and methods, in order to leverage the benefits of CI. They have to learn a different style of development based on the guiding principles:
At first the development group will likely moan and complain that this approach to development (particularly the requirements to make code always releasable and making test automation part of the development job) involves more work by developers and is less efficient. They will argue that the straightforward design-code-test pattern for developing features (or systems) is much more efficient. On the surface, they are absolutely correct. However, when they experience the whole life-cycle guided by these principles, then they will realize that this new approach is actually much faster and more efficient overall. This new style of development paired with Continuous Integration will make the entire cycle of getting changes to the product fully integrated much more efficient by eliminating the endless rework cycles.
In a similar manner the culture of the "test group" needs to shift. First of all the line between development and test will erode and fade. That line is significantly blurred when we require that test automation be a part of every piece of development. This means that test automation is not a separate activity or an after-thought. Rather the test automation is built in to the entire process. The best example of this is Acceptance Test Driven Development.
While there is still a place for manual and exploratory testing of the system, the primary focus of the people skilled in testing shifts to test automation. The tester's role shifts from designing and executing comprehensive test scenarios to designing and implementing small automated tests that are:
This situation was a tragedy. Sadly we have seen this tragedy play out several times. However, the bigger problem that we see is that organizations are not willing to move towards Continuous Integration. I addressed why people are reluctant in my blog "Why Continuous Integration is like Broccoli" and I addressed how to help your executives recognize the tremendous benefits they can get from CI in "The Secret to Implementing Continuous Integration - CHEESE!"
Please do your part in making the best possible use of the limited budgets that we often face to implement Continuous Integration. You can do this, by making sure that your organization understands and faces the cultural shifts needed for CI. Please write and let me know your story for implementing Continuous Integration.