What It Takes to Do Large Scale Technical Agile

Agile requires attention to quality code.  When multiple teams are involved in building technical products the challenges become even greater.  And, if you are working on seriously complex problems (such as virtual operating systems), test-first is really essential.

To do Agile well while building technical products, teams must do the following:

  • Use test-first methods
  • Do emergent design
  • Break work down into small stories
  • Do continuous integration
  • Use shared backlogs
  • Use proper release planning

We’ll go through each of these topics to illustrate why they are so essential.

Test-First

Using Test-first methods means to define your tests prior to writing code.  There are two levels of test-first:

  • Unit Test-driven Development (UTDD)
  • Acceptance Test-Driven Development (ATDD)

Unit Test-Driven Development

This is the classic TDD.  When Agile just existed at the team level unit and acceptance test driven development were done virtually concurrently and the distinction was not as important as it is now.  Unit TDD means that before methods are written, tests for the methods are defined and implemented.  This is the famous red-to-green transition. Define the test, write it, fail the test (red), write the code, pass the test (green).  The importance of this is significant. It:

  • requires validating what is to be built prior to building it
  • requires designing from a behavioral perspective (the first mandate of design patterns)
  • requires creating a framework for automated testing
  • encourages using small stories
  • promotes discipline

Acceptance Test-Driven Development

Almost the worst thing a team can do is build something it doesn’t need. Not only does this waste time, it adds to the complexity of the system increasing future time to develop useful functionality while increasing maintenance costs.  This, unfortunately, happens all too often – mostly due to assumptions made or the wrong people making decisions. ATDD requires product owners, developers and testers to jointly create requirements by defining acceptance tests for them.  This joint conversation identifies and eliminates assumptions and creates clarity of what is to be built prior to writing code.  Serendipitously, ATDD does not actually take more time than normal methods, rather it takes better timing of when the work takes place.

Do Emergent Design

Emergent design means to avoid a big upfront design but instead have the design evolve (emerge) as the application is being built. At Net Objectives we’ve been teaching emergent design by integrating the thought process of Design Patterns with the test-first approach of UTDD and ATDD.  Design patterns are often described as “solutions to recurring problems within a context.” However, a deeper understanding of patterns is that they provide a design approach that can mitigate risk and enable designs to emerge with minimal refactoring of design. By integrating the small steps of TDD with the thought process of quality design as espoused by patterns, emergent design enables quick validation of both requirements and the code used to achieve it.

Break work down into small stories

Many developers of technical products don’t believe you can break their work down into small steps.  Fortunately you can, and even better, it doesn’t take too much to learn how.  This is an essential skill in that it is one of the key enablers of emergent design.  It is often suggested that stories be written in the form:

“As a <user role>, I want <goal> so that <value>”

This is a good first start.   However, to continue splitting stories up into smaller pieces, a better format is:

“Given a <initial state>, and <event occurs>, then <asserted final state>”

By making the initial state more and more constraining, successively smaller stories can be written.

Do continuous integration

There are two common myths in software development:

  1. Developers spend a lot of time fixing bugs
  2. Integration errors are frequent and unavoidable

Developers actually don’t spend a lot of time fixing bugs. They spend a lot of time finding bugs.  This is not merely semantics.  It will take much longer for a developer to “fix” a bug if it is detected days after it is written than if it is detected immediately.  In both cases, the “fix” is the same.  But in the case where a lot of time passes, the finding and deciding on the fix takes much longer. This is one other reason automated testing is so essential.  When errors are made, the quick detection of them eliminates most of the work in finding them since it is easier for the developer to remember what he just did.

In the case of integration errors, they are actually rare.  Virtually all “integration errors” are detecting errors in the agreed upon communication of the different components being integrated.  They represent either communication errors or errors that arose when one team didn’t implement the agreement properly. By continuously validating that work spread across teams actually works across teams, much of the extra work one typically finds in building software is eliminated. 

Continuous integration can eliminate the finding of bugs and the thrashing that occurs during integration that most developers spend most of their time on.

Use shared backlogs

Using one backlog to drive several teams keeps the work of the teams in synch with each other.  Delays between making errors and detecting errors causes lots of challenges in finding and fixing the errors.  Shared backlogs ensures teams will be working on related features at nearly the same time. This helps eliminate integration errors and thrashing.

Use proper release planning

Proper release planning can help break the overall problem up into small chunks – a prime method of reducing simplicity.  Consider this, say your team had to build an application that had 50 features and you initially estimated it as 6 months of work.   Now, let’s say that before beginning work you realized you needed to add another 50% functionality.  Would you guess it would take just another 50% of time (e.g., 3 months) or would you guess that it would take more than that?  Most people would guess the later because complexity of code does not increase linearly with the amount of functionality but exponentially with it.  Thus, as we add features, the percentage increase in time will likely go up as a higher percentage that the % increase in functionality required.

The converse is also true.  If we remove 1/3 of the functionality from a system’s scope before starting, we can expect to remove more than 1/3 of the work.  This implies that an effective method of managing product development is to build core functionality in stages while attending to the extensibility of these core stages.  This has the advantage of also having a releasable product sooner and reduces the risk of running out of time without anything to release.

In Summary

Developing technical products with Agile methods is typically more difficult than building well-defined business applications. Technical products require different risk mitigation efforts and certain technical Agile skills.  The return on investment for these are well worth it, however, both near term and long term.  Near term because products can be delivered quickly, long term because technical debt can be managed allowing for future Agile development.