Manual Testing and Acceptance Test Automation

June 1, 2017 — Posted by Scott Bain

Acceptance tests should always be considered "executable specifications" of the system. This is equally true whether that "executable" part is an automated aspect of your process or not.

Indeed many organizations find that in their initial adoption of acceptance testing, attempting to add automation at the same time is simply too large a bite to take. They find that the biggest "bang for the buck" is to start by ensuring that everyone knows how to write, read, change, and understand acceptance tests, and that attending to their automation is a secondary concern.

In fact, it might be a mistake to leap to automation too quickly. Automating ATDD is generally speaking a tools issue, and different tools have different strengths and weaknesses in terms of the way they automate various forms of specification. Any tool can be used for any expression, but some are a more natural fit than others.

All specification is about behavior, and rooted in the concepts "Given, When, Then", but that does not mean that the best expression of a behavior is always clearest using this language.

For example, do you see a Given, When, Then here?

Benzine


No? Me neither. But for a group of chemists this might be much clearer than expressing a behavior in English words. Why force them to write their specifications in Gherkin or a Fit table if that's always going to stand in their way? We would always want the automation technique/tool/strategy to follow the most valuable expression of the behavior, once we know what that is. We may even have to create some custom tools if that makes the most sense.

Thus, for many the initial task is to figure that out. The first few attempts may be wrong-headed or lack sufficient specificity, or may take some time to get done. With something like acceptance testing, which ideally should be a collaborative process involving many different parts of the organization (developers, testers, project managers, product owners, subject matter experts, etc...) it may take quite a while to get everyone trained and comfortable with the chosen expression. This is important. There is no point in automating the wrong thing.

Because of this, we may well prefer to start with the "automation" part of ATDD to be a job for manual testers. Humans can form the glue between the expression of a behavior of the system and the system's execution, just as our tradition would have it. We'd expect that a manual tester, an intelligent and experienced person, would be able to take any reasonable test delineation and make sense of it, even if we have not yet figured out its best, most clear form.

Furthermore, there may well be some aspects of a system that will always have to be tested manually. Is this "pleasant" to use? Does this "make sense" to a person of a specific culture? Is this shade of blue "different enough" from that one to prevent misunderstandings? Some things are pretty tough for a computer to do for us.

Does this mean manual testing can not be part of the coordination of an automated ATDD process either before we've settled on it, or once we have it established? Not all all.

Before you have any automation, the obvious thing you can do is to simply have testers read the acceptance tests and execute them as they always have. Once you have some automation in place, however, you can still bring them along into the effort.

Because my blog series has been using SpecFlow, I'm going to use it here in my example. Please keep in mind that this is an arbitrary choice for our purposes. I could have picked anything and my point would be exactly the same.

One bit of convenience for us, here, is that when you create a new "feature file" with this particular tool, it provides an example Given, When, Then, just to remind you of the particular syntax details the tool expects (in this case, Gherkin). That's makes things easy for me in showing an example, and for you if you want to play with this.

Feature: Calculator
    In order to avoid silly mistakes
    As a math idiot
    I want to be told the sum of two numbers

@mytag
Scenario: Add two numbers
    Given I have entered 50 into the calculator
    And I have entered 70 into the calculator
    When I press add
    Then the result should be 120 on the screen

I am not saying this would be particularly difficult to automate, I just want to show how easy it is to dovetail this specification, even though we are using an automation tool, with a manual testing effort. If, for example, the "calculator" in question is, at this point, an actual physical device that we plan to replace, later, with a software component.

Remember that an acceptance test is really two things: the expression of the behavior (shown above) and the "glue" or "binding" of this to the system. If I know a person is going actually conduct the test, we can still create a binding for them.

[Binding]
public class CalculatorSteps{
    [Given("(.*)")]
    public void GivenAnything(string toBeEnsured) {
        MakeThisTrue(toBeEnsured);
    }

    [When("(.*)")]
    public void WhenAnything(string toDo) {
        DoThis(toDo);
    }

    [Then("(.*)")]
    public void ThenAnything(string toCheck) {
        ThisShouldBeTrue(toCheck);
    }
}

This binding would match any simple Given, When, and Then statements, no matter what they were about. Again, we are intending for a person to execute the actual behavior of the system, so we would expect them to act intelligently. We can assume they know how to find and turn on the calculator, enter numbers into it, and so forth. The helper methods "GivenAnything()" etc would be implemented to pass the information on to the tester in any way we find convenient. A message box on the screen might be easiest, for example:

private static void MakeThisTrue(string toBeEnsured){
    MessageBox.Show(toBeEnsured, "Setup:", MessageBoxButtons.OK);
}

private static void DoThis(string toDo){
    MessageBox.Show(toDo, "Trigger:", MessageBoxButtons.OK);
}

private static void ThisShouldBeTrue(string toCheck){
    var result = MessageBox.Show(toCheck, "Verify: Is This True?",
                 MessageBoxButtons.YesNo);
    Assert.AreEqual(DialogResult.Yes, result);
}

Again, this is only an example. The separation here is the key. If I change the way I am expressing the behavior, I could still use this binding to organize the manual tester's actions. If I want to switch from manual to automated tests, or from the physical calculator to the system component, then I only change the binding and not the behavioral expression. Adding is adding, after all.

So, if your organization or team is considering adopting acceptance testing as part of your process, remember that you can take it on a bit at a time. There is no need to become overwhelmed or to feel that this is "too much change". You can incrementally integrate ATDD into your existing processes and take on each aspect as you see the value and achieve the comfort level you need.

The most important part of ATDD is doing it, achieving the collaboration that ensures the entire organization is aligned around real business value, that we have a clear and shared understanding of what that value is. Everything else must follow.

Subscribe to our blog Net Objectives Thoughts Blog

Share this:

About the author | Scott Bain

Scott Bain is an consultant, trainer, and author who specializes in Test-Driven Development, Design Patterns, and Emergent Design.



        

Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Guy Beaver
Business and Strategy Development, Executive Management, Management, Operations, DevOps, Planning/Estimation, Change Management, Lean Implementation, Transitioning to Agile, Lean-Agile, Lean, SAFe, Kanban, Scrum
Israel Gat
Business and Strategy Development, DevOps, Lean Implementation, Agile, Lean, Kanban, Scrum
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, SAFe, Kanban
Ken Pugh
Agile Design and Patterns, Software Design, Design Patterns, C++, C#, Java, Technical Writing, TDD, ATDD, Certifications, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Marc Danziger
Business and Strategy Development, Change Management, Team Agility, Online Communities, Promotional Initiatives, Sales and Marketing Collateral
Max Guernsey
Analysis and Design Methods, Planning/Estimation, Database Agility, Design Patterns, TDD, TDD Databases, ATDD, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile
Steve Thomas
Business and Strategy Development, Change Management, Lean Implementation, Team Agility, Transitioning to Agile
Tom Grant
Business and Strategy Development, Executive Management, Management, DevOps, Analyst, Analysis and Design Methods, Planning/Estimation, Innovation Games, Lean Implementation, Agile, Lean-Agile, Lean, Kanban