Layers in Tests

June 18, 2010 — Posted by Ken Pugh

Dale Emery wrote an excellent article on Writing Maintainable Automated Acceptance Tests. It's at http://dhemery.com/pdf/writing_maintainable_automated_acceptance_tests.pdf. He showed how to do the testing in Robot. Bob Martin gave an alternative way to do the tests in Fit at http://fit.c2.com/wiki.cgi?CommandLineFixture.

I'd like to show another way to design the tests.

The Problem

Dale's original problem statement was:

Suppose we are testing a system's account creation feature. The create command creates a new account, given a user name and a password. One of the account creation feature's responsibilities is to validate passwords. That is, it must accept valid passwords and reject invalid ones. To be valid, a password must be from 6 to 16 characters long and include at least one letter, at least one digit, and at least one punctuation character. If the submitted password is valid, the create command creates the account and reports Account Created. If the password is invalid, the create command refrains from creating the account and reports Invalid Password.

In short:

Create a new account given a user name and a valid password.

My Approach

Here's my approach.

I am tool agnostic. I do favor tables to specify examples. They can be clearer and easier to read by the customers who provide the requirements. The tables could be implemented as tests by any number of tools. Table driven requirements and tests help reduce ambiguity. For example, I'm not sure what is exactly meant by punctuation characters.

Why is account creation given the responsibility to validate passwords? A password is a data type that should contain its own validation. The create command should create an account given a user name and a valid password. It should not create the account and report an error if given an invalid password.

I like Bob's approach. But I often think in the opposite direction, from the smallest concept to larger ones. You can be a top-downer or a bottom-upper or a bit of both. Some people like me work both ways. Sometimes I work from the large to the small, other times in the opposite direction.

As soon as I identify a component that has a well-defined contract, I make up tests for that contract. This particular requirement stated the validation rules for a password as part of the story. So working those rules first seems easier.

My first question is what are punctuation characters? To clear this up, I made up a table. I've only put a few characters in this example, as to include the entire list would simply waste electrons. I'd check with the customer to make sure that I captured the entire set or maybe included too many. The question of what is a letter also arises. Is it a character recognized in ASCII or can it be one in a larger group, such as European language characters?

There are additional questions. Can you have a space in a password? Are all punctuation characters allowed? Should you filter out characters that could possibly inject SQL into a password or user name? These characters include a single quote, double quote, slash, back slash, and semi colon.

Password Valid Characters

   

Characters

Valid ?

Notes

0-9

Yes

Digit

a-z, A-Z

Yes

Letter

, &

Yes

List of punctuation characters should be supplied here

Anything else

No

 

This table (or a variation of it) could be used as the tests for a password character validation routine.

Now that the characters are valid, the same sort of tests that Dale and Bob had in their examples can be shown in a table. For example:

PasswordValidation

   

Password

Valid?

Notes

Abc0,

No

Too short

Abc0,f

Yes

Shortest allowed

Abc0,fghijklmnop

Yes

Longest allowed

Abc0,fghijklmnopq

No

Too long

A0,&a9

Yes

All valid characters (may be redundant test)

A0,&~9

No

Invalid character

A0B&a9

No

No punctuation

AB,,&&

No

No digit

,,&&09

No

No letter

If the valid characters changed, it's possible that some of these tests might have to change as well. Are we missing anything from password validation tests? Completeness tests might include tests to ensure that password consisting of only letters, only numbers, or only punctuation are invalid. The Notes column could be turned into a results column that gives the reason for why a password is invalid. The result could then be incorporated in an error message that is displayed to the user (as shown later for the create account test).

The password validation is reusable. The tests can be re-used for creating passwords for any application if the passwords are to obey the same business rules.

Now the next rule is that you can only create an account if you have a valid password. We need only show two tests – one with a valid password and one without. That shows the intent. Does this test have to change if the business rules for password changes? Only if the test breaks so that one password becomes either valid or invalid. Does that mean there is duplication? Yes. But is the duplication acceptable if it makes it easier for the customer to understand the tests? That's a question for the project team (customer, tester, developer) to answer. Tests are not just for checking that the code works; they are for communication between the three roles.

Create Account

     

User Name

Password

Account Created?

Notes

Fred

A0,&a9

Yes

 

Fred

A0,&~9

No

Invalid Password

If the customer understood the concept of variables, then you could have a table that set up variables for later reuse in other tests. This would make the other tests less coupled to the details of an invalid or valid password. For example,

Password Variables

 

Name

Value

ValidPassword=

A0,&a9

InvalidPassword =

A0,&~9

Create Account

     

User Name

Password

Account Created?

Notes

Fred

=ValidPassword

Yes

 

Fred

=InvalidPassword

No

Invalid Password

Now one measure for tests is how much they have to change if an additional business rule is added. That can give an indication of coupling. Suppose that the customer specified the rules for a username as given in the following tables. First the character table:

Username Valid Characters

   

Characters

Valid?

Notes

0-9

Yes

Digit

a-z, A-Z

Yes

Letter

@

Yes

0 or 1 times

.

Yes

Cannot appear twice in a row. Cannot appear before or after @. Cannot appear as first or last character

Anything else

No

 

Now email address gurus may note that email addresses have more potential characters. There are also more rules (e.g. there must be at least one character before the @, and at least one dot (.) after the @. But let's keep the table simple. (A more complete version is appearing in my book on Acceptance Test Driven Development. You can see a preliminary copy at https://www.netobjectives.com/resources/books/lean-agile-acceptance-test-driven-development. You need to register.) The validation tests for a username include:

Username Validation

   

Usename

Valid?

Notes

a@b.com

Yes

 

.a@b.com

No

Starting period

a.@b.com

No

Period next to @

a@.b.com

No

Period next to @

a@b.com.

No

Ending period

a..a@b.com

No

Two periods in a row

&@b.com

No

Invalid character

A@@b.com

No

Two @

Now the test for creating an account becomes: If the original choice for a user name was invalid based on this new business rule, we would have had to change that value. Or we could create variables – ValidUser and InvalidUser

Create Account

     

User Name

Password

Account Created?

Reason?

Fred

A0,&a9

Yes

Account_Created

Fred

A0,&~9

No

Invalid_Password

.Fred

A0,&a9

No

Invalid_User _Name

We could have a fourth test that enters both an invalid user name and an invalid password for completeness. Is that test really necessary? Only if the customer and/or tester wants to see it or there is a different error message that should be produced.

One More Abstraction:

The result shown is the reason why an account was not created. The result does not have to be the message that is displayed to the user. You can abstract the result from the actual message that is displayed. This allows you to change the message without changing the business logic tests. You are only changing a textual element. To test the textual element, you could use a table as:

Reason Display

 

Reason

Display Message?

Invalid_Password

You have entered an invalid password. A valid password is …"

Invalid_User _Name

You have entered an invalid user name. A valid user name is …"

These tables do not drive how the underlying code is structured, other than to make password responsible for validating itself. The Password class might have a parse(String input) method that returns a Password or throws an exception. Or it might have a Password(String input) constructor that throws an exception. Or a constructor that does no checking and an isValid() method that indicates whether the corresponding Password is valid or not. The tests in the tables are decoupled from the implementation.

A Final Note

Since the customer has specified all these requirements (tests), then I call these customer acceptance tests. You may implement them in an acceptance test framework (Fit, Robot), or a xUnit test framework, but they are still acceptance tests.

Subscribe to our blog Net Objectives Thoughts Blog

Share this:

About the author | Ken Pugh

Ken Pugh was a fellow consultant with Net Objectives (www.netobjectives.com). He helps companies transform into lean-agility through training and coaching. His particular interests are in communication (particularly effectively communicating requirements), delivering business value, and using lean principles to deliver high quality quickly.



        

Blog Authors

Al Shalloway
Business, Operations, Process, Sales, Agile Design and Patterns, Personal Development, Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Cory Foy
Change Management, Innovation Games, Team Agility, Transitioning to Agile
Guy Beaver
Business and Strategy Development, Executive Management, Management, Operations, DevOps, Planning/Estimation, Change Management, Lean Implementation, Transitioning to Agile, Lean-Agile, Lean, SAFe, Kanban, Scrum
Israel Gat
Business and Strategy Development, DevOps, Lean Implementation, Agile, Lean, Kanban, Scrum
Jim Trott
Business and Strategy Development, Analysis and Design Methods, Change Management, Knowledge Management, Lean Implementation, Team Agility, Transitioning to Agile, Workflow, Technical Writing, Certifications, Coaching, Mentoring, Online Training, Professional Development, Agile, Lean-Agile, SAFe, Kanban
Ken Pugh
Agile Design and Patterns, Software Design, Design Patterns, C++, C#, Java, Technical Writing, TDD, ATDD, Certifications, Coaching, Mentoring, Professional Development, Agile, Lean-Agile, Lean, SAFe, Kanban, Kanban Method, Scrum, Scrumban, XP
Marc Danziger
Business and Strategy Development, Change Management, Team Agility, Online Communities, Promotional Initiatives, Sales and Marketing Collateral
Max Guernsey
Analysis and Design Methods, Planning/Estimation, Database Agility, Design Patterns, TDD, TDD Databases, ATDD, Lean-Agile, Scrum
Scott Bain
Analysis and Design Methods, Agile Design and Patterns, Software Design, Design Patterns, Technical Writing, TDD, Coaching, Mentoring, Online Training, Professional Development, Agile
Steve Thomas
Business and Strategy Development, Change Management, Lean Implementation, Team Agility, Transitioning to Agile
Tom Grant
Business and Strategy Development, Executive Management, Management, DevOps, Analyst, Analysis and Design Methods, Planning/Estimation, Innovation Games, Lean Implementation, Agile, Lean-Agile, Lean, Kanban