Agile Unit Testing

This paper explains one of the prime goals of Cairngorm; which is to allow developers to follow agile testing of Flex RIAs. Agile testing combines engineering with testing activities into one activity and can be expressed by following Test Driven Development where a developer would write tests first. In particular, this paper focuses on unit testing where units represent objects or single architectural layers in contrast to functional testing (customer tests) where a unit represents the complete application (many architectural layers). Agile functional testing is also vitally important to Flex project but the focus of a separate paper.

Why to Isolate - Focus of Unit Tests

Most objects of a system depend on other objects (depended-on objects). Objects contain other objects and delegate work to them. When unit testing one object the decision is up to the developer to touch the depended-on object with the same unit test or substitute the depended-on object with an alternative object (test double).

The motivations to substitute (isolate) depended-on objects with test double are manyfold and have to be contrasted with the motivations against it.

Motivations for isolation

Motivations for isolation (concrete depended-on objects are substituted with test doubles) are:

• Simulate hard to test behaviour
  If the object under test contains behaviour that is hard to simulate in a unit testing environment (i.e. specific error conditions, or asynchronous events), a substituted test double could simulate it and could allow more test scenarios and faster test runs.
• Granular defect localization
  If an implementation changes and a test flags up an unexpected result, it is easy to localize the defect if the object under test is isolated from it's environment.
• No triggering of unwanted behaviour
  Without any isolation, the behaviour of the object under test might trigger behaviour on the depended-on object that cause other unexpected effects in unit tests. With substituting the depended-on object, the behaviour the object under test triggers is controlled.
• Less setup code to satisfy dependencies of depended-on objects
  Without any isolation, depended-on objects might require further dependencies in order to execute without errors. These dependencies would need to be satisfied by the unit test, increasing the knowledge of the unit test unnecessarily. Would the object under test use test doubles instead of concrete depended-on objects, the test doubles can be designed not to require further dependencies, reducing setup code and making the test easier to read and maintain.
• Higher coverage through testing interactions
  In order to test how an object under test interacts with depended-on objects, the unit test needs access to the depended-on objects. Test doubles can be designed to offer flexibility on how unit tests can observe the interaction with their object under test (mocks). Depended-on objects are often not designed for it, and developers are in danger of sacrificing encapsulation of a concrete implementation of a depended-on object.
• Code coverage numbers are more significant
  When concrete depended-on objects are substituted, they don't get coverage through unit tests that focus on other objects. Code coverage tools can easier identify the need for additional tests of the concrete depended-on objects when no code coverage on concrete depended-on objects has been registered via other unit tests.

Motivations against isolation

When concrete depended-on objects are executed by the object under test, then isolation is not performed and the motivations against isolation can be:

• Over-specified unit tests hinder refactorings
  If all interactions with depended-on objects are tested, refactorings on objects under test that change any of the interactions also need to change the unit tests and this increases the overall effort of refactorings. To counter this danger, unit tests should only focus to test one piece of behaviour and not repeat tests of other tests and not test interactions that are not significant.
• More difficult to write and read unit tests
  If the object under test has to create test doubles for all it's depended-on objects, it increases the size of the unit test and makes it more difficult to read. However, automatic test double creation libraries such as ASMock ease the time it takes to write test doubles.
• Higher code coverage with fewer tests
  While objects under test might not be able to and want to test all interactions with depended-on objects, they automatically cover some behaviour of depended-on objects and can achieve a higher code coverage with fewer tests. However, the higher code coverage through unit tests that triggered code of depended-on objects might lead to a false sense of security as the focus of the unit test was not on the depended-on object but on the object under test.
• Danger of substituting behaviour to test
  When isolating too much developers might accidently substitute objects they actually would want to test and the risk of integration increases. Watch code coverage results to ensure behaviour continuous to be tested.

The more depended-on objects the unit test touches, the more it steers testing efforts away from unit testing towards functional testing. At some point, different tooling than unit testing frameworks, additional qualified personnel such as Quality Engineers and separate test runners are more appropriate for complete functional and black box testing (See Agile Functional Testing).

When to Isolate - Isolation by Architectural Layer

As shown above, motivations to isolate objects exists as motivations against it exists. When should a developer decide to isolate an object and when not?

As a general rule of thumb, this paper recommends isolating at each client side architectural layer. Presentation layer, domain layer, application layer and integration layer. Often developers can isolate objects beyond that and prevent the dangers of over-specified tests that hinder refactorings with focusing each test on a new, small piece of behaviour that is not repeatedly covered by another test. Also, interactions do not always need to be tested to the full API including parameters. A test might gain enough realism by just observing one method call that identifies an interaction with a depended-on object instead of observing all method calls of similar type including full parameter sets. Mock frameworks in Flex today (i.e. ASMock and mock-as3) allow this flexibility to prevent having to specify every detail.

How to Isolate - Substitution Patterns and Test Doubles

Isolation can be achieved with substituting the depended-on object with an object that is controlled by the unit test suite; a test double. The test double takes the same interface as the production object but does have a different implementation, usually much simplified and targeted for a particular test case or suite.

Test doubles can come in many flavours. The most important test doubles are stubs and mocks. A stub directs inputs into the object under test. A mock object observes outputs of the object under test.

Substituting - Dependency Injection and Dependency Lookup

Two of the most popular approaches to get substituted behaviour into objects under test is dependency injection and dependency lookup. Dependency injection injects the test double via the API of the object under test. Dependency lookup retrieves the test double via another object. The other "locator" object should then provide some mechanism to configure it during unit tests with a test double.

Substituting - Overriding and Overwriting

Substitution by overriding can be achieved with subclassing the depended-on object with an object only used in tests that provide an alternative implementation for the behaviour not of interest in unit tests of the object under test.

Substitution by overwriting simply sets (overwrites) a depended-on object with a test double. However, this is only feasible if the depended-on object is exposed via an API of the object under test and typed as an interface or common base class.

Substituting - Test Hocks

A Test Hock is a control point (i.e. if-statement) within the object under test that decides to behave differently based on if it's being exercised during unit tests or during production. This can require a known object with global state to tell objects if they are in a unit tests or not. However, instead of hardcoding a conditional statement in production code, this could also be done using Flex's conditional compilation. In any case, this approach adds risk, size and complexity to production code that would not need to be there following the approaches above and should be used with caution.

How to Test - Applying Testing Knowledge

More information

Unit Test Organization

While there are many solutions to how to organize unit tests, Cairngorm attempts to provide a convention to improve consistency between projects.

Same Project

Prefer to keep test code in the same project as source code, separated by source paths.

Same Package

Application code and test code are separated within the same project but use the same package. This is possible by putting them into different source paths. The application code remains in Flash Builder's standard "src" source path, while the test code is moved to the additional source path "test". This can also make searches for objects easier.

Test Case Naming

Prefer "Test" suffix instead of "Test" prefix. A suffix can ease type-ahead searches via Flash Builder as the test is immediately alongside the object under test. If a test class becomes too large, analyse if the design of the object under test can be improved. If not, multiple test classes can be created with the naming "Type_Feature_Test".

Test Method Naming

Self-documenting tests i.e. should or Given/When/Then naming.
http://dannorth.net/introducing-bdd

Summary - Test Infected Teams

Test-infected developers never write their tests days after their code. Test-infected developers want to write tests, because that's the way they think about software development. They don't want to think otherwise. Test-infected developers never have excuses not to test. They are never too busy to test, their environments never take up too much time to create test data, and their customer never complain that testing is too expensive because it takes too much time.

If it is difficult to create an environment of test infected developers,

• analyze the design of the application for testable code, prevent repetitive and over-specified tests, keep tests small, easy and close to the way objects are used in production.

• don't attempt to test every single line of code in your Flex application. Realize that functional tests have a role. Focus your unit tests to test behaviour and not structure and wiring.

• ensure libraries, frameworks, API projects, any other code that is shared by multiple developers achieve the highest coverage and quality of tests.

• keep quality of test code as high as application code. Maintain tests with refactorings but realize that it's sometimes easier to throw away and rewrite tests as it's sometimes easier to throw away and rewrite code.

References

Meszaros, G. (June 2007). xUnit test patterns: refactoring test code. Addison-Wesley.

Beck, K. (2002). Test Driven Development: By Example. Boston, MA: Addison-Wesley Longman Publishing Co., Inc.

Williams, P. (2009). Unit Testing User Interfaces. Retrieved May 05, 2009 from http://weblogs.macromedia.com/paulw/archives/2008/03/unit_testing_us.html

http://blog.james-carr.org/2006/11/03/tdd-anti-patterns/

Feathers, M. (June 2008). The Flawed Theory Behind Unit Testing. Retrieved September 07, 2008 from http://michaelfeathers.typepad.com/michael_feathers_blog/2008/06/the-flawed-theo.html