By failing to prepare you’re preparing to fail

The repetition problem

Some time ago I had a chance to witness the truth of this universally applicable sentence on a set of integration tests. Our integration tests were taking definitely longer than we expected them to and it was hard to pin to down the exact few cases, which were driving the performance of the whole test set down.

After looking at the tests, I noticed some suspicious lines of code in the test setup (we were using SpecFlow):

If you’re familiar with SpecFlow, maybe you've already noticed the issue. There’s a common rule taught to all young programmers: if your program is going to use a calculated value multiple times, be sure to calculate it just once.

If we translate the previous example from SpecFlow to nUnit Framework, it would look similar to that:

This code does the all the initialization before every test being run, multiplying the work by the number of tests. However, being hidden behind a neat attribute, it’s easy to be overseen.

Know your framework

Every test framework uses its own ways of making all the bits come together, but the usual structure of a testing framework supports at least the following test setup hierarchy:

The difference can be insignificant for a small number of tests, but with a growing system and a growing size of test coverage, it can become more apparent with time.

To avoid this problem, you should think about this structure when writing your tests.

C# Framework comparison

Framework Scope	MsTest	nUnit	xUnit	SpecFlow
Per Test	TestInitialize	SetUp	Ctor()	BeforeScenario
Per Class	ClassInitialize	OneTimeSetUp	IClassFixture	BeforeFeature1)
Per Domain/ Namespace	-	SetUpFixture3)	ICollectionFeature2)	BeforeFeature1)
Per Assembly	AssemblyInitialize	SetUpFixture3)	ICollectionFeature2)	BeforeTestRun

^{1)SpecFlow offers initialization ‘per Feature’ ensuring that all the tests in a specific domain are initialized once, together}

^{2)xUnit offers a generic interface, which in combination with attributes Collection and CollectionDefinition allows for multiple tests to have shared context}

^{3)nUnit offers a SetupFixture attribute that allows a one time setup for unit tests in a given namespace}

A short comparison of functionality offered by those frameworks can be found in the git repository. The solution in the repo shows how you can initialize your tests using different frameworks.

Be sure to enable ‘Redirect all Output Test to the Immediate Window’ to see when each method is called in the immediate window.

Know your borders

Precisely defining not only scope for your tests (what method do I test?), but also for groups of tests  (what feature do I test?) can help you set up a concise architecture with general initialization on top (for the whole assembly) and getting fluently more specific, as the scope is being reduced.

By grouping the tests thematically you should be able to extract at least part of the initialization to the higher levels.

• by module/class for unit tests, 
• by domains/subdomains for integration tests

Even if you follow a well-designed test architecture, especially the higher-level tests (feature tests, integration tests) tend to have additional dependencies to other modules, which are only needed in some cases. 
In this case injecting a static dependency for needed tests could solve the issue. Consider creating a static context, which will be initialized only once per whole assembly and passing it to the needed tests. Another solution would be grouping tests in a module according to the external dependencies and initializing the context once per group.

Keep ‘em separated

There are two things to keep in mind when generalizing test setup: test isolation and domain consistency. Generalization of test setup shouldn’t be done, if it is a risk to tests’ isolation. Keeping tests independent from each other should always be priority over speed gains.

Similarly, if an object can be reused by two tests which cover totally different subdomain of the product, it’s better to initialize the class twice (if the definitions of domain and subdomain are unclear, I highly encourage you to get familiar with Domain Driven Development principles). Still, each of two objects can be reused by multiple tests covering functionalitz in same subdomain.

Once and for all

Like well written code is designed to reuse once written methods, well written architecture should reuse already calculated results. It’s often easy to spot efficiency problems in the production code – usually thanks to the voice of dissatisfied users.

Such problems in test coverage can easily go unnoticed, causing slow decay of the whole test coverage. Slow tests are being run more rarely, which lowers their usefulness to the developers’ team. If you want to maximize gains from your test set, it’s vital to keep it fast and well organized. Taking advantage of what your test framework offers and using configuration options for test initialization is one of the important things to keep in mind while designing your test architecture.