Together with our great team of test framework and automation developers at Unity, I spent last week on Test Driven Development (TDD) training with Roy Osherove, speaker at several conferences plus e.g. author of the book The Art Of Unit Testing. This post serves primarily as my personal notes from the training, although I of course hope others find it useful as well.

The topic of this post is defining properties/characteristics of good unit tests, however first we need to get an understanding of what a "unit" is.

Output from a unit (of work)

Roy defines a unit, i.e. the subject being tested by a unit test, as a "unit of work", rather than just a method or class. A unit of work accepts input/gets invoked, and has as output:

  • Return value / exception
  • Noticeable system state change
  • 3rd party call

For 3rd party calls, we implement fakes (mocks) which can help us verify that the expected calls actually were executed.

Test public API

Private members are always called in a context, which is important to include when testing our code from a unit test. If you have private methods containing functionality you would like to test separately, you can make them internal and test them using [InternalsVisibleTo] attribute in C#. However it makes sense to me, primarily focusing on testing the public API, since this first of all gives a well-defined layer of separation for what should be tested.

In relation to this, consider the database as "private" to the implementation, and therefore should not be tested explicitly. Including validation of values in the database, means that your tests suddenly spans multiple layers, which at some point will decrease maintainability of your tests, and most likely also violates e.g. http://en.wikipedia.org/wiki/Single_responsibility_principle. Any software development principle and guidelines for production code also holds true for test code. As mentioned earlier, your test code should have the same (hopefully) high quality as your production code, otherwise you won't trust your tests (more on this later).

Unit vs. Integration tests

In a unit test, you need to have control over all parts of the system under test. If not, it's an integration test.

Writing unit tests is an investment up front

For me, the primary reason for writing unit tests, is that it's an investment you make up front, to ensure a maintainable product of high quality on the longer run. As Roy added, "you never get/have time to write tests later in a project", so simply just do it right from the beginning. Roy presented one study of two teams working on similar project in same company, and while the TDD team released their feature a few days later than the non-TDD team (in a project of approx. one "team-month), they won on the long run by having 90% fewer bugs in their feature, resulting in fewer support-request and bugfixes. Given how TDD encourages teams e.g. to pair up and consider use cases before jumping into development, I can, based on my experience, easily see how they end up with a higher quality feature in the TDD team.

Unit test naming convention and structure

Suggested naming convention for tests: [UnitOfWork_StateUnderTest_ExpectedBehavior], e.g: Add_TwoNumbers_SumsThemUp, see also http://osherove.com/blog/2005/4/3/naming-standards-for-unit-tests.html

Writing readable/maintainable tests using "3A" pattern:

  • Arrange
  • Act (invoking the entry point)
  • Assert

An example of a unit test written this way (from our daily TDD Kata, which we started doing every morning for 30 mins during the week):

[TestCase(3, "1,2")]
[TestCase(6, "1\n2,3")]
public void Add_MultipleNumbers_ReturnsSum(int expected, string input)
{
	// Arrange
	var calc = GetDefaultStringCalculator();

	// Act
	int result = calc.Add(input);

	// Assert
	Assert.AreEqual(expected, result);
}

The comments above are simply included for the purpose of explaining the structure. You shouldn't have boilerplate comments like these in your code ever :)

Also note the use of a factory method above for instantiating the object being tested. This way your test code becomes more maintainable, e.g. if adding a constructor argument at a later point, you only have to add this one place.

Fakes: Mocks vs. stubs

Fake is the generic term for mocks and stubs, and is defined as "something that looks like something else, but isn't". General rule is that you assert against a mock, but only use stubs for providing assumptions for your tests.

If you are asserting against multiple mocks in a test, it's a smell that the test violates single responsibility principle, and is likely a candidate for splitting into separate tests.

Characteristics of good unit tests

While this training specifically was targeting writing unit tests, my personal observations tells me that the properties of good unit tests also is true for any other automated test, including integration and especially UI tests (which from my experience is the most unstable type of tests, why people easily loose faith in the results)

Trustworthy

  • If a test fails randomly people will start ignoring it. As soon as you have one red test, it's easy to let another red test slip in (http://blog.codinghorror.com/the-broken-window-theory)
  • Run tests as part of your automated build process, otherwise people won't write tests
  • Test smell: If a test fails, can you easily say why it fails
  • Start by writing a failing test, which gives you faith that the test will actually fail if the feature is broken at some point
  • Generally "high coverage + reviewed tests = good quality"

Maintainable

  • No logic inside a unit test, e.g. no loops, if statements etc.
  • Only test public APIs
  • Only test one thing per test. E.g. if asserting on multiple objects, you risk getting unexpected behaviors. If it's a matter of reducing amount of code, you can likely refactor your test code to contain e.g. a factory method for setting up classes used in the test.
  • Isolated / independent, i.e. each test has to be able to run separately
  • Deterministic, not depending on external factors
  • Don't use magic numbers, except e.g. "42", "user" or other values which obviously doesn't have any business meaning. Magic numbers (or strings) tend to "get children", meaning they get copied to other tests, and suddenly your tests are becoming polluted with numbers which you don't know why was originally introduced. Use well-named constants instead

Readable

  • Don't use magic numbers (see above)
  • Don't reuse production code for calculating an expected value. You risk duplicating a bug from production code, plus makes your test harder to read. Put simply, assert on the value "4" instead of "2+2". Also makes another reader of your tests, know what to expect
  • Follow the Arrange/Act/Assert pattern for structuring tests. Don't mix asserts, i.e. only have asserts at the end of your tests

Quotes from the training

During the week, Roy came up with several fun and/or inspiring statements, of which I managed to note the following. Hope they make sense without further explanation:

"All software problems are people problems"

"If you hate it, automate it!"

"Change where you work, or change where you work"

"A problem can be solved, a limitation has to be worked around"

"For each behavior, the world is perfectly designed for that behavior to happen"

And not least I will recommend this TDD training to every developer out there. Roy is a really good speaker and is able to explain the concepts of TDD to the audience, in a clear, concise and inspiring way. I knew writing tests is important and valuable, but this week actually showed me that TDD, done right, simply results in a much nicer, cleaner and higher quality code base. I caught myself having several "wow!" experiences during the week, from the beauty and simplicity of the code we did as part of the exercises.