Rasmus Møller Selsmark

On software test and test automation

Book review: Lean from the Trenches - Managing Large-Scale Projects with Kanban

clock January 4, 2013 07:33 by author rasmus

Last time I read a “The Pragmatic Programmer” series book was “Ship It!: A Practical Guide to Successful Software Projects” some years back, which I remember as being a well-written practical oriented book on developing software. I spent some time in the Christmas holiday reading “Lean from the Trenches: Managing Large-Scale Projects with Kanban” which is also a short, extremely well-written but still comprehensive book describing the authors experiences from building a nation-wide software system for the Swedish police.

Being a tester, I especially found the sections on how they did test and quality assurance on the project interesting. Along with my earlier experiences, this book confirms me that test is becoming an integrated part of software development, and that the role of the tester in future is moving towards more quality assurance, rather than primarily having a focus on executing tests. I still believe there is a future for execution of exploratory tests in future, but scripted tests seems in many cases to be best handled by the developers, even functional tests.

Minimum Viable Product or “How to slice the elephant”

The book starts out on page 6 by describing how they divided the system into small deliverables, in this case

  • Start by deploying the system to one small region
  • First version only supports a small amount of case types, rest is handled the old way
  • Having the customer in-house

A few simple rules, but as the rest of the book shows, they play a very important role in order to ensure a successful project.

Cause-Effect Diagrams

First part of the book describes the project and processes on a more overall level, whereas the last third of the book goes into more details on the techniques used by the team. One of the chapters in this section contains an introduction on getting started with Cause-Effect Diagrams/Analysis, for which the book describes the following basic process (page 134):

1. Select a problem – anything that’s bothering you – and write it down
2. Trace “upward” to figure out the business consequences , the “visible damage” that your problem is causing
3. Trace “downward” to find the root cause (or causes)
4. Identify and highlight vicious cycles (circular paths)
5. Iterate these steps a few times to refine and clarify your diagram
6. Decide which root causes to address and how (that is, which countermeasures to implement)

One of the examples from the book is “Defects Released to Production” from which the problem “Angry Customers” is identified as well as “Lack of Tools and Training in Test Automation” as a root Cause, as shown in the following example:


Taken from page 139 although the diagram in the book is more comprehensive, e.g. by identifying multiple root causes. In relation to Cause-Effect Analysis, the book says on page 52 that

“Bugs in your product are a symptom of bugs in your process

Which is another way to phrase the well-known statement that you cannot test quality into a product, but needs to be an integrated part of the software development process.

From a testing perspective, this book contains a lot of useful information, e.g. chapter 9 “Handling Bugs” visualizes how “continuous testing” helps minimizing the total time spent on testing and bug-fixing, by having bugs fixed as early as possible. It’s been a long time since I’ve met a team only doing test at the end of a release cycle, but still the following figures shows how we can benefit from having “continuous testing”, in order to reduce total cost of developing software.

Traditional waterfall approach:


Using continuous testing:


Even though we spend more time on testing, it pays off by lowering the time spent on bugfixing:


Limit to 30 active bugs

Another interesting idea from this chapter is to set a WIP (Work in Progress) limit on 30 bugs. Bugs that don’t make it into the top-30, are automatically marked as “deferred” immediately and won’t be fixed. If a bug is found important, one of the existing 30 bugs is removed from the list. This is an effective way of limiting the number of bugs, which I think also helps on keeping focus on fixing bugs immediately (even without necessarily reporting them), since you don’t have the option of just piling the bug along the existing probably several hundreds of existing reported bugs. You easily get “bug-blind” when you have reported more than 50-100 bugs in your system, and setting a hard limit to e.g. 30 seems to be a good way to mitigate this issue.

Reducing the test automation backlog

Chapter 18 is dedicated to describing how to handle lack of test automation for a legacy product, where the advice is to have the whole team increase the test automation a little each team. It’s important to notice that it should be the whole team, and not e.g. a separate team, as the whole team should learn and be responsible for writing tests. For identifying which test cases to automate first, the book prioritizes the tests by

  • How risky the test case is, i.e. will it keep you awake if this test is not executed?
  • Cost of executing test manually
  • Cost of automating test case

Beside what I’ve described in this review, the book also contains valuable tips on how to organize your scrum boards, standups etc.

In short this is a really good book, which is furthermore quickly read, so I will recommend it to everyone participating in software development projects.

Experiences of Test Automation

clock August 9, 2012 21:56 by author rasmus

Finally finished reading the book Experiences of Test Automation by Dorothy Graham and others. The book is a kind of “essay collection” of case studies, with people describing their approaches and experiences in implementing test automation for various kinds of software systems.

Being composed of different stories, of course there will be some stories relevant and some less relevant. I found the following chapters to be the most interesting ones:

  • Chapter 9 – Model-Based Test-Case Generation in ESA Projects
    It was actually not the model-based content of this chapter I found most interesting, rather its focus on calculating ROI for test automation projects, and when to expect break-even. Always a good idea to include ROI when doing test automation, in order to set stakeholder expectations.

    One of the charts showing break-even for an automation project after 4 test repetitions/cycles, looks like this:

  • Chapter 10 – Ten years and still going
    This was also a case study in their previous book Software Test Automation from 1999. The chapter includes the original case study and then ends with a short conclusion on the project status now. Must say I’m impressed to hear that they now run 5 million automated tests each month.

  • Chapter 15 – Test Automation of SAP Business Process and 16 – Test Automation of a SAP Implementation
    These two chapters gives a good insight into how SAP have automation as a feature of their product (eCATT), i.e. allowing partners to use the built-in automation features to test customizations. Found this especially relevant, as I’m working on making our test automation, running on the standard version of our product, usable for our consultants on their customization projects. I certainly see automation as a feature of a product, and not just being executed by the testers.

    And then I strongly agree with the first statement in the chapter on page 278: “Test automation is nothing but software development” followed by “(…) any test automation project that did not follow software development processes and standards failed or ended up with huge maintenance efforts.”.

  • Chapter 21 – Automation through the Back Door (by Supporting Manual testing)
    With so many different sources for the book, all chapters of course doesn’t have the same level of technical depth. This chapter though contained the following diagram on their Keyword-driven framework implementation. I've not specifically worked with keyword-driven testing, nevertheless I think the diagram gives a good overview of the different components of an automated test case, i.e. test logic/code, input for test case and a test execution engine.

    And then I like the concept of focusing on increasing efficiency of manual test through automation. Scripted test cases tends to find relatively few bugs, so freeing manual testers to do more exploratory testing, by automating any scripted manual test case like BVT, regression test etc., is almost certain to increase the effectiveness of your manual test

  • Chapter 27 – BHAGs, Change, and Test Transformation
    What I find interesting in this chapter, is the following table at bottom of page 503, containing metrics of which test cases that finds bugs from their project:

    Automation Defects Manual Scripted Defects Exploratory Test Defects Fix Verification Defects
    9.3% 24.0% 58.2% 8.4%

    Also matches my experience well, that you shouldn’t expect to find many bugs with automated test cases, but it can free resources to find more bugs using other approaches. The automated tests still brings a lot of value value to a project this way, even if they don't find many bugs.

  • Chapter 28 – Exploratory Test Automation: An Example Ahead of Its Time
    Final chapter is by Harry Robinson, who has pioneered model based testing. Besides from being an introduction into model based testing, the chapter also describes they managed to improve quality of the product, by working close with developers and adjusting models based on changes to requirements. The system test ran for 8 months after release, before the first (low-severity) bug was found. Very impressive indeed, and I found this chapter really interesting to read.

In general, it’s an ok book. As the book presents a wide range of different solutions to test automation projects, and in general lacks technical details of the implementations, I wouldn’t recommend the book to someone new to develop test automation, from a technical point of view. But from a more organizational/theoretical point, this book is relevant, with it’s general conclusions on

  • Test automation is software development, and to increase the probability of being successful you need to follow same rules as for any other software development project
  • You need management support to be successful with test automation
  • Test automation is a long-term investment. You should estimate ROI before starting, in order to set stakeholder expectations
  • Define an architecture for your test automation code.