Finally finished reading the book Experiences of Test Automation by Dorothy Graham and others. The book is a kind of “essay collection” of case studies, with people describing their approaches and experiences in implementing test automation for various kinds of software systems.

Being composed of different stories, of course there will be some stories relevant and some less relevant. I found the following chapters to be the most interesting ones:

• Chapter 9 – Model-Based Test-Case Generation in ESA Projects
  It was actually not the model-based content of this chapter I found most interesting, rather its focus on calculating ROI for test automation projects, and when to expect break-even. Always a good idea to include ROI when doing test automation, in order to set stakeholder expectations.
  
  One of the charts showing break-even for an automation project after 4 test repetitions/cycles, looks like this:
  
  
  
• Chapter 10 – Ten years and still going
  This was also a case study in their previous book Software Test Automation from 1999. The chapter includes the original case study and then ends with a short conclusion on the project status now. Must say I’m impressed to hear that they now run 5 million automated tests each month.
  
  
• Chapter 15 – Test Automation of SAP Business Process and 16 – Test Automation of a SAP Implementation
  These two chapters gives a good insight into how SAP have automation as a feature of their product (eCATT), i.e. allowing partners to use the built-in automation features to test customizations. Found this especially relevant, as I’m working on making our test automation, running on the standard version of our product, usable for our consultants on their customization projects. I certainly see automation as a feature of a product, and not just being executed by the testers.
  
  And then I strongly agree with the first statement in the chapter on page 278: “Test automation is nothing but software development” followed by “(…) any test automation project that did not follow software development processes and standards failed or ended up with huge maintenance efforts.”.
  
  
• Chapter 21 – Automation through the Back Door (by Supporting Manual testing)
  With so many different sources for the book, all chapters of course doesn’t have the same level of technical depth. This chapter though contained the following diagram on their Keyword-driven framework implementation. I've not specifically worked with keyword-driven testing, nevertheless I think the diagram gives a good overview of the different components of an automated test case, i.e. test logic/code, input for test case and a test execution engine.
  
  

  

  
  And then I like the concept of focusing on increasing efficiency of manual test through automation. Scripted test cases tends to find relatively few bugs, so freeing manual testers to do more exploratory testing, by automating any scripted manual test case like BVT, regression test etc., is almost certain to increase the effectiveness of your manual test
  
  
• Chapter 27 – BHAGs, Change, and Test Transformation
  What I find interesting in this chapter, is the following table at bottom of page 503, containing metrics of which test cases that finds bugs from their project:
  
  

  Automation Defects	Manual Scripted Defects	Exploratory Test Defects	Fix Verification Defects
  9.3%	24.0%	58.2%	8.4%

  
  Also matches my experience well, that you shouldn’t expect to find many bugs with automated test cases, but it can free resources to find more bugs using other approaches. The automated tests still brings a lot of value value to a project this way, even if they don't find many bugs.
  
  
• Chapter 28 – Exploratory Test Automation: An Example Ahead of Its Time
  Final chapter is by Harry Robinson, who has pioneered model based testing. Besides from being an introduction into model based testing, the chapter also describes they managed to improve quality of the product, by working close with developers and adjusting models based on changes to requirements. The system test ran for 8 months after release, before the first (low-severity) bug was found. Very impressive indeed, and I found this chapter really interesting to read.

In general, it’s an ok book. As the book presents a wide range of different solutions to test automation projects, and in general lacks technical details of the implementations, I wouldn’t recommend the book to someone new to develop test automation, from a technical point of view. But from a more organizational/theoretical point, this book is relevant, with it’s general conclusions on

• Test automation is software development, and to increase the probability of being successful you need to follow same rules as for any other software development project
• You need management support to be successful with test automation
• Test automation is a long-term investment. You should estimate ROI before starting, in order to set stakeholder expectations
• Define an architecture for your test automation code.