I’ve been so fortunate to have been invited as official blogger at the GOTO Aarhus 2012 conference this year, which most likely means a number of new readers will be reading this blog post, so I’ll start by giving a short introduction to myself.

10 years ago I shifted from “regular” software development to software testing with a job as SDET (Software Development Engineer in Test) in Microsoft Vedbæk (Denmark), after which a couple of years followed as software developer (but still with focus on test and quality assurance) until I started in my current job as technical lead for test automation in ScanJour two years ago. We run Scrum in teams of 5-8 persons, with usually two domain (manual) testers and one automation tester in each team. The role of the automation tester ranges from automating already defined manual functional test cases to performing load and performance tests.

Now I’ve got the chance to participate in a GOTO conference, which seems to be a true developer conference, by not having “Tester” as an option in the Role field when you register – I registered myself as “Other”

So what’s the state of test automation among developers?

During the last 10-12 years in software development, I’ve seen how unit tests has helped to increase the quality of the products we ship, by often ensuring the most obvious bugs being caught early, but not least ensuring the individual classes/units are actually testable (otherwise it wouldn’t be possible to write unit tests against them).

But I still face problems when trying to write automated tests, due to lack of “testability” of the products in the following areas:

1. No silent installation/uninstallation, or not officially supported (so if you report bugs, they are closed “by design”)
2. Missing unique id’s on controls in webpages when doing UI tests, which makes it harder to write test automation and makes test cases less stable
3. No clear testable layers in the application, which often means you have to resort to doing automation on the UI layer
4. Test is not done in parallel with development, but sometimes when the developer has started working on a new task or even in a later sprint (unfortunately)

#4 is not only a developer-issue, but still a problem for the team, because it means that you risk finishing/shipping code, which either contains bugs or you find out is not testable.

Inspired by ISTQB (a certification for testers), I see the following levels of software tests (simplified, there also exists load test etc.), with unit tests being closest to the code and of course automated, and release acceptance testing often (but not necessarily) being a manual testing activity.

I highly value that testing should be independent from development (even if part of same Scrum team), simply to get another pair of eyes to look at the system; I’ve seen many “idea-bugs” (not understanding customer needs correctly) being caught early by domain testers, that could otherwise have easily slipped all the way to the customer (where the cost of fixing it is higher).

Regarding the issues I raise above concerning lack of testability, I think many of these could be solved by having the software developers developing more automated functional testing, in the same way that unit tests ensures some degree of testability of units/classes. After all it’s in the interest of the whole team to deliver working software to the customers, as effectively as possible.

So the questions I’ll try to get answers to while being at GOTO Aarhus 2012 conference are:

• What’s the state of automated functional test at all?
• Are software developers already automating non-unit tests like e.g. functional UI tests?
• Is it common to have automation testers in the industry (like my current job), is automation a part of usual software development activities or have you actually been able to successfully do capture/replay test automation maintained by domain/manual tester? (I would be surprised if the latter is case)

You are certainly already now welcome to give me input to these topics, by leaving a comment below, thanks.

PS. If you are interested in what GOTO conference sessions a tester finds interesting, you can see my schedule here.

Finally finished reading the book Experiences of Test Automation by Dorothy Graham and others. The book is a kind of “essay collection” of case studies, with people describing their approaches and experiences in implementing test automation for various kinds of software systems.

Being composed of different stories, of course there will be some stories relevant and some less relevant. I found the following chapters to be the most interesting ones:

• Chapter 9 – Model-Based Test-Case Generation in ESA Projects
  It was actually not the model-based content of this chapter I found most interesting, rather its focus on calculating ROI for test automation projects, and when to expect break-even. Always a good idea to include ROI when doing test automation, in order to set stakeholder expectations.
  
  One of the charts showing break-even for an automation project after 4 test repetitions/cycles, looks like this:
  
  
  
• Chapter 10 – Ten years and still going
  This was also a case study in their previous book Software Test Automation from 1999. The chapter includes the original case study and then ends with a short conclusion on the project status now. Must say I’m impressed to hear that they now run 5 million automated tests each month.
  
  
• Chapter 15 – Test Automation of SAP Business Process and 16 – Test Automation of a SAP Implementation
  These two chapters gives a good insight into how SAP have automation as a feature of their product (eCATT), i.e. allowing partners to use the built-in automation features to test customizations. Found this especially relevant, as I’m working on making our test automation, running on the standard version of our product, usable for our consultants on their customization projects. I certainly see automation as a feature of a product, and not just being executed by the testers.
  
  And then I strongly agree with the first statement in the chapter on page 278: “Test automation is nothing but software development” followed by “(…) any test automation project that did not follow software development processes and standards failed or ended up with huge maintenance efforts.”.
  
  
• Chapter 21 – Automation through the Back Door (by Supporting Manual testing)
  With so many different sources for the book, all chapters of course doesn’t have the same level of technical depth. This chapter though contained the following diagram on their Keyword-driven framework implementation. I've not specifically worked with keyword-driven testing, nevertheless I think the diagram gives a good overview of the different components of an automated test case, i.e. test logic/code, input for test case and a test execution engine.
  
  

  

  
  And then I like the concept of focusing on increasing efficiency of manual test through automation. Scripted test cases tends to find relatively few bugs, so freeing manual testers to do more exploratory testing, by automating any scripted manual test case like BVT, regression test etc., is almost certain to increase the effectiveness of your manual test
  
  
• Chapter 27 – BHAGs, Change, and Test Transformation
  What I find interesting in this chapter, is the following table at bottom of page 503, containing metrics of which test cases that finds bugs from their project:
  
  

  Automation Defects	Manual Scripted Defects	Exploratory Test Defects	Fix Verification Defects
  9.3%	24.0%	58.2%	8.4%

  
  Also matches my experience well, that you shouldn’t expect to find many bugs with automated test cases, but it can free resources to find more bugs using other approaches. The automated tests still brings a lot of value value to a project this way, even if they don't find many bugs.
  
  
• Chapter 28 – Exploratory Test Automation: An Example Ahead of Its Time
  Final chapter is by Harry Robinson, who has pioneered model based testing. Besides from being an introduction into model based testing, the chapter also describes they managed to improve quality of the product, by working close with developers and adjusting models based on changes to requirements. The system test ran for 8 months after release, before the first (low-severity) bug was found. Very impressive indeed, and I found this chapter really interesting to read.

In general, it’s an ok book. As the book presents a wide range of different solutions to test automation projects, and in general lacks technical details of the implementations, I wouldn’t recommend the book to someone new to develop test automation, from a technical point of view. But from a more organizational/theoretical point, this book is relevant, with it’s general conclusions on

• Test automation is software development, and to increase the probability of being successful you need to follow same rules as for any other software development project
• You need management support to be successful with test automation
• Test automation is a long-term investment. You should estimate ROI before starting, in order to set stakeholder expectations
• Define an architecture for your test automation code.

At TechEd 2012 session DEV340 - Taking Your Application Lifecycle Management to the Cloud With the Team Foundation Service (which I recommend watching) the presenter mentions that TFS 2012 has been used internally at Microsoft for a while. This is described in further details at http://blogs.msdn.com/b/buckh/archive/2012/06/08/developer-division-is-running-on-tfs-2012-rc.aspx, where it is also mentioned that they are running with 3600+ users currently. I think (hope…) this will result in a more stable TFS, especially in Lab Management. Looking forward to get onto TFS 2012