The good thing, for me as a tester, by attending a developer conference like GOTO, is that I get a different perspective on my profession, which I probably wouldn’t have gotten by attending a usual test conference. Today I saw a couple of sessions on Continuous Deployment, most remarkably a presentation by http://www.etsy.com/ in which it was shown how they do 30 deployments to production daily.

A common anti-pattern in relation to deployment, is the concept of a “deployment army” as presented by Michael T. Nygard in his presentation, where as a result of having few deployments, each release is so large that you need an “army” of people to deploy it. This gives a high cost on deployment, so you cannot afford to increase the number of deployments, and the result is a kind of dead-locked situation that you need to get out of. Continuous Deployment may solve this, as soon as you have accepted that you have a problem.

The benefits of doing Continuous Deployment is that the software development team gets almost immediate feedback on deployed changes, without having to wait for some time before releasing into production. Below is an example from Etsy, where they after a release noticed an increase in the number of warnings (the vertical red line at approx. 16:05). Within 10-15 minutes they reacted to the situation and was able to release a new version which fixed most of it, and 30 minutes after the incident, the issue was solved completely. Of course we want to avoid such issues, but when they happen, it’s important that the organization can react fast. So the benefits are basically that we get really fast and relevant feedback to changes to code. I’ll get back to what implications this has to an organization.

As stated by several speakers today, this of course requires the architecture to be prepared for this way of deployment, especially not having one big monolithic system, but being able to deploy individual components of the system. For the people being skeptical to how Continuous Deployment can actually work, Jez Humble mentioned in his presentation that when Flickr was acquired by Yahoo in 2005, they compared Flickr (which were deploying continuously to production) against all other Yahoo services, and it turned out that Flickr had the highest uptime. So Continuous Deployment certainly can work for high uptime sites, and after hearing the presentations today, it is certainly the case.

But lets take a look at a deployment process in general:


(NFR Test: Non-functional requirements testing)

This looks very much similar to a deployment cycle that I’m used to, although having some of the steps being manual or partly manual in my situation. In Continuous Deployment this must be performed several times per day, which leaves very limited time (actually none) for any manual task involved. So moving towards Continuous Deployment means that all of the processes, including test, must be automated. As it’s the developers that are monitoring the production site (performance, errors etc.) there are even no dedicated system operators involved. The DevOps (developer/operations) role is a part of the individual development team.

To follow up on the title of this blog post, I got a couple of minutes to talk with Michael T. Nygard (independent consultant) and Mike Brittain (Etsy), about the role of test automation. The very interesting answer from them was that they don’t have dedicated test automation engineers in Etsy and Michael said that he doesn’t see this role in new organizations, but rather see a role of SRE (software reliability engineer) around occasionally. Amazon is also present at the conference with a booth, which basically seems to be a recruitment campaign among Danish developers. I took a look at their job postings, and it turns out that they have no test automation engineer jobs either. I heard from others that Netflix (a TV/video streaming site) in their presentation also mentioned that they have a very limited number of testers.

Conclusion on state of test automation in the industry

I started out my series of GOTO blog posts at http://rasmus.selsmark.dk/post/2012/08/27/GOTO-Aarhus-2012-Is-it-time-for-developers-to-move-beyond-unit-tests.aspx by asking the question

“Is it common to have automation testers in the industry (like my current job) or is automation a part of usual software development activities?”

After today, I must say that the conclusion is that it’s not common to have dedicated testers (at least not in newer companies) and in order to do Continuous Deployment fully, dedicated testers becomes a bottleneck, and therefore this role does not exists in such an organization. And in the case of Etsy, they even have very few (<5) people doing manual exploratory testing out of a total of 350 employees. So the game is certainly changing for testers, if you move towards more often releases. I don’t feel concerned as such, as I’m sure any good tester has valuable domain knowledge, which can be used elsewhere in the organization, but we should be aware that the world is changing for us.