The GOTO Aarhus 2012  conference is over, and I can look back at some really good and inspiring days where I had the pleasure of hearing speakers like Martin Fowler talking about NoSQL databases.

As the title says, it was an introduction to NoSQL databases, but with references back to relational databases and even network databases, which was the predecessor to the relational databases. I’m old enough to have been taught about network databases in university, but nevertheless really interesting to see how a computer science field like databases is evolving. As with many of the other speakers at the conference, Martin Fowler did a very good job of keeping the right balance between theory and practice.

Scott Hanselman was talking about how the browser and the Web has evolved so it’s now possible to even run a Commodore 64 or Linux emulator using JavaScript. As he said, “JavaScript is The Assembler Language of the Web”:

This was followed up Anders Hejlsberg and some of his team members from Microsoft, when they presented TypeScript, which is a new type-safe language with classes that compiles into JavaScript. Of course the TypeScript compiler is written in JavaScript (Which is quite useful, as it then easily can run on different operating systems)

Michael T. Nygaard and several others had some really good sessions on Continuous Deployment. Michael talked about the concept of “deployment army”. Think we all know of situations where a deployment have required special skills.

One of the purposes of Continuous Deployment is to eliminate the need for big releases, by building up confidence around the software product, so it really always is in a releasable state. Then you can decide to release it or not, depending on your type of software and customers. And this is only a few of all the speakers I’ve heard in the last three days (actually the ones where I had some decent pictures...)

As said, it has been some fantastic days at the GOTO conference. The conference has been able to find just about the right balance between theory and practice in the sessions. Only very few sessions have been either too theoretical or practical oriented. Also when comparing this conference to Microsoft TechEd, which I attended earlier this year, I find the GOTO conference to have a much higher level, both when it comes to speakers and topics.

So if you are considering only one conference for the next year (and GOTO is able to keep the high professional level of speakers and topics) I will definitively recommend that you consider attending the GOTO conference. At least I certainly hope that I can be part of it again next year.

The good thing, for me as a tester, by attending a developer conference like GOTO, is that I get a different perspective on my profession, which I probably wouldn’t have gotten by attending a usual test conference. Today I saw a couple of sessions on Continuous Deployment, most remarkably a presentation by http://www.etsy.com/ in which it was shown how they do 30 deployments to production daily.

A common anti-pattern in relation to deployment, is the concept of a “deployment army” as presented by Michael T. Nygard in his presentation, where as a result of having few deployments, each release is so large that you need an “army” of people to deploy it. This gives a high cost on deployment, so you cannot afford to increase the number of deployments, and the result is a kind of dead-locked situation that you need to get out of. Continuous Deployment may solve this, as soon as you have accepted that you have a problem.

The benefits of doing Continuous Deployment is that the software development team gets almost immediate feedback on deployed changes, without having to wait for some time before releasing into production. Below is an example from Etsy, where they after a release noticed an increase in the number of warnings (the vertical red line at approx. 16:05). Within 10-15 minutes they reacted to the situation and was able to release a new version which fixed most of it, and 30 minutes after the incident, the issue was solved completely. Of course we want to avoid such issues, but when they happen, it’s important that the organization can react fast. So the benefits are basically that we get really fast and relevant feedback to changes to code. I’ll get back to what implications this has to an organization.

As stated by several speakers today, this of course requires the architecture to be prepared for this way of deployment, especially not having one big monolithic system, but being able to deploy individual components of the system. For the people being skeptical to how Continuous Deployment can actually work, Jez Humble mentioned in his presentation that when Flickr was acquired by Yahoo in 2005, they compared Flickr (which were deploying continuously to production) against all other Yahoo services, and it turned out that Flickr had the highest uptime. So Continuous Deployment certainly can work for high uptime sites, and after hearing the presentations today, it is certainly the case.

But lets take a look at a deployment process in general:

(NFR Test: Non-functional requirements testing)

This looks very much similar to a deployment cycle that I’m used to, although having some of the steps being manual or partly manual in my situation. In Continuous Deployment this must be performed several times per day, which leaves very limited time (actually none) for any manual task involved. So moving towards Continuous Deployment means that all of the processes, including test, must be automated. As it’s the developers that are monitoring the production site (performance, errors etc.) there are even no dedicated system operators involved. The DevOps (developer/operations) role is a part of the individual development team.

To follow up on the title of this blog post, I got a couple of minutes to talk with Michael T. Nygard (independent consultant) and Mike Brittain (Etsy), about the role of test automation. The very interesting answer from them was that they don’t have dedicated test automation engineers in Etsy and Michael said that he doesn’t see this role in new organizations, but rather see a role of SRE (software reliability engineer) around occasionally. Amazon is also present at the conference with a booth, which basically seems to be a recruitment campaign among Danish developers. I took a look at their job postings, and it turns out that they have no test automation engineer jobs either. I heard from others that Netflix (a TV/video streaming site) in their presentation also mentioned that they have a very limited number of testers.

Conclusion on state of test automation in the industry

I started out my series of GOTO blog posts at http://rasmus.selsmark.dk/post/2012/08/27/GOTO-Aarhus-2012-Is-it-time-for-developers-to-move-beyond-unit-tests.aspx by asking the question

“Is it common to have automation testers in the industry (like my current job) or is automation a part of usual software development activities?”

After today, I must say that the conclusion is that it’s not common to have dedicated testers (at least not in newer companies) and in order to do Continuous Deployment fully, dedicated testers becomes a bottleneck, and therefore this role does not exists in such an organization. And in the case of Etsy, they even have very few (<5) people doing manual exploratory testing out of a total of 350 employees. So the game is certainly changing for testers, if you move towards more often releases. I don’t feel concerned as such, as I’m sure any good tester has valuable domain knowledge, which can be used elsewhere in the organization, but we should be aware that the world is changing for us.

Mondays keynote at the GOTO Aarhus conference this year was held by Rick Falkvinge who is leader of the Swedish Pirate Party, a political party working for free information and a free Internet. Title of the keynote was “Red flags on the internet”, which refers to the “Red Flag Act” of 1865, where a law was introduced in United Kingdom, which required that a person should walk in front of every car waving with a red flag, to warn pedestrians.

It turned out that this law was lobbied by the British Railways in order to secure their interests. But the result was that Germany this way got 20 years advantage for their automobile industry, so it ended by hurting Britain more than it helped. Rick went on to other examples from the history, ending with Kodak, who actually invented the digital camera back in 1976, but because their income depended heavily on their analog film products, they didn’t develop this new digital technology further and eventually went bankrupt in January 2012. So the point here is that it doesn’t help protecting/hiding information and inventions, as it will at at some point emerge anyways.

Even though the Swedish Pirate Party is currently mostly a protest party, he drew lines back to the Green politics 40 years ago, which is now a common part of the official program for all political parties. He didn't give a good answer to how we can actually still get paid if all information is freely available, but I still find it interesting to see how history has shown that information and new technologies cannot be hidden away.

New infrastructure introduces new types of companies

With the shift from e.g. water mills to electricity, it was no longer necessary to build factories close to the energy source, but could rather be built where the actual need was. Unfortunately I didn’t get an image of the actual slide, but I do find it thoughtful this similarity with how we are still constrained today, e.g. of where companies can be built, so I think we need an image of a good old watermill-powered factory from Wikipedia. I’ll get back to this shortly.

RubyMotion - a company of the Internet age

This brings me on to Laurent Sansonetti, founder of RubyMotion which is a platform for developing Ruby-based applications on e.g. iPhone. I was so fortunate to have a talk with Laurent about his company.

Laurent have worked as developer for several large companies, including 7 years at Apple, before he decided to work on his own open-source project RubyMotion. Moving from a secure job at Apple and starting his own, did result in a couple of very insecure months with different challenges, and sometimes fear that the product would never made it. But it went well, and they now have several large customers and are making money. Congratulations, as it’s good to see that there is always room for new innovative products.

The structure of the company is what Laurent calls a “distributed startup”, meaning that the current three employees are spread across the entire world (Belgium, United States and Japan) and don’t have any office yet. They work from where they decide, more or less at the time they decide and communicate using CampFire, which is a group-based chat system, where they can have ongoing discussions, and see what has been talked about while they were gone. Because being spread across the world this way, means that the company is more or less “open” always. Then they meet every third month for a week physically, in order to actually meet and have face-to-face meetings.

Other advantages of being distributed this way, is that you actually get easier access to talent, as they don’t need to relocate in order to work for you, and perhaps otherwise wouldn’t be able to take the job. As Laurent said, it’s hard to find good programmers and even harder to have them relocate.

I think this is a good example of a company that leverages the possibilities of the internet age, to build a startup company that a few years ago would have been difficult to imagine. Especially it’s worth noting that with only 3 people they are able to keep the company running most of the day. Like back when factories had to built close to an energy source, I still think that (IT) companies of today are constrained to some extend of having access to talents or other kind of infrastructure. Although I actually like going to the office (working zone) every day, I think it’s worth paying attention to the possibilities that are available for starting up a new company with very little “deadweight” like physical building, network, servers etc., when all of this can be distributed/hosted on the net.

Even though this is a part of my “warm-up” blog posts for GOTO Aarhus 2012 Conference, I’ll start out referring to a related event this week. At our latest meeting in the Danish TFS User Group, held at the Microsoft office in Hellerup, Rune Abrahamsson from BRFkredit had a presentation on how they are using http://cuite.codeplex.com/ for developing functional tests for one of their systems. If you are interested, Mads (our agile coach at ScanJour) has also blogged from the user group here.

Although the people attending the TFS Users Group meetings are mostly developers, we often have at least one testing-related topic on the agenda, and this time it was clear that many of the present developers does have experience with developing functional tests, often using a UI automation framework like Microsoft Coded UI. So even before attending the GOTO Aarhus 2012 Conference, it seems I can conclude that developers are writing functional tests, which pleases me, as I think the people writing the software, also are the best at writing automation for it. The testers can then do actual quality assurance, by ensuring that the customer requirements have been automated, as well as performing exploratory testing to exercise the software in new ways.

Sorry for the bad picture quality; the slide shows at the bottom which areas (including functional UI tests) are covered by developer tests, whereas e.g. exploratory tests are performed by their manual testers (which are actually domain experts, not full-time testers). I find it very positive to see a development team take quality seriously, by including test automation, when they are in a situation where they are lacking testers. I guess this is not an uncommon scenario in the industry, that you have to get test assistance from other teams/departments.

And it makes me wonder if manual testers should be forced to not do manual scripted test at all, but only do exploratory testing and quality assurance, since the developers will fill the gap themselves, by automating the scripted test cases?

To put this in the context of this years GOTO conference, I have been looking at the biographies of some of the speakers, and found a video by Steve Freeman on Sustainable Test-Driven Development, where he touches topics like:

• Too coupled production and test code, which makes refactoring difficult (beginning of video)
• Test code structure (similar to Given/When/Then pattern of e.g. SpecFlow)
• “You come back to the code after 6 months, and forgot why you did this” (approx. 13:00)
• Patterns for writing test code. As simple as good variable naming, using DSL syntax to have more more readable test code
• Prepare for your test to fail at some point, simply by having clear error messages when it happens (“Explain yourself”, “Describe yourself” and “Tracer Objects” around 23 mins into the video).
• Make your tests robust, e.g. “Only Enforce Order When It Matters” (around 40:00)
• “Tests Are Code Too” (43:35), which is the last slide and also seems to be the headline for this presentation

The last slide is shown here:

Although all four bullets here are important, I have myself faced the “If it’s hard to test, that’s a clue” problem, when developing unit tests, both for production code as well as test automation framework code. As an automation tester, I also regularly have the problem when writing automated functional tests against production code, e.g. missing id’s on UI controls, no clear interface for testing etc. When the development team is writing automated functional tests as part of definition of done, it should hopefully result in code better suited for automation. And then the general conclusion, that test automation should be treated like any other code activity.

This year Steve Freeman has a session on Raspberry Pi with extra toppings Monday at 13:20, which unfortunately is a timeslot where I have also some other sessions I would like to see (What is value and Mythbusting Remote Procedure Calls), but it might be that I have to change my mind; as it could also be fun to get an introduction to the Raspberry Pi.

After watching this video, I feel confident that I will meet developers at the GOTO Aarhus 2012 Conference this year with a quality/testing mindset, and I’m looking forward to talk with you about your view on test automation, and how I as an automation tester can bring value and increase the quality of our software products, even if it's no longer a dedicated test automation developer writing the functional tests.

Feel free to comment on this post below.

I’ve been so fortunate to have been invited as official blogger at the GOTO Aarhus 2012 conference this year, which most likely means a number of new readers will be reading this blog post, so I’ll start by giving a short introduction to myself.

10 years ago I shifted from “regular” software development to software testing with a job as SDET (Software Development Engineer in Test) in Microsoft Vedbæk (Denmark), after which a couple of years followed as software developer (but still with focus on test and quality assurance) until I started in my current job as technical lead for test automation in ScanJour two years ago. We run Scrum in teams of 5-8 persons, with usually two domain (manual) testers and one automation tester in each team. The role of the automation tester ranges from automating already defined manual functional test cases to performing load and performance tests.

Now I’ve got the chance to participate in a GOTO conference, which seems to be a true developer conference, by not having “Tester” as an option in the Role field when you register – I registered myself as “Other”

So what’s the state of test automation among developers?

During the last 10-12 years in software development, I’ve seen how unit tests has helped to increase the quality of the products we ship, by often ensuring the most obvious bugs being caught early, but not least ensuring the individual classes/units are actually testable (otherwise it wouldn’t be possible to write unit tests against them).

But I still face problems when trying to write automated tests, due to lack of “testability” of the products in the following areas:

1. No silent installation/uninstallation, or not officially supported (so if you report bugs, they are closed “by design”)
2. Missing unique id’s on controls in webpages when doing UI tests, which makes it harder to write test automation and makes test cases less stable
3. No clear testable layers in the application, which often means you have to resort to doing automation on the UI layer
4. Test is not done in parallel with development, but sometimes when the developer has started working on a new task or even in a later sprint (unfortunately)

#4 is not only a developer-issue, but still a problem for the team, because it means that you risk finishing/shipping code, which either contains bugs or you find out is not testable.

Inspired by ISTQB (a certification for testers), I see the following levels of software tests (simplified, there also exists load test etc.), with unit tests being closest to the code and of course automated, and release acceptance testing often (but not necessarily) being a manual testing activity.

I highly value that testing should be independent from development (even if part of same Scrum team), simply to get another pair of eyes to look at the system; I’ve seen many “idea-bugs” (not understanding customer needs correctly) being caught early by domain testers, that could otherwise have easily slipped all the way to the customer (where the cost of fixing it is higher).

Regarding the issues I raise above concerning lack of testability, I think many of these could be solved by having the software developers developing more automated functional testing, in the same way that unit tests ensures some degree of testability of units/classes. After all it’s in the interest of the whole team to deliver working software to the customers, as effectively as possible.

So the questions I’ll try to get answers to while being at GOTO Aarhus 2012 conference are:

• What’s the state of automated functional test at all?
• Are software developers already automating non-unit tests like e.g. functional UI tests?
• Is it common to have automation testers in the industry (like my current job), is automation a part of usual software development activities or have you actually been able to successfully do capture/replay test automation maintained by domain/manual tester? (I would be surprised if the latter is case)

You are certainly already now welcome to give me input to these topics, by leaving a comment below, thanks.

PS. If you are interested in what GOTO conference sessions a tester finds interesting, you can see my schedule here.