GTAC

Say what?!
But before making my “Halleluja!”, I have a bone to pick. I’m highlighting the automation in that sentence above, because I came away with a very weird feeling of witnessing a group of developers padding themselves on the back for being developers instead of testers. Yes, they were in many ways awesome guys, trying very hard to find solutions where bug prevention is the key. This is natural in a test automation conference. But I had two moments where I had to blink a few times to let it sink in.

First one was when Simon Stewart from Facebook told us about the cool work he has done in the build system, which he admitted looked a lot like the Google build system where he came from, and it was really inspiring, but he commented on the fact that there is no test organisation in Facebook, there are only developers. Say what?! 

To be fair, Simon did not seem to agree with the intelligence in that decision, and I certainly don’t. So yeah, you can have devs make tests, but do they LIVE for it? Are they PASSIONATE about testing? Will they actively research better ways to write tests and will they pay all the attention to the suites and how they execute? Will they wake up in the morning and think of testing or will they think of features? I want testers to live and breathe for this, not have it as a secondary afterthought.

I do, however, agree with the common sentiment that developers must write tests along with their production code. Especially unittests. As a Googler, Mark Trostler, said in his talk about making code testable “if you follow all these principles you can get away with not writing tests first, but why would you?”

Second one was a casual conversation I had with a Google Software Engineer in Test (sorry, can’t remember your name if you read this), who worked on Chrome OS. I asked how big his department was, he said there were about 400 developers and 15 testers. And no test engineers at all. No test organisation either. I’m afraid I impolitely stared at him for a few more moments than what was comfortable, because that just blew my mind. The ratio is insane in itself and the organization leaves the testers without a proper discipline. The SET didn’t seem particularly happy either.

In my book these examples are less than optimal, to say the least. When you test software you have a gigantic surface to cover and you need a lot of diversity in the way you attack the system. Diversity means you need people with different tools, different focuses and different skills and that usually means different backgrounds. If you hire only developers, you lose out on a valuable testing ressource: Experts in structured and exploratory testing. These guys find different types of bugs, often a lot of them and we need them desperately.

Whether we have the correct balance in Unity, where it is 1:1 between automation and manual or if it needs to be different, I can’t say for sure, but I surely know I will never have a department without test engineers.

Continuous Integration
Ok, enough ranting. Now to the awesome. Google’s first keynote about their build system and all the data they collect from it was just awe-inspiring. The amount of work they put into this thing is unbelievable, but the benefits they reap in terms of productivity is also huge. And the ability to have 15000+ developers working on one source repository in one branch with one head is a dream come true from a testers point of view. And this is on a codebase of >100 mio. lines of code where 50% are changing every month.

So what is the magic? Speed. Everything has to be fast enough to facilitate a pre-commit test, which uses trickery like incremental builds, caching of artifacts and massive parallelism to achieve the goal of being able to pre-commit test up to 60+ commits per minute during peak hours. This really makes it a great incentive for everyone to write tests for their code.

But that’s not enough. They also harvest all the data about the system and uses it avidly to both fame and shame developers into writing tests and making better code. Automated code analysis, code review systems, reporting to monitors with faces of people doing good or bad. Anything goes in the quest for increasing quality. That really tickled my nerve, so I’m going to do what I can to make this a reality in Unity.

Flaky Tests
Another big theme was flaky tests. Tests which returns false positives and are non-deterministic. A huge theme for any organisation which has a lot of automation, including Unity. At the talks there was a difference in how these were treated, going from tolerating them for shorter periods to complete non-acceptable. In Unity we adopt zero tolerance for flaky tests, so anything has to be investigated immediately and if there is a suspicion of having a bad test, we delete it or mark it as a known failure and decomission it until we can fix it. Flaky tests are toxic for automation; it’s a slow killer which deprives it of validity and renders everything useless.

Performance
When you have companies like Twitter, Google and Facebook speaking, you can’t avoid talking about scale and performance. The talk done by James Waldrop from Twitter was about how he created performance tests and made a framework which was able to create Twitter scale load on the system: 2.000.000 requests per second! This is an outrageus number and the fact that their systems are able to handle this kind of load is mindblowing.

Aside from direct performance tests, there was a theme all the way of speed in the tests. More tests, more parallellism, cloud execution and for Android: Emulators. The exponential growth of tests and test execution requires us to do something other than brute force, which was a very interesting topic for a talk by a Ph.D. student who worked as an intern at Google to make it possible to use heuristics to find culprit changeset in long running test suites. This is the kind of trickery we need to employ in the future to cope with the ever increasing amount of tests.

Worth it?
Hell yes. Conferences which gets you inspired are an investment you get back tenfold. Ideas are surging and I’m inspired to go home and DO something, which is great. I hope I will get a chance of attending again.