Ep 3: We Don’t Need No Stinkin’ Standards

This week I discuss eBay UK’s announcement that they will not be supporting or adhering to ISO29119.

This is not the episode I originally was going to release but that one needs more work, and this piece of news came across my twitter feed this weekend just came out of the Lets Test conference in Stockholm.

Lets Test is a context-driven testing testing conference – the idea behind context driven testing is that testing should be relevant to the context that the project, and only through judgement and skill, exercised throughout the entire project, are we able to to choose the right tools and actions to effectively test and deliver quality on the project. And that makes sense, some projects require focus on different areas, some projects – particularly government projects – have certain rules and guidelines to adhere to, and that has to be tested and delivered. Different stakeholders have different needs, things like that.

There’s a whole wiki page dedicated to Software Testing Controversies[1] (it has issues, but its there), and the intro is dedicated to Context Driven Testing, and how proponents maintain there are no best practises for testing, just good practices depending on project context, and the rejection of standards on the premise of a cost benefit analysis – essentially, one size does not fit all.

So, what happened at Lets Test? Well, among other things, the head of Software Testing and Development at eBay UK announced they would not be implementing or supporting ISO29119 at eBay. The full statement is short, and there’s a text version on Google Drive[2].

Firstly, to view the full standard you have to pay roughly £122 (or 178 Swiss Francs), which is a whole other issue, especially for software testers just starting out or freelance/contractors, but broadly ISO29119 is a set of standards (obvs) that encompasses 5 standards[3]:

ISO/IEC 29119-1: Concepts & Definitions
ISO/IEC 29119-2: Test Processes
ISO/IEC 29119-3: Test Documentation
ISO/IEC 29119-4: Test Techniques
ISO/IEC 29119-5: Keyword Driven Testing

And there are lots of techniques in part 4, that the standard covers and section 5 (unpublished) will contain a section that will help to apply test case designs and techniques to types of testing, like Accessibility and Performance testing.

So it seems like there will be some flexibility in the standard to pick what methods are best placed to serve your needs. However, there’s not much in the way for things like Exploratory Testing (and in fact, pretty much all the techniques seem to be automated, with little to no manual testing). I found a blog post where someone quoted the standard in a critique and it does cover exploratory testing, but notes that unscripted, exploratory testing is unrepeatable[4], which, is not the case? I generally can repeat tests I’ve carried out, and I can take notes with reproducible steps if needed. Which is a weird thing to say – lack of documentation at the start of testing does not mean no documentation can be produced.

This standard worries me for a couple of reasons. Firstly, exploratory and manual testing are strong testing methods, providing greater confidence in the software produced, and you’re not limited by the test cases. Secondly, it seems very easy to imagine a situation where government or similarly public funded projects require ISO29119 testing, which means that smaller companies can’t compete, because meeting these standards might be too prohibitive in many ways.

As a non-ISTQB qualified, manual tester, I’m pretty biased here, but I have tried to find an argument for these standards in the interest of learning and balance, and I’m having issues finding that view – the closest I can find is a blog post around a statement released by the ISO about the backlash this was receiving[5]. The statement isn’t good, and this blog post tears it apart pretty easily. If anyone is for this standard, or knows of somewhere I can find an argument for it, please let me know, I’m interested in hearing about it.


Leave Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.