I felt a bit sciency, so let’s discuss the scientific method.
We all use it, whether we realise it or not. The basics:
- Oh, this is odd, is it because of this?
- Update your hypothesis based on experimental results
- Keep going til you have nailed down a cause and effect.
As testers, we can’t prove there are no bugs, we can only say we’ve not discovered or encountered any, backed up with the evidence of the tests we’ve done.
And then you have to gather the evidence of bugs you have found, and maybe use that evidence to advocate for those bugs. Not all bugs will be fixed, or even can be fixed, and so a decision has to be made. As well as what makes sense technologically or for the budget, we also need to choose what bugs to fix on the basis of what will be most useful to the users or clients, and the way to gauge this is to have evidence about the bug.
So the first step to this is, once you’ve got a bug, information gathering. Sometimes this is easy, and if you know the system you may know what the bug is (permissions issue, something missing, fairly obvious stuff), or you may have to do some digging to find out the details of the bug, At minimum I like to include steps to reproduce, what I expect, what actually happens, with screengrabs if appropriate. If I need to put my reasoning as to why I expected what I expected, I’ll add that. That report is my findings and essentially justification as to why I think this bug should be fixed. Then I can either make the bug a blocker to the story or not.
This then goes to the devs and if a conversation is needed about the bug it can happen with all the information documented in a (hopefully) clear and helpful manner. The exception to this is if I’m not sure if the bug is actually a bug, or if I’m unsure if it should be a blocker, in which case I may talk to the lead dev and go from there, sometimes talking to the client if we want their opinion on the bug.
So that’s the basics, what else can we learn from the scientific community with regards to testing.
Scientific papers go through peer review before publishing. The idea is that the work is independently reviewed before it’s published (deployed to live), and then the world gives feedback (UAT).
In the software dev world, this is provided both by the tech and code review by another dev, and then testing by the testers*.
There is one section of the scientific method that testing (and exploratory testing especially) does deviate from. The experiment design or plan. A lot of work goes into an experiment plan, including setting the statistical significance level, p values against the null hypothesis, or confidence intervals (these are all ways of determining the level of confidence needed that what you’re seeing is not chance, essentially). And all this has to be done before you start, because this can determine your sample size.
None of this stuff is done in testing outside of a lab, really, and even a formal test plan isn’t really done in exploratory testing, or it’s there but not as rigorous.
That’s not to say we don’t have a plan. I use the AC plus my knowledge of the system and the feature I’m testing, plus any feedback or results I get from the feature as I’m testing. We know the areas that have the most risk and can focus on those. At the end of the session, we can review our notes and see if there’s any areas we’ve skipped over and go back and do further testing if needed.
What happens essentially is we build a robust plan we go along through the testing process, building and adding as we go. We may start with a framework, but we end up with a full test plan and execution report to use as evidence to support our hypothesis, which is there are no bugs that we can find in the system we are testing.
*Let’s not talk about the failings of peer review in the scientific community, or the worrying trend of burying results that don’t fit the hypothesis – if you’re interested, read Ben Goldacre’s stuff, it’s fascinating and angering in equal measure:
http://www.badscience.net/
http://www.amazon.co.uk/Bad-Pharma-How-Medicine-Broken/dp/000749808X