Let’s talk about Oracles.
Oracles determine whether a program or piece of software has passed or failed a test. In mythology they are also just that, mythological. And whilst, unlike seers who divined via interpretation of signs, they were spoken to directly by Gods, they didn’t share their knowledge often, were over subscribed, and seemed to be bribable by animal sacrifices, especially if they were Greek Oracles.
Oracles were often misinterpreted – for example the Greek tragedy of the original motherfucker Oedipus, King Laius is told he will die at the hands of his son by an Oracle, so he tells his wife (Queen Jocasta) to kill their baby son. She tells a servant to do it, and through various means depending on the version, Oedipus ends up being adopted. Oedipus hears a rumour of his adoption and asks an Oracle who his real parents are. Instead he is told is will kill his father and mate with his mother. Oedipus flees from his home, taking this to mean his adoptive parents are his real parents.
He meets King Laius and Queen Jocasta on the rod, falls in love with the queen, kills the king, and via a run in with a Sphinx, ends up with her hand in marriage. Thus the prophecy is fulfilled but via a roundabout kind of way.
(Yes this is mostly an excuse to talk about Greek mythology, something I am a huge fan of. Relatedly, if you’re also interested in Greek mythology, and want a comic rec, there is a great comic called ODY-C, which is a gender swapped retelling of homer’s odyssey, set in space and its amazing)
A fitting name, I think. Making decisions is hard. As humans we are horrifically bad at making decisions – there is a whole range of ways we can mess up making a simple decision (side note, I finally got the audiobook version of Thinking Fast and Slow, which is on basically every tester’s reading recommendations I’ve ever seen – I feel like listening to it is some form of initiation), but let’s take the two most likely here:
False Positive: We think all is well, when actually, the program has failed somewhere
False Negative: We’re pretty sure that there is a fail, but the program is fine
Generally speaking, this pitfalls affect automated checks more than humans.
So, maybe instead of viewing oracles as well – oracles – mouthpieces of the Almighties, maybe we should come at this from a heuristic, divination sense. Seers, if you will.
When we start a project, chances are the client wants something new, a new system, or at least a new design. This means that our heuristics are quite basic – we know what the client wants to achieve – and what they want their clients or users to achieve – but we have very few oracles to go off.
This stretches the definition of oracle a little, which is based on actual testing of software, but we have to test the plan first – is what we plan to deliver going to pass the tests we set? Only then can we build the basics, and start testing from there. I put together an incomplete list of the oracles we can get when starting to plan a project.
Company branding and house style
Previous versions; what they liked, what they didn’t
What examples of competitors or other similar functionalities and apps/sites they like
Third party integration they need/want, and the documentation there
These can be used to help us decide if our ideas and plans meet the needs of the client and their users.
Then we can get more specific, though still incomplete by necessity:
A set of preconditions that specify the state of the software and system when you start the test (The scenario)
A set of procedures that specify what you do when you do the test (When I do this)
A comparison of what the software under test does with a set of postconditions: the predicted state of the system under test after you run the test. This set of postconditions make up the expected results of the test. (Then I should see this vs. what the test actually produces)
Oracles are, or probably kind of have to be, heuristic as opposed to prescriptive – they enable people to discover or learn, they’re not a script. Michael Bolton has a long series of posts on this which helped me understand these concepts a bit better1. This is because context applies to software and testing – things have to be correct and fit for purpose, not definitively right (see essentially any other episode – I’ve spoken about this a lot).
If you apply a test (or an oracle) to a piece of software and you review the results, you can then decide if the results is correct or not. Is it a problem? Is what you expect? This is the true value of oracles – giving you information that you can use to test more and better.
So where am I going with this? Oracles help you decide what is a problem. Because, as I said, humans are terrible at decisions. Even if you end up applying the wrong oracle, you can still learn from applying it – you get information from the results and you can hone from there.
So oracles are terrible at telling the future, okay at telling you when a piece of software has passed the test, but best at helping you decide if any problem is actually a problem.
Also, if you’re in ancient Greece, they are bribable with animal sacrifices.