Ep 51: Leave No Trace

If you’ve been following the soccer/football news last month1, you’ll have seen that that last match of the season at Old Trafford was suspended due to a suspicious package found in a toilet. After a controlled explosion, it was discovered that the suspicious package was one that was left in the stadium after a training exercise. Now I can think of at least three points of failure here (the training people for not counting devices in and out of exercises, the people being trained not finding the device, and any of the other staff who missed it between the training session and that match day). And it happened on the last match day of the season.

It was a clusterfuck, and it was a clusterfuck that reminded me of the importance of tidying up after yourself.

We try to silo our test environments away from the client – we have an internal test environment, and a staging one that the client works on. The stage site is where UAT happens, and possibly content population prior to go live depending on set up. We also tell the client to play with the site we’re building on stage – this is their opportunity to get to grips with the site before they have to worry about it going live.

The dev site is essentially a bit of a playground – we know the client can’t see it, so we know we can try things out that may break that environment or the client might not ever see, but we want to see what it looks like.

But we still are careful with what content goes on there. Example content should be at the very least bog standard lorem ipsum. Occasionally we’ll put a copy of live content on the staging site, if available, or we’ll make some relevant example content. This is because, firstly, it is a lot easier to test the usability and look and feel of the site with live data, secondly, it gives context to the client, who may finding looking at a bare or unfinihsed site difficult. It’s so easy to get distracted with the bareness of the site that what’s there doesn’t get a proper look over. You’ll note I said content there, not data. And that’s for a reason:

You’ve got to be careful regarding test data. If you need to use live data, and the live data is customer details, then you’ve potentially got some legal issues to contend with. If you’re using credit card information, then the PCI DSS (Payment Card Industry Data Security Standard) will apply to your test environment, and that is a pain in the arse from my brief dealings with PCI compliance. The UK Data Protection Act states that you have to tell people upfront specifically why you want to collect their data and what you plan to do with it. So unless you tell people you’ll be using their data for testing? You can’t do that.

The law sees no difference between live and test environments when it comes to data protection and breaches, and according to one source I found, 70% of data breaches are inside jobs2.

So what do you do?

Sometimes you need live data in order to get the job done, whether that be finding a bug or load testing. You can mask or anonymise data, scrub the sensitive bits out. IF you’re not using credit card details that takes a lot of the data out of the way, but if you are you shouldn’t be able to access it anyway. So that leaves names, addresses, emails, etc. The issue with that method is that can remove any usefulness from the data.

You can generate data; and this can usually be automated. The issue there can be relevance or context of data. Generating data that makes sense within the specific context could be anything from really hard to impossible, and so this might make the data useless.

Or, you make sure your test environments are up to scratch, privacy wise, and see if you can add a clause into your Terms and Conditions to mention testing using data.

Or do what the majority of companies seem to do and just…use the data and hope for the best? No the best plan, I don’t think.

My motto is always don’t test with any content you wouldn’t want a client to see. So I use bugmagnet to fill things with lorem ipsum and email addresses etc. Most of the time it’s not needed but I’d rather not have something just confusing rather than anything problematic.

So I feel that, as testers, we should try to adhere to the Leave No Trace3 set of ethics where it makes sense to:

    1. Plan ahead and prepare

I assume we all do this anyway, but it doesn’t hurt to have a plan, even if its just in your head, of what content or inputs you need to use for this testing. Plan for what you need to do when you need live data or something close to live data.

    1. Travel and camp on durable surfaces

Check your environment and/or devices are ready for you to test. Nothing like starting to test then realising the developers weren’t finished deploying and you end up losing a load of work, or, as in my case more than once, freaking out because the site looks wrong then realising I’ve still got javascript turned off from testing my last story.

    1. Dispose of waste properly

If you need to enter data that’s invalid to test something, clear it out. If you’ve got live data on a test environment, scrub it and/or delete it after use. Unless:

    1. Leave what you find

If you find a bug, sometimes leaving the content or data that caused it (if you can) can be useful for the dev to review. Try not to delete anything you didn’t make in case someone else is testing something.

    1. Minimize campfire impacts

If you do need to stress test, or load test, or security test something, make sure that you don’t break everything? Or give people a heads up so they know what to expect.

    1. Respect wildlife

Yeah, I got nothing. I guess if you have to test on a production environment, don’t change anything that might fuck up the live environment or content?

    1. Be considerate of other visitors

I don’t often test at the same time as other testers or users, but when we have we’ve let each other know and collaborated to ensure we don’t trip over each other while testing things. I try to make it as obvious as possible that I’m making test data and who I am so people don’t get weirded out by random content or changes being made.

If I do have to do something on live, I try to either keep it as inconspicuous as possible (most of the sites we do have the ability to view content without actually publishing it), and I ensure the username and content all scream THIS IS A TEST. And we inform the client beforehand, so everyone knows what to expect. That is rare in my line of work though, so I can’t speak to techniques here.

It’s something I’ve never really considered before, not in this depth, so I thank the three levels of fuckery that inspired this episode, as it’s surprisingly interesting.






Leave Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.