Can I say it’s one of my least favourite part of testing? Absolute pain in the arse. We’ve been revisiting our devices and device testing policy (we try to revisit either ad hoc when something new is announced and on a regular basis regardless). We’ve had some interesting bugs from users on an old Nokia Lumia or clients using an older version of android and a stock browser.
Not only do you have the various combinations of hardware, screen size, OS, and browser combinations when on a mobile device you’ve got things like networks, and connectivity to consider.
There are tools like Browserstack which give you all the combos you could need, plus mobile emulators for all of the major mobile operating systems out there. Now, I always feel a bit weird testing in emulators. I know browserstack says that the emulators are the same as actual devices1, but they’re clean installs on devices. They aren’t devices that are used on a regular basis and visiting many sites and downloading things, so I do wonder how realistic they are but I also realise that realism is hard to get in devices, so you can hit most issues using these tools.
There are also tools like Ghostlab, which allows you to hook up multiple devices and browsers, and have the site you’re testing displaying on all of them at the same time. They are linked, so selecting a link on the android phone you’ve got hooked up presses the same link on the other devices, and browsers you’ve got hooked up. This means you’ve got the snapshot of realism in testing a selection of real devices, but without needing a day to do device/cross browser testing.
Combined I think this is a decent testing strategy. But how do you decide what devices you need?
For phones it’s pretty easy. Apple + Samsung is a hefty chunk of the smartphone market, then you can add some of the rarer devices in (a HTC windows phone, for example), and you’re done on the phones. We’ve got a couple of iPhones and Samsungs for greater coverage.
Then a few tablets, following the same methods as above.
Android and iOS versions are another story. iOS has a decent matrix for what versions are supported on what devices2, and windows is similar, with IE linked to their OS version (though they’ve gone from version 8 to version 10 to match their windows pc version numbers, which is annoying), but Android has different stock browsers, and different devices get the version updates at different times. I have a Wileyfox that runs bloody CyanogenOS so I’m part of the problem. You can do the market research to see what sort of devices are hitting your site, or use something like browserstack’s guide to testing the correct devices3.
We also get some feedback during UAT testing as clients will test using whatever devices or browser/OS setup they have, and then we can decide whether it’s something that we missed and need to fix, or if it’s not in our list of set ups we’ll support.
So that’s tools sorted, what about the actual testing?
We generally do wireframes and then component-based designs. So we’ll have a styletile that has examples of all the elements – menus, logos, blocks – and then wireframe the pages using desktop/mobile breakpoints to illustrate layout. This is what I’ll use to visually check the designs on different devices.
As I test websites, not apps, there is less for me to worry about functionalty-wise. I don’t have to worry about security, or access to various parts of the mobile device, or push notifications or things like that. I need to test if the website looks and works as expected on a smaller viewport, with different interactions.
That’s not to say it’s easy, or unimportant – last year Ofcom released a report that Smartphones were the most popular way to access the internet in the UK4, so it’s important that users can get to your site on those platforms.
Once you’ve decided on what you’re going to test on, and how you’re going to get the devices (actual or in the cloud), you then need to decide how to actually test.
You can apply whatever heuristic you normally use for testing on the web to mobile web testing, with some extra stuff for touch and gyroscopic testing.
Session based testing works best with this – taking notes and screengrabs as you go along and then creating bugs as needed afterwards rather than as you go.