Ep 56: And our survey says

An interview with Rosie Hamilton!

This year Rosie did a large survey of software testers and then did the number crunching to produce a snapshot of testers in 2016. All the data and the scripts she used are up on GitHub for transparency, and for others to do their own analysis if they wish.

Rosie has now made a web app for people to explore the data she found without having to run scripts or learn R. She has just published a blog post about how she did this, with a link to the app.

I’ve been interested in the make up of testers; where we come from, when and how we decide to become testers for a while now, and I knew when I saw the survey that I wanted to invite Rosie on to the show to chat about it.

We talk about the survey process, from inception to analysis; what the survey taught her about testers, and about surveys, where testers come from, and the plans for the next iteration of the survey.

Summarised transcription:

The inception, the decision making, whys, whens etc. Did you have it all planned out from the beginning and have the analyses you wanted to do before you formulated the questions or what it more ad hoc than that?

  • It kind of happened accidentally. A lot of people had been asking me the same questions. I tried to answer from personal experience but couldn’t, I reached out to some friends on social media to get their views, but still wasn’t any close to answers.
  • Had a few doubts about asking people to fill out a form, mostly I wasn’t sure how I was going to analyse the results to find answers, but decided to go for it.
  • I spent about 4 weeks planning the survey. I had a list of questions I wanted answered and I worked backwards from there. I tested the survey on the testers I work with. The main concern which was raised while testing the survey was keeping it totally anonymous. Some of the questions got changed before it was unleashed on the community.
  • While the survey was open and collecting data I started planning how I was going to deal with the results. I started learning R which was a very steep learning curve at first.

Lessons learned, both from the survey and from doing the survey?

What I learned about surveys:

  • Open ended text boxes are a real pain to clean up and make sense of the results.
  • I should have asked more demographic questions which would have allowed the results to be split into more groups. From the start I didn’t want to divide the sample by gender as I thought gender was irrelevant but I wish I had asked which country or continent people were working in as it would have been nice to try find some patterns there.
  • The questions which were crystal clear with closed answers like yes/no true/false gave the best results.
  • The more data collected the better. You can never have too much data.
  • I learned that R is REALLY powerful. It lets you start writing the analysis before all the results are in. You can just keep feeding it data as the collection of responses grows.
  • I think being transparent from the start about why the data was being collected, what it was going to be used for and also making the results publicly available was what has made this study quite unique. I know there are other surveys about testing like the ‘state of testing’ but the raw data from that one is not in the public domain, only someone’s interpretation of the results is available.

What I learned about testers:

  • We are a really diverse group and we all have our own stories.
  • I spoke to one tester while the survey was going on, they said they had struggled to answer the what made you apply for your first testing job. They said they came to work one day and were told ‘you are a tester now, your old job doesn’t exist’.
  • I feel testing is still has a bit of stigma around being an unglamourous job. It is very hard to hire testers. I also feel that testing is not a really aspirational career choice given how few testers wanted to test while in Education.
  • One of the results which surprised me was that 2 out of 3 testers which started testing in the last two years didn’t study computing. I think students which study computing are mostly moving into development jobs. Which means junior testing jobs are being filled by people without computing backgrounds.
  • There was one person that completed the survey that when asked what the liked about testing answered ‘none of the above’. One in every four testers aren’t happy in their jobs so I think testing isn’t for everyone. I also think there are some really bad testing jobs out there. In the worst workplaces we have testers getting blamed for missing bugs, management not understanding what testers do and forcing testers to sign disclaimers stating there are no bugs.

Feedback from other people?

The feedback has been really positive. I’ve also realised even though the blog posts were written for a testing audience, other groups have been reading about the survey.

Future plans?

I have this crazy plan to turn all the data from this survey into a web application. I think this will be good because it will provide an interface for people to explore the data themselves. If I manage to create this web app that I’m dreaming about, I am certainly going to blog about it.

I definitely think I am going to survey testers again next summer, which should give me enough time to design a much better survey for 2017.