Ep 65: A Gordian Knot of Numbers

Or: This week Matt and I talk ourselves in circles about metrics:

Opening thoughts:

  • Why measure? Because everyone wants to improve.
  • What to measure? Very very difficult to measure “soft skills” like testing.
  • Several perspectives on this.


I’ve worked in places where they counted bugs and I’ve made the mistake of counting test cases.

Further thoughts:

  • Power to change the need for overly simplistic metrics?
  • Asked for metrics but allowed to choose what to measure?
  • Try to identify why they want those metrics and use critical thinking to explain why it is flawed.
  • Counting bugs – project that is near completion vs a new project?
  • Counting test cases – no test case is equal, all written differently. What is a “test case”?
  • To the junior testers – help your test lead
  • To the test managers – don’t rely on metrics
  • Consider the danger of KPIs, what behaviours do you want to encourage?

Testing could be measured by:

  • Are customers happy?
  • Is the product selling?
  • Are there very few critical issues found in production?
  • Are budgets and schedules being met?
  • Are developers happy?
  • Are product managers happy?
  • Are testers happy?
  • If you have to count bugs then can you add some context to those metrics?

Asked around testersio slack team:

Feelings of metrics not making sense for individuals, only departments
Could measure % of production deployments without a critical defect reported in 24 hours/week?

Leave Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.