2023-05-16 · Software Testing

Run a triage workflow for flaky tests

Quarantine, diagnose, delete: a repeatable routine for tests that lie.

Introduction

Flaky tests train teams to ignore red builds. This checklist covers detecting flakes automatically, a quarantine lane with a named owner, and the four root causes behind most of them.

Field notes

A few patterns we kept seeing while puttering around Software Testing problems:

  • The simple version survived every refactor; the clever version didn’t.
  • Documentation written during the work aged far better than documentation written after.
  • The second time a problem shows up, it is already a pattern — name it.

A checklist you can steal

  • The failure is reproducible with one command.
  • The fix has a test that fails without it.
  • Someone other than the author has run it on their machine.

Closing

Write the short version future-you will actually read. That is the whole trick.