The standup had a ritual by March. Priya, the QA lead, would read out the overnight result, and whoever’s name sat nearest the failing test would sigh and say the thing everybody already knew: “just re-run it, it’s probably fine.” The product was good. The pipeline was haunted. Twenty-plus developers were spending their mornings as ghost hunters, chasing failures that vanished on the second attempt. Nobody had said the word TestOps out loud yet, but over one quarter — on purpose, and mostly by the book — this team rebuilt its relationship with its own test suite. Here is how it went, told the way it actually felt.
The week the TestOps ghost got a name
Tomás, a backend engineer with a taste for spreadsheets, did the unglamorous thing first: he counted. For two weeks he logged every red run and whether a retry turned it green. The main branch, he reported, had gone green on fewer than twenty out of every hundred runs. Nineteen-point-eight percent, to be exact — a number the team received the way patients receive X-rays. Worse, nearly every “failure” he traced showed the signature of bad instrument rather than bad code: identical code, opposite outcomes on two attempts. A published account from Slack’s mobile team later gave the team comfort that this was survivable — their comparable green rate sat at 19.8 percent before repair and reached 96 percent after. But comfort comes later. First, evidence. The team adopted one rule that changed everything: a test only earns the label flaky when the very same commit has passed once and failed once. No more folklore, no more vibes. The results store — every attempt recorded against its commit — made the rule enforceable with a query.
Quarantine, with names and deadlines
Confirmed flakes left the gating path immediately, but the team wrote three conditions on the whiteboard before the first one moved. Every quarantined test gets an owner. Every owner gets a ticket. Every ticket gets a deadline, after which the test is either fixed or deleted. The first week felt radical — eleven tests stepped aside, and the pipeline’s mood changed overnight. The graveyard temptation arrived quietly in week three, when someone suggested quarantine was “basically a junk drawer”. Priya killed the idea in one sentence: a junk drawer with eviction notices is a queue, and queues drain. By the end of the month the quarantine list held four tests, two fixed, two deleted, none forgotten.
The data nobody suspected
Then came the finding that embarrassed everyone. One of the “fixed” flakes kept returning, and the trail led somewhere unexpected: two tests were quietly sharing a customer record. Whichever ran second inherited whatever mood the first had left behind. The team rewrote its data rules from scratch, and the new ones read like commandments. Each test creates its own records through the API in setup. Identifiers carry the run number and the worker’s name, so collisions cannot hide. Teardown deletes everything, no exceptions. And when real production data is genuinely needed, it arrives masked and in slices — never as a full copy, which brings risk without honesty. The mystery flake never returned. Roughly a third of the remaining “random” failures vanished the same week, which taught everyone a lesson in humility: some ghosts are just roommates.
The gate that failed in silence
In week nine, an incident taught the most expensive lesson of the quarter. A shard job crashed mid-run; the CI system, following its default instructions, skipped everything after it — including the job that was supposed to certify the merge. Branch protection saw no failing verdict and waved the change through. The change was fine, that time. The silence was not. Dev opened the workflow file, added the condition that forces the deciding gate to run even when earlier jobs die — the if: !cancelled() pattern — and then deliberately broke a job on a throwaway branch to watch the gate catch it. It did. “A gate that only fires on sunny days,” Priya wrote in the incident review, “is a decoration.” The team now tests the gate the way firefighters test alarms: on purpose, on schedule.
Ninety-six and holding
By the end of the quarter, Tomás’s spreadsheet told a different story. Main went green on ninety-six runs out of a hundred — the same figure the Slack mobile team reported after taking flake control seriously, which the team took as reassurance that their path was the ordinary one. The wait for a trustworthy answer dropped from hours to minutes. Retries, once a reflex, became rare enough that each one now signals something real. And the emotional change outlasted the numeric one: developers stopped bracing when they pushed. One of them described the difference in the retro as “the pipeline went from weather forecast to verdict” — which is, in one sentence, what running tests as a service is supposed to feel like. Teams wanting the wider context for these practices will find how DevSecOps transforms release safety a natural extension, and the DevSecOps roadmap for developers and security teams extends the same pipeline discipline into security checks; anyone starting from a rougher place can learn from the mistakes companies commonly make adopting DevOps.
What this TestOps story teaches
Five lessons, stated plainly for the next team on the same road:
- Count before you fix — beliefs about a suite are usually wrong in both directions.
- Confirm before you label — a flake verdict is only as good as its evidence.
- Contain with owners and deadlines — containment without consequence is just storage.
- Treat test data as a first-class citizen — contamination impersonates coincidence.
- Make sure the final decision actually executes — silence is not approval.
If you want the discipline these practices belong to, its history and its research base, the Ops Atlas page dedicated to TestOps is the neutral starting point. The team in this story is fictional. The quarter it had is not.
