A Retry Pass Is a Bug

Last Saturday I opened the CI history for the Go codebase behind Symbol and listed every test job from the last two weeks that failed once and passed on retry. Then I spent a few hours with agents making each one pass on the first run.

My favorite one was a company list test. It filtered companies with keyword=parent and expected a specific result. Most days it passed. Some days the random test-data generator invented a company called “New Media Parents,” the filter matched it, and the test failed. Nobody wrote a bug. A fake company with a plausible name walked into the wrong test.

That is what flakes look like up close. They are rarely mysterious. They are small, specific mistakes that only show up when the dice land a certain way.

Retry hides the bug

Most CI setups retry failed jobs, and I understand why. Nobody wants a red build because of a network blip. Some of our retry failures really were that: the Go module proxy dropped a stream. Those are not test bugs.

The rest were. A test that can fail and then pass on retry is telling you that its outcome depends on something it does not control. Retry turns that signal into noise. The build goes green, the team learns to click “re-run,” and the flake stays in the suite until it fails on the day you need a hotfix out.

So the rule I wanted was simple: a retry pass is a bug. Find it, name it, fix it.

What the flakes actually were

Once I stopped calling them “flaky” and read them one by one, they fell into a short list:

1. Random text in places that need exact text. Generated names in keyword filters, substring matches, and unique columns. New Media Parents was not alone.
2. A “unique” ID that was shared. The factory library kept a named UUID for the whole test process, so two companies could end up with the same generated name and hit a unique index.
3. Random time zones. Factories picked a random zone, which made “current month” move at month boundaries. Tests passed most of the month and failed near the edges.
4. Counts that randomness could change. A scheduling test counted slots, but random assignment could put two emails in one slot.
5. Helpers that were not safe in parallel. A mock counted calls with count++ while webhooks ran in parallel workers, and lost updates. Another test sent two concurrent requests through one shared session map and crashed the process.
6. Escaping. A mail test HTML-escaped a company name and then searched the plain text body for it. Generated names can contain &, +, and '. The fix used a name with an apostrophe on purpose: “How’s My Offer?-” plus an ID.
7. State that leaked when a test failed. A shared mail mock was cleared at the end of each test, after the assertions. When an assertion failed, the clear never ran, and the next test started with someone else’s mail.

None of these needed a clever fix. Fixed strings, a new UUID per name, UTC unless the test is about time zones, atomic counters, one session per request, and defer for cleanup.

How I made the agents prove it

Agents are very good at this kind of work, and they are also very good at sounding finished. Two habits kept the session honest.

Say what you did not run. The environment the agents worked in had no Postgres and no Docker. The packages compiled, but the suites could not run there. Every pull request said that in plain words, and CI was the judge. An agent claiming “tests pass” without running them is worse than a flake, because it looks like evidence.

Include a case that fails without the fix. Each pull request listed its test cases as GIVEN, WHEN, THEN, and at least one case was written to fail if the fix was missing. For the keyword test, that meant a child company named “Parent Tenant” on purpose, so the filter had a known match and an ignored filter would show up. A fix that cannot fail is just a hopeful edit.

Even then, fixing flakes produced a flake. One pull request failed CI because a suite’s setup cleared the mail mock but skipped the database cleanup, so a later test found leftover company rows. The agent fixed the setup order and CI went green. It was a small reminder that the suite is a shared system, and every test that leaves state behind is writing a bug for a stranger.

Then write the rules down

The fixes cleaned up two weeks of history. They did nothing for the tests that agents will write next week.

That part goes in AGENTS.md. Our coding agent loads that file for every task in the repo, so its Tests section now opens with two lines I like a lot:

A test that can fail and then pass on retry is a flake. Do not add one.

Below that is a short list. Each line names the failure mode and then the rule: no generated text for unique columns or keyword filters, pin the time zone to UTC, do not assert on the current month without a mocked clock, do not assert counts that randomness can change, make parallel helpers safe, and unescape HTML before you match a name.

I wrote last week about putting AI context in the repo. This is what that looks like on a normal Saturday. Agents write a lot of tests, and they write them in whatever style the repo allows. If the repo allows random names in filter tests, an agent will produce them at a rate no human could match. If the repo says no, in one short line, the agent follows it on the first try.

The discipline

I do not think flaky tests are an AI problem. We had them before agents and we will have them after. What changed is the volume. When an agent can write fifty tests in an afternoon, a bad habit in the test suite scales the same way a good one does.

So the discipline I am keeping is small. Treat a retry pass as a bug. Make every fix prove it can fail. Write the failure mode into the file the agent reads before it starts. And keep an eye out for New Media Parents. They are still out there in the generator, waiting for someone to search for parent.