Artificially Designed · Issue 12: The False Pass

Issue 12 · The False Pass

Hi friend,

Last week I asked which job on your desk has two tools holding it right now. That question is still open, so if you have been meaning to hit reply, it is not too late, and the answers are going into Last Tool Standing.

This week is about the opposite failure. Not two tools fighting over one job, but one job that everybody believes is being done and nobody is actually doing. I published a long piece on it Saturday and the short version is here, because the pattern turned out to be much bigger than code.

People build this wrong constantly, and I built it wrong too. You add an automated check, and then you quietly stop doing the thing the check replaced, without ever deciding to.

The clearest version is code review. Qodo’s product telemetry, published alongside their 2025 developer survey, reports that when an AI review tool is turned on, 80% of pull requests get no human comment or review at all. Not fewer comments. None. That is vendor telemetry with no published method, from a company that sells AI code review, and it is still the closest public measurement of this behaviour that exists.

The reason it happens is worse than laziness. The tool was installed to catch things, and it does catch things, and the comments are usually right. Every correct comment teaches the team that the tool has this handled, so opening the diff yourself starts to feel like duplicated work. The team is responding sensibly to evidence. The mechanism that produces the rubber stamp is the tool working well, not the tool working badly, which is why waiting for a better model does not fix it.

There is a structural problem underneath it too. The model reviewing the code and the model that wrote the code come from the same training distribution and carry the same blind spots. Left unread, that is not a second opinion. It is the first opinion wearing a badge.

The pattern that holds is one line, and it goes in the ticket before anyone builds anything. Under your acceptance criteria, write the specific way a plausible implementation could satisfy every one of them and still be wrong. I call it the wrong-if line and my own specs reject a story that is missing it. Acceptance criteria describe success, which is the easy half. The wrong-if line names the misunderstanding a competent person is most likely to make, and it turns review from “does this look right” into “here is the exact scenario I am going to try to trigger.” A reviewer with that line has an assignment. A reviewer without it has an impression.

I got to test this on myself last week and I did not enjoy it. I have an audit script that checks whether my documentation has gone stale, and it was reporting zero stale documents while a specification sat five days out of date. The staleness checker was iterating a hardcoded list of two filenames, and the stale file was not one of them. The check was passing because it was not looking.

That is the same failure as the rubber stamp, one level up. I had built a verification layer and then trusted its green light without ever asking what it examined. No check at all leaves you appropriately nervous. A false pass sends you home.

I laid the rule out properly last issue, so the one-line version. Every job already has something doing it, and a new tool has to take that slot rather than move in beside it. Name the job first and most tool arguments end there.

Here is this week’s, and it is my own SEO stack. The job is telling me what to fix next on a site almost nobody has found yet. OpenSEO holds that slot. The tool I tried and put down is Semrush.

Being straight about what that means: I ran Semrush’s free tier for a while and liked it, and the price is what stopped me, so I do not have a full performance verdict and I am not going to pretend otherwise. What I have is a decision I actually made, which the rule is designed to interrogate rather than justify.

So I went and got the numbers before writing this, because the whole point of the second criterion is that it can do the job, not that it can do more. Here is what Google Search Console says for July against August, and it is not a flex.

charmthirteen.com went from 51 impressions in July to 81 across the first 27 days of August, and one click to two. orinyx.io went from 76 to 131, and three clicks to five. On a daily rate both roughly doubled, and Orinyx’s last four days in that window ran 11, 14, 11 and 11, which is about five times its July pace.

I looked at that shape and felt good for a second, and then read the base. One hundred and thirty-one impressions is not an audience. It is a rounding error with a trend line on it. Which is the answer to whether the expensive tool takes the slot, and the answer is no, because my constraint is not the quality of my keyword data. My constraint is that I have not published enough yet. Better data about a search I am nowhere near winning does not change a single thing I would do on Monday.

What would have to be true for Semrush to take the slot back, since that is the fourth criterion and it deserves a real answer? Enough volume that the difference between good keyword data and excellent keyword data actually changes a decision. That is a real condition and I am nowhere near it, and I would rather say the number out loud than imply otherwise.

One flaw in the winner, because a directory that only says nice things is an ad. OpenSEO’s Google Analytics connection reports not-connected for both of my sites, so my real traffic numbers live somewhere else entirely. The panel is shaped exactly like the place my numbers should be. It is not that place. Search Console, which is what the numbers above came from, works fine.

That row and the others go up at charmthirteen.com/last-tool-standing, including the unflattering ones.

Strong engineering practices do not protect you from this, and there is now telemetry saying so.

Faros AI published its 2026 AI Engineering Report, drawn from two years of telemetry across 22,000 developers and more than 4,000 teams. Median time in code review is up 441.5%. Pull requests merged with no review at all, human or automated, are up 31.3%. Incidents per merged pull request are up 242.7%.

The part that made me sit up is a direct contradiction. DORA’s 2025 report concluded that strong engineering foundations offer protection against AI’s downsides, and I leaned on that framing in my own post on Saturday. Faros looked at teams with mature practices and high DORA scores and found them deteriorating at the same rate as everyone else. Their explanation is that surveys measure how developers feel and telemetry measures what shipped.

Faros sells engineering intelligence, so read the numbers with that in mind. Read the contradiction anyway.

Pick the one automated check you trust most. The deploy gate, the lint rule, the spam filter, the alert that would tell you a payment failed. Any of them.

Now go find out what it does when it cannot run.

Not what it does when it fails. What it does when it errors out, times out, hits a permissions problem, or silently skips the thing it was supposed to examine. If the answer is that it goes green, or that it stays quiet, you have a false pass sitting in your stack and you have been reading it as good news. Fix that before anything else on your list.

The fix is one rule. Every check reports one of three things, pass, fail, or did-not-run with a reason, and never two of them collapsed into one. The moment “did not run” can quietly render as “fine”, you can no longer tell a check that passed from a check that never executed. Unknown and safe are different words.

It takes about ten minutes and I have never once run it without finding something.

Drag the wrong-if line out of a task you have already described. It works on anything with a definition of done, not only code.

WHEN TO USE IT: before you hand a task to a contractor, an employee, or an agent. Before you approve work someone else did.

WHAT IT DOES: most definitions of done are satisfiable by something you would reject on sight, and you cannot see that from inside your own description. This makes the model argue for the bad version so you can look at it.

TIP: the step people skip is the “how would I find out” question, and it is the whole exercise. If your honest answer is that you would notice something felt off, you do not have a check, you have a hope. Push until the answer names a thing someone could actually go and look at.

— Lex

September 13. K-State’s Center for Advancement of Entrepreneurship is taking applications for its Fall 2026 Accelerator. Two tracks, and the constraint that decides it for most people is that you have to be Kansas-based. The growth track is the one worth your attention if you already have sales, since it runs online from September 29 to November 17 with an in-person closing week, takes ten entrepreneurs, and puts up to $50,000 in front of top performers. The idea track is six weeks in person in Manhattan, takes ten to fifteen people, and tops out at $10,000. Growth-track interviews are September 14, selections are announced September 23, and you have to confirm by September 25. Criteria and application: cba.k-state.edu/academics/departments/center-advancement-entrepreneurship/accelerator.html

September 18. Final-round proposals close for DeveloperWeek 2027, which runs February 9 to 11 at the Santa Clara Convention Center. Four parallel conferences share the process and you pick which one you are submitting to, including AI DevWorld. Two things worth knowing before you spend the evening on it. Every talk is live and in person, so budget the trip before you apply, and if you submitted to the first round in August and heard nothing, your proposal already carried over and you do not need to resubmit. Apply to speak: developerweek.com/speakers/apply-to-speak/

Tool verdicts, templates, and the reference material that does not belong in an email are all on the site at charmthirteen.com

I am speaking at the Kansas City Developer Conference, September 9 to 11.

What is the last green check you trusted without knowing what it actually looked at? Reply with the check, and I will tell you whether mine was worse.

Share the newsletter: artificiallydesigned.beehiiv.com