Your workflow runs on a model you can't pin

Most teams pin every dependency except the one they use all day.

Almost every project I've worked on in the last twenty years has had a lock file, because we all learned (usually the hard way) that an upstream change you didn't ask for can break a build that worked fine yesterday. So we pin versions, we read changelogs, and we upgrade when we decide to, not when someone else does.

Then we opened an AI coding tool and handed the most important dependency in our working day to someone else, without any version number on it.

The dependency that moves on its own

Fortune reported on Anthropic's explanation for the weeks this spring when Claude Code got worse, after users had complained for a while. There were three separate changes behind it.

On March 4, the default reasoning effort went from high to medium to cut latency, and Anthropic later said that had been the wrong tradeoff. From March 26, a bug made the model keep discarding its own reasoning history partway through a session, so it came across as forgetful and erratic, and usage limits drained faster than people expected. On April 16, an instruction in the system prompt started capping responses at 25 words between tool calls, and it was reverted four days later because it measurably hurt coding quality.

Anthropic said all three were resolved as of April 20, and it reset usage limits for all subscribers on April 23. None of these changes is exotic, a default got changed, a bug shipped, a prompt was edited, and that happens to all software.

The difference is that nothing in my repository changed. There was no diff, no version bump and no failing test, and the only signal I had was a vague sense that the tool was having a bad week. I'm not picking on Anthropic, they explained what went wrong, which is more than most vendors do. But the behavior I depend on lives on someone else's servers and changes on someone else's schedule.

Upgrades break things too. Steve Yegge built Gas Town, an elaborate setup for running coding agents, and wrote about how it ended in a piece quoted by Simon Willison. He says it worked brilliantly up through Opus 4.6 and then fell apart with Opus 4.7, because the new model picked up what he calls a "just two more things" tic. It never converged on being ready to do real work and always wanted to go back and fiddle with Gas Town itself. He also admits that the system was meant to be reusable, but he only ever used it to build itself.

So a newer model broke a working system, not because it was worse but because it was different. Anyone who has upgraded a framework and discovered that some behavior they relied on was never really part of the contract knows this. With models there is hardly a contract at all, so almost everything you rely on is in that category.

The third kind of change has nothing to do with the model. In June, TechCrunch reported that SpaceX had agreed to acquire Cursor in a $60 billion stock deal, which SpaceX said was likely to close in the third quarter. I have no view on whether that's good for Cursor users, I'm only noting that an editor plenty of developers spend their whole day in is set to belong to a rocket company, and none of those developers got a say.

Pricing moves with ownership and strategy as well. The same Fortune piece mentions that The Information reported Anthropic moving enterprise pricing to a consumption-based model, which one analyst estimated could potentially triple costs for heavy users. That is a dependency change too, it just shows up on the invoice instead of in the output.

What I do about it

I'm not saying anyone should stop using these tools, I use them. But treat them the way you treat any other dependency you can't control, i.e. assume they will change and make sure you notice when they do. For me that means three habits.

First, I keep a handful of saved tasks and rerun them after any change. Real tasks from my own codebase with an answer I know is good, five or ten of them, are enough. When a model version changes, a tool updates, or the output feels off, I run them and compare. It's a small eval, not a benchmark, but it turns "it feels worse this week" into something I can check. Don't waste time arguing about feelings when you can rerun ten tasks.

Second, I keep prompts, instructions and harness config in the repo. If the behavior of your setup depends on a system prompt, then that prompt is code. It needs a history and a diff, so you can at least rule out your own side.

Third, I try not to build anything that only works with one model. If a workflow falls over when you switch versions or providers, you've found the fragile part, and it's much better to find it on a quiet afternoon than during the week a model is quietly degraded.

None of this is new. It's what we already do with libraries, databases, cloud APIs, etc., we just haven't applied it to the part of the stack that now writes a lot of our code. It's also why we build Workbricks apps as visible steps instead of one long prompt. When the model underneath changes, you can retest one step instead of guessing at the whole thing.

The tool is going to change either way. Pin what you can, test what you can't, and make sure you're the first to find out when it does.