Build log

Catch Up, Gotcha, Recycle

A month lost to ungovernable models, a Codex farewell, and why I am benchmarking Devin and Factory back to back before I trust either of them with a production Rust codebase.

Big inference is playing a game right now and the cycle goes like this.

One month OpenAI is ahead. The next month Anthropic closes the gap or opens one of their own. Someone releases a benchmark, someone else disputes it, a new model drops and the leaderboard reshuffles again. Catch up, gotcha, recycle. If you are building on top of any of this, you are not building on a foundation. You are building on a conveyor belt.

I lost a month to it.

The Codex Problem Was Not Codex

I want to be precise about this because the distinction matters.

The harness was not the problem. Codex as an environment for agentic engineering is reasonable. The problem was the model layer underneath it, specifically what happened when the combination of Astra and the 6 Sol model landed in a Rust codebase with any meaningful complexity.

I had hard safeguards in place. Anti-drift measures. Explicit constraints designed to keep the agent within the boundaries of what I actually needed done. None of it held. The models became, and I am choosing this word carefully, ungovernable. Not occasionally wrong. Not producing suboptimal output that needed review. Actively doing more harm than good, in ways that required more time to remediate than the original work would have taken without any agentic assistance at all.

The 5.6 model had been working well with Rust. The combination that followed it was catastrophic enough that I mothballed several active projects and redirected to building essential internal tooling for Panamorphix, work where the blast radius of a misbehaving model was contained and the SQL complexity that tends to cause the most damage simply does not exist.

So today, with some sadness, I am leaving Codex behind. Not because the harness failed. Because the model it is running on right now is not the model I need for the work I do.

The Want Your Cake Problem

Here is the underlying issue that no single harness solves cleanly.

I want the best model available at any given moment. Not the best model that was available when I chose my tooling. The best one right now, today, this week, which is a moving target that the inference providers themselves cannot predict with any certainty.

The harness locks you in at some level. Either to a provider, or to a model family, or to an integration pattern that makes switching expensive enough that you end up running a model that has been superseded because the cost of migration is higher than the cost of the degraded output. That is the gotcha. You chose well six months ago and the choice is now working against you.

The honest answer is that there is no harness that fully solves this yet. The best you can do is choose a harness with the most model flexibility and the clearest path to switching when the model layer moves, which it will, probably before the end of the quarter.

Devin and Factory

I have been looking at both seriously.

Devin has genuine strengths for my workflow. The on-call model fits how I actually work, which is not always at a desk, not always in a single focused session, often across multiple parallel workstreams that need to be picked up and put down on short notice. The mobile app is a real win for that pattern. Factory has a PWA which is, to be direct about it, not the same thing.

Where Factory pulls ahead is granularity of model choice. The ability to be specific about which model runs against which task matters enormously when the model layer is as volatile as it currently is. If I can point different tasks at different models based on what I know about each model's current strengths, I am less exposed to the conveyor belt problem. I am not locked to the inference provider's current best guess about what I should be running.

Replit, for what it is worth, had a class-leading mobile experience when I used it for web work about a year ago. Neither Devin nor Factory is quite there yet, though Devin is closer.

I am torn. Both have real limitations alongside the genuine strengths.

The Only Honest Answer

I am going to run Devin for a month on real production workloads, then run Factory for a month on the same class of work, and benchmark the findings properly. Not synthetic tests. Not the tasks each tool markets itself against. The actual work: governed Rust codebases, complex graph logic, multi-agent coordination across a portfolio of products that do not forgive drift.

Then I will report back here with what I actually found.

That is the only honest answer when the marketing for both is optimistic and the reality is unknown until you are inside it. A month of real use tells you more than any comparison post written by someone who has not committed production code through either of them.

The catch up, gotcha, recycle cycle keeps running. The only way to stay ahead of it is to keep testing, keep switching when the evidence says switch, and never confuse the harness with the model underneath it.

They are not the same thing. That distinction cost me a month to learn properly.