Senior Engineer vs. 6 AI Agents: A Real-World Enterprise Codebase Experiment

This blog is based on an article originally written in February 2026, so some information may now be outdated.

There’s no shortage of flashy demos of AI agents spinning up apps from a single prompt. But there’s a massive gulf between generating a standalone landing page and shipping a tightly integrated, type-safe feature inside a massive enterprise codebase.

To see how modern agents handle real-world engineering constraints, I ran a rigorous head-to-head experiment. I pitted myself, a Senior Developer, against 6 AI-agent implementations (Claude Max, GPT, Claude AWS, Kimi, MiniMax, and Mistral).

The assignment was to add a user management page (/ai/team) based on a highly aspirational UX mockup. To test how the agents navigated real-world obstacles, the backend legacy API was left with massive blank spots, including a user role property that was a generic string with no available options.

The Environment & The Stack

I wanted to eliminate the “it works on my machine” factor and force the models to reason about modern, strict TypeScript architectures.

• The Core: React 19, TanStack Query (React Query), Effect, Waku, @base-ui/react, and Tailwind CSS.
• The Rules: Ultra-strict TypeScript configs, specifically enabling strict: true, noUncheckedIndexedAccess: true, and exactOptionalPropertyTypes: true.
• The Data Layer: Automated TypeScript types generated directly from OpenAPI and GraphQL schemas.

First, I established the baseline or “gold standard”, taking 14 hours of net development time to implement the page flawlessly, complete with idiomatic state management and robust error handling. Then, I handed the exact same repository and prompt to the agents.

The Raw Metrics

On paper, the agents’ initial outputs looked impressive. They generated thousands of lines of code, wrote End-to-End (E2E) specs, and executed for pennies in compute costs.

But as any developer who has inherited a legacy codebase knows, lines of code added are often a liability rather than an asset. When I opened the pull requests, I found major architectural divergence.

Where the Agents Diverged from Idiomatic Code

While almost all the agents managed to render a UI that visually worked, their underlying architectural choices fell into a few classic patterns of technical debt.

1. The Form Handling Trap: Neighboring Code Beats Documentation

I had placed extensive, highly detailed markdown documentation in the repository outlining coding standards, including explicit instructions to use react-hook-form and Zod for forms. Every single agent ignored those written instructions. They all defaulted to standard LLM training data patterns: tracking form inputs via manual useState strings and validating emails with fragile, copy-pasted regular expressions.

Yet all the models successfully built complex Effect-based GraphQL resolvers. Why? Because examples of those resolvers already existed in neighboring files. The core takeaway is that written documentation is far less important to an AI than the code it runs alongside. Models will ignore your markdown guidelines, but they will perfectly replicate the architectural patterns they see in nearby files.

2. Optimistic UI: Syncing Cache vs. “Faking It”

When a user updates a teammate’s role, the UI should ideally update instantly, rolling back only if the API call fails. I observed two radically different architectural patterns emerge here:

• The Right Way: My self, Claude Max, Claude AWS, and Kimi, used TanStack Query’s cache properly. We manipulated onMutate, took a cache snapshot, updated via setQueryData, and handled the rollback on error. Claude Max even abstracted this into a highly clean, reusable useOptimisticMutation hook.
• The Shortcuts: GPT, MiniMax, and Mistral bypassed the cache entirely, tracking the mutation state locally in the component. Worse, MiniMax and Mistral literally hardcoded setTimeout(() => {}, 1000) blocks to simulate network latency instead of integrating with the actual promise lifecycle.

3. Type Drift and Schema Mismatches

With automated OpenAPI code generation, you should inherit types, not write them. I didn’t manually declare a single type that already existed in the schema. The agents, however, struggled with strict types:

• GPT manually re-declared MemberRole and TeamMember types, but changed the data shapes, such as defining an ID as a string when the backend schema explicitly demanded a number.
• Kimi introduced a silent runtime bug by declaring a user role as an all-caps string ('ADMIN'), while the backend code generator expected PascalCase ('Admin').

The Remediation Tax: Calculating True TCO

Evaluating an AI agent solely on token cost misses the point. The metric that actually matters for engineering teams is Remediation Time: How long does it take a senior dev to refactor the agent’s code to meet production standards?

Assuming a standard engineering cost of $100/hour, I calculated the Total Cost of Ownership for each run. This includes the initial token costs, the time I spent nudging the agent, and the time required to rip out bad abstractions, like fake timeouts and manual state tracking.

The Breakdown:

• The Outlier: Claude Max was incredibly efficient. For a raw token cost of $4 and about 2.5 hours of human cleanup, mostly removing some minor feature creep, it yielded an 80% net savings. Its component separation was clean, and it understood the data layer.
• The Monoliths: GPT and Claude AWS saved some time but required heavy refactoring. GPT, for example, dumped 1,045 lines of code into a single, massive file that had to be manually decomposed.
• The Net-Negatives: MiniMax and Mistral actually cost more than writing the feature from scratch. Because they faked the API mutations with local states and timeouts, a senior engineer would have to throw away their entire data-fetching architecture and rebuild it properly.

The Takeaway

Agents aren’t taking senior engineering jobs anytime soon, mostly because they struggle with contextual awareness. They don’t look at your package.json to see how you handle forms, and they don’t look at your existing patterns to see how you handle state. They write the most generic version of the code requested, which often violates strict linting and architectural rules.

However, my experiment shows that if you establish a highly rigid baseline with strict TypeScript flags, automated code generation, and tight linting constraints, the top-tier models like Claude Max have fewer paths to wander off-track.

The real job of a senior engineer right now is not to write endless text prompts or massive documentation files. The job is to craft an impeccable, rigid initial scaffold. If you give the models a high-quality pattern in the code itself, they will replicate it endlessly.

Try it Yourself

Want to run agents against your own stack, but keep model access controlled?

The amazee.ai AI Gateway provides teams with a single integration point for multiple frontier models while keeping usage private and under governance.

• Multi-model routing behind a single endpoint, so you can evaluate and swap models without refactoring your toolchain.
• Policy-controlled data flow for prompts, code, and artifacts so sensitive context stays protected.
• Centralized access control and auditability to prevent model experimentation from turning into unmanaged shadow tooling.

If you are standardizing secure access to frontier models for engineering workflows, get your key now.