Blog / AI-SDLC
In AI-SDLC , Spec and Context Matter the Most , Not the Model
Vasanthavishnu Vasudevan
Chief Product Officer
01 · Executive summary - Why we ran this study
Teams everywhere have already integrated AI into their software lifecycle — autonomous agents, LLMs, copilots in the IDE. But the ROI is still hard to prove. Governance is difficult. So is measuring real productivity, reining in architectural drift, and keeping cost under control.
This study looks at what actually changes when leveraging Opsera’s Software Factory (Forge) within the SDLC, using data from real projects, through four lenses: architecture drift, productivity, governance, and savings. Rework per task fell from 3.2 tries to 1.3. Delivery time dropped 70-85% against a 10-hour/story-point baseline. Every work order stayed traceable through the RTM, with security and compliance checks gated at commit. Token spend per work order fell roughly 75%.
That pattern traces to one practice: giving every agent, regardless of which model runs it, a bounded work order with explicit scope, acceptance criteria, and the architecture already resolved. Plan the work up front, and drift, speed, governance, and cost move together in the right direction.
What is Forge? Forge is Opsera's AI software factory (softwareforge.ai): the first Intent & Context-aware Software Factory, not a faster IDE. It turns intent into living specifications, keeps context in one place across the whole build, and runs any model or agent against human-auditable Work Orders. Because the plan exists before the code, every output is aligned, governed, and production-ready by design. Zero drift. No rework. |
02 · Methodology - How the tests were run
First, the ground rules. Every number here is one of three things: measured in Forge, modeled from that data, or taken from an outside benchmark. We flag which is which up front, so you always know what you're looking at.
Measured in Forge (real)
937 Forge projects; per-language figures from representative initiatives sampled across them
Work-order counts, story points, risk tiers
10 hrs / story-point estimation baseline
Acceptance-criteria density (~5–11 / WO, avg ~8)
Native agent integrations & commit-time quality gate
Modeled or external (labeled)
Token-burn reduction (transparent model)
Time- & cost-savings % (modeled vs. baseline)
Model rankings (published benchmarks, e.g., SWE-bench)
Agent/model radar profiles (directional)
Important: Forge work orders record language, complexity, effort, and quality criteria — not which model or agent authored each line. Model/agent comparisons therefore rest on published benchmarks and Forge's real integration surface, and are labeled as such throughout. This separation keeps the operational claims grounded in data and the competitive claims clearly attributed. |
Why a 10-hour / story-point baseline?
Any savings figure is only as honest as what you compare it to. We use 10 human-hours per story point — not COCOMO II or cost-per-line — for three reasons. It is the number Forge itself generated (nearly every project estimates effort at story points × 10), so it was not imposed. Story points already fold in complexity and risk, and multiplying by 10 turns that into a familiar unit (about 1.25 engineer-days per point). And it treats every language the same, where a per-line baseline would favor some over others. One caveat, plainly: 10h/SP is a planning estimate, not measured time — so we treat it as a conservative, human-effort reference.
Baseline model | What it measures | Why we didn't lead with it |
|---|---|---|
10 hrs / story point (used here) | Complexity-weighted human effort, stack-neutral | — (Forge-native, uniform, reproducible) |
COCOMO II (effort from size + drivers) | Person-months for a whole system | Needs calibrated size/driver inputs Forge doesn't store; hard to reproduce per-project |
Cost per KLOC | Effort per 1,000 lines of code | Language-biased; Forge records no LOC — would require post-hoc measurement |
Function points | Effort per unit of delivered function | Requires manual FP counting; not captured in work orders |
% of full-rewrite cost | Modernization vs. rebuild-from-scratch | Useful for exec framing; we surface it in the scenario (§8) rather than as the base unit |
Bottom line: the savings percentages are anchored to a real, conservative, stack-neutral denominator. Swap in a richer modernization model and the absolute hours shift, but the relative pattern across languages holds.
Acceptance criteria — what done means
Forge plans the whole initiative before any code exists: intent becomes requirements, then an architecture, then an ordered list of Work Orders — each linked back to a requirement and each carrying clear acceptance criteria, the checkable "definition of done." That single artifact does four jobs, which are the four themes of this piece: it says what to build (drift), lets an agent check its own work (productivity), carries the compliance rules (governance), and cuts rework (savings). The Avg AC / WO column below shows how detailed that "done" list is.
Stack / language | Modernization pattern | Work orders | Story pts | Baseline hrs | Avg AC / WO |
|---|---|---|---|---|---|
Kotlin / Java + .NET 8 | .NET Framework + WCF monolith → cloud-native microservices | 169 | 1,428 | 14,280 | 11.3 |
Python / Vue 3 / JS | Legacy framework + jQuery → Vue 3 + hardened runtime | 103 | 660 | 6,600 | 8.9 |
Java (Spring/Hibernate) | In-place security & architecture hardening (healthcare) | 42 | 296 | 2,960 | 8.5 |
SQL · T-SQL → PL/pgSQL | Commercial DB → PostgreSQL (schemas + procedures) | 38 | 258 | 2,580 | 10.6 |
SQL · T-SQL → MySQL | Commercial DB → Aurora MySQL (DMS pipeline) | 26 | 168 | 1,680 | 10.4 |
Node/TS + React | Java servlets/JSP + MySQL → Node/TS + PostgreSQL + React | 43 | 186 | 1,860 | 6.6 |
TypeScript / React | JavaScript → typed React SPA | 33 | 117 | 1,170 | ~9 |
C# / .NET | .NET Core 2.1 (EOL) → .NET 8 API | 24 | 67 | 91* | 5.0 |
Swift / Kotlin / IaC | Greenfield multi-platform + Terraform + microservices | 29 | ~230 | ~2,300 | ~3 |
* One initiative used a tightened per-point estimate rather than the 10× baseline — included to show Forge adapts estimation to task granularity. All other rows use the standard 10 hrs/story-point baseline.
What is Avg AC / WO? The average number of explicit, checkable "definition-of-done" statements Forge attaches to each work order — e.g. "the endpoint rejects unauthenticated requests with 401" or "unit-test coverage is ≥ 90%." It is the key quality-density metric: more criteria = a tighter contract the agent must satisfy and a human can verify against. Across the sampled projects it runs ~5–11 criteria per work order (averaging about 8), rising on high-risk, correctness-critical tasks. |
We look at Forge through these four lenses because they are the questions every engineering leader is actually asking about AI in the SDLC. Drift asks whether autonomous agents stay true to the architecture. Productivity asks whether work actually ships faster. Governance asks whether that speed costs you security or compliance. Savings asks whether the token and licence bills pay off. Together they turn a vague "is AI worth it?" into four things you can measure — all driven by the same spec.
03 · Architecture drift — keeping agents on target
Left alone, an agent rediscovers the world every task — and drifts. A spec points it at a target.
OUTCOME — rework iterations per task: 3.2 → 1.3 — objective criteria cut retry loops and keep autonomous agents from drifting
Left to itself, an agent starts every task by rediscovering the world: grepping the repo, opening dozens of files, inferring the architecture, guessing requirements, then iterating when the guess is wrong. In agentic coding, this exploration-plus-rework loop — not code generation — is where most tokens, wall-clock, and money go. Forge changes the input: instead of a one-line prompt against an unknown repo, the agent receives a bounded work order with explicit in/out scope, ~8 acceptance criteria, the relevant RTM rows, and injected coding standards. The expensive discovery work is done once, deterministically, and reused by every agent on every task.
Take Devin, a fully autonomous coding agent that runs a task from start to finish with little human input. That independence is what makes it powerful — and what makes drift dangerous: given a vague goal, it can confidently build the wrong thing. So it is the sharpest test of whether a clear spec keeps an agent on track.
Which agents Forge drives
Forge is agent-agnostic: it integrates natively with IDE and CLI agents and autonomous runners, supplying each one the same context and enforcing the same definition of done. The table shows where each fits in a modernization workflow.
Agent | Category | Models it runs | Forge integration | Best-fit modernization role |
|---|---|---|---|---|
Claude Code | Agentic CLI | Claude Opus 4.8 / Sonnet 4.6 / Haiku 4.5 | Native hooks + MCP | Complex, multi-file refactors and migrations from the terminal. |
Cursor | AI IDE | Multi-model: Claude Opus 4.8 · GPT-5 · Gemini 3 | Native hooks + MCP | Interactive, human-steered execution of work orders. |
Windsurf (Cascade) | AI IDE | Multi-model: Claude Opus 4.8 · GPT-5 · Gemini 3 | Native hooks + MCP | Flow-based multi-file changes with inline review. |
Antigravity | Agent-first IDE (Google) | Gemini 3 (primary) · Claude Opus 4.8 · GPT-5 | Native + MCP | Autonomous, multi-surface execution against a Forge backlog. |
Codex | Agentic coding agent (OpenAI) | GPT-5 / GPT-5-Codex | via MCP | Autonomous cloud + CLI execution of scoped work orders. |
VS Code + Copilot | Inline + agent mode | Multi-model: GPT-5 · Claude Opus 4.8 · Gemini 3 | via MCP | Completion-assisted edits; smaller, well-scoped work orders. |
Devin | Autonomous cloud SWE (Cognition) | Proprietary orchestration over frontier models | via MCP work orders | Parallel, unattended execution of an independent work-order set. |
How Devin becomes dramatically more efficient on Forge
Without Forge
Open-ended goal, implicit scope
Agent infers requirements & architecture
Success is subjective → many review rounds
No traceability; regressions hard to attribute
With Forge
Bounded work order, explicit in/out scope
Architecture & RTM supplied as context
~8 acceptance criteria on avg = objective "done"
Quality gate + telemetry on every commit
Net effect: autonomy is aimed. Any autonomous agent executes a dependency-ordered backlog where each task is independently verifiable, so first-pass success rises, token/compute spend falls, and a human reviews against criteria instead of re-reading the diff cold.
Agent capability profile (illustrative, directional). All agents sit high on Forge-native context — that's the equalizer.
How to read it — Each spoke is a separate capability, scored from 0 at the center to 10 at the outer edge — the further a line reaches on a spoke, the stronger that agent is there. Read it one spoke at a time rather than by overall size; a bigger shape is not automatically better, since what matters is the mix that fits the job. Here, every agent reaches far on “Forge-native context,” which is the point: Forge is the equalizer.
The token cost of that rediscovery-and-rework loop is quantified under 06 · Savings.
04 · Productivity — plan once, run in parallel
Plan once, in detail. Then the work moves — across dozens to 100+ work orders no human could hold at once.
OUTCOME — time saved vs. the 10h/story-point baseline: modeled 70–85% reduction, largest where model maturity is highest
A single initiative routinely expands into dozens to 100+ work orders across many epics — one enterprise strangler-fig migration planned out to 169 work orders / 1,428 story points / 14,280 baseline hours, roughly 6.9 person-years of traditional effort. No human holds that plan in their head; no un-guided agent executes it coherently. The chart below shows how much breadth (work orders) and depth (story points) Forge plans per initiative — the raw material agents then execute in parallel.
Volume & complexity by stack — work-order count (bars) vs. story points (line).
A modernization walk-through (archetype, anonymized)
Consider a legacy .NET Framework + WCF/SOAP monolith on an aging commercial database, powering a regulated domain, that must become a cloud-native, event-driven microservices platform. Step 0: the team states intent; Forge generates PRD, BRD, architecture, and a dependency-ordered backlog with an RTM — nothing coded yet, but the multi-quarter plan is explicit and traceable. Step 1: high-capability models take the high-risk domain logic while cheaper, faster models take scaffolding, and an autonomous agent drains the independent backlog in parallel. Step 2: each agent gets a work order with in/out scope, ~11 acceptance criteria, its RTM rows, and injected standards, so it doesn't re-discover the architecture per task. Step 3: shipping agents generate CI/CD, wire security/quality scans, and canary-roll each service.
14,280h traditional baseline (~6.9 yrs) → ~2,100–4,300h modeled AI-assisted (70–85% ↓) · 100% of work orders traceable via RTM. Baseline hours are Forge-generated estimates (10h/SP); the AI-assisted range is modeled, not a measured outcome of this archetype.
The per-language time bars behind the 70–85% range are under 06 · Savings.
05 · Governance — security & compliance, built in
The rules ride along in the spec — gated at commit, not bolted on in a review at the end.
OUTCOME — 100% RTM traceability, gated SAST/SCA: every work order traced via RTM; security, dependency and secret scans gated in-pipeline. Compliance injected at spec
Work orders are risk-tagged (low / medium / high) and carry dense, checkable acceptance criteria. Higher acceptance-criteria density correlates with the tasks a human reviewer most wants an agent not to improvise on — security, data integrity, compliance. Regulated modernizations carry a heavy medium/high-risk load, exactly where AC density and human review matter most.
Risk profile across three regulated initiatives (stacked work-order counts by tier).
Shipped, tested, and observed
Shipping agents (CI/CD): detect or generate pipelines, trigger and monitor runs, canary-roll with automatic rollback — Day-0 code becomes Day-1 deployment without a hand-built pipeline.
Testing in the loop: the pre-commit gate auto-generates missing unit tests, runs the suite, and blocks the commit until every acceptance criterion is validated.
Security posture: SAST, dependency/SCA and container scanning, secret detection, and compliance checks gated in-pipeline; modernizations in the dataset carry explicit security work orders (injection remediation, audit logging, RBAC).
Quality posture: static analysis, coverage thresholds, and architecture-fitness rules so 100+ agent-written work orders converge on one consistent codebase.
Insights & the closed loop: dev-activity telemetry and DORA signals flow back on every commit; learnings feed forward into future specs.
06 · Savings — context and model economics
Context is cheaper than rediscovery. And no single model is cheapest at every job.
OUTCOME — cost per work order: ~75% modeled token reduction; realized cost savings vary by language and model choice
Two levers move cost. First, front-loading discovery once so agents stop re-paying the exploration tax on every task. In the model below, discovery and rework collapse (38→7 and 22→4 thousand tokens per work order) while code generation is roughly unchanged — a net ≈ 72k → 21k tokens/WO, about 71% fewer.
Modeled token spend per work order — unstructured prompt vs. Forge-supplied context. Illustrative model, not Forge telemetry.
Honesty note. The token figures are a transparent model, not Forge telemetry. What is measured is the structure that drives the model: bounded scope, dependency graphs, and ~8 acceptance criteria per work order. The mechanism — front-loading discovery so agents don't re-pay it — is architectural, and the direction of the effect is not in question. |
The second lever is matching the model to the job. Savings are not the same across languages: well-supported ones like TypeScript and Python compress the most, while tricky, high-stakes work (T-SQL → PL/pgSQL, regulated Java) needs more human review and saves less — though still a lot. That gap is the case for staying model-agnostic. Cost savings run a few points below time savings everywhere, since the model subscription and tokens eat into part of the labor saved.
Modeled time-savings % and cost-savings % by language, vs. the 10h/SP baseline.
Benchmarks — which model for which job
Forge is model-agnostic: it produces the spec, you choose the engine. The guidance below combines published coding benchmarks (e.g., SWE-bench Verified) with stack-fit patterns observed across the dataset. Benchmark standings are external and move quickly — treat this as a decision framework, not a leaderboard.
Model | Model ID | Context | $ / 1M in | $ / 1M out | Best for |
|---|---|---|---|---|---|
Claude Fable 5 | claude-fable-5 | 1M | $10 | $50 | Hardest long-horizon, autonomous modernization runs |
Claude Opus 4.8 | claude-opus-4-8 | 1M | $5 | $25 | High-risk domain logic, complex multi-file refactors (default) |
Claude Sonnet 4.6 | claude-sonnet-4-6 | 1M | $3 | $15 | High-volume execution, balanced cost/quality |
Claude Haiku 4.5 | claude-haiku-4-5 | 200K | $1 | $5 | Scaffolding, boilerplate, classification, speed-critical steps |
Language / stack | Primary recommendation | Strong alternative | Why |
|---|---|---|---|
TypeScript / JavaScript / React | Claude Opus 4.8 / Sonnet 4.6 | GPT-5-Codex | Highest training-data density; excellent multi-file refactors and tool use. |
Python | Claude Opus 4.8 · GPT-5 | Gemini 3 (large ctx) | Both frontier families are near-ceiling; pick on price/latency. |
Java / Spring (enterprise) | Claude Opus 4.8 | Gemini 3 (huge repos) | Strong on long, verbose enterprise code and framework idioms. |
Kotlin / Android / KMP | Claude Opus 4.8 / Sonnet 4.6 | GPT-5 | Good idiomatic Kotlin + coroutines; reliable multiplatform reasoning. |
C# / .NET | Claude Opus 4.8 · GPT-5 | — | Mature ecosystem coverage; strong on hosting/DI migrations. |
Swift / iOS | Claude Opus 4.8 · GPT-5 | — | Solid SwiftUI; verify against latest SDK changes. |
SQL dialects · T-SQL → PL/pgSQL | Claude Fable 5 / Opus 4.8 | GPT-5 | Semantics-preserving procedure translation is high-stakes — use the top tier; still review-heavy. |
Large multi-repo / legacy sprawl | Claude Fable 5 | Gemini 3 (context) | 1M-token context + long-horizon reasoning hold cross-repo state. |
Infrastructure-as-Code (Terraform) | Claude Opus 4.8 · Gemini 3 | GPT-5 | Good HCL + provider knowledge; pair with policy scanning. |
Model suitability by dimension (illustrative, directional). Read per-axis, not by area.
How to read it — Same idea: each spoke is one capability, scored 0 at the center to 10 at the edge, and a model's line reaches further out where it is stronger. Compare the models spoke by spoke — the aim is the best fit for a given task, not the largest overall shape.
Because Forge is the spec layer, model choice is a runtime decision, not a rewrite. Teams routinely run a high-capability model on high-risk work orders and a cheaper, faster model on low-risk scaffolding — the acceptance criteria are identical either way.
|
07 · Key takeaways
Across all four lenses the pattern is the same: put the plan and the context up front, then let any model or agent execute against a clear, checkable spec. That is what turns modernization into a measurable lifecycle.
Architecture drift — Clear, checkable Work Orders keep agents on target instead of wandering. Modeled rework drops from 3.2 to 1.3 tries per task (about 60% less).
Productivity — Planning the whole initiative up front lets work run in parallel across 100+ Work Orders. Modeled delivery time drops 70–85% against the 10-hour/story-point baseline.
Governance — Security and compliance ride inside the spec and are gated at commit, not bolted on later. Every Work Order stays traceable, with SAST/SCA and secret scans in-pipeline.
Savings — Setting up context once instead of every prompt cuts modeled token spend about 75% per task, and staying model-agnostic lets you pick the cheapest capable model for each language.
The through-line: one spec, set before the code, drives all four — aligned, governed, and production-ready by design. Zero drift, no rework.
The percentage figures above are modeled estimates, not measured telemetry, and are labeled as such throughout.
Next steps
Get started building or modernizing using Opseras Software Factory. Learn more at app.softwareforge.ai. Learn more about the Opsera offering at Opsera.ai
