Pin Your AI Models Like You Pin Your Dependencies

Last Tuesday a workflow that had been quietly running for six weeks started returning JSON with an extra field. No code changed. The model name in the request was claude-sonnet-4. The pinned alias rolled forward from 4.5 to 4.6 and the new version preferred to include a confidence key whether you asked for it or not.

That is a tiny example of a category. Production AI calls drift because the model identifier you wrote three months ago does not point at the same weights anymore. Some vendors version explicitly. Some advance silently under "latest". Some quietly retire endpoints with 30 days notice posted on a status page nobody reads.

You already pin react, axios, and next to exact versions in your lockfile. AI models deserve the same discipline. Here is the playbook we run on every production project.

TL;DR

| Practice | Stops | Cost | | --------------------------------------- | -------------------------------------- | ------ | | Use dated model IDs, never aliases | Silent rollouts breaking JSON shape | 5 min | | Log model + version + cost per call | Surprise bills, drift detection | 30 min | | Wrap calls with a fallback chain | Provider outages, deprecated endpoints | 1 hour | | Snapshot test the prompt response shape | Output schema regressions | 1 hour | | Set hard cost ceilings per route | Runaway agent loops | 20 min |

1. Use the dated model ID, always

Most providers publish two kinds of model IDs:

| Type | Example | What you get | | ----- | ---------------------------- | ------------------------------------------------- | | Alias | claude-sonnet-4 | Whatever Anthropic decides "Sonnet 4" means today | | Dated | claude-sonnet-4-6-20250929 | The exact weights from that release |

The alias is convenient. It is also the source of every "it worked yesterday" Slack message in production AI work. Use the dated form everywhere except prototypes.

// don't
const MODEL = "claude-sonnet-4";

// do
const MODEL = "claude-sonnet-4-6-20250929";

OpenAI does the same with snapshots: gpt-5-mini-2026-04-15 versus gpt-5-mini. Google has gemini-2.5-pro-001. The pattern is consistent. The vendors all publish the snapshot IDs. Most tutorials still use the aliases because they read better in marketing copy.

Put the model ID in a single config file:

// src/config/models.ts
export const MODELS = {
  classifier: "claude-haiku-4-5-20251001",
  coder: "claude-sonnet-4-6-20250929",
  planner: "claude-opus-4-7-20260201",
  embedder: "text-embedding-3-large",
} as const;

export type ModelRole = keyof typeof MODELS;

Now upgrades are a single PR with a single line diff and a clear blast radius.

2. Log model, version, and cost on every call

You cannot detect drift if you cannot see it. Wrap every call so the response always carries { model, inputTokens, outputTokens, costUsd } and log it.

// src/lib/ai-call.ts
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

const PRICING: Record<string, { in: number; out: number }> = {
  "claude-haiku-4-5-20251001": { in: 0.8, out: 4 },
  "claude-sonnet-4-6-20250929": { in: 3, out: 15 },
  "claude-opus-4-7-20260201": { in: 5, out: 25 },
};

export async function call(model: string, prompt: string, maxTokens = 4096) {
  const start = Date.now();
  const r = await client.messages.create({
    model,
    max_tokens: maxTokens,
    messages: [{ role: "user", content: prompt }],
  });
  const p = PRICING[model] ?? { in: 0, out: 0 };
  const cost = (r.usage.input_tokens * p.in + r.usage.output_tokens * p.out) / 1_000_000;
  const result = {
    text: r.content[0]?.type === "text" ? r.content[0].text : "",
    model,
    inputTokens: r.usage.input_tokens,
    outputTokens: r.usage.output_tokens,
    costUsd: cost,
    latencyMs: Date.now() - start,
  };
  console.log(JSON.stringify({ event: "ai_call", ...result, prompt_chars: prompt.length }));
  return result;
}

Pipe those JSON logs to whatever you already use for observability. After a week of data you have a baseline. After a month, you can spot a drift the day it happens because the cost per call or the output token count shifts off the baseline distribution.

The cost logging is also the only honest way to see what AI features actually cost you in production. Vendor dashboards aggregate across the whole org. Per route logging tells you that the search autocomplete is 60% of your inference bill, which is the number that funds the conversation about whether to rip it out.

3. Build a real fallback chain

Anthropic had a 4 hour outage on April 9. OpenAI's GPT-4o endpoint was throttled to single digit RPS for two hours on April 17. If your product depends on a single model from a single vendor, your uptime is bounded by their worst day.

A fallback chain is not failover engineering. It is one function:

// src/lib/ai-fallback.ts
import { call } from "./ai-call";

interface FallbackOptions {
  primary: string;
  fallbacks: string[];
  timeoutMs?: number;
}

export async function callWithFallback(prompt: string, opts: FallbackOptions) {
  const chain = [opts.primary, ...opts.fallbacks];
  let lastErr: unknown;
  for (const model of chain) {
    try {
      return await Promise.race([
        call(model, prompt),
        new Promise((_, reject) =>
          setTimeout(() => reject(new Error("timeout")), opts.timeoutMs ?? 20_000),
        ),
      ]);
    } catch (e) {
      lastErr = e;
      console.warn(`[ai] ${model} failed, trying next:`, e);
    }
  }
  throw lastErr ?? new Error("all models failed");
}

Use it like this:

const result = await callWithFallback(prompt, {
  primary: MODELS.coder, // sonnet 4.6
  fallbacks: ["claude-opus-4-7-20260201"], // pricier but available
  timeoutMs: 30_000,
});

The fallback model deliberately does not match the primary for one or both of cost and output shape. That is fine. The point is that the request returns successfully and you handle the slight quality difference downstream, instead of returning a 500.

4. Snapshot test your prompt response shape

When the model rolls and the JSON shape shifts, you want CI to find out, not your error tracker.

// __tests__/ai-shape.test.ts
import { describe, expect, it } from "vitest";
import { call } from "../src/lib/ai-call";
import { MODELS } from "../src/config/models";

describe("classifier output shape", () => {
  it("returns the keys we depend on", async () => {
    const r = await call(
      MODELS.classifier,
      'Classify the following as one of [bug, feature, question]: "the button does not work". Return JSON: {"category": ...}',
    );
    const parsed = JSON.parse(r.text);
    expect(parsed).toHaveProperty("category");
    expect(["bug", "feature", "question"]).toContain(parsed.category);
    expect(Object.keys(parsed)).toHaveLength(1);
  });
});

Run this on a schedule, not just on CI. A nightly job that hits every pinned model with a known prompt and asserts the shape gives you a 24 hour heads up on drift, even when the model ID has not changed. Vendors patch in place sometimes.

5. Hard cost ceilings per route

Agent loops run away. A retry on a tool that returns garbage can spin a model into 40 calls before something stops it. Put a ceiling at the route level, not just the global account level.

// src/lib/ai-budget.ts
const BUDGETS_USD: Record<string, number> = {
  "search-autocomplete": 0.001,
  "code-review-agent": 0.5,
  "doc-summary": 0.05,
};

export class BudgetExceededError extends Error {}

export function enforceBudget(route: string, costUsd: number) {
  const max = BUDGETS_USD[route];
  if (max && costUsd > max) {
    throw new BudgetExceededError(
      `route ${route} exceeded budget: $${costUsd.toFixed(4)} > $${max}`,
    );
  }
}

Wire it at the call site. Throw the error, catch it at the request boundary, log it loudly, and return a degraded response. The first time this fires in production you will discover an agent loop you did not know you had. That is the value.

The rolled model story is real

The opening anecdote is real. An internal article quality scorer started returning a confidence field after Anthropic rolled the alias. Three downstream consumers parsed strict JSON without additionalProperties: false, so nothing crashed, but the dashboard started showing a metric that nobody had wired in. We chased the bug for an afternoon before someone checked the model ID and found it was an alias.

We pinned everything to dated IDs the next morning. The lock file equivalent is now a models.ts constant. We update it on a schedule, not on a vendor's schedule.

The bottom line

Pin the version. Log the cost. Wrap with a fallback. Snapshot the shape. Cap the budget. Five practices, none of them clever, all of them missing from most AI codebases we see.

The reason they are missing is the same reason your first Node project did not use a lockfile: nothing burned yet. The fix is also the same: write the playbook once, copy it to every new project, and never think about it again until you need to upgrade.

What is the oldest model ID in your production code right now? If you do not know, that is the first hour to spend.