Impact on Product Development

Nondeterminism, the property that changes everything downstream

Traditional software is a function. Same input, same output, every time. Nearly everything about how teams build, test, price and ship software rests on that, usually without anyone noticing it is an assumption.

A model-backed feature is not a function. The same input produces different output, and not as a defect. That is the mechanism working. Even at temperature zero, in a vendor's own words, "the results will not be fully deterministic." Sampling parameters explains why pinning the knob does not pin the output.

Correctness becomes a distribution instead of a value, and once it is a distribution, testing, cost, failure handling and planning all change shape.

Testing when the same input gives different answers

expect(output).toBe(expected) does not work. That is not a small inconvenience. It is the assertion most test suites are built out of.

What replaces it is a different kind of check, and a different grain. Instead of asserting the output, you assert properties of it: the JSON parses against the schema, the cited document is the right one, the refund amount is 40, the reply declines to give medical advice. Instead of a verdict per run, you get a pass rate over a set of cases, and you compare rates rather than strings. Deterministic evals covers writing the checks and regression testing covers comparing runs.

The organizational consequence is larger than the technical one. In traditional development, the test suite is a gate a developer can run alone in thirty seconds. Here the equivalent is slower, costs money each time, and produces a number that needs interpreting rather than a green tick.

That changes who is involved. Deciding whether 84 percent is acceptable is a product judgment, not an engineering one, and it recurs on every change. Teams that do not plan for this end up with an engineer quietly making product decisions in a pull request, or with nobody making them at all.

Cost becomes a design decision

In conventional software, the cost of a feature is roughly the cost of building it. Serving it is a rounding error, and the engineer writing the endpoint does not think about unit economics.

Here it is per request, it is visible, and it is set by design choices rather than by an infrastructure setting chosen later.

Prices are per million tokens and split between input and output. Claude Opus 5 is listed at $5 per million input tokens and $25 per million output tokens (checked 2026-09-16), so each output token costs five times an input token. So the bill depends on how much context you send and how long an answer you ask for, and both of those are decided by whoever designs the feature. Retrieving ten documents instead of three is a product decision with a line-item consequence. So is "summarize in a paragraph" versus "summarize in a page."

So "what does this cost per user per month" becomes a question to answer during design, at a rough order of magnitude, before building. It is unfamiliar and it is not optional: a feature whose unit cost exceeds what the user is worth does not become viable later.

The mitigations are design decisions too. Batch processing is documented at 50 percent off; prompt caching cuts the cost of a reused prefix substantially. Both shape the architecture, which means both have to be considered early rather than applied as an optimization afterward.

Model latency you cannot optimize away

A model call takes seconds. Not milliseconds you can profile away, but seconds, set largely by how many tokens are generated, on someone else's hardware.

You can pick a faster model, send less context, ask for shorter output, or run calls concurrently. What you cannot do is make it fast the way you would make a database query fast, and the floor is high enough to change the interface.

Which is the point: this stops being an engineering concern and becomes a design one. A three-second wait needs a designed waiting state, not a spinner. Streaming responses exists because showing the answer as it arrives changes the perceived wait more than any optimization would. A multi-step agent taking thirty seconds is not a slow page; it is a different interaction, and it needs a different screen.

Designers and engineers have to have this conversation early, because the honest answer, "this takes four seconds and always will", changes what gets designed. Inference separates the latencies worth measuring separately.

Designing for a confident wrong answer

Conventional failures announce themselves. An exception, a 500, a timeout, something to catch and something to log.

The characteristic failure here announces nothing. The model returns a fluent, well-formatted, confident answer that is wrong. It is the same shape as a correct answer. There is no exception, no status code, and no signal your code can branch on. In an agent, the same property compounds: Anthropic's guidance on autonomous loops warns of "the potential for compounding errors", where each step builds on the last one's mistake.

So the failure has to be handled in the design rather than in a catch block.

Make the answer checkable. Show sources, quote the passage, link the record. A user who can see where an answer came from can catch what your code cannot. This is the strongest argument for retrieval over the model's own knowledge, and it is a product argument rather than a technical one.

Put a person where the stakes are. Draft, do not send. Suggest, do not execute. Review, do not commit. The tasks where confident-and-wrong is catastrophic are the tasks where a review step belongs, and deciding which those are is a product decision made at design time.

Design the uncertain state. "I could not find this in your documents" is a valid outcome, and the interface has to have somewhere to put it. Applications without a designed uncertain state force the model to answer anyway, and it will.

Your feature inherits the model's retirement date

The one nobody schedules.

Every model ID is "a pinned snapshot", and vendors publish retirement dates for them. One current model's commitment is "Not sooner than October 15, 2026". Your feature depends on an artifact with an end date attached.

No conventional dependency behaves like this. A library you pinned keeps working indefinitely; upgrading is optional and you do it when convenient. Here the upgrade is scheduled by someone else, and the replacement is not a drop-in: a newer model is better on average and different in specifics, which means prompts tuned against the old one need re-verification.

So model migration is recurring maintenance and belongs on a roadmap. The concrete thing to do about it is the cheap one: keep the saved cases from regression testing, and keep the model identifier in configuration rather than scattered through the codebase. Then a migration is a measurement. Without them it is a guess shipped to production.

What changes for people who are not engineers

This is not only an engineering shift, and the teams that struggle are usually the ones that treated it as one.

Product specifies acceptable behavior over a distribution of inputs rather than defining correct output. That means supplying examples, the twenty real cases that define what good looks like, which is a different and more demanding artifact than a requirements document.

Design designs waiting, uncertainty, correction and review as first-class states rather than edge cases, because all four are on the common path.

Support fields reports of a kind that do not reproduce. "It gave me a wrong answer" with no reliable way to see it again requires tracing, so that the actual request can be retrieved and turned into a case.

Legal and compliance get involved earlier, because what the model says is attributable to you, and because prompts leaving your infrastructure is a data question before it is a technical one.

The common thread: several things that used to be engineering details (cost per request, failure handling, what "done" means) became decisions the whole team has to make. That is the actual impact on product development, and it is more about where decisions get made than about which model you picked.

Further reading

• Anthropic, Messages API reference: the temperature-zero caveat that starts this whole chain.
• Anthropic, Models overview: the pricing split and the retirement commitments.
• Anthropic, Building effective agents: compounding errors in autonomous loops.
• What is an AI engineer?: the role itself, rather than the process around it.