Bias and Fairness

Three places bias enters a feature you did not train

You called an API. You did not choose a training corpus, run a fine-tune, or set a loss function. It is tempting to conclude that bias is upstream and therefore not your problem.

NIST's framework identifies three categories to manage: systemic, computational and statistical, and human-cognitive. Mapped onto an application you did not train, they land in three different places, and two of them are yours.

Inherited from the model. Patterns in the training data, which you cannot inspect and cannot change. Real, and upstream.

Introduced by your system. Your prompt, your retrieval, your examples. A few-shot prompt whose examples all describe one kind of customer teaches the model what a typical customer looks like. A corpus that documents one product line thoroughly and another thinly produces confident answers about the first and vague ones about the second. This is yours entirely.

Created by the use. The same output used differently has different consequences. A ranked list shown as suggestions is not the same as a ranked list that decides who gets contacted. This is a product decision, and it decides how much the first two matter.

NIST also notes that bias is broader than demographic balance and that AI systems can increase the speed and scale of it. The practical reading: a small inconsistency in a feature that runs a million times is a different object from the same inconsistency in a feature that runs twice.

What you can measure from outside the model

Most fairness metrics you will find need labelled demographic data and known outcomes across a population. Most application teams have neither, and cannot ethically collect the first merely to run a metric.

What you can do without any of that is test for inconsistency: does the system treat equivalent inputs equivalently?

That is a narrower question than fairness, and it is answerable today with the same machinery you already use for evaluation. It will not tell you whether your feature produces just outcomes. It will tell you whether it produces different outcomes for inputs that should be the same, which is a defect regardless of which definition of fairness you hold.

Building matched inputs for your own feature

The method is pairs that differ in one attribute and nothing else.

Take a real input from your system. Copy it. Change one thing that should not affect the answer, and hold everything else fixed.

A: "Hi, I'm Aisha Rahman. My order hasn't arrived and I'd like a refund."
B: "Hi, I'm James Whitfield. My order hasn't arrived and I'd like a refund."

Run both, many times, and compare the outputs on whatever your feature decides: was a refund offered, what tone was used, was escalation suggested, how long was the reply.

Three things make this work rather than produce noise.

Vary one attribute at a time. Change the name and the phrasing together and you cannot attribute a difference to either.

Run each pair repeatedly. Output varies between identical calls, so a single pair proves nothing. You are comparing distributions, which is the same problem regression testing solves by comparing rates rather than single verdicts.

Test what the feature decides, not how it reads. If your system produces a structured decision, compare the decisions. Comparing prose is where disagreements about interpretation come in.

Conducting adversarial testing follows the same shape from a published taxonomy: derive the surface, build the cases, run them. This is that method with a different taxonomy.

Reading a difference without overclaiming

Suppose the refund rate differs between A and B by six points across two hundred runs. What have you learned?

You have learned that the system is inconsistent with respect to that attribute. That is a real finding and a defect worth fixing, because a decision should not depend on a name.

You have not learned the size of the effect in production, because your matched pairs are not your traffic. You have not learned that the system is unfair to a group, because one attribute in one feature is not a claim about outcomes. And you have not learned the cause, which may be the model, your examples, your retrieval, or all three.

Say what you measured. "Refund offers differed by six points between otherwise identical tickets when only the customer name changed, over two hundred runs per variant" is a sentence you can defend and act on. "The model is biased" is not, and it is less useful because it points at nothing you can change.

The number above is illustrative. Nothing here was measured.

What you can change downstream, and what you cannot

Once you have a finding, the options are the ones pre-trained models lays out, and they are limited in a specific way.

Your prompt and examples. If the few-shot examples skew, fix them. This is the cheapest change and often the effective one, because you wrote the skew.

Your retrieval. If the corpus covers some cases better, that is a content problem with a content fix, and it will show up as vagueness rather than as anything dramatic.

Where the decision sits. Move the consequential part into code. A model that drafts and a rule that decides is a different risk profile from a model that decides, and it is the same conclusion constraining inputs and outputs reaches for other reasons.

Which model. If the inconsistency survives everything above, it is upstream, and your remaining lever is choosing a different one and measuring again.

What you cannot do is fix the model's priors. Accepting that is what makes the rest of the list worth working through rather than a consolation prize.

Further reading

• NIST, AI Risk Management Framework: the three bias categories and the govern, map, measure, manage structure.
• Conducting adversarial testing: the same derive-and-run method against a security taxonomy.
• LLM evaluations: the machinery for running a case set repeatedly.
• Pre-trained models: which stage produced a behaviour, and what can reach it.