AI Safety and Ethics

The harms are yours even though the model is not

You did not train the model, and you are still the one shipping the feature. When it gives a wrong medical answer, refuses one group of users more often, or persuades someone it is a person, the complaint arrives at your product.

This page is the frame around the specific practices. Four articles here cover the mechanics: bias and fairness for measuring differential treatment, content moderation APIs for classifying content, conducting adversarial testing for finding failures on purpose, and prompt injection for the attack. What follows is the decisions none of them make for you.

A structure to hang it on

NIST's AI Risk Management Framework organises this work into four functions: Govern, Map, Measure, and Manage. It is voluntary, and the reason to borrow it is that the order is right.

Govern is who decides and who signs off. Map is working out what could go wrong in your specific feature. Measure is producing numbers for those things. Manage is what you do about them.

Most teams start at Measure, because it looks like engineering. Mapping first is cheaper: you find out which harms are plausible for what you are building before you build instruments for harms that are not.

Four decisions no library makes for you

Does the user know they are talking to software? Disclosure is a product decision with regulatory weight in some jurisdictions. The failure mode is not lying; it is a friendly assistant whose interface never says, and a user who assumes.

Which outputs need a person before they land? OpenAI's guidance is to have "a human review outputs before they are used in practice", particularly "in high-stakes domains, and for code generation". Note the second half. Code generation is named alongside high-stakes domains, and it is the one teams ship unreviewed.

Also from that guidance, and easy to skip: the reviewer needs access to the source material and an understanding of the system's limits. A human rubber-stamp is worse than no review, because it transfers accountability without adding judgement.

What will you refuse to build? Some features are a bad idea at any quality level. Scoring people for decisions they cannot appeal, inferring protected characteristics, generating a real person's likeness or voice without consent. Deciding this before a deadline is easier than during one.

Who is accountable when it goes wrong? A named person, not a team. This is the Govern function and the one most often left implicit.

Measure what your feature can be unfair about

"Is it biased" has no answer. "Does it answer questions about one product line worse than another, or treat one dialect differently" does.

Bias and fairness covers building matched inputs and reading a difference without overclaiming. The ethical contribution here is picking which comparisons to run, and that follows from knowing who your users are, which is the scope work in know your customers and use cases.

Run the comparison you would least like to see the result of. That is where the information is.

Say what it cannot do, in the interface

The most effective safety work is usually the least technical: telling users what the feature is not reliable for, where they will read it.

A confidence figure nobody understands is not this. A line saying the assistant can be wrong about pricing and linking to the page that is authoritative is. So is showing the retrieved source next to the answer, which turns a claim into something the reader can check.

This costs less than any model change and moves more of the real risk, because it converts a silent wrong answer into a verifiable one.

Write down what you decided

The output of this work is a short document, not a feeling: what the feature is for, what it refuses, which harms you mapped, what you measured, what the residual risk is, and who accepted it.

Two reasons. It is what you produce when someone asks, whether that someone is a customer, a regulator or your own security review. And it is the only way the decision survives the person who made it leaving.

Further reading

• NIST, AI Risk Management Framework: the Govern, Map, Measure and Manage functions.
• OpenAI, Safety best practices: human review, including for code generation.
• Bias and fairness: measuring differential treatment from outside the model.
• Conducting adversarial testing: finding the failures before a user does.