Audio Processing

Two ways audio reaches a model

Audio does not go into a language model. Something has to happen first, and there are two somethings.

Transcribe, then prompt. Send the recording to a speech-to-text model, get text back, put the text in a prompt. Two calls, two bills, and the second half is ordinary text work you already know how to do.

Send the audio. Some models accept an audio file as a content block beside your text instruction, the way an image would go. One call, and the model works from the recording rather than from a transcript of it.

Which one fits depends on a question worth asking early: does your task need anything that is in the sound but not in the words?

Transcribing first, and what that discards

Transcription is lossy in a specific direction. It keeps the words and drops everything about how they were said.

Gone: tone, hesitation, emphasis, the pause before "sure", sarcasm, overlapping speech, background sound, and by default who was speaking. For a task like pulling action items out of a meeting, none of that matters. For "was this customer satisfied", the transcript may say "fine" where the recording says something else entirely.

The practical details, from OpenAI's speech-to-text guide as of 2026-09-16. It documents several transcription models, gpt-transcribe as the general recommendation, gpt-4o-transcribe and gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize, and whisper-1. Uploads are capped: "Files can be up to 25 MB", with supported formats "mp3, mp4, mpeg, mpga, m4a, wav, and webm."

Transcription is cheap, fast, and produces text you can store, search, diff, and feed to anything. When it fits, it fits well.

Sending the audio itself

The alternative keeps the sound. Google's Gemini documentation describes models that "analyze and understand audio input and generate text responses", handling description, transcription, speaker identification, and reasoning about what was heard.

Because there is no transcript in between, you can ask questions a transcript could not answer: whether a speaker sounded uncertain, what the non-speech sounds were, who talked more. You can also ask for a transcript, since transcription is one of the things it does.

There are constraints in exchange, and they are different in kind from a file-size cap. Audio is "Downsampled to 16 Kbps" and "Multi-channel audio combined to single channel", so a stereo recording separating two speakers onto two channels loses that separation on the way in. Google's own page points elsewhere for "real-time transcription", recommending the Cloud Speech-to-Text API, which is a useful signal about what this path is for.

Limits: a file-size budget against a token budget

The two paths meter you differently, and this is the practical difference nobody mentions.

Transcription has a file-size budget. Twenty-five megabytes, and audio compression decides how much time that buys. A well-compressed voice recording fits hours; an uncompressed WAV fits minutes. Crossing the line is a hard rejection.

Audio-native has a token budget. Gemini documents "32 tokens per second of audio (1 minute = 1,920 tokens)" with a ceiling of "9.5 hours of audio per prompt". Crossing the line is not a rejection. It is a bill.

The arithmetic surprises people. A ten-minute support call is 19,200 tokens of input before you have asked anything about it. An hour is roughly 115,000. At those sizes the model call is no longer a rounding error next to the transcription call, and for a pipeline processing thousands of recordings, the choice between paths is a budget decision rather than a technical one.

The rule of thumb that falls out: transcribe first when you are processing volume and only need the words. Send the audio when the sound carries the answer, or when one extra call would cost more than the tokens.

Splitting a long recording without cutting a sentence

Past the size limit, you split. How you split affects accuracy.

OpenAI's guidance: "use a compressed audio format or split the file into 25 MB or less chunks". Then the part people skip: "Avoid splitting in the middle of a sentence, which can remove context and reduce accuracy."

A transcription model uses surrounding audio to disambiguate. Cut mid-sentence and both halves lose the context that would have resolved a homophone or a proper noun, so you pay twice for one cut. Split on silence instead. Any voice-activity detection tool will find the gaps, and a few seconds of overlap between chunks costs almost nothing and saves the boundary words.

Then you have a second problem, which is that you now hold several transcripts and want one. Concatenating is usually right. Summarizing each chunk and combining summaries is not, unless you accept that anything spanning a boundary is gone.

Timestamps and speakers are separate features

These are the two things people assume come free, and neither does.

Timestamps are model-specific. On OpenAI, they come from whisper-1 through the timestamp_granularities[] parameter, "structured timestamp data for captioning and video editing." The newer transcription models are not documented as offering them, so a captioning pipeline may want the older model on purpose.

Speaker labels are a different model again. gpt-4o-transcribe-diarize handles diarization, working out who spoke when, returning diarized_json with "segments with speaker, start, and end metadata." For audio over thirty seconds it wants a chunking_strategy set.

So on that provider, timestamps and speaker labels currently live in two different models, and a pipeline needing both needs to plan for it. Gemini references segments in "MM:SS format" and lists diarization as something to request through structured output, which is a different shape again. You are asking rather than setting a flag.

One caution about asking a general model for timestamps. It will produce them, formatted correctly, whether or not they are accurate. Structured output gets you a valid shape, not a true value. If timestamps matter, use a model documenting them as a feature, and spot-check against the audio.

Further reading

• OpenAI, Speech to text: models, the 25 MB limit, chunking advice, timestamps, and diarization.
• Google, Audio understanding: duration limits, the per-second token cost, and what audio-native models can be asked.
• Text to speech: the same pipeline in the other direction.
• Multimodal AI: why one model takes more than text.