Video Understanding

Video is frames plus audio, and you pay for both

A model does not watch video. It receives sampled frames and a transcribed audio track, and the sampling rate decides both what it can see and what it costs.

Gemini's default is one frame per second. At that rate the published arithmetic is:

Which lands at roughly 100 tokens per second of video at low resolution, or 300 at high. A ten-minute clip is therefore around 60,000 tokens at the low setting and 180,000 at the high one.

Run that against your own footage before designing anything. Video is the modality where a feature that works in a demo becomes unaffordable at volume, because the cost scales with duration rather than with the number of requests.

Getting a file in, and the size cliff

Four routes, and the one you pick is mostly decided by file size.

Inline is the tempting default and the one that breaks first. Under a minute it is fine; past that, the File API is the route, and it has the additional property that an uploaded file can be reused across requests rather than re-sent.

Accepted types are the usual set: video/mp4, mpeg, mov, avi, x-flv, mpg, webm, wmv and 3gpp.

YouTube has its own rules

Passing a YouTube URL works and comes with limits that are easy to hit in testing. Free tier allows no more than 8 hours of YouTube video per day. Paid is unlimited. Only public videos work, so anything unlisted or private fails.

Request counts differ by model generation: one video per request before Gemini 2.5, up to 10 on 2.5 and later.

Three parameters that change the bill

Static processing accepts a processing configuration, and this is where cost control lives.

fps overrides the one-frame-per-second default. Halving it halves the frame tokens. For a lecture or an interview, where the picture barely changes, a low rate loses nothing. For sport or a screen recording of fast interaction, it loses the thing you were looking for.

start_offset and end_offset, in milliseconds, clip to an interval. If you know the relevant part is in the third minute, send the third minute. This is the largest single saving available and the most often skipped.

Both apply to static processing only.

Ask in timestamps

Queries can reference a specific moment in MM:SS form, such as 01:15, and answers can be asked to cite them. That turns a vague question into a checkable one.

It matters more than it first appears. "Summarize this video" produces prose nobody can verify. "List each point where the speaker changes topic, with timestamps" produces claims you can click on and confirm, which is the difference between a demo and something you can put in front of a user.

Agentic processing, where it is available

Beyond the default static mode, Gemini offers agentic processing, which the documentation describes as up to 88% more token-efficient, with usage varying by content complexity rather than by duration.

The caveats are worth reading in that order. It is available on a subset of models, currently the Flash line, and its cost is no longer predictable from duration. That is a good trade for exploratory queries over long footage, and a worse one when you need to know what a job costs before running it.

Further reading

• Google, Video understanding: the token rates, upload routes, YouTube limits and the processing parameters.
• Multimodal AI: how media reaches a request in the first place.
• Image understanding: the same cost question for single frames.
• Audio processing: the track that comes with the picture.