Hugging Face Inference SDK

One client in front of several ways to run a model

InferenceClient, from the huggingface_hub package, sends HTTP calls to models on the Hugging Face Hub. The point is that the same client talks to a serverless partner, a dedicated endpoint, or a server on your laptop, so trying a different backend does not mean rewriting the call.

That matters when you are evaluating open models. Comparing four of them across three hosting arrangements is a lot of client code if each combination has its own SDK.

The three backends and the parameter that selects them

Inference Providers is serverless access through partner companies. This is the rename to know about: it "builds on our previous Serverless Inference API," so older tutorials use a name that no longer exists. The provider default is auto, which picks the first available provider for that model according to your configured order.

from huggingface_hub import InferenceClient

client = InferenceClient(provider="together", model="meta-llama/Llama-3.1-8B-Instruct")
client.chat_completion(messages, max_tokens=100)

Inference Endpoints is dedicated managed infrastructure. You deploy a model, get a URL, and pass it as model. The rest of your code does not change.

A local server works the same way, as long as it exposes an OpenAI-compatible API. llama.cpp, Ollama, vLLM, and TGI all qualify:

client = InferenceClient(model="http://localhost:8080")

One constraint catches people: you cannot pass both a URL and a provider. They are mutually exclusive, because a URL already says where the request goes.

A surface organized by machine learning task

Most provider SDKs give you chat and embeddings. This one is organized by machine learning task, and the list is long: chat_completion sits beside text_to_image, feature_extraction, automatic_speech_recognition, summarization, image_classification, translation, token_classification, and about twenty more.

client.text_to_image("A flying car crossing a futuristic cityscape.")
client.automatic_speech_recognition("meeting.flac")
client.feature_extraction("text to embed")

Provider support varies by task. Hugging Face publishes a matrix, and the pattern is that chat completion is widely supported while the specialized tasks are served by fewer providers. Check the matrix for the task you need rather than assuming a provider covers everything, and expect the matrix itself to change.

Moving from an OpenAI client

If you already call OpenAI, the migration is smaller than you would expect. chat_completion follows the OpenAI Chat Completions shape, and client.chat.completions.create is an alias for it:

- from openai import OpenAI
+ from huggingface_hub import InferenceClient

- client = OpenAI(
+ client = InferenceClient(
    base_url=...,
    api_key=...,
)

base_url and api_key are aliases for model and token, added to make this diff small. Parameters and response format match, stream=True behaves the same, and AsyncInferenceClient mirrors the async client.

That is the portability argument made concrete. Your call site stays put while the model behind it changes.

The timeout default that surprises people

By default the client waits indefinitely for a response.

On a first request to a cold provider, or a long generation, that means a call that appears to hang. Set a timeout and handle the error:

from huggingface_hub import InferenceClient, InferenceTimeoutError

client = InferenceClient(timeout=30)
try:
    client.text_to_image(...)
except InferenceTimeoutError:
    ...

Do this before you put the client behind a web request. A handler with no timeout inherits the client's, and an indefinite wait ties up a worker until something else gives up.

Further reading

• Hugging Face, Run inference on servers: the three backends, the task matrix, and the timeout behavior.
• Self-hosted models: operating the local server this client talks to.
• Choosing the right model: what to compare once switching is cheap.