from litellm import completion
resp = completion(
model="gpt-4o-mini", # or "anthropic/claude-3-5-sonnet", "mistral/mistral-large"
messages=.
---
## Streaming, Tools, and JSON Mode
### Streaming Responses
```python
from litellm import completion
for chunk in completion(
model="gpt-4o-mini",
messages=.
### Cost and Token Usage
LiteLLM can track token usage and estimate cost per request, model, or project. With the proxy, you can export usage to logs, dashboards, or a billing sink. This is invaluable when you mix vendors with different pricing.
---
## The LiteLLM Proxy (LLM Gateway)
If you’re a team or platform, the proxy is the real superpower: a central service with routing, auth, rate limits, logging, and observability. You interact with it using the OpenAI API surface so your app code barely changes.
### Start the Proxy
```bash
# simplest local run
litellm --port 4000

from openai import OpenAI
client = OpenAI(base_url=" api_key="svc-app-123")
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=.
---
## Advanced Routing: Latency, Cost, or Reliability
You can implement routing strategies like:
- Weighted round-robin to A/B models
- Lowest-latency-first by region
- Cost-aware routing for non-critical endpoints
- Fallback-on-error/retry across providers
With a router policy, you can say “prefer cheap, fall back to premium for tough prompts.” This offers high availability and predictable budgets.
---
## Guardrails, Moderation, and Safety
Add pre- and post-processing middleware to strip PII, enforce safety filters, or moderate outputs before returning to clients. Combine provider-native moderation (e.g., OpenAI, Google) with your own policy checks in the proxy. Example: require JSON schema validation and re-ask when invalid.
---
## Observability and Logging
- Enable request/response logging with redaction.
- Export metrics to Prometheus/Grafana or your APM.
- Trace latency, tokens, and cost by endpoint and user.
This turns “model roulette” into a managed service with SLOs and budgets.
---
## Real-World Usage Patterns
1) Multi-vendor resilience
- Primary: fast/cheap model; Fallback: high-accuracy model on 429/5xx.
- Benefits: better uptime, cost control, and stable quality.
2) Feature flag model upgrades
- Use router weights to canary a new model to 5% of traffic; monitor metrics; ramp up when stable.
3) Product tiers
- Free tier routed to small models; Pro tier to premium models.
4) Prompt registries and templates
- Centralize prompts in the proxy so services inherit improvements without redeploys.
5) Team billing and budgets
- Track spend by API key; enforce soft and hard limits per team or product.
---
## Security and Compliance Checklist
- Store provider keys in your secret manager; reference via env vars in config.
- Turn on request redaction and PII scrubbing in logs.
- Use per-service API keys for the proxy; rotate regularly.
- Set org-wide rate limits and quotas.
- Add allowlists/denylists for models and endpoints.
---
## Troubleshooting: Fast Fixes
- “Unauthorized” via proxy: Check `auth.api_keys` and that your client uses `base_url` + correct key.
- Model not found: Ensure `model_list` contains the friendly name you’re calling.
- Timeouts: Increase `timeout` or route to a lower-latency provider region.
- Weird outputs: Enable JSON schema + validation; add retries and fallbacks.
- Cost spikes: Turn on caching; route bulk traffic to cheaper models; set per-key quotas.
For deeper dives and latest features, the official docs are updated frequently and worth bookmarking. Tutorials like DataCamp’s guide are great for hands-on patterns, and the beginner crash course video can help you see the concepts in action.
---
## Put It All Together: Reference App Skeleton (Python FastAPI)
```python
# app.py
from fastapi import FastAPI
from pydantic import BaseModel
from litellm import completion
import os
class ChatReq(BaseModel):
question: str
app = FastAPI
@app.post("/ask")
async def ask(req: ChatReq):
resp = completion(
model=os.getenv("DEFAULT_MODEL", "gpt-4o-mini"),
messages=.
### FAQ
Q1:What is LiteLLM and why use it over direct provider SDKs?
LiteLLM is an OpenAI-compatible gateway for 100+ LLMs, giving you one API and one mental model. It reduces vendor lock-in, simplifies routing, and adds ops features like caching, retries, and cost tracking.
Q2:How do I use LiteLLM with the OpenAI SDK?
Point the SDK’s base URL to the LiteLLM proxy and use your proxy API key. Your code can stay the same while the proxy swaps providers or models behind the scenes.
Q3:Can LiteLLM stream responses and return JSON?
Yes. Use `stream=True` to get token streams, and `response_format` with JSON schema to enforce structured outputs across providers.
Q4:How do I control costs across different LLM providers?
Enable usage logging and cost estimation, add caching, set rate limits, and route bulk traffic to cheaper models via the proxy. Monitor with dashboards for budgets and SLOs.
Q5:Is LiteLLM suitable for production teams?
Yes. The proxy provides auth, rate limits, routing, observability, and safety middleware. It’s designed as an LLM gateway that centralizes governance while keeping your app OpenAI-compatible.