# pip install litellm
from litellm import completion
response = completion(
model="gpt-4o", # or "anthropic/claude-3.5-sonnet", "google/gemini-1.5-pro" etc.
messages=.
## LiteLLM vs. OpenRouter
OpenRouter aggregates many models behind a single token and offers simple routing, public rate limits, and a marketplace-like feel. LiteLLM, by contrast, is open-source and often runs in your infra.
- Control: LiteLLM gives you private control; OpenRouter is a hosted aggregator.
- Cost transparency: With LiteLLM, you bring your own provider keys; with OpenRouter, you pay OpenRouter, which may bundle fees.
- Compliance: Self-hosting LiteLLM can simplify data residency and compliance.
TrueFoundry’s breakdown of LiteLLM vs OpenRouter highlights these strategic differences and when each is a better fit.
## How It Compares to LangChain and LlamaIndex
- LangChain: Broader orchestration framework (chains, agents, tools, memory). You can use LiteLLM inside LangChain to abstract models.
- LlamaIndex: Data-oriented RAG framework. LiteLLM can act as the LLM layer underneath.
- Native SDKs (OpenAI, Anthropic, Google): Best for full-feature parity and newest features; worst for multi-provider switching.
If you just need model interchangeability and clean governance, LiteLLM is the specialized tool. If you want agent frameworks or complex RAG pipelines, pair LiteLLM with LangChain/LlamaIndex.
## Performance and Reliability
- Latency: You’ll see minimal overhead vs direct calls, but routing/proxy logic does add a small tax. In exchange, you gain fallbacks and policy controls.
- Reliability: Centralized retries + provider fallback improves uptime for production apps.
- Cost optimization: Route cheap models for routine tasks; keep premium models for critical paths.
Tip: Instrument with logs and tracing. Many teams forward LiteLLM gateway logs to their observability stacks.
## Security and Compliance Considerations
- Key management: Store provider keys securely; use environment variables or vaults.
- Auditability: Central proxy logs queries, response metadata, and spend.
- Data handling: Self-hosting helps with data residency and privacy guarantees.
## Pricing and Licensing
- LiteLLM itself is open-source; you pay providers directly. That often means lower lock-in, clearer costs, and the freedom to switch.
- Operational cost: If you run the proxy, budget for the service (containers, monitoring, on-call).
## When to Choose LiteLLM
Choose LiteLLM if you:
- Need to support multiple providers now or soon
- Want centralized rate limits, budgets, and observability
- Prefer OpenAI-style ergonomics across the board
- Need fallbacks for production reliability
Consider alternatives if you:
- Only call a single provider with cutting-edge features
- Need deep, provider-specific capabilities immediately at launch
- Don’t want to operate a gateway service
## Implementation Patterns That Work
1) Simple abstraction for a single app
- Use the Python SDK
- Configure 1–2 fallback models
- Log responses and costs to your app’s telemetry
2) Organization-wide gateway
- Run the proxy in Kubernetes or ECS
- Enforce org-wide rate limits and quotas
- Tag requests by team/project for chargeback
- Add routing policies (e.g., low-cost defaults, high-accuracy for production paths)
3) RAG + LiteLLM
- Use LlamaIndex/LangChain for retrieval logic
- Plug LiteLLM in as the model layer to keep provider flexibility
## Developer Experience: The Good and the Rough Edges
- Good: Dead-simple migration from OpenAI SDKs, robust model coverage, fallbacks.
- Rough: Occasional provider-specific quirks leak through; tool/function calling parity can be imperfect. Keep an eye on release notes and pin versions for stability.
The developer community has called for stronger engineering rigor across fast-moving LLM libraries; that’s a healthy lens to bring to any adoption, including LiteLLM.
## Verdict: Is LiteLLM Worth It?
LiteLLM is one of the easiest ways to build a multi-model strategy without rewriting your code for every provider. If your roadmap includes vendor flexibility, uptime via fallbacks, and spend control, LiteLLM earns a strong recommendation. If you’re all-in on a single vendor and need every new feature on day one, go native.
By the way, if you want a friendly interface to experiment with prompts and models side-by-side, a tool like [Sider.AI](https://sider.ai) can accelerate your workflow while you’re evaluating providers—handy for prompt trials and quick comparisons before you lock in routing policies .
## Quick Start Checklist
- Install LiteLLM SDK or proxy
- Define primary and fallback models
- Set org-wide rate limits and budgets
- Centralize logs and traces
- Test provider-specific features (tools, images, JSON modes)
- Pin versions and document your model matrix
## What’s Next
- Add health checks and circuit breakers per provider
- Build an A/B router to compare model quality vs cost in production
- Create a migration playbook for adding/removing providers
### FAQ
Q1:What is LiteLLM and why use it?
LiteLLM is an open-source gateway and SDK that lets you call 100+ LLMs through a single, OpenAI-compatible API. Use it to switch providers easily, set fallbacks, and centralize rate limits and cost tracking.
Q2:How does LiteLLM compare to OpenRouter?
LiteLLM is self-hostable and open-source, giving you private control with your own provider keys, while OpenRouter is a hosted aggregator. Choose LiteLLM for compliance and control, OpenRouter for quick access to many models.
Q3:Can LiteLLM replace LangChain or LlamaIndex?
No. LiteLLM focuses on model abstraction and routing. Pair it with LangChain or LlamaIndex if you need agents, chains, or RAG pipelines while keeping provider flexibility.
Q4:Does LiteLLM support fallbacks and retries?
Yes. You can define fallback chains so if one provider fails or rate-limits, LiteLLM automatically tries the next model, improving reliability for production workloads.
Q5:Is LiteLLM free to use?
LiteLLM is open-source, so there’s no license cost, but you’ll pay for the underlying model usage. If you run the proxy, factor in operational costs for hosting and monitoring.