Price-performance arbitrage for LLM inference. One OpenAI-compatible API that sends every request to the best of 313 endpoints across 19 providers, in under 10 ms.
Before anything is sent, an ML classifier predicts how much the prompt needs of quality, speed and cost. It runs in milliseconds and adapts to your traffic over time.
Refactor this 2,000-line payments module and keep its public API stable.
Endpoints that score poorly on evals for this kind of prompt are filtered out. Of the rest, Cailos picks the best fit, blending eval scores, live latency, reliability and your feedback. The whole decision takes under 10 ms.
For the refactor, that's the best fit, not the most expensive model on the list.
Email addresses, phone numbers, card numbers and IP addresses are replaced with placeholders before a request leaves Cailos, and restored in the reply. Providers only ever see the placeholders.
Every request carries a ranked list of fallbacks. If the first choice errors, times out or hits a rate limit, the next endpoint takes it, usually within 50 ms. After three consecutive failures a circuit breaker opens and traffic stays away until probes confirm recovery.
The best-fitting endpoints start together. The first to stream tokens serves the reply and the others are stopped. If that stream stalls mid-answer, the next-fastest picks it up without breaking the response.
Two different models answer in full. A third, different again, reads both and picks one. You receive that answer word for word, so it still traces back to a single endpoint in response.cailos.endpoint.
Two models draft. A third reads both and writes the reply you receive. Where the drafts disagree it has to settle the disagreement rather than pick a side. It's the slowest mode and the most careful.
Simulated traffic across the providers Cailos routes to. Turn any of them off: requests already headed there move to the next endpoint on their list, the circuit opens, and nothing reaches your users as an error.
The last 40 requests. Outlined ones failed over to their next endpoint.
Drop Cailos into any OpenAI-compatible agent framework. The SDK doesn't change, only the endpoint. Model selection becomes automatic.
from agents import Agent triage = Agent( name="triage", model="gpt-4o-mini", instructions="Classify: billing, technical, or escalate.",) resolver = Agent( name="resolver", model="gpt-4o", instructions="Draft resolution from KB.", tools=[search_kb, lookup_customer],)Triage is cheap but may misclassify. The resolver is expensive for simple tickets. If GPT-4o goes down, the pipeline breaks.
from agents import Agent, set_default_openai_api, set_default_openai_clientfrom openai import AsyncOpenAI cailos = AsyncOpenAI(base_url="https://cailos.com/v1", api_key="cai_...")set_default_openai_client(cailos, use_for_tracing=False)set_default_openai_api("chat_completions") triage = Agent( name="triage", model="auto", # fastest cheap model instructions="Classify: billing, technical, or escalate.",) resolver = Agent( name="resolver", model="auto", # best tool-calling model instructions="Draft resolution from KB.", tools=[search_kb, lookup_customer],)model="auto": each agent gets the right model for its job, with automatic failover across 19 providers.
One format for every provider. Format translation, tool schemas and vision payloads are handled for you.
Provider APIs change and models are deprecated. Cailos absorbs every breaking change so your integration doesn't.
Switch providers in seconds. Circuit breakers fail over when a provider goes down, so your on-call doesn't get paged.
Every endpoint is evaluated on intelligence, tool calling, structured output and vision. Routing reflects the current model landscape.
llmshield swaps email addresses, phone numbers, card numbers and IP addresses for placeholders before a request leaves Cailos, then restores them in the reply.
Every request's endpoint, latency, cost and routing trace, live in your dashboard and broken down per API key.
Every request is processed by llmshield before it reaches any provider. Email addresses, phone numbers, card numbers and IP addresses are replaced with placeholders before transit, then restored in the response on its way back, so providers never see them.
Cailos does not log the text of your prompts. We keep output metadata only: which endpoint was picked, latency, token counts, cost, and the routing decision trace. The only derived signal we retain is a small set of decoupled keyphrases used to improve routing over time. No data reaches any third party other than the provider itself.
Every endpoint in the registry runs an eval suite covering intelligence, tool calling, structured output and vision before it's eligible for routing. Endpoints that score below threshold for a kind of prompt are filtered out when that kind arrives. Scores update as providers ship new model versions, and the full per-endpoint breakdown is in your dashboard.
Routing decisions take under 10 ms end to end. Time to first token is set by the provider, not Cailos, and the endpoint Cailos picks is often faster than the one you'd have hard-coded, which more than offsets the routing overhead.
Every request has a ranked list of fallback endpoints. If the first choice errors, times out or hits a rate limit, Cailos retries against the next-ranked endpoint, usually within 50 ms. After three consecutive failures a circuit breaker opens and traffic stays away from that provider until probes confirm it has recovered.
Yes. Pass a model name instead of auto and Cailos becomes a transparent gateway. You still get the unified API, failover across providers, PII protection and the dashboard; routing only happens when you ask for it.
cailos.com/v1
A drop-in replacement for any OpenAI SDK. Change two lines and reach 313 endpoints from 19 providers.