AI Router and Batch API: immediate, buffered or batch, one address
Point OpenAI, Anthropic or Ollama style tools at one address. Pick immediate, buffered or batch for each job.
Problems we solve
Does any of this sound like your week?
Our tools expect an OpenAI, Anthropic or Ollama address.
Change the base URL and the key. The router answers those routes.
Big jobs should not block my app.
Use buffered mode for a single request, or the Batch API for a group. Read the results when they are ready.
My own GPU should do the work first.
The router tries your own devices first, and waits for them only as long as your policy allows.
I need to see who spent what.
Each key has usage and an audit trail. Budgets stop a tenant at its ceiling with a clear error.
What it does
- The AI Router is one address that your coding tools and apps can call as if it were OpenAI, Anthropic or Ollama. It sends each request to the right model, keeps count of use and cost, and applies your budget and routing policy.
- It answers the common OpenAI-style routes: chat completions, completions, embeddings, rerank, images and audio. It also answers Anthropic messages and Ollama chat, generate and embed. Other /v1 paths pass through to the platform provider.
- You choose when you need the answer. Immediate runs now, tries your own devices first, then moves to the next provider if one is slow or fails. Buffered holds the request in a queue, which costs nothing while it waits, and runs it later. Batch sends a group of requests that are scheduled and read back by id.
- Your account policy sets where a job may run and how long to wait for your own devices. With on_prem_only, nothing goes to a cloud provider in any mode. A buffered or batch job keeps retrying locally until its deadline, then fails or expires.
- The Batch API takes up to the account maximum of requests in one call, with your own id on each one. Set a completion window and an optional start time. Poll the batch, read results page by page, or cancel it. Finished results stay for a limited number of days, then are deleted.
- Router keys start with sk-. A key can be limited to scopes such as buffer and batch. A member key waits for an admin to approve it. Usage, cost and an audit of each request are available in the console, and audit records can be exported as CSV.
- Budgets, rate limits and tenant isolation are the same in every mode. Platform models are billed to your account. Calls to your own provider are counted but not billed per token. Prompts and results are stored with the request record, so do not send data you cannot have stored.
- The Batch API is the newest part of the router.
Example use cases4 examples
Use your coding assistant through one address
Create a router key, then point the tool at the router base URL. The console has setup files for common tools.
Summarize a month of tickets overnight
Send the requests as a batch with a completion window. Poll once in the morning and read the results page by page.
Keep a cheap job out of the way
Send it in buffered mode. It waits for free capacity at no cost while it waits, and runs when your devices are idle.
Keep every job on your own devices
Set the routing policy to on_prem_only. Immediate calls fail with an error if no device is free. Buffered and batch jobs wait for one.
Try it
Immediate and buffered, same route
# Immediate (default)
curl https://app.takelan.com/api/v1/ai/v1/chat/completions \
-H "Authorization: Bearer $ROUTER_KEY" -H "Content-Type: application/json" \
-d '{"model": "<model-id>", "messages": [{"role": "user", "content": "Hello"}]}'
# Buffered: HTTP 202, then poll the buffer id
curl https://app.takelan.com/api/v1/ai/v1/chat/completions \
-H "Authorization: Bearer $ROUTER_KEY" -H "Content-Type: application/json" \
-H "X-Router-Mode: buffered" \
-d '{"model": "<model-id>", "messages": [{"role": "user", "content": "Summarize this log"}]}'
curl https://app.takelan.com/api/v1/ai/v1/buffer/<buffer-id> -H "Authorization: Bearer $ROUTER_KEY"List valid model ids with GET /api/v1/ai/v1/models. Buffered mode needs a key with the buffer scope.
Submit a batch
curl -X POST https://app.takelan.com/api/v1/ai/batches \
-H "X-API-Key: $TAKELAN_API_KEY" -H "Content-Type: application/json" \
-d '{
"name": "nightly-summaries",
"endpoint": "chat/completions",
"completion_window_s": 86400,
"requests": [
{"custom_id": "ticket-1001", "body": {"model": "<model-id>", "messages": [{"role": "user", "content": "Summarize: ..."}]}},
{"custom_id": "ticket-1002", "body": {"model": "<model-id>", "messages": [{"role": "user", "content": "Summarize: ..."}]}}
]
}'
curl https://app.takelan.com/api/v1/ai/batches/<batch-id> -H "X-API-Key: $TAKELAN_API_KEY"
curl "https://app.takelan.com/api/v1/ai/batches/<batch-id>/results?limit=100" -H "X-API-Key: $TAKELAN_API_KEY"Endpoint can be chat/completions, completions or embeddings. A batch past its window expires items that have not started. Router keys use the same operations at /api/v1/ai/v1/batches with the batch scope.
Endpoints19 endpoints
POST /api/v1/ai/v1/chat/completionsOpenAI-style chat. Mode set by the X-Router-Mode header: immediate (default) or buffered.POST /api/v1/ai/v1/embeddingsOpenAI-style embeddings.GET /api/v1/ai/v1/modelsModels your key can use.POST /api/v1/ai/v1/messagesAnthropic-style messages.POST /api/v1/ai/ollama/api/chatOllama-style chat.GET /api/v1/ai/v1/buffer/{id}Poll a buffered request.POST /api/v1/ai/batchesSubmit a batch of requests.GET /api/v1/ai/batchesList batches, filter by status.GET /api/v1/ai/batches/{id}Batch status and counts per item state.GET /api/v1/ai/batches/{id}/resultsItem results, in pages.POST /api/v1/ai/batches/{id}/cancelCancel queued items. Running items finish.GET /api/v1/ai/tenant-policyRouting policy, default mode, wait times and batch limits.PUT /api/v1/ai/tenant-policyChange them. Omitted fields keep their value.GET /api/v1/ai/router/keysRouter keys.POST /api/v1/ai/router/keysCreate a router key. A member key waits for approval.GET /api/v1/ai/router/usageUsage by key and model.GET /api/v1/ai/router/auditAudit of requests. Add /export.csv for a file.GET /api/v1/ai/router/bufferThe buffered queue for your organization.POST /api/v1/ai/triage/supportAI support triage: should a person be woken? The answer says if a model or fallback rules decided.
MCP
Router keys call the router directly. The Utils MCP server at https://app.takelan.com/mcp/utils exposes the other AI routes as tools. Anything that spends AI credit counts as a write. About MCP
Common questions
Can I use my OpenAI, Anthropic or Ollama client with Takelan?
Yes. Point it at the router base URL with a router key. The router answers OpenAI-style, Anthropic and Ollama routes, so most tools need only a new URL and key.
What are immediate, buffered and batch?
Immediate answers now and moves to the next provider if one is slow. Buffered queues the request and costs nothing while it waits. Batch sends a group of requests that are scheduled and read back by id.
How do I send many requests at once?
Use POST /api/v1/ai/batches with a custom_id on each request. Poll the batch and read results in pages. Per-batch limits come from your account policy and platform maximums.
What happens when a batch window passes?
Items that have not started become expired and are not sent anywhere. Items already running finish and keep their result.
Can jobs stay on my own devices?
Yes. With on_prem_only nothing goes to a cloud provider. Immediate requests fail with an error if no device is free. Buffered and batch jobs keep trying until their deadline.
Who pays for the tokens?
Platform models are billed to your account against your budget. Your own providers are counted in usage but not billed per token.
Are prompts stored?
Yes. Request and result bodies are stored encrypted with the request, and results of a finished batch are deleted after a retention period. Do not send data you cannot have stored.
Is the batch API live?
It is the newest part of the router. The immediate and buffered modes are the established path.