tashfeenahmed--freellmapi/docs/clients.md

Clients & coding agents

← Back to README · Documentation index

• OpenAI-compatible clients
• Coding agents
• Native Gemini clients
• Ollama clients
• Headerless clients
• MCP server
• VS Code ghost-text autocomplete (Continue)
• Context Handoff

OpenAI-compatible clients

Any client that can target an OpenAI-compatible base URL can use FreeLLMAPI:

• LangChain, LlamaIndex, official OpenAI SDKs: set base_url to http://localhost:3001/v1 and use the unified key from the dashboard.
• Local GPU boxes: add custom OpenAI-compatible endpoints for Ollama, llama.cpp, LM Studio, vLLM, or an internal gateway.

Coding agents

Use the generator instead of hand-editing a client configuration:

export FREELLMAPI_API_KEY=<unified-key>   # or pass --api-key on each command
npx freellmapi setup-claude --url http://localhost:3001 --dry-run
npx freellmapi setup-claude --url http://localhost:3001

--dry-run prints a diff. Real writes merge with the existing configuration and create a timestamped backup first. --profile <name> creates a named Claude/Codex profile. The live /v1/models catalog supplies the model ids and context windows.

Agent	Automated command	Manual base URL	Wire
Claude Code	setup-claude or credential-free-on-disk launch	http://localhost:3001	Anthropic Messages
Codex CLI	setup-codex or launch-codex	http://localhost:3001/v1	Responses (wire_api = "responses")
Cline	setup-cline	http://localhost:3001/v1	OpenAI Chat
Continue	setup-continue	http://localhost:3001/v1	OpenAI Chat / legacy Completions
Aider	setup-aider	http://localhost:3001/v1	OpenAI Chat
OpenCode	setup-opencode	http://localhost:3001/v1	OpenAI Chat
Goose	setup-goose	http://localhost:3001/v1	OpenAI Chat
Qwen Code	setup-qwen	http://localhost:3001/v1	OpenAI Chat (native Gemini also works)
Roo Code	setup-roo	http://localhost:3001/v1	OpenAI Chat
Kilo Code	setup-kilo	http://localhost:3001/v1	OpenAI Chat
Crush	setup-crush	http://localhost:3001/v1	OpenAI Chat
Cursor	setup-cursor prints the guide	public https://…/v1	OpenAI Chat
Anything else	setup-generic prints a ready block	http://localhost:3001/v1	OpenAI Chat

The root-vs-/v1 distinction matters: Claude Code expects the server root because it appends the Anthropic Messages path. OpenAI-compatible clients in this table—including Cline, Aider, Goose, Codex, Continue, OpenCode, Qwen, Roo, Kilo, and Crush—expect their configured base URL to include /v1.

Native Gemini clients

Gemini CLI and Gemini-lineage clients can speak Google's wire format directly:

export GOOGLE_GEMINI_BASE_URL=http://localhost:3001
export GEMINI_API_KEY=freellmapi-your-unified-key
gemini

The native surface implements GET /v1beta/models, model metadata, generateContent, streamGenerateContent (including ?alt=sse), and countTokens. Authentication accepts x-goog-api-key, Bearer, or Gemini's ?key= fallback. Prefer the header: query credentials leak into history and proxy logs.

The Keys → Agents tab maps Gemini Pro, Flash, and Flash-Lite family names to Auto or a pinned catalog model.

Ollama clients

Ollama emulation is off by default. Enable one of these modes on Keys → Agents:

• open-loopback: no key on this machine only. The socket peer must be 127.0.0.1/::1; enabling desktop LAN access does not widen it. Docker note: inside a container the socket peer is the Docker bridge IP, not loopback, so this mode refuses even host-local traffic through a published port — use key-required for Docker deployments.
• key-required: clients must send Authorization: Bearer <unified-key>.

The exact endpoints are /api/tags, /api/chat, /api/generate, /api/show, /api/version, /api/embed, and legacy /api/embeddings. Streaming uses newline-delimited JSON, not SSE. Point Zed, JetBrains AI Assistant, or another Ollama-capable client at http://localhost:3001.

Headerless clients

If a client cannot set headers, create a separately revocable token on Keys → Agents and use:

http://localhost:3001/v1/t/<token>/chat/completions
http://localhost:3001/v1/t/<token>/responses
http://localhost:3001/v1/t/<token>/models

The same prefix exposes /api/chat and /api/tags. Never put the unified API key in a URL. URL tokens have independent hashes and immediate revocation because URLs routinely leak into shell history, reverse-proxy logs, and telemetry.

MCP server

On top of inference, the router is an MCP server: agents can introspect it mid-session (usable models and the params each one honors, provider health, usage and cache stats, routing strategy). For Claude Code:

claude mcp add --transport http freellmapi http://localhost:3001/mcp \
  --header "Authorization: Bearer freellmapi-your-unified-key"

Any MCP client that speaks Streamable HTTP works the same way: point it at /mcp with the unified key as a Bearer token.

FreeLLMAPI is local-first and single-user by design. Your provider keys stay in your SQLite database, encrypted at rest, and requests go from your machine to the upstream providers you enabled.

VS Code ghost-text autocomplete (Continue)

FreeLLMAPI exposes /v1/completions for editor autocomplete clients that send legacy OpenAI prompt/suffix requests. Example Continue config:

models:
  - name: FreeLLMAPI Autocomplete
    provider: openai
    model: auto
    apiBase: http://localhost:3001/v1
    apiKey: freellmapi-your-unified-key
    useLegacyCompletionsEndpoint: true
    roles:
      - autocomplete

Context Handoff

When FreeLLMAPI falls over to a different model mid-conversation (quota, rate limit, cooldown), the new model has no idea it is picking up someone else's task. Context handoff adds a single compact system message to the outbound request that tells the new model exactly that:

FreeLLMAPI context handoff:
You are taking over an ongoing conversation from another model (groq:llama-3 → google:gemini-flash).
Continue the user's task using the conversation context already provided in this request.
Do not restart the task, re-ask already answered setup questions, or discard prior tool results.
Respect the user's latest message as the highest-priority instruction.

Recent session summary:
User: …
Assistant: …

Enable it in .env:

FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

How it works:

• Messages per session are stored in memory (TTL: 3 hours).
• Only injected when the selected model changes for a given session key.
• Not injected on the first request, on same-model continuations, or if a handoff message is already present.
• Session key: X-Session-Id header if present, otherwise SHA-1 of the first user message (same as sticky sessions).
• Storage is in-memory only. Nothing is written to disk or logged.

Important: Context Handoff improves continuity for conversations routed through FreeLLMAPI. It cannot recover provider-internal hidden state or messages that were never sent to the proxy.