AnythingLLM OpenAI-compatible API setup: base URL, model and embeddings

Use AnythingLLM's OpenAI (Generic) provider for a chat API that matches the OpenAI request and response format. The chat connection and the model used to embed uploaded documents are separate settings, so test them as two jobs.

Set the chat provider

In Settings, choose the generic OpenAI LLM provider and enter the values supplied by your API provider:

Provider: OpenAI (Generic)
Base URL: The provider's OpenAI-compatible API root, often ending in /v1
API key: A key issued by that provider
Model: The exact chat model ID that provider accepts

Enter the API root rather than the full /chat/completions request URL unless the provider specifically documents a full endpoint field. Do not guess the model ID from its display name.

For a self-hosted Docker setup, AnythingLLM documents the equivalent environment variables. Keep the real key in your local .env or deployment secret store, and keep that file out of Git:

LLM_PROVIDER=generic-openai
GENERIC_OPEN_AI_BASE_PATH=https://api.example.com/v1
GENERIC_OPEN_AI_API_KEY=replace-with-your-provider-key
GENERIC_OPEN_AI_MODEL_PREF=your-exact-chat-model-id

Select the model that will answer

AnythingLLM distinguishes System, Workspace and Agent LLMs. The System LLM is the default; a Workspace LLM overrides it for that workspace, and an Agent LLM can be configured separately. If a connection works in one workspace but another still uses a different provider, check the workspace's own model selection.

Start with one short chat prompt. A model appearing in a selector does not establish that the provider accepts its ID. If the request returns an error, compare the captured base URL, model ID and HTTP status with the provider's current API documentation.

Keep document embeddings separate

Uploading documents for retrieval-augmented generation (RAG) uses an embedding model to turn text into vectors. This is configured separately from the chat LLM. A chat endpoint that accepts Chat Completions does not automatically provide an embeddings endpoint.

For a self-hosted setup, AnythingLLM lists separate generic embedder settings:

EMBEDDING_ENGINE=generic-openai
EMBEDDING_BASE_PATH=https://api.example.com/v1
GENERIC_OPEN_AI_EMBEDDING_API_KEY=replace-with-an-embedding-capable-key
EMBEDDING_MODEL_PREF=your-exact-embedding-model-id

Use these only if the provider documents a compatible embeddings route and the model ID you entered is an embedding model. Otherwise, keep AnythingLLM's built-in embedder or choose a separately supported embedding provider. The embedder is system-wide; changing it after documents have been embedded can require deleting and re-embedding those documents.

If replies fail while streaming

AnythingLLM streams by default. For a self-hosted Generic OpenAI provider whose endpoint does not support streaming, the official configuration guide documents this Docker setting:

GENERIC_OPENAI_STREAMING_DISABLED="true"

It affects the Generic OpenAI provider only. Restart the self-hosted service after changing its environment. The setting is not a fix for an incorrect URL, key or model ID.

If chat works but an agent does not

Check the Agent LLM separately and confirm that the selected model supports the tool-calling behavior you need. A successful ordinary chat does not prove an agent workflow is supported by that model or provider.

Official references

Continue with free and trial OpenAI-compatible API routes, or use the local request builder to check your JSON and URL before sending a request yourself.

Checked 2 November 2026 against the linked AnythingLLM documentation and server example. This is a configuration guide; no AnythingLLM instance, API key, provider request or model inference was tested.

Free API access and limits

Check the site free API access and limits before trying to obtain a site key. Read the current status and request limits before using a site key. It is not a source of official provider keys.

ChinaGenAI is not affiliated with the providers named here. This release contains setup guidance and explicitly labeled fictional examples, not claimed live model benchmarks.

Connect AnythingLLM to an OpenAI-compatible API.