Configure Promptfoo’s target and grader separately.

Promptfoo has two provider roles here: providers sends the test prompt to the model under test; defaultTest.options.provider sends model-graded assertions to a judge. A compatible gateway can serve the first model and still reject the grader's different model ID. Use openai:chat: when the custom endpoint accepts Chat Completions; a bare openai: ID selects Promptfoo's Responses route for supported recent OpenAI models.

Set the model under test

Use the exact model ID your gateway documents or lists. For a custom OpenAI-compatible Chat Completions API, set apiBaseUrl to its API root, including a required prefix such as /v1. Do not append /chat/completions or /responses; Promptfoo builds the operation path.

apiHost is a hostname setting that builds the HTTPS /v1 root. Use apiBaseUrl when the endpoint has a custom path, protocol, or port.

Point model-graded assertions to a supported judge

Only model-graded assertions, such as llm-rubric, need a grader model. Deterministic checks such as contains do not need one. If the model output was returned but the evaluation fails with an unsupported model or model-not-found message, check which provider made the grading call.

providers:
  - id: openai:chat:YOUR_TARGET_MODEL_ID
    config:
      apiBaseUrl: https://api.example.com/v1
      apiKeyEnvar: MODEL_API_KEY

defaultTest:
  options:
    provider:
      id: openai:chat:YOUR_GRADER_MODEL_ID
      config:
        apiBaseUrl: https://api.example.com/v1
        apiKeyEnvar: MODEL_API_KEY
  assert:
    - type: llm-rubric
      value: The answer addresses the question accurately

Replace both model IDs with IDs supported by the endpoint. They may be the same, but they do not have to be. Keep MODEL_API_KEY in the environment of the process running Promptfoo; the official configuration says a missing apiKeyEnvar value does not fall back to OPENAI_API_KEY.

Separate a path failure from a grader failure

Promptfoo issue #10526 describes one report where an origin-only apiBaseUrl omitted /v1 and the default grader requested a model the gateway did not serve. The issue remained open when checked on 27 October 2026. This report does not show how common the failure is or prove that every current installation behaves the same way.

Check the current Promptfoo configuration

• OpenAI-compatible provider, base URL and grader configuration
• LLM rubric and model-graded assertions
• Compare the captured and expected URL locally if the exact request path is available.

Documentation and the public issue checked 27 October 2026. No Promptfoo run, account, API key, or provider request was used.