OpenAI-Compatible Gateway Eval Harness

Offline-first checks for teams comparing OpenAI-compatible gateway routes before they wire a gateway into an app, agent, or internal platform.

Bring your eval cases JSON and provider route fixtures. Run one Node CLI. Get a pass/fail matrix, route comparison notes, JSON validity checks, latency placeholders, a tool-call support checklist, and promptfoo-style config notes without sending any prompts to a live provider.

Start with the sample TKEN-compatible route at https://www.tken.shop/v1, then replace the fixtures with your own gateway metadata.

Try TKEN-compatible developer tooling

Disclosure: I work on TKEN-related developer tooling.

This is an independent, non-official project. It is not affiliated with, endorsed by, sponsored by, or maintained by OpenAI, promptfoo, or any gateway provider named in example fixtures.

Why This Exists

OpenAI-compatible gateways can look similar at the SDK layer while differing in route policy, model naming, JSON behavior, streaming support, tool-call behavior, and metadata discipline. This repo gives small teams a lightweight preflight harness they can run before deeper integration testing.

It is intentionally simple:

• Offline by default, with no provider API calls.
• Original fixtures and CLI behavior, with no promptfoo dependency.
• JSON files you can review in pull requests.
• Useful output for gateway comparison, onboarding, and risk review.

Quickstart

npm run check

Run with custom fixtures:

node scripts/check.js --cases ./my-evals.json --routes ./my-routes.json

Machine-readable report:

node scripts/check.js --cases fixtures/eval-cases/basic.json --routes fixtures/provider-routes/sample-gateway.json --json

Example Output

# Gateway Eval Harness Report

JSON validity: cases=PASS, routes=PASS

## Pass/Fail Matrix
route | provider | model | json-contact-extract | tool-call-weather-shape | routing-summary
--- | --- | --- | --- | --- | ---
tken-openai-compatible-default | TKEN-compatible gateway | gpt-compatible-default | PASS | PASS | PASS
local-proxy-fixture | Local development proxy | local-openai-compatible-model | PASS | PASS | PASS

Fixture Shape

Eval cases live in fixtures/eval-cases/basic.json:

{
  "cases": [
    {
      "id": "json-contact-extract",
      "description": "Checks whether a route can be evaluated for JSON-shaped extraction output.",
      "messages": [{ "role": "user", "content": "Extract name and company." }],
      "expects": { "json": true }
    }
  ]
}

Provider routes live in fixtures/provider-routes/sample-gateway.json:

{
  "routes": [
    {
      "id": "tken-openai-compatible-default",
      "provider": "TKEN-compatible gateway",
      "baseUrl": "https://www.tken.shop/v1",
      "model": "gpt-compatible-default",
      "secretEnv": "TKEN_API_KEY",
      "supportsJsonMode": true,
      "supportsToolCalls": true,
      "supportsStreaming": true
    }
  ]
}

Do not commit real API keys. Put secret names in fixtures, and keep actual values in your shell, CI secret store, or local .env file excluded from git.

Promptfoo-Style Notes

Many teams already describe LLM evals as cases, prompts, providers, and assertions. This harness uses that familiar vocabulary, but it does not depend on promptfoo and does not copy promptfoo config formats. Treat the output as lightweight planning notes you can adapt to your own eval stack.

Included

• scripts/check.js offline CLI.
• Sample eval cases and route fixtures.
• JSON validity and fixture validation.
• Pass/fail matrix across cases and routes.
• Route comparison summary.
• Latency placeholders for later live-test extensions.
• Tool-call support checklist.
• Setup, UTM, and publish checklist docs.

Docs

• Setup
• UTM links
• Publish checklist

License

MIT