ClinicaLLM-OmniBench

Project Overview

ClinicaLLM-OmniBench is a lightweight benchmark framework for medical exam and medical specialty question-bank evaluation.

This repository is a refactor and upgrade of an older internal project. Instead of continuing to extend patch-based legacy scripts, the current version rebuilds the workflow around a cleaner evaluation skeleton with a stronger emphasis on readability, maintainability, and resumable execution.

To keep the repository clean and easier to publish, the legacy old/ reference directory has already been removed. The current workflow does not depend on the previous implementation, and future maintenance should follow the new structure described in this repository.

Repository Structure

ClinicaLLM-OmniBench/
  README.md
  main.py
  ollama_pull_llm.py
  dataset/
    general/
    2012/
      attending/
      senior/
    ...
    2026/
      attending/
      senior/
  result/
    general/
    2012/
      attending/
      senior/
    ...
    2026/
      attending/
      senior/
  docs/
    PRD.md
    DOC_TREE.md
    REFACTORING_GOALS.md
  models/
    ollama_models.json
    model_modes.example.json
  src/
    dataset_utils.py
    judge_utils.py
    model_utils.py
    provider_utils.py
    result_utils.py
    runner_utils.py

Dataset Layout

dataset/

dataset/ stores the source question files. The directory is organized in two groups:

• general/ Stores datasets that are not managed by year, such as united_states_medical_licensing_exam.json, kidney_disease_specialty_questions.json, and health_classification.json.
• 2012/ through 2026/ Each year contains:
  • attending/
  • senior/

Each dataset JSON is expected to follow a unified document structure:

{
  "dataset_meta": {
    "dataset_name": "Internal Medicine",
    "year": 2025,
    "subject": "internal_medicine",
    "title_level": "attending",
    "source_note": "To be filled: source note",
    "license_note": "To be filled: license and usage note",
    "usage_note": "To be filled: usage scope note",
    "maintenance_note": "To be filled: maintenance note"
  },
  "items": [
    {
      "question_id": "internal_medicine_000001",
      "question_text": "...",
      "options": {
        "A": "...",
        "B": "..."
      },
      "answer": "A",
      "model_results": {}
    }
  ]
}

If a legacy dataset still uses a raw list[question] format, the runtime automatically wraps it into the unified document format.

result/

result/ mirrors dataset/ exactly.

For example:

• dataset/2025/attending/internal_medicine.json
• result/2025/attending/internal_medicine.json

The difference is that result/ appends model outputs into the model_results field of each question instead of splitting one output file per model.

Example:

{
  "question_id": "internal_medicine_000001",
  "question_text": "...",
  "answer": "A",
  "model_results": {
    "qwen3_8b": {
      "provider": "ollama",
      "model": "qwen3:8b",
      "raw_text": "A",
      "normalized_answer": "A",
      "status": "ok",
      "judge": {
        "is_correct": true,
        "score": 1,
        "judge_method": "rule_based_v1"
      }
    }
  }
}

Model Configuration

models/ollama_models.json

This is the primary model registry used by the benchmark.

Each model entry contains:

• model_id
• model_key
• provider
• family
• size
• mode
• enabled

Notes:

• model_id is the request-time model name, for example qwen3:8b.
• model_key is the stable underscore-style field name used inside result JSON files, for example qwen3_8b.
• mode currently supports think and non_think.
• Models with enabled=true run by default when --models is not specified.

models/model_modes.example.json

This is an optional example file, not the primary configuration.

Its purpose is to show how model mode overrides can be declared separately if the project later decides to externalize mode control more aggressively. The current runtime still prefers the mode field inside models/ollama_models.json, and the example file can be missing without breaking execution.

ollama_pull_llm.py

This script prints an Ollama pull list derived from models/ollama_models.json.

Behavior:

• Prints ollama pull ... commands grouped by think and non_think
• Uses model-family comments to keep the output readable
• Does not download anything automatically
• Leaves the actual download decision to the user

Because many listed models are large, you should review the output first and then selectively run only the commands you really want. This is intentional and helps avoid filling up disk space by accident.

Example:

python3 ollama_pull_llm.py

If you want a text file copy of the generated output:

python3 ollama_pull_llm.py > ollama_pull_llm.txt

Provider Contract

The project currently supports three provider presets:

• ollama
• openrouter
• vllm

All three are routed through the same OpenAI-compatible request adapter.

That means switching providers should mainly require changing:

• --provider
• --base-url
• --api-key

The benchmark flow itself should not need a rewrite for that change.

Provider-specific default environment variables:

• ollama
  • OLLAMA_HOST
  • OLLAMA_BASE_URL
  • OLLAMA_API_KEY
• openrouter
  • OPENROUTER_BASE_URL
  • OPENROUTER_API_KEY
• vllm
  • VLLM_HOST
  • VLLM_BASE_URL
  • VLLM_API_KEY

Shared benchmark environment variables:

• BENCHMARK_PROVIDER
• BENCHMARK_JUDGE_MODEL

Usage

Default run

python3 main.py

Recommended default command for the current Ollama-based setup:

python3 main.py \
  --provider ollama \
  --base-url http://localhost:11434/v1 \
  --api-key ollama \
  --judge-model gpt-oss:120b

Example command for OpenRouter:

python3 main.py \
  --provider openrouter \
  --judge-model gpt-oss:120b

Example command for vLLM:

python3 main.py \
  --provider vllm \
  --base-url http://localhost:8000/v1 \
  --api-key EMPTY \
  --judge-model gpt-oss:120b

For Ollama and vLLM, the default host is localhost. You can keep local development simple with the default .env, or point the benchmark to a different machine by changing either:

• OLLAMA_HOST / VLLM_HOST
• or the full OLLAMA_BASE_URL / VLLM_BASE_URL

Examples for remote servers:

OLLAMA_HOST=10.0.0.25 python3 main.py --provider ollama

VLLM_HOST=10.0.0.30 python3 main.py --provider vllm --api-key EMPTY

If you already know the full endpoint path, setting the full base URL still wins:

OLLAMA_BASE_URL=http://10.0.0.25:11434/v1 python3 main.py --provider ollama

Default behavior:

• Scan dataset/
• Load all models with enabled=true from models/ollama_models.json
• Write results into result/
• Judge answers through the same provider, preferring gpt-oss:120b before falling back to regex/rule-based comparison

Run specific models

python3 main.py --models qwen3:8b,deepseek-r1:8b

Point to a specific Ollama or OpenAI-compatible endpoint

python3 main.py \
  --base-url http://localhost:11434/v1 \
  --api-key ollama \
  --provider ollama

Limit how many new questions each model processes

python3 main.py --limit 100

Disable LLM judging and use regex/rule-based judging only

python3 main.py --judge-model re

Judging And Outputs

Judging

The current implementation uses LLM-first judging with rule-based fallback.

• By default, the runner uses the current provider to call gpt-oss:120b as the judge model.
• The judge model compares the expected answer against the raw model response and must answer only TRUE or FALSE.
• If the judge model is unavailable, returns invalid output, or is explicitly disabled, the system falls back to local regex/rule-based judging.
• Multiple-choice fallback logic extracts uppercase option letters such as A or B,C.
• Health classification fallback logic extracts POSITIVE or NEGATIVE.
• If --judge-model is set to an empty string, re, or regex, the runner skips LLM judging and uses rule-based judging only.
• The judge model uses the same configured provider endpoint as the answer model. This keeps local Ollama-based setups and other OpenAI-compatible deployments operationally consistent.

The judge writes:

• is_correct
• score
• judge_method
• reason

Depending on the path taken, judge_method will look like one of these:

• llm_judge_gpt_oss_120b
• rule_based_v1
• rule_based_fallback_v1

Configuration precedence

The runtime resolves configuration in this order:

• Explicit CLI arguments win first
• Environment variables provide defaults second
• Registry files provide model metadata and mode behavior after the CLI has selected the relevant config paths

Examples:

• --base-url overrides OLLAMA_BASE_URL or OPENAI_BASE_URL
• --api-key overrides OLLAMA_API_KEY or OPENAI_API_KEY
• --judge-model overrides BENCHMARK_JUDGE_MODEL
• --provider overrides BENCHMARK_PROVIDER

When --provider is selected but --base-url / --api-key are omitted, the runner uses provider-specific environment defaults:

• ollama -> OLLAMA_BASE_URL, otherwise http://{OLLAMA_HOST}:11434/v1, plus OLLAMA_API_KEY
• openrouter -> OPENROUTER_BASE_URL, OPENROUTER_API_KEY
• vllm -> VLLM_BASE_URL, otherwise http://{VLLM_HOST}:8000/v1, plus VLLM_API_KEY

Current failure behavior

The current runner is intentionally fail-fast for provider request errors during answer generation. If the main provider call fails, the run stops instead of silently continuing with partially missing outputs. Judge-model failures are treated differently: they fall back to local regex/rule-based judging so the benchmark can remain runnable when the preferred judge model is not installed or does not follow the required TRUE / FALSE contract.

Provider Smoke Tests

The repository includes three separate smoke-test scripts under tests/:

• tests/test_ollama_hello.py
• tests/test_openrouter_hello.py
• tests/test_vllm_hello.py

Each script sends a simple hello request to the provider using gpt-oss:20b and prints the returned text. They are intended as connectivity checks, not as benchmark-quality evaluations.

Token limits

The current defaults are:

• think models: max_tokens = 131072
• non_think models: max_tokens = 4096

Raw result preservation

Each question keeps:

• raw_text
• normalized_answer
• judge

This makes it possible to review whether the model answered correctly and whether it followed the required output format.

Resume Support And Atomic Writes

Resume behavior

The project resumes at the question level.

If a question already contains a valid result for a given model, that question-model pair is skipped on the next run instead of being requested again.

Atomic result writes

Result files are written through a same-directory temporary file flow:

• Write *.temp.json
• Atomically replace the target *.json

This reduces the chance of corrupting the main result file if a run is interrupted.

Current Scope

The current version already covers:

• Standard dataset directory scanning
• Model registration and selection
• OpenAI-compatible provider requests
• Rule-based answer normalization
• Question-level judging
• Incremental result write-back
• Resume support

The current version does not yet cover:

• Concurrent scheduling
• A richer multi-provider registry
• More advanced free-text judging
• Exported benchmark summary reports

Maintenance Guidance

If you keep extending this repository, the safest conventions are:

• dataset/ manages source questions only
• result/ manages outputs only and should not overwrite source question files
• models/ manages model registry data only
• Directory names, file names, JSON field names, and placeholder text should stay in English
• Documentation, comments, and top-level explanations should remain explicit enough that external readers do not need to infer benchmark behavior from the code alone