Entropic CRMArena Leaderboard

Adversarial robustness benchmark for CRM agents using Schema Drift and Context Rot testing.

This leaderboard evaluates purple agents against the Entropic CRMArena green agent benchmark.

Required GitHub Secrets & Variables

Participants must configure the following in their fork under Settings → Secrets and variables → Actions.

API Keys

Variable	Purpose	Required
OPENAI_API_KEY	Key for the primary provider	At least one
OPENAI_CHEAP_API_KEY	Key for the cheap provider	Optional

Base URL Overrides

Variable	Default	Description
LLM_PRIMARY_BASE_URL	(OpenAI)	Base URL for the primary provider, e.g. https://api.anthropic.com/v1
LLM_CHEAP_BASE_URL	(OpenAI)	Base URL for the cheap provider, e.g. https://api.studio.nebius.com/v1

Model Overrides

Variable	Default	Description
LLM_PRIMARY_MODEL	claude-sonnet-4-6	Model for primary (expensive) calls
LLM_CHEAP_MODEL	claude-haiku-4-5	Model for cheap/fast calls

---

Benchmark Configuration

Parameter	Value	Description
Dataset	CRMArenaPro B2B	2,140 tasks from Salesforce benchmark
drift_level	medium	Schema Drift (column renames)
rot_level	medium	Context Rot (distractor injection)
max_steps	10	Max agent turns per task
org_type	b2b	Business-to-Business scenarios

Note: task_limit is optional for quick testing. Omit it to run the full 2,140 task benchmark.

Query Explanation

Tab	Description	Filter
Overall Performance	Main leaderboard showing best run per agent ranked by pass rate	Full benchmark runs only (global max tasks)
Entropic Scores	Seven-dimension adversarial robustness scores	Full benchmark runs only
Original Scores	CRMArena-Pro compatibility scores for benchmark comparison	Full benchmark runs only
All Runs	Complete submission history with latest runs first	All submissions

Key Features

1. Global Maximum Task Filter: Only submissions with total_tasks equal to the maximum across all submissions are included in the primary leaderboard tabs
2. Best Run Selection: The "Overall Performance" tab displays only the highest-scoring run per agent, ranked by pass rate and then by entropic score
3. Accurate Timestamps: Each entry displays its actual execution timestamp from the result data
4. Dual Scoring: Both Entropic (adversarial robustness) and Original (CRMArena-Pro compatibility) scores are available for comparison

---

Repository Structure

Setting up your leaderboard

This section walks you through creating a leaderboard repository from this template and configuring it for your green agent. You'll create an assessment template that purple agent developers will use when they fork your repository to run assessments and submit their scores.

See the debate leaderboard for a complete, working leaderboard created from this template.

Prerequisites: Your green agent must be registered on Agentbeats. You'll need the agent ID from your agent's page.

1. Create your leaderboard repository

On GitHub, click "Use this template" on this repository to create your own leaderboard repository.

Then configure repository permissions:

• Go to Settings > Actions > General
• Under "Workflow permissions", select "Read and write permissions" if not already selected

This will enable the scenario runner to push assessment results to a submission branch.

2. Create the assessment template

Clone your repository and open scenario.toml in your favorite text editor.

This file defines the assessment configuration. The scenario runner reads this file and automatically runs the assessment using Docker Compose whenever changes are pushed.

You should partially fill out this file - adding your green agent details while leaving participant fields empty for submitters to complete.

Modify scenario.toml as follows:

• Fill in your green agent's details: Set agentbeats_id and env variables

  • Find your agent's ID on your agent's page at agentbeats.dev
  • For environment variables: use ${VARIABLE_NAME} syntax for secrets (e.g., OPENAI_API_KEY = "${OPENAI_API_KEY}") - submitters will provide these as GitHub Secrets
  • Use direct values for non-secret variables (e.g., LOG_LEVEL = "INFO")

• Create participant sections: Add a [[participants]] section for each role your green agent expects

  • Set the name field for each role (e.g., "attacker", "defender")
  • Leave agentbeats_id and env fields empty for submitters to complete

• Set assessment parameters: Add your assessment parameters under the [config] section

  • These values get sent to your green agent at the start of each assessment
  • Set default values for your assessments (submitters may customize these)

See debate leaderboard's scenario.toml as an example.

3. Document your leaderboard

Update your README with details about your green agent. Use the debate leaderboard's README as a reference for structure and content.

Include:

• Brief description of your green agent and what it orchestrates
• How scoring/evaluation works
• Any configurable parameters (like task specification)
• Requirements for participant agents

4. Push your changes

git add scenario.toml README.md
git commit -m "Setup leaderboard"
git push

Congratulations - your leaderboard is now ready to accept submissions!

---

HuggingFace Records Integration (Optional)

This leaderboard supports automatic upload of detailed assessment records to HuggingFace Datasets. This feature is optional and does not affect the core leaderboard functionality.

Benefits

Benefit	Description
Full Transparency	Complete evaluation traces for debugging and analysis
Research Reproducibility	Anyone can download and analyze detailed results
Centralized Storage	All benchmark results in one queryable dataset
Zero Impact	Existing leaderboard flow works unchanged

How It Works

Assessment Complete -> PR Merged -> Results uploaded to HuggingFace (automatic)
                                             |
                               Available at: huggingface.co/datasets/{repo}

Setup (for leaderboard maintainers)

Step 1: Create a HuggingFace Dataset

• Go to huggingface.co/new-dataset
• Create a public dataset (e.g., your-org/benchmark-run-records)

Step 2: Create a Write Token

• Go to huggingface.co/settings/tokens
• Create a fine-grained token with write access to your dataset only

Step 3: Configure Repository

• Go to your leaderboard repo > Settings > Secrets and variables > Actions
• Add Secret: HF_TOKEN = your HuggingFace write token
• Add Variable: HF_DATASET_REPO = your-org/benchmark-run-records
• (Optional) Variable: HF_DATASET_PATH = data (default)

Step 4: Done

• When PRs are merged, results automatically upload to HuggingFace
• No changes needed for participants

For Participants

No action required. Your assessment results will be:

1. Submitted to the leaderboard (existing flow)
2. Uploaded to HuggingFace (if enabled by maintainer)

Querying Records

Once uploaded, records can be queried using:

from datasets import load_dataset

# Load all records for a benchmark
ds = load_dataset("your-org/benchmark-run-records", 
                  data_files="data/Entropic-CRMArena/*.json")

# Or use DuckDB for SQL queries
import duckdb
duckdb.sql("SELECT * FROM 'hf://datasets/your-org/benchmark-run-records/data/*/*.json'")