Aztec Butler

A tool for helping out with managing multiple aztec nodes with an opinionated workflow and set-up.

Requirements

• Node.js v22.0.0 or higher

Development

Running CLI Commands

The shell scripts in scripts/ use tsx for TypeScript execution (compatible with Node.js v22):

# Run any script directly
./scripts/scrape-coinbases.sh --network mainnet
./scripts/get-queue-stats.sh --network mainnet

# Or run CLI commands directly with tsx
npx tsx cli.ts scrape-coinbases --network mainnet
npx tsx cli.ts get-queue-stats --network mainnet --json

Building for Production

npm run build
npm start -- serve --network mainnet

Attester State Diagram

stateDiagram-v2
    [*] --> NEW: Attester keys created

    NEW --> IN_STAKING_PROVIDER_QUEUE: isInProviderQueue = true<br/>(attester added to provider queue)

    IN_STAKING_PROVIDER_QUEUE --> ROLLUP_ENTRY_QUEUE: !isInProviderQueue && onChainView.status = NONE<br/>(left provider queue, now in rollup entry queue)
    IN_STAKING_PROVIDER_QUEUE --> ACTIVE: !isInProviderQueue && onChainView.status = VALIDATING<br/>(left provider queue, went directly active)

    ROLLUP_ENTRY_QUEUE --> ACTIVE: onChainView.status = VALIDATING<br/>(began validating)

    ACTIVE --> NO_LONGER_ACTIVE: onChainView.status = EXITING or ZOMBIE<br/>(attester withdrawn/slashed)

    NO_LONGER_ACTIVE --> [*]

Loading

State Descriptions:

• NEW: Initial state for newly discovered attesters (not yet in any on-chain queue)
• IN_STAKING_PROVIDER_QUEUE: Attester address exists in the staking provider's queue on the staking contract
• ROLLUP_ENTRY_QUEUE: Attester has an on-chain view in the rollup contract with status = NONE (in global entry queue, waiting to become active)
• ACTIVE: Attester is actively validating on the rollup contract (status = VALIDATING)
• NO_LONGER_ACTIVE: Attester has been withdrawn or slashed (terminal state)

State Transitions (purely on-chain):

State transitions are determined exclusively by on-chain data:

• isInProviderQueue: Attester address exists in the staking provider's queue array (from staking contract)
• onChainView: Attester data from rollup contract (includes status and other attester info)
• onChainView.status: On-chain status from rollup contract (NONE, VALIDATING, ZOMBIE, EXITING)

Coinbase Tracking (separate from state):

Coinbase addresses are tracked separately for operational awareness but do not affect state transitions. The entry queue scraper reports which attesters are missing coinbase configuration via:

• providerNextMissingCoinbaseArrivalTimestamp: When the next attester without coinbase will reach the front of the queue
• providerNextMissingCoinbaseAddress: Address of that attester

See src/server/state/index.ts and src/server/state/transitions.ts for implementation details.

Documentation

• Operator Guide - Complete guide for validator key management (generate, deploy, register)
• Daemon Setup - Run aztec-butler as a systemd service with Prometheus metrics

Configuration

Staking Registry Targeting

Proposal-related CLI commands support selecting which staking registry to target:

• native (default)
• olla

Supported commands:

• add-keys
• get-provider-id
• get-create-staking-provider-calldata
• new-publisher-keys
• process-private-keys
• prepare-deployment

Use --registry <native|olla> (defaults to native if omitted).

Example:

# Default (native)
aztec-butler get-provider-id 0xYourAdminAddress --network mainnet

# Explicit Olla target
aztec-butler get-provider-id 0xYourAdminAddress --network mainnet --registry olla

For Olla target support, configure:

OLLA_AZTEC_STAKING_REGISTRY_ADDRESS=0x...
OLLA_AZTEC_STAKING_PROVIDER_ADMIN_ADDRESS=0x...
OLLA_REWARDS_COINBASE_ADDRESS=0x...

If required Olla variables are missing for the selected command and --registry olla is used, the CLI fails fast with a clear error.

For process-private-keys, GCP Secret Manager naming uses Ethereum network naming derived from ETHEREUM_CHAIN_ID (not NETWORK):

• 1 -> mainnet
• 11155111 -> sepolia
• any other chain -> chain-<id>

Example: with NETWORK=testnet and ETHEREUM_CHAIN_ID=11155111, secrets are created as web3signer-sepolia-....

Secret naming format:

• Attester keys (ETH/BLS): web3signer-<network>-<eth|bls>-att-<id>-<publicKey>
• Publisher keys (ETH): web3signer-<network>-eth-pub-<publicKey>

Publisher secrets are keyed by publisher public address (no validator index in the secret name), so shared publishers across multiple validators reuse the same secret.

process-private-keys is now interruption-safe for GCP secret uploads: if a secret exists for a key but has no enabled versions, rerunning the command appends the missing version instead of skipping it.

Duplicate checks for add-keys and process-private-keys are performed across both registries (where available), not only the selected target. If one registry is unavailable or unconfigured, Butler warns and continues checking the other registry.

Olla ABI Sync

Olla ABI support is sourced from local olla-core artifacts via:

npm run sync:olla-abi

By default, the sync script reads from ../olla-core. You can override this using OLLA_CORE_PATH.

Note: scraper support/refactors are intentionally out of scope for this change.

Unified Keys File Format

Aztec Butler uses a unified configuration format for both validator nodes and the monitoring server. Keys files follow the naming convention:

[network]-keys-[serverId]-v[version].json

Examples:

• mainnet-keys-A-v1.json
• mainnet-keys-B-v2.json
• testnet-keys-validator1-v3.json

File structure:

{
  "schemaVersion": 1,
  "remoteSigner": "https://signer.example.com:8080",
  "validators": [
    {
      "attester": {
        "eth": "0x...",
        "bls": "0x..."
      },
      "coinbase": "0x...",
      "feeRecipient": "0x...",
      "publisher": "0x..."
    }
  ]
}

Server Auto-Discovery:

The monitoring server automatically discovers and merges all keys files matching the pattern {network}-keys-*.json in the data directory. For each server ID, only the highest version number is loaded to avoid conflicts.

Workflow:

1. Generate keys files:

   aztec-butler prepare-deployment \
     --production-keys existing-keys.json \
     --new-public-keys new-keys.json \
     --available-publishers publishers.json \
     --network mainnet

2. Populate coinbase addresses:

   # Scrape coinbase addresses from on-chain events
   aztec-butler scrape-coinbases --network mainnet
   
   # Fill coinbases into keys files
   aztec-butler fill-coinbases --network mainnet --keys-file mainnet-keys-A-v1.json

3. Deploy:

   # Copy keys files to monitoring server's data directory
   scp mainnet-keys-A-v1.json server:~/.local/share/aztec-butler/
   
   # Start monitoring server (auto-discovers keys files)
   aztec-butler start-server --network mainnet

Note: The coinbase field is optional in keys files. New validators created by prepare-deployment won't have coinbase addresses initially. Use the fill-coinbases command after running scrape-coinbases to populate them.

Running as a Service

To run aztec-butler as a systemd service, see the daemon setup guide. The daemon runs the butler in server mode, providing Prometheus metrics and automated monitoring for your Aztec nodes.

Roadmap

1. merge attester-scraper and entry-queue stats scraper. (they both scrape from the same resource and can be done in one go)