hercules-system-design.md

Hercules Job System Design

Overview

The Hercules job system is a powerful background task processing system designed to handle asynchronous tasks with features like prioritization, throttling, and scheduling. This document outlines the design for a Node.js/TypeScript implementation that can be open-sourced.

Core Components

1. Models

Task Model

interface Task {
  id: string;
  name: string;
  moduleName: string;
  functionName: string;
  args: any[];
  kwargs: Record<string, any>;
  state: TaskState;
  priority: Priority;
  runAt: Date;
  createdAt: Date;
  updatedAt: Date;
  completedAt?: Date;
  throttleKey?: string;
  uniqueKey?: string;
  retries: number;
  maxRetries: number;
  executionNamespace: string;
  timeoutLimit: number;
  isRecurring: boolean;
  exception?: string;
  exceptionTrace?: string;
  tags: Record<string, string>;
  criticality: Criticality;
}

Throttle Model

interface Throttle {
  key: string;
  concurrent: number;
  executionNamespace: string;
}

Callback Model

interface Callback {
  id: string;
  moduleName: string;
  functionName: string;
  args: any[];
  kwargs: Record<string, any>;
}

2. Enums

Task State

enum TaskState {
  WAITING = 'W',
  READY = 'R',
  EXECUTING = 'X',
  COMPLETE = 'C',
  FAILED = 'F',
  THROTTLED = 'T'
}

Priority

enum Priority {
  VERY_LOW_PRIORITY = -10000,
  NICE_TO_HAVE = -1000,
  DEFAULT = 0,
  HIGH_PRIORITY = 100,
  ALMOST_LIVE = 900,
  LIVE = 1000
}

Criticality

enum Criticality {
  CRITICAL = 'CRITICAL',
  HIGH = 'HIGH',
  MEDIUM = 'MEDIUM',
  LOW = 'LOW',
  INFO = 'INFO'
}

3. Core Services

Task Service

Responsible for creating, updating, and retrieving tasks.

interface TaskService {
  createTask(task: Partial<Task>): Promise<Task>;
  updateTask(id: string, updates: Partial<Task>): Promise<Task>;
  getTask(id: string): Promise<Task | null>;
  getTasks(filter: Partial<Task>, limit?: number, sort?: Record<string, 1 | -1>): Promise<Task[]>;
  deleteTask(id: string): Promise<boolean>;
}

Runner Service

Executes tasks from the queue.

interface RunnerService {
  start(): Promise<void>;
  stop(): Promise<void>;
  runOne(): Promise<boolean>;
  popTask(): Promise<Task | null>;
  executeTask(task: Task): Promise<any>;
}

Scheduler Service

Schedules recurring tasks.

interface SchedulerService {
  start(): Promise<void>;
  stop(): Promise<void>;
  scheduleTask(task: Task, schedule: Schedule): Promise<Task>;
  unscheduleTask(taskId: string): Promise<boolean>;
}

Task Controller Service

Manages task state transitions.

interface TaskControllerService {
  start(): Promise<void>;
  stop(): Promise<void>;
  moveWaitingToReady(): Promise<number>;
  moveThrottledToWaiting(): Promise<number>;
  moveStuckExecutingToWaiting(): Promise<number>;
}

Throttle Service

Controls concurrency of task execution.

interface ThrottleService {
  shouldThrottle(task: Task): Promise<boolean>;
  throttleTask(task: Task): Promise<Task>;
  unthrottleTask(task: Task): Promise<Task>;
  getThrottleKey(task: Task): string;
  setThrottleLimit(key: string, concurrent: number): Promise<void>;
  getThrottleLimit(key: string): Promise<number>;
}

4. Decorators

Task Decorator

function herculesTask(options?: {
  priority?: Priority;
  timeout?: number;
  maxRetries?: number;
  throttle?: { concurrent: number, throttleKey?: string };
  uniqueKey?: string | ((args: any[]) => string);
  executionNamespace?: string;
  criticality?: Criticality;
  completedCallback?: Function;
  failedCallback?: Function;
}): MethodDecorator;

Recurring Task Decorator

function recurringTask(options: {
  schedule: {
    cron?: string;
    daily?: string[];
    hourly?: string[];
  };
  uniqueKey: string;
  priority?: Priority;
  timeout?: number;
  maxRetries?: number;
  executionNamespace?: string;
  criticality?: Criticality;
}): MethodDecorator;

Architecture

Database Layer

• PostgreSQL for task storage
• Redis for throttling and distributed locking

Queue Layer

• Redis-based queue for task prioritization
• SQS-like interface for task distribution

Worker Layer

• Multiple worker processes/threads for task execution
• Graceful shutdown and restart capabilities

API Layer

• REST API for task management
• WebSocket for real-time task status updates

Implementation Plan

Phase 1: Core Models and Database Layer

• Implement Task, Throttle, and Callback models
• Set up PostgreSQL connection and repositories
• Implement basic CRUD operations

Phase 2: Queue and Worker Layer

• Implement Redis-based queue
• Implement worker process for task execution
• Implement basic throttling mechanism

Phase 3: Scheduler and Controller

• Implement scheduler for recurring tasks
• Implement task controller for state transitions
• Implement heartbeat mechanism

Phase 4: API and Monitoring

• Implement REST API for task management
• Implement WebSocket for real-time updates
• Implement monitoring and metrics

Phase 5: Decorators and Client SDK

• Implement task decorators
• Implement client SDK for easy integration
• Write documentation and examples