AGOUTIC: Automated Genomic Orchestrator

Release: 3.6.6 Status: Active Prototype

🧬 Overview

AGOUTIC is a general-purpose agent for analyzing and interpreting long-read genomic data (Nanopore/PacBio). It uses a Dual Interface architecture (REST + MCP) to allow both human users and AI agents to orchestrate complex bioinformatics pipelines.

The system is composed of:

Cortex: Agent Engine - AI-powered orchestration and user interaction
Atlas: Consortium Data Integration - Registry-driven ENCODE and IGVF retrieval via MCP
Launchpad: Execution Engine - Dogme/Nextflow pipeline management (local + remote SLURM)
Analyzer: Analysis Engine - Results analysis and QC reporting
edgePython: Differential Expression — Bulk/single-cell RNA-seq DE via edgePython
XgenePy MCP: Cis/trans regulatory modeling — local XgenePy execution with canonical artifacts
UI: Web interface for monitoring and control

Current status: database infrastructure centralized in common/database.py with Alembic migrations. Gene annotation and enrichment tools moved from edgePython to Analyzer. The analyzer/server4 adapter layer proxies remaining edgePython MCP calls upstream. Atlas now exposes both ENCODE and IGVF consortium MCP servers through the same schema-aware routing and formatting layer.

🖥️ Execution Modes

AGOUTIC supports dual execution modes:

Local: Runs Nextflow/Dogme pipelines directly on the local machine (default, original behavior)
Remote SLURM: Submits jobs to a remote SLURM cluster via SSH

Remote execution features:

Saved SSH profiles — per-user connection profiles with secure key references (no raw secrets stored). Supports local OS user key access through a per-session broker launched under that Unix account with su (password used transiently, never stored)
SLURM resource management — configurable account, partition, CPUs, memory, walltime, GPUs with validation
Shared OpenChromatin GPU runtime defaults — DNA SLURM runs now default to the shared Dogme OpenChromatin GPU container and task-scoped runtime wiring instead of the older custom host-mounted modkit path
Remote base path model — a single remote_base_path anchors ref/, data/, and per-workflow remote directories
Remote browsing and stage-only intake — browse saved-cluster paths and stage references/input data without submitting a job
Stage transfer controls — running stage-only transfers can be refreshed, cancelled, resumed, and failed staging cards can delete their reserved local workflow folders directly from the UI
Result destination policy — keep results remote-only, copy back locally, or both
Staged approval prompts — Cortex collects details progressively, presents summary before submission
Run and staging status tracking — dedicated staging tasks plus remote execution stage labels through completed, including byte-level transfer progress, current-file details, and faster live refresh while transfers are active
Scheduler integration — SLURM job ID tracking, state polling via sacct/squeue, cancellation via scancel

Phase 1 limitation: Analyzer operates on local-accessible files only. Remote results must be copied back before downstream analysis.

See docs/remote_execution_architecture.md for architecture details, docs/cluster_slurm_setup.md for setup, docs/user_guide_execution_modes.md for usage, and TUTORIAL.md for an end-to-end user walkthrough.

🔬 Analysis Capabilities

AGOUTIC combines computational workflow execution with agent-guided biological interpretation. After a pipeline finishes, the platform helps users move from raw output folders to scientific insight, including isoform behavior, modification patterns, pathway shifts, and gene-level functional context.

Analysis Goals

AGOUTIC is designed to help users:

Interpret isoform discovery and transcript structure outputs from long-read workflows
Summarize RNA and DNA modification signals from workflow artifacts
Review QC metrics across runs and identify quality or completeness issues
Inspect gene-level and transcript-level result tables with context
Run downstream differential expression and enrichment analysis
Compare outputs across samples, conditions, and workflows

Supported Analysis Types

Quality Control (QC) analysis
- Parse run summaries, count/stat tables, alignment summaries, and basecalling outputs
- Validate run completeness and surface troubleshooting context from logs and artifacts
- Generate QC summaries for quick review across workflow outputs
Transcriptomic analysis
- Explore gene- and transcript-level quantification outputs
- Support isoform-aware interpretation from long-read RNA/cDNA pipelines
- Review splice-aware and transcript-structure-relevant outputs
Epitranscriptomic and epigenomic analysis
- Summarize RNA modification outputs from direct RNA workflows
- Summarize DNA modification outputs from DNA workflows
- Parse and interpret bedMethyl outputs, including region- and gene-linked review where applicable
Differential analysis
- Run bulk and single-cell RNA-seq differential expression through edgePython
- Compare grouped samples directly from reconciled workflow abundance tables or saved dataframes
- Use the stateful DE flow: load → filter → normalize → design → fit → test → results
- Prefer workflow-local reconciled.gtf annotation when present so transcript-aware plots and summaries stay aligned with the active workflow output
- Filter by FDR/logFC and report annotated top genes for interpretation
Functional interpretation
- Run GO enrichment (BP, MF, CC)
- Run Reactome and KEGG pathway enrichment
- Translate Ensembl IDs and normalize symbols for human and mouse datasets
- Build and reuse colocated GTF-backed gene/transcript caches for workflow-local or custom annotations

For deeper tool-level details, see analyzer/README.md, skills/differential_expression/SKILL.md, skills/enrichment_analysis/SKILL.md, and SKILLS.md.

Typical Analysis Workflow

Pipeline execution ↓ Result discovery ↓ QC parsing and summary generation ↓ Expression / isoform / modification result extraction ↓ Optional differential expression analysis ↓ Functional enrichment ↓ Agent-guided visualization and biological interpretation

What Happens Next in Analysis

Once parsing and summaries are complete, AGOUTIC can pivot into follow-up interpretation tasks such as cross-workflow comparison, condition-focused differential analysis, and targeted functional hypotheses (for example, pathway-level shifts or biologically coherent gene programs).

Analysis Inputs

The analysis layer consumes:

Pipeline result folders (for example workflow1/, workflow2/)
CSV/TSV/BED/bedMethyl files
Counts and summary/statistics tables
Annotation and quantification outputs
User-selected files from workflow subdirectories

Analysis Outputs

The analysis layer returns:

Parsed result tables
QC summaries and run-validation context
Annotated gene/transcript lists
Differential expression result tables
GO/pathway enrichment tables
Interactive plots and chart-ready summaries
Chat-readable scientific interpretation for downstream decisions

Example Analysis Requests

Summarize the QC for workflow2
List the important files in workflow1/annot
Parse the bedMethyl output and summarize methylation patterns
Show the top expressed genes from this result file
Run differential expression between control and treatment
Compare the treated samples treated_1 and treated_2 to the control samples ctrl_1 and ctrl_2
Compare treated_1 and treated_2 to ctrl_1 and ctrl_2 from DF1 at transcript level
/de treated=treated_1,treated_2 vs control=ctrl_1,ctrl_2
Annotate these Ensembl IDs
Run GO enrichment on the upregulated genes
Compare workflow1 and workflow2 outputs

Visualization Support

Inline Plotly visualizations directly in chat
Interactive bar, scatter, heatmap, box, histogram, pie, venn, and upset plots from conversation dataframes
Automatic plotting from parsed tables when chartable data are detected
Cross-workflow open-chromatin overlap requests can run as approval-gated background workflows that write workflow-scoped overlap CSVs and manifests for downstream plotting
Saved venn/upset overlap plots over earlier chat dataframes reload the full source table from analyzer provenance or matching project files when only preview rows are available, so overlap counts stay accurate and two-set venn diagrams remain readable for large overlaps
edgePython-backed DE and enrichment plots now default to 600 dpi raster export with an SVG companion for project-scoped artifact output
Publication-style volcano and MD plots route through edgePython, while generic dataframe charts remain an interactive Plotly path rather than a server-side publication export path

Current Analysis Limitations

Analyzer currently requires local-accessible files
Remote-only results must be copied back before downstream analysis
Some analysis pathways are file-format dependent and assume expected output conventions
Cross-run comparison is strongest when workflows use consistent references and naming
Interpretation depth depends on pipeline completeness and annotation availability

See docs/remote_execution_architecture.md for remote execution constraints and staging/copy-back behavior.

🔒 Security & Multi-User Isolation

AGOUTIC enforces access control at every layer:

Authentication: Google OAuth 2.0 with session cookies (httponly, samesite=lax, secure in production)
Authorization: Role-based access (owner / editor / viewer) checked on every endpoint via require_project_access(). Admins bypass all project-level checks; public projects allow viewer access.
Job ownership: Each job records the submitting user_id. require_run_uuid_access() verifies ownership before exposing debug info or analysis results.
File isolation: User-jailed paths (AGOUTIC_DATA/users/{username}/{project-slug}/) with input sanitization and jail-escape guards; legacy {user_id}/{project_id} paths are still supported for backward compatibility.
Server-side project IDs: UUIDs generated server-side via uuid4() — clients never control the ID.
Project management: Full dashboard for browsing projects, viewing stats/files/jobs, renaming, archiving, and permanent deletion with cascading cleanup.
Bootstrap & admin scripts: Run python scripts/cortex/init_db.py for a fresh database bootstrap, python scripts/cortex/set_usernames.py auto to derive usernames from email addresses on an existing instance, and python scripts/cortex/bootstrap_project_tasks.py to seed persistent project tasks from existing workflow history.

🚀 Quick Start

Installation

# Create environment
conda env create -f environment.yml
conda activate agoutic_core

Run the System

# Recommended: start the full backend stack
./agoutic_servers.sh --start

# Then start the UI separately
streamlit run ui/appUI.py --server.address 0.0.0.0 --server.port 8501

For local development, you can still run services manually:

# Terminal 1: Start Launchpad REST
uvicorn launchpad.app:app --host 0.0.0.0 --port 8003 --reload

# Terminal 2: Start Launchpad MCP
python -m launchpad.mcp_server --host 0.0.0.0 --port 8002

# Terminal 3: Start Cortex
uvicorn cortex.app:app --host 0.0.0.0 --port 8000 --reload

# Terminal 4: Start UI
cd ui && streamlit run appUI.py

Note: running python ui/appUI.py directly will not work correctly because the UI auth flow depends on Streamlit request context and browser cookies.

Verify Installation

# Check Cortex health
curl http://localhost:8000/health

# Check Launchpad health
curl http://localhost:8003/health

# Test Atlas connection
python cortex/atlas_mcp_client.py

# Expected: Connection success and K562 search results

Task System

AGOUTIC now maintains a persistent project task list instead of relying on a hard-coded checklist in the UI.

Tasks are projected from durable workflow records, mainly ProjectBlock state plus job progress payloads.
The chat page groups tasks into pending, running, follow-up, and completed sections.
Parent tasks can include child tasks for workflow stages, per-file download progress, analysis completion, and result review.
Existing history can be backfilled safely with:

python scripts/cortex/bootstrap_project_tasks.py

# Optional: seed only one project
python scripts/cortex/bootstrap_project_tasks.py --project-id <project_id>

📋 Architecture Overview

┌──────────────────────────────────────────────────────────────┐
│                   AGOUTIC System v3.3.2                     │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────┐                                               │
│  │  Web UI  │ (Streamlit)                                   │
│  └────┬─────┘                                               │
│       │ REST API                                            │
│       ↓                                                      │
│  ┌────────────────────────────────────────┐                │
│  │        Cortex (Agent Engine)         │                │
│  │     AI Orchestration + Coordination    │                │
│  └────┬────────────┬────────────┬─────────┘                │
│       │            │            │                           │
│       │ MCP        │ REST       │ MCP                       │
│       ↓            ↓            ↓                           │
│  ┌────────┐  ┌──────────┐  ┌──────────┐                   │
│  │Atlas│  │ Launchpad │  │ Analyzer │                   │
│  │ENCODE  │  │ Nextflow │  │ Analysis │                   │
│  │ Portal │  │ Pipeline │  │  Engine  │                   │
│  └────┬───┘  └─────┬────┘  └─────┬────┘                   │
│       │            │              │                         │
│       ↓            ↓              ↓                         │
│    ENCODE      Dogme          Results                      │
│    Portal    Pipelines         Files                       │
│                                                              │
└──────────────────────────────────────────────────────────────┘

🔧 System Components

Cortex: Agent Engine (Port 8000)

Role: Central orchestrator with LLM reasoning
Tech: FastAPI + OpenAI-compatible LLM
Features:
- Chat interface with skill-based workflows
- Coordinates Atlas, Launchpad, and Analyzer
- Block-based project timeline
- Persistent project task list with child-task projection for downloads, workflow stages, analysis, and follow-up review
- Background job monitoring with stop/cancel buttons
- Download cancel button — "🛑 Cancel Download" on running downloads with partial-file cleanup
- "List my data" command — chat-based central data folder listing (DB + disk fallback)
- Post-cancel workflow management — Delete / Resubmit buttons on cancelled jobs; chat-based deletion via natural language
- User authentication
- Role-based authorization gates on all endpoints
- Server-side project CRUD (POST/GET/PATCH /projects)
- [[PLOT:...]] tag parsing → AGENT_PLOT blocks for inline Plotly charts (histogram, scatter, line, area, bar, box, violin, strip, heatmap, pie, venn, upset)
- Per-message and per-conversation token tracking — every LLM response records prompt_tokens, completion_tokens, total_tokens, and model_name in the database; exposed via GET /user/token-usage (own data) and GET /admin/token-usage (all users)
- find_file echo recovery — when a weak model emits a find_file JSON result verbatim instead of a [[DATA_CALL:...]] tag, the pipeline intercepts the response, auto-chains to parse_csv_file/parse_bed_file/read_file_content, and strips the bad block from conversation history to prevent looping
- ENCODE tool routing guards — structural checks prevent LLM misrouting (e.g. cell-line names sent to get_experiment); assay-only queries are routed to search_by_assay; assay name aliases (e.g. RNA-seq → total RNA-seq, ChIP-seq → TF ChIP-seq) are resolved before MCP dispatch
- IGVF dispatch guards — malformed IGVF calls now backfill missing required sample_term/assay_title values from the user message when possible, and schema validation drops still-invalid calls before they hit MCP
- Tool Schema Contracts — machine-readable JSON Schema for every MCP tool, fetched at startup from /tools/schema endpoints on all servers. Injected into the system prompt as a compact reference and used for pre-call param validation (strip unknown params, check required fields, normalise enums).
- Structured Conversation State — typed ConversationState JSON (skill, project, sample, experiment, dataframes, workflows) built each turn and injected as [STATE]...[/STATE] so the LLM always sees current context
- Error-Handling Playbook — deterministic failure rules in the system prompt + structured [TOOL_ERROR] blocks + single-retry for transient failures
- Output Contract Validator — post-LLM validation catches malformed DATA_CALL tags, duplicate APPROVAL_NEEDED, unknown tools, and mixed sources
- Provenance Tags — [TOOL_RESULT: source, tool, params, rows, timestamp] headers on every tool result for auditability; persisted in AGENT_PLAN blocks
- Plan-Execute-Observe-Replan — structured multi-step planning layer that now runs through manifest-first classification and composition for core deterministic flows. SkillManifest metadata supplies planner triggers, expected inputs, required services, runtime hints, and MCP tool chains; plan_composer.py builds DE, enrichment, and XgenePy plans from that metadata; legacy templates remain as deterministic fallback for unmigrated flows; and CHECK_EXISTING guards still skip expensive operations when results already exist.
- Deterministic skill-management commands — /skills, /skill <skill_key>, and /use-skill <skill_key> expose the live skill catalog, describe individual skills, and persist manual skill switching across turns without relying on a freeform model response.
- Gene Annotation & ID Translation — offline Ensembl gene ID ↔ symbol translation (human + mouse) via pre-built lookup tables. Auto-annotates gene symbols when DE data is loaded; all downstream outputs (top genes, heatmaps, summaries) automatically use readable symbols instead of raw Ensembl IDs. Bidirectional lookup_gene tool answers "what is the Ensembl ID for TP53?" style queries. Pre-LLM auto-skill detection routes gene questions to the correct skill from any context (including Welcome). MCP tools: annotate_genes (edgePython, DE-stateful), translate_gene_ids and lookup_gene (Analyzer).
- Robust DATA_CALL tag parsing — bracket-aware parameter parser handles JSON arrays inside DATA_CALL tags (e.g. gene_symbols=["TP53", "BRCA1"]). Mistral-native [TOOL_CALLS]DATA_CALL: format is auto-normalized to standard [[DATA_CALL:...]] tags.
- Skill-defined plan chains — skill authors can declare multi-step workflows in skill Markdown files under a ## Plan Chains section. A single message like "get K562 experiments and make a plot by assay type" is detected at classify-time and produces both a data search and a visualization. Trigger phrases support multi-phrasing (AND/OR keyword groups) for flexible matching. See SKILLS.md for the full authoring guide.
- Skills system documentation — new top-level SKILLS.md documents the complete skills framework: skill file structure, routing patterns, [[DATA_CALL:...]] / [[PLOT:...]] tag system, plan chains format, and a step-by-step guide for creating new skills.
- Inline Plotly visualizations with deduplication — [[PLOT:...]] tags produce interactive bar, scatter, line, area, violin, strip, pie, histogram, heatmap, venn, and upset charts rendered directly in chat. A three-layer pipeline guarantees chart generation: chain context injection → second-pass PLOT tag instructions → post-DataFrame fallback. Deduplication and prompt-intent selection prevent duplicate/overlapping traces and stale style leakage across turns. Supports explicit colors (color=green), grouped or stacked bar modes, and boolean-column overlap plots for presentation-friendly set comparisons.
- DF inspection quick commands — list dfs lists all dataframes in the conversation with their metadata; head df1 (or head df3 5) shows the first N rows as a markdown table. Both bypass the LLM entirely — zero token cost.
- In-memory dataframe actions — filter, subset, select columns, rename, sort, melt, aggregate, join, and pivot existing conversation dataframes without going through analyzer file calls. Saved transforms can also appear as block-specific PENDING_ACTION controls in the UI.

Atlas: Consortium Data MCPs (ENCODE 8006 / IGVF 8009)

Role: Registry-driven public consortium data retrieval
Tech: fastmcp + Atlas registry/configuration with ENCODE and IGVF MCP servers under atlas/
Features:
- Search ENCODE experiments by biosample/organism/target
- search_by_assay — assay-first ENCODE search (e.g. "how many RNA-seq experiments") across both organisms, returning combined counts and per-organism lists
- Search IGVF measurement sets, analysis sets, prediction sets, files, genes, and samples
- Download and file-metadata helpers for both consortium integrations
- Metadata caching, result formatting, and tool/parameter alias repair
- 30+ MCP tools across the current consortium integrations
- Agent routing guards in Cortex prevent structural misrouting and block invalid required-param calls before MCP execution
Extension pattern: atlas/mcp_server.py extends ENCODELIB's FastMCP server, while atlas/igvf_mcp_server.py and atlas/launch_igvf.py provide the parallel IGVF HTTP MCP server path.
Tool schemas: atlas/tool_schemas.py and atlas/igvf_tool_schemas.py define JSON Schema contracts for the ENCODE and IGVF tools, served via /tools/schema GET endpoints.
Docs: ATLAS_IMPLEMENTATION.md

Launchpad: Execution Engine (Ports 8003 REST / 8002 MCP)

Role: Nextflow pipeline execution
Tech: FastAPI + Nextflow + Dogme
Features:
- Submit Dogme DNA/RNA/cDNA pipelines
- Shared OpenChromatin GPU container defaults for SLURM DNA runs with task-scoped runtime injection for OpenChromatin work
- Real-time job monitoring
- Log streaming
- User-jailed working directories
- Job cancellation — SIGTERM-based cancel with cooperative .nextflow_cancelled marker; properly displays CANCELLED (not FAILED) in UI
- Workflow folder deletion — DELETE endpoint removes work directory and sets status to DELETED; block status updated immediately so UI reflects deletion
- Job resume — resubmit cancelled/failed jobs with Nextflow -resume flag to reuse cached task results in the same workflow directory instead of starting fresh
- Stage-only transfer lifecycle controls — refresh, cancel, resume, and failed-stage cleanup actions are available in both the workflow UI and Task Center
- Live staging telemetry — running transfers surface current file, transferred bytes, total size, and faster refresh cadence during brokered or direct rsync activity
- delete_job_data MCP tool — enables chat-based deletion ("delete workflow1")
Docs: launchpad/README.md

edgePython: Differential Expression (Port 8007)

Role: Bulk and single-cell RNA-seq differential expression analysis
Tech: FastMCP + edgePython
Features:
- Full DE pipeline: load → filter → normalize → design → dispersion → fit → test → results → plots
- Gene list filtering from DE results by FDR, logFC, and direction (up/down/all)
- Gene annotation (annotate_genes) on DE results in-place
- Workflow-local reconciled.gtf preference for annotation when available, with shared reference caches as fallback
- Stateful pipeline — each step builds on previous results within a session
- Volcano, MDS, MA, BCV, heatmap plot generation
- TSV/CSV/JSON result export
- JSON Schema tool contracts via /tools/schema
Docs: edgepython_mcp/

Analyzer: Analysis Engine (Ports 8004 REST / 8005 MCP)

Role: Results analysis, QC reporting, gene annotation, and GO/pathway enrichment
Tech: fastmcp + Python analysis tools + g:Profiler
Features:
- Parse pipeline outputs (CSV, TSV, BED files)
- Generate QC reports and analysis summaries
- File discovery and content reading
- Workflow folder browsing via list_job_files
- Gene ID translation (translate_gene_ids) and bidirectional lookup (lookup_gene) via Ensembl reference tables
- GTF-backed gene and transcript annotation with colocated caches for shared references, workflow-local outputs, or custom user-provided GTFs
- GO enrichment (BP/MF/CC) and pathway enrichment (KEGG/Reactome) via g:Profiler
- Per-conversation enrichment state management
- Species auto-detection from gene ID prefixes (ENSG → human, ENSMUSG → mouse)

Workflow Directory Layout:

$AGOUTIC_DATA/users/{username}/{project-slug}/
├── data/              # Uploaded input data
├── workflow1/         # First job's output
│   ├── annot/         # Annotations, final stats, counts
│   ├── bams/          # BAM alignment files
│   ├── bedMethyl/     # Methylation BED output
│   ├── fastqs/        # FASTQ files
│   └── ...
└── workflow2/         # Second job's output

Agent Commands (handled automatically by Cortex's safety net):
- list my data / list my files — lists all files in your central data folder
- list workflows — lists all workflow folders in the project
- list files / list files in workflow2/annot — lists files in a workflow or subfolder
- parse annot/File.csv — finds and parses a file by relative path
- parse workflow2/annot/File.csv — parses a file in a specific workflow
Docs: analyzer/README.md

📁 Project Structure

agoutic/
├── README.md                     # This file
├── environment.yml               # Conda environment specification
├── alembic.ini                  # Alembic migration configuration
├── CONFIGURATION.md              # Path configuration guide
├── ATLAS_IMPLEMENTATION.md     # Atlas integration guide
├── ATLAS_QUICKSTART.md         # Atlas quick reference
│
├── cortex/                      # Agent Engine
│   ├── README.md                # Cortex documentation
│   ├── app.py                   # FastAPI application
│   ├── agent_engine.py          # AI agent orchestration
│   ├── skill_manifest.py        # Skill capability registry for routing + planning
│   ├── plan_classifier.py       # Manifest-first request classification
│   ├── plan_composer.py         # Manifest-driven deterministic plan builder
│   ├── planner.py               # Planner orchestration + fallback selection
│   ├── plan_executor.py         # Deterministic step execution engine
│   ├── plan_replanner.py        # Failure recovery + plan adjustment
│   ├── dependencies.py          # Auth gates (require_project_access, require_run_uuid_access)
│   ├── user_jail.py             # Path traversal guards & file isolation
│   ├── auth.py                  # Google OAuth 2.0 + cookie hardening
│   ├── models.py                # Database models
│   ├── schemas.py               # Request/response schemas
│   ├── config.py                # Configuration
│   ├── db.py                    # Database connection
│   ├── prompt_templates/        # LLM system prompts (first-pass, planning, second-pass)
│   └── routes/                  # Extracted REST route modules
│
├── launchpad/                      # Execution Engine
│   ├── README.md                # Launchpad documentation
│   ├── app.py                   # FastAPI application
│   ├── nextflow_executor.py    # Nextflow wrapper
│   ├── mcp_tools.py            # MCP tool definitions
│   ├── mcp_server.py           # MCP server
│   ├── models.py               # Database models
│   ├── schemas.py              # Request/response schemas
│   ├── config.py               # Configuration
│   ├── db.py                   # Database connection
│   ├── quickstart.sh           # Quick start setup
│   ├── DUAL_INTERFACE.md       # REST + MCP architecture
│   └── IMPLEMENTATION_SUMMARY.md # Implementation details
│
├── scripts/                     # Manual admin and operational utilities
│   ├── cortex/
│   │   ├── init_db.py           # Fresh database bootstrap utility
│   │   ├── set_usernames.py     # Username/slug admin CLI
│   │   └── bootstrap_project_tasks.py # Backfill persistent project tasks
│   ├── launchpad/
│   │   ├── debug_job.py         # Job inspection helper
│   │   └── submit_real_job.py   # Manual job submission helper
│   └── build_gene_reference.py  # One-time Gencode GTF → TSV builder
│
├── ui/                          # Web Interface
│   ├── README.md               # UI documentation
│   ├── app.py                  # Streamlit main app (chat, sidebar, auto-refresh)
│   └── pages/
│       ├── projects.py         # Projects dashboard (stats, files, bulk actions)
│       ├── results.py          # Job results analysis (auto-lists project jobs)
│       └── admin.py            # Admin user management
│
├── atlas/                      # ENCODE MCP Extension
│   ├── launch_encode.py        # HTTP launcher (imports mcp_server for extensions)
│   ├── mcp_server.py           # Extends ENCODELIB FastMCP with search_by_assay + /tools/schema
│   ├── tool_schemas.py         # JSON Schema contracts for all 16 ENCODE tools
│   ├── config.py               # Atlas configuration
│   └── result_formatter.py     # Result formatting helpers
│
├── edgepython_mcp/              # edgePython DE Server
│   ├── edgepython_server.py    # FastMCP tool definitions (DE + filtering)
│   ├── mcp_server.py           # Server wrapper + /tools/schema endpoint
│   ├── launch_edgepython.py    # HTTP launcher
│   ├── tool_schemas.py         # JSON Schema contracts for DE tools
│   └── config.py               # Configuration
│
├── common/                      # Shared Utilities
│   ├── database.py            # Centralized DB infrastructure (Base, engines, sessions)
│   ├── gene_annotation.py      # Ensembl gene ID ↔ symbol translation (bidirectional)
│   ├── mcp_client.py           # Shared MCP HTTP client
│   ├── logging_config.py       # Structured logging setup
│   └── logging_middleware.py   # Request logging middleware
│
├── skills/                      # Workflow Definitions
│   ├── welcome/SKILL.md                 # New-user onboarding
│   ├── ENCODE_Search/SKILL.md           # ENCODE search skill + routing rules
│   ├── ENCODE_LongRead/SKILL.md         # ENCODE pipeline definition
│   ├── run_dogme_dna/SKILL.md           # DNA pipeline definition
│   ├── run_dogme_rna/SKILL.md           # RNA pipeline definition
│   ├── run_dogme_cdna/SKILL.md          # cDNA pipeline definition
│   ├── analyze_local_sample/SKILL.md    # Sample intake workflow
│   ├── analyze_job_results/SKILL.md     # Post-pipeline results analysis
│   ├── download_files/SKILL.md          # File download workflow
│   ├── differential_expression/SKILL.md # edgePython DE skill
│   ├── enrichment_analysis/SKILL.md     # GO & pathway enrichment skill
│   ├── remote_execution/SKILL.md        # Remote SLURM workflow
│   └── shared/
│       ├── SKILL_ROUTING_PATTERN.md     # Shared skill routing reference
│       └── DOGME_QUICK_WORKFLOW_GUIDE.md # Shared workflow parsing guide
│
└── data/                        # Data & Database (created at runtime)
    ├── database/
    │   └── agoutic_v24.sqlite
    ├── reference/               # Gene annotation reference files
    │   ├── human_genes.tsv
    │   └── mouse_genes.tsv
    ├── launchpad_work/            # Job execution directories
    ├── launchpad_logs/            # Server logs
    └── users/                   # Per-user jailed project dirs

🔑 Key Concepts

Dual Interface Architecture

AGOUTIC provides two complementary interfaces:

REST API - For web clients, dashboards, and scripting
- Traditional HTTP endpoints
- Easy integration with existing tools
- Language-agnostic clients
MCP Protocol - For LLM agents and AI orchestration
- Model Context Protocol (MCP)
- Tools exposed as structured capabilities
- Seamless AI agent integration

See launchpad/DUAL_INTERFACE.md for detailed architecture.

Job Submission Workflow

User Request
    ↓
Cortex (Agent)
  - Interprets intent
  - Plans workflow
  - (Optional) Requests approval
    ↓
Launchpad (Executor)
  - Receives job
  - Generates Nextflow config
  - Submits to cluster/local
  - Monitors progress
    ↓
Dogme Pipeline
  - Basecalling
  - Alignment
  - Quantification
  - Modification calling
    ↓
Results & Reports
  - Return to Agent
  - Display in UI

Supported Analysis Modes

DNA Mode

Genomic DNA and Fiber-seq analysis
Includes modification calling (5mC, 6mA, etc.)
Full basecalling → alignment → quantification pipeline

RNA Mode

Direct RNA-seq analysis
Native RNA modification calling (m6A, pseU, etc.)
Splice-aware alignment

cDNA Mode

Polyubiquitin cDNA and isoform analysis
No modification calling (faster processing)
Transcript quantification focus

⚙️ Configuration

AGOUTIC uses two root path variables with sensible defaults:

# Where source code lives (auto-detected)
export AGOUTIC_CODE=/path/to/agoutic

# Where data/database/jobs live (defaults to $AGOUTIC_CODE/data)
export AGOUTIC_DATA=/path/to/storage

No required environment variables for default local setup. Defaults work automatically, but AGOUTIC_CODE and AGOUTIC_DATA can be overridden if needed. See CONFIGURATION.md for detailed configuration options.

Path Resolution

AGOUTIC_CODE/
├── cortex/          # Agent engine
├── launchpad/          # Execution engine
├── ui/               # Web interface
└── skills/           # Workflow definitions

AGOUTIC_DATA/
├── database/         # SQLite database
├── launchpad_work/     # Job working directories
├── launchpad_logs/     # Server logs
├── logs/             # Structured logs (all servers)
└── users/            # Per-user jailed project dirs
    └── {username}/   # e.g. eli/
        └── {project-slug}/  # e.g. k562-atac-seq/
            ├── data/
            ├── results/
            └── workflow1/

� Logging & Observability

AGOUTIC uses structlog for unified structured logging across all servers. Every log entry is a JSON object written to both per-server and unified log files.

Log Files

Logs are written to $AGOUTIC_DATA/logs/:

$AGOUTIC_DATA/logs/
├── agoutic.jsonl          # Unified log (all servers)
├── cortex.jsonl          # Cortex only
├── launchpad-rest.jsonl     # Launchpad REST API
├── launchpad-mcp.jsonl      # Launchpad MCP
├── analyzer-rest.jsonl     # Analyzer REST API
├── analyzer-mcp.jsonl      # Analyzer MCP
├── encode-mcp.jsonl       # ENCODE MCP server
├── *.log                  # Raw stdout/stderr (safety net)
└── *.YYYYMMDD_HHMMSS.*   # Rotated previous logs

Reading Logs

# Stream the unified log
tail -f $AGOUTIC_DATA/logs/agoutic.jsonl | jq .

# Filter by server
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.server == "cortex")'

# Filter by log level
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.level == "error")'

# Filter requests by path
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.path == "/chat")'

# Find slow requests (>1s)
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.duration_ms > 1000)'

# Trace a request across servers by request_id
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.request_id == "some-uuid")'

Request Tracing

Every HTTP request receives a unique X-Request-ID header. This ID is:

Bound to all log entries emitted during the request
Returned in the response X-Request-ID header
Available at request.state.request_id in route handlers

Log Rotation

When servers are started or restarted via agoutic_servers.sh, existing log files are automatically renamed with a timestamp (e.g., cortex.20260213_143052.jsonl). Empty log files are skipped.

Environment Variables

Variable	Default	Description
`AGOUTIC_LOG_FORMAT`	`json`	Set to `dev` for coloured human-readable console output

�🚀 Running the System

Development Mode

# Recommended for a full local stack
./agoutic_servers.sh --start

# Start the UI separately
streamlit run ui/appUI.py --server.port 8501

If you need manual development startup:

# Terminal 1: Start Launchpad
cd /path/to/agoutic
uvicorn launchpad.app:app --port 8003 --reload

# Terminal 2: Start Launchpad MCP
python -m launchpad.mcp_server --host 0.0.0.0 --port 8002

# Terminal 3: Start Cortex
uvicorn cortex.app:app --port 8000 --reload

# Terminal 4: Start UI (if using Streamlit)
cd ui && streamlit run appUI.py

Using MCP Interface

from cortex.mcp_client import LaunchpadMCPClient

# Connect to MCP server
client = LaunchpadMCPClient()
await client.connect()

# Submit a job
job = await client.submit_dogme_job(
    project_id="proj_001",
    sample_name="liver_dna",
    mode="DNA",
    input_directory="/data/pod5"
)

# Monitor progress
status = await client.check_nextflow_status(job["run_uuid"])

Using REST API

# Submit job
curl -X POST http://localhost:8003/jobs/submit \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "proj_001",
    "sample_name": "liver_dna",
    "mode": "DNA",
    "input_directory": "/data/pod5"
  }'

# Check status
curl http://localhost:8003/jobs/{run_uuid}/status

# Get results
curl http://localhost:8003/jobs/{run_uuid}

📚 Documentation

Component Documentation

cortex/README.md - Agent Engine documentation
launchpad/README.md - Execution Engine documentation
ui/README.md - Web UI documentation

Configuration & Setup

CONFIGURATION.md - Full configuration guide
QUICK_REFERENCE.md - Path configuration quick start
docs/DATAFRAMES.md - Dataframe commands, transforms, plotting, and memory guide

Architecture & Details

docs/DATAFRAMES.md - Dataframe commands, transforms, plotting, and memory guide
launchpad/DUAL_INTERFACE.md - REST + MCP architecture
launchpad/IMPLEMENTATION_SUMMARY.md - Implementation details

🧪 Testing

Run All Tests

The project has 1068 tests providing comprehensive coverage.

# Run the full test suite (1068 tests)
pytest tests/ -q

# Cortex tests only
pytest tests/cortex/ -q

# With coverage report
pytest tests/cortex/ --cov=cortex/app --cov-report=term-missing

# Other components
pytest tests/atlas/ tests/common/ tests/analyzer/ tests/launchpad/ tests/ui/ -q

# Single test file
pytest tests/cortex/test_chat_data_calls.py -x -q

# Focused task lifecycle and hierarchy coverage
pytest tests/cortex/test_project_endpoints.py -q

Test Architecture

In-memory SQLite with StaticPool for fast, isolated tests
Mocked LLM via AgentEngine patches (no real model calls)
Mocked MCP via MCPHttpClient patches (no real service connections)
37 cortex test files covering: chat endpoint, approval gates, background tasks, project management, block endpoints, conversations, auth, admin, downloads, uploads, pure helpers, tool routing, skill detection, validation, planning
Task coverage includes persistent task projection, task actions, download-file children, and workflow-stage children in tests/cortex/test_project_endpoints.py
Shared fixtures in tests/conftest.py for DB engine, sessions, mock users

Bootstrap Existing History

If you deploy this release onto a server that already has projects and stored workflow blocks, run the task bootstrap once after the application starts:

python scripts/cortex/bootstrap_project_tasks.py

The script is idempotent: it reconciles the current workflow state and is safe to re-run.

Run Demo

# Interactive demo for Launchpad
python launchpad/demo_launchpad.py

🐛 Troubleshooting

Server Won't Start

Check port availability: lsof -i :8003 (Launchpad REST), lsof -i :8002 (Launchpad MCP), or lsof -i :8000 (Cortex)
Check database connectivity: python -c "from launchpad.db import SessionLocal; SessionLocal()"
Check Python version: python --version (requires 3.12+)

Job Stuck in RUNNING

Check Nextflow process: ps aux | grep nextflow
Check logs: tail -f $AGOUTIC_DATA/launchpad_logs/*.log
Cancel job: curl -X POST http://localhost:8003/jobs/{run_uuid}/cancel

Configuration Issues

Verify configuration: python -c "from launchpad.config import *; print(f'Code: {AGOUTIC_CODE}')"
Check paths: ls -la $AGOUTIC_DATA/launchpad_work

📊 Performance

GPU Concurrency (Per-Pipeline)

Control the number of simultaneous GPU tasks (dorado basecalling, openChromatin) within a single pipeline run. Configurable in the approval form or via environment variable:

export DEFAULT_MAX_GPU_TASKS=8  # Optional explicit limit. Leave unset for no maximum. Range: 1-16

If DEFAULT_MAX_GPU_TASKS is unset, Launchpad omits Nextflow maxForks for GPU-bound Dogme processes and lets Nextflow manage concurrency. Users can also override per-job in the approval form dropdown or via chat ("limit dorado to 3 concurrent tasks").

Concurrent Jobs

Limit concurrent jobs to avoid resource exhaustion:

export MAX_CONCURRENT_JOBS=2  # Adjust based on server capacity

Database Selection

Development: SQLite (default, no setup required — create_all() at startup)
Production: PostgreSQL with Alembic migrations (alembic upgrade head)

# Set via environment variable
DATABASE_URL = "postgresql://user:pass@localhost/agoutic"

📝 Workflow Skills

Pre-defined bioinformatics workflows are available in skills/:

welcome/SKILL.md - New-user onboarding
ENCODE_Search/SKILL.md - ENCODE search and data discovery
ENCODE_LongRead/SKILL.md - ENCODE consortium workflow
run_dogme_dna/SKILL.md - Genomic DNA analysis workflow
run_dogme_rna/SKILL.md - Direct RNA-seq workflow
run_dogme_cdna/SKILL.md - cDNA isoform workflow
analyze_local_sample/SKILL.md - Sample intake and validation
analyze_job_results/SKILL.md - Post-pipeline results analysis
download_files/SKILL.md - File download orchestration
differential_expression/SKILL.md - edgePython DE pipeline (with gene annotation)
enrichment_analysis/SKILL.md - GO & pathway enrichment analysis
remote_execution/SKILL.md - Remote execution workflow
shared/SKILL_ROUTING_PATTERN.md and shared/DOGME_QUICK_WORKFLOW_GUIDE.md - Shared reference docs

🤝 Contributing

Code Structure

Use type hints throughout
Write tests for new features
Document configuration changes
Update this README for major changes

Testing Requirements

# Run full test suite (1040+ tests)
pytest tests/ -q

# With coverage
pytest tests/ --cov=cortex --cov=launchpad --cov-report=html

📞 Support

Check CONFIGURATION.md for configuration issues
Check launchpad/README.md for execution engine issues
Check cortex/README.md for agent engine issues

📦 Version Information

Release: 3.6.6 — rerun-heavy Streamlit polling paths now close short-lived API responses eagerly, auth helper calls use the same eager-close path, publication controls no longer leak threads or file descriptors on figure-heavy project pages, and projects with saved figures render chats and plots correctly on first visit
Python: 3.12+
FastAPI: Latest (from environment.yml)
SQLAlchemy: 2.0+
Nextflow: >= 23.0
Status: Active Development

🗓️ Development Timeline

complete: Core infrastructure, dual interface, MCP integration
complete: Web UI job monitoring, approval gates, project management
complete: Plan-execute-observe-replan, gene annotation, expanded templates
complete: Centralized DB, Alembic migrations, enrichment tools in Analyzer
complete: Cortex modularisation and DE adapter integration
current: Manifest-driven planning, workflow-local annotation, overlap workflows, and remote execution hardening
next: Production deployment preparation

Name		Name	Last commit message	Last commit date
Latest commit History 269 Commits
.vscode		.vscode
alembic		alembic
analyzer		analyzer
archive		archive
atlas		atlas
common		common
cortex		cortex
data		data
docs		docs
edgepython_mcp		edgepython_mcp
launchpad		launchpad
scripts		scripts
skills		skills
tests		tests
ui		ui
xgenepy_mcp		xgenepy_mcp
.coverage		.coverage
.env		.env
.gitignore		.gitignore
AGOUTIC_CODE_PATHS.md		AGOUTIC_CODE_PATHS.md
CHANGELOG.md		CHANGELOG.md
CONFIGURATION.md		CONFIGURATION.md
LICENSE		LICENSE
QUICK_REFERENCE.md		QUICK_REFERENCE.md
README.md		README.md
SKILLS.md		SKILLS.md
TUTORIAL.md		TUTORIAL.md
VERSION		VERSION
agoutic_servers.sh		agoutic_servers.sh
alembic.ini		alembic.ini
environment.yml		environment.yml
load_env.sh		load_env.sh
pytest.ini		pytest.ini

Uh oh!

Folders and files

Latest commit

History

Repository files navigation