GraphRAG Kit

English | 中文

---

Introduction

GraphRAG Kit is a universal knowledge-graph (GraphRAG) pipeline built on LightRAG (HKUST-NLP), covering the full chain from raw documents to a queryable knowledge graph:

Document parsing (MinerU) → Preprocessing → Graph building / incremental updates → Multi-mode retrieval → Reasoning-path visualization → Entity governance

Features

• Full pipeline: PDF/DOCX/PPTX/images → Markdown → cleaned text → knowledge graph + vector indexes → QA / deterministic lookup / visualization.
• MinerU client: MinerU v3 HTTP API adapter with skeleton-output detection, HTTP 409 retry, and local fallbacks (pdfplumber for PDF, python-docx for DOCX).
• Incremental updates: content-hash change detection (add/modify/delete/reindex), doc_id indexing, delete-failure rollback, query-cache invalidation, storage backups, and a watchdog daemon.
• Multi-mode retrieval: naive / local / global / hybrid plus auto intent routing (keyword rules) and a circuit breaker for local-mode timeouts (30-min TTL).
• Query enhancement: exact-match short-circuit for code-like terms (model numbers, standard IDs) with controlled fuzzy correction, context noise filtering, and file-catalog injection for provenance questions.
• Reasoning-path visualization: every query can print text reasoning chains (A --[relation]--> B); Mermaid graph generation included.
• Entity governance: merge normalized duplicate entities, purge single-char / spec-value pseudo entities (online API and offline batch paths), and verify embedding compatibility when switching embedding backends.
• Deterministic lookup: graphrag lookup queries 1-hop relations with no LLM and no network, emitting structured JSON for embedding into any pipeline.

Architecture

┌──────────────────────────────────────────────────────────────┐
│ Raw documents (PDF/DOCX/PPTX/images)                          │
└──────────────────────────┬───────────────────────────────────┘
                           │ ① graphrag parse (MinerU v3 API / local fallback)
                           ▼
┌──────────────────────────────────────────────────────────────┐
│ Raw Markdown layer   ./raw_docs/*.md                          │
└──────────────────────────┬───────────────────────────────────┘
                           │ ② Preprocessing (building/preprocess.py, pre_update.py)
                           ▼
┌──────────────────────────────────────────────────────────────┐
│ Processed text layer   ./data/processed/*.txt + manifest.json │
└──────────────────────────┬───────────────────────────────────┘
                           │ ③ Ingest / incremental update (building/ingest.py, update.py, incremental.py)
                           ▼
┌──────────────────────────────────────────────────────────────┐
│ LightRAG index layer   ./data/storage/                        │
│   graph_chunk_entity_relation.graphml (knowledge graph)       │
│   vdb_entities / vdb_relationships / vdb_chunks (vector DBs)  │
│   kv_store_*.json (KV cache & doc status)                     │
└──────────────────────────┬───────────────────────────────────┘
                           │ ④ Retrieval (querying/)
                           ▼
┌──────────────────────────────────────────────────────────────┐
│ graphrag query (LLM QA + reasoning chain) / graphrag lookup   │
└──────────────────────────┬───────────────────────────────────┘
                           │ ⑤ Governance & diagnostics
                           ▼
┌──────────────────────────────────────────────────────────────┐
│ inspection/doctor.py, inspection/trace.py,                    │
│ maintenance/normalize_entities.py, purge_noise_offline.py,    │
│ verify_embedding_compat.py                                    │
└──────────────────────────────────────────────────────────────┘

Code Structure

src/graphrag_kit/
├── cli.py                     # CLI entry (thin argparse subcommand wrappers)
├── client.py                  # Shared layer: config loading, LLM/embedding adapters, RAG init, service probing
├── parsing/
│   └── mineru.py              # MinerU v3 API client + skeleton detection + 409 retry + local fallback
├── building/
│   ├── preprocess.py          # Full preprocessing (HTML tables/LaTeX/TOC/image-ref cleanup)
│   ├── pre_update.py          # Incremental preprocessing (changed files only)
│   ├── ingest.py              # Full ingest (schema-driven entity/relation extraction)
│   ├── update.py              # Incremental update (change detection, delete-then-insert, rollback)
│   ├── incremental.py         # Change detection, doc_id index, backups, query-cache cleanup
│   └── watch_update.py        # Watchdog daemon (debounce + file lock)
├── querying/
│   ├── query.py               # Multi-mode retrieval entry (with query-enhancement orchestration)
│   ├── query_refine.py        # Exact-match short-circuit, fuzzy correction, context filtering, prompt assembly
│   └── lookup.py              # Deterministic graph lookup (no LLM/network, 1~N hops)
├── inspection/
│   ├── trace.py               # Reasoning-path text chains + Mermaid graphs
│   └── doctor.py              # Consistency diagnostics + MinerU/LLM/embedding reachability
└── maintenance/
    ├── normalize_entities.py  # Fragment-entity merge + noise cleanup (LightRAG API, dry-run default)
    ├── purge_noise_offline.py # Offline batch noise purge (single read/write per file)
    └── verify_embedding_compat.py # Embedding-switch compatibility check (read-only)

Quick Start

1. Install

pip install -e .            # core features
pip install -e ".[parse]"   # plus pdfplumber / python-docx (fallback when MinerU is unreachable)

2. Deploy MinerU (document parsing service, optional but recommended)

MinerU is OpenDataLab's open-source document parsing project; this tool is only an HTTP API client for it.

• Project: https://github.com/opendatalab/MinerU
• Start a v3-series HTTP service per its docs (e.g. docker); default assumed address is http://localhost:18000.
• Health check: curl http://localhost:18000/health
• MinerU is not required: graphrag parse falls back to local parsers for PDF/DOCX (needs the [parse] extra), or you can drop your existing .md files into ./raw_docs/ and skip parsing entirely.

3. Configure

cp config/env.template .env     # fill in API keys and service URLs

Edit config/config.yaml (paths, models, concurrency, backup policy, MinerU address) and config/schema.yaml (rewrite entity/relation types and examples for YOUR domain — schema quality directly determines extraction quality).

Three OpenAI-compatible services are needed (they may be the same endpoint):

1. Extraction LLM (ingest/update): DeepSeek, GPT, local vLLM, etc.;
2. Query LLM (query): local/private deployment works too;
3. Embedding service: bge-m3, text-embedding-3-small, etc. — dimension must match embedding_dim.

CLI Reference

Command	Description
graphrag parse <file|dir>	Parse documents to Markdown via MinerU; batch mode preserves directory structure. Options: --output-dir, --api-url, --timeout, --max-retries, --no-fallback
graphrag ingest	Full index build. Options: --dry-run, --resume, --config
graphrag update	Incremental update. Options: --dry-run, --no-backup, --no-clear-query-cache
graphrag watch	Watchdog daemon. Options: --oneshot, --dry-run
graphrag query "..."	Retrieval QA. --mode auto|local|global|hybrid|naive (default: auto routing); local-timeout circuit breaker (exit code 4)
graphrag lookup <entity>	Deterministic lookup (no LLM). Options: --depth N, --storage-dir; JSON output, exit codes 0/2
graphrag trace "..."	Query and print knowledge-graph reasoning chains (A --[relation]--> B)
graphrag normalize	Entity merge + noise cleanup (dry-run by default, --apply to execute)
graphrag doctor	Consistency diagnostics (storage/manifest/doc_index) + MinerU/LLM/embedding reachability; --reconcile rebuilds indexes from storage

Acknowledgements

• LightRAG — the knowledge-graph RAG framework by HKUST-NLP;
• MinerU — the document parsing project by OpenDataLab.

License

MIT © ariasu