Diagram Proposal: Data Pipeline Diagram for ETL and Data Flow Visualization

[BARBOUCHEEMNA]

Proposal

Mermaid.js should add a Data Pipeline diagram type because current diagram types (e.g., flowcharts) are not expressive enough to clearly represent ETL processes and data workflows.

Data engineering and system architecture diagrams often require domain-specific elements such as data sources, transformations, and storage systems. Using generic flowcharts for these purposes leads to ambiguity and reduced readability.

This new diagram type would introduce dedicated components for:

• Data sources (APIs, databases, streams)
• Transformations (filtering, aggregation, processing steps)
• Data sinks (data warehouses, storage systems)
• Directed data flow between stages

By providing a specialized syntax, Mermaid would better support modern data workflows and improve clarity in technical documentation.

Use Cases

The Diagram could be used for:

• Visualizing ETL pipelines (Extract, Transform, Load)
• Documenting data engineering architectures
• Representing real-time data streaming systems
• Teaching data flow concepts in education
• Explaining backend data processing in system design

Screenshots

Conceptual layout:

• Sources (cylinders or labeled nodes) on the left
• Transformations (rectangular processing blocks) in the middle
• Storage systems (warehouse/lake) on the right
• Arrows indicating clear directional data flow

Example structure:

[API] ---> [Clean Data] ---> [Aggregate] ---> [Data Warehouse]

Syntax

Proposed syntax example:

pipeline
source API
source Database

transform CleanData
transform Aggregate

sink DataWarehouse

API --> CleanData
Database --> CleanData
CleanData --> Aggregate
Aggregate --> DataWarehouse

Extended example with labels:

pipeline
source API
source Logs

transform ParseLogs
transform FilterErrors

sink Storage

API --> ParseLogs : JSON data
Logs --> ParseLogs
ParseLogs --> FilterErrors
FilterErrors --> Storage : cleaned data

Implementation

None

[OliverWilliamsDataManager]

Upvote - I would find this very useful as well!

[BARBOUCHEEMNA]

Thanks for the support! Glad to hear this would be useful to others as well @OliverWilliamsDataManager

[ashishjain0512]

@BARBOUCHEEMNA It would be helpful if, along with the suggested syntax, you could add the expected rendered output for when someone starts contributing.

[BARBOUCHEEMNA]

Thanks for the feedback, @ashishjain0512! You're absolutely right. Here are some rendered output examples for the proposed syntax:

Example 1: Basic ETL Pipeline

pipeline
  source API
  source Database
  transform CleanData
  transform Aggregate
  sink DataWarehouse
  
  API --> CleanData
  Database --> CleanData
  CleanData --> Aggregate
  Aggregate --> DataWarehouse

Loading

Expected visual:

• API (left, cylinder shape) ──→ CleanData (center, rectangle)
• Database (left, cylinder shape) ──→ CleanData
• CleanData ──→ Aggregate (center, rectangle)
• Aggregate ──→ DataWarehouse (right, warehouse icon/shape)

Example 2: Real-time Streaming Pipeline

pipeline
  source Kafka
  transform ParseJSON
  transform FilterErrors
  transform Enrich
  sink RedisCache
  sink Analytics
  
  Kafka --> ParseJSON : raw events
  ParseJSON --> FilterErrors
  FilterErrors --> Enrich
  Enrich --> RedisCache : cleaned data
  Enrich --> Analytics : metrics

Loading

Expected visual:

• Kafka (source) flows into ParseJSON
• ParseJSON → FilterErrors → Enrich (transformation pipeline)
• Enrich branches to both RedisCache and Analytics (dual sinks)
• Edge labels showing data types/formats