MDL Emergent Language

Language born from compression. Multi-agent systems develop structured symbolic communication through information-theoretic pressure alone — no language supervision required.

Core Innovation

When agents must coordinate through a bandwidth-limited channel, the pressure to compress observations into minimal messages forces the emergence of compositional, hierarchical proto-language. This mirrors how human language evolved: limited vocal bandwidth + social coordination → structured grammar.

Key Insight: Minimum Description Length (MDL) pressure — minimizing the bits needed to transmit task-relevant information — is sufficient to produce discrete, compositional symbol systems. No explicit linguistic supervision, no reward shaping, no architectural bias.

What Makes This Novel

Approach	Communication	Structure Source	Limitation
CommNet	Differentiable averaging	Implicit	No discrete symbols
TarMAC	Attention-based	Architecture	Still continuous
Referential Games	Speaker-listener	Task reward	Simple referents only
EGG	Gumbel-softmax	Game structure	Requires game design
Ours (MDL)	Discrete bottleneck	Compression	Novel mechanism

The Science

Problem Definition

N agents observe local states $s_i$ and communicate through discrete channels $m_i \in {1,...,K}^L$ to coordinate actions.

Hypothesis

Multi-agent systems trained with an information bottleneck on communication channels will develop discrete, compositional symbol systems whose structure reflects the causal structure of the environment.

Training Objective

$$\min_{p(m|s)} ; \lambda \cdot I(s; m) - \beta \cdot \mathbb{E}[R_{\text{coord}}] + \alpha \cdot H_{\text{symbol}}(m)$$

Where:

• $I(s;m)$: mutual information between state and message (compression)
• $R_{\text{coord}}$: coordination reward (utility)
• $H_{\text{symbol}}(m)$: per-symbol entropy (discreteness pressure)

Predictions

As channel capacity decreases:

1. Discretization — symbols become categorical
2. Compositionality — complex meanings built from combinations
3. Hierarchy — frequent meanings get shorter codes (Zipf's law)

Architecture

┌─────────────────────────────────────────────────┐
│              GRID-WORLD ENVIRONMENT             │
│   ┌───────┐  ┌───────┐  ┌───────┐  ┌───────┐   │
│   │ NW 🔴 │──│ NE 🔵 │──│ SW 🟢 │──│ SE 🟡 │   │
│   └───┬───┘  └───┬───┘  └───┬───┘  └───┬───┘   │
│     Agent A    Agent B    Agent C    Agent D     │
└─────────────────────────────────────────────────┘
         │           │           │           │
         ▼           ▼           ▼           ▼
┌─────────────────────────────────────────────────┐
│           OBSERVATION ENCODER (CNN)             │
│    Local 5×5 View → Conv2d → FC → Latent z     │
└─────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────┐
│        COMMUNICATION CHANNEL (MDL)              │
│   Encoder: z → MLP → Gumbel-Softmax → symbols  │
│   Aggregator: mean pool → context               │
│   Decoder: [z; context] → action logits         │
└─────────────────────────────────────────────────┘

Quick Start

Installation

# Clone the repository
git clone https://github.com/yourusername/mdl-emergent-language.git
cd mdl-emergent-language

# Install dependencies
pip install -r requirements.txt

Run Tests

python tests/test_all.py

Train

# Full training (50k episodes, ~2 hours on CPU)
python train.py

# Quick demo (100 episodes, ~1 minute)
python demo.py

Analyze

# Run inference on trained model
python inference.py models/checkpoints/checkpoint_ep50000.pt

Project Structure

mdl-emergent-language/
├── config.yaml              # All hyperparameters
├── train.py                 # Training loop
├── inference.py             # Run trained agents
├── demo.py                  # Interactive demo
├── requirements.txt
├── LICENSE
├── README.md
├── .gitignore
├── src/
│   ├── __init__.py
│   ├── environment.py       # Grid-world with rooms + objects
│   └── agents.py            # MDL agent architecture
├── analysis/
│   └── language_analysis.py # Zipf, topographic similarity, etc.
├── models/                  # Checkpoints (gitignored)
│   ├── checkpoints/
│   └── logs/
└── tests/
    └── test_all.py          # Full test suite

Evaluation Metrics

Metric	What It Measures	Ideal Value
Coordination Accuracy	Task completion rate	High
Topographic Similarity	Compositionality (Spearman ρ)	> 0.3
Zipf Exponent	Hierarchical structure	≈ 1.0
Symbol Discreteness	Entropy per symbol position	Low
Information Density	Bits per symbol	Moderate
Vocabulary Utilization	Used symbols / total symbols	High

Results (Expected)

After training:

• Symbols become discrete: agents converge to using specific symbols for specific meanings
• Compositionality emerges: room position → symbol[0], object color → symbol[1]
• Zipf's law appears: most common symbols are used much more frequently
• Communication helps: agents with communication outperform those without

The Science Behind It

Why Compression Creates Structure

Consider an agent that needs to communicate "red object in room NW" to another agent. With unlimited bandwidth, it could send the raw 5×5 observation grid. But with only 2 symbols from a vocabulary of 4, it must compress.

The MDL pressure forces the agent to find the most efficient encoding. If "red" always appears in the NW room, the agent learns:

• Symbol 0 = "NW room" (most common → shortest code)
• Symbol 1 = "red object"

This is exactly how human language works: frequent words are short (Zipf's law), and meaning is compositional (adjective + noun).

Connection to Information Theory

The Information Bottleneck principle (Tishby et al., 1999) states that optimal representations compress input while preserving task-relevant information. Our work extends this to communication: agents must compress observations into messages that preserve coordination-relevant information.

Future Work

1. Scale to more agents (8, 16, 32) — does language complexity grow?
2. Harder tasks — navigation, construction, negotiation
3. Language transfer — can emergent language help in new environments?
4. Human interpretability — can humans learn the emergent language?
5. Compositional generalization — do agents generalize to unseen combinations?
6. Connection to natural language — does the structure mirror human language?

References

• Tishby, N., Pereira, F.C. & Bialek, W. (1999). The Information Bottleneck Method.
• Kirby, S., Griffiths, T. & Smith, K. (2008). Cumulative Cultural Evolution.
• Sukhbaatar, S., Szlam, A. & Fergus, R. (2016). Learning Multiagent Communication with Backpropagation.
• Kharitonov, E. et al. (2019). EGG: Emergence of lanGuage in Games.
• Lazaridou, A. et al. (2018). Emergence of Linguistic Communication from Referential Games.

Citation

@software{mdl_emergent_language,
  title={MDL Emergent Language: Language Born from Compression},
  year={2026},
  url={https://github.com/yourusername/mdl-emergent-language}
}

License

MIT License — see LICENSE

---

Built with the conviction that language isn't taught — it's compressed into existence.