SWIFT-LLM

Semantic-Aware Intelligent Fast Inference with Tiered Routing

A production-grade LLM optimization framework achieving 3000x latency reduction through semantic caching, intelligent query routing, and multi-tier model orchestration.

Overview

SWIFT-LLM is a multi-layer optimization framework designed for production LLM systems. It combines semantic caching, intelligent query routing, and multi-tier model selection to dramatically reduce latency and costs while maintaining response quality.

The Problem

Traditional LLM inference suffers from:

High latency: API calls take 1-3 seconds
High costs: Premium models are expensive at scale
Redundant computation: Similar queries recomputed from scratch
One-size-fits-all: Simple queries use same resources as complex ones

The Solution

SWIFT-LLM addresses these through:

graph LR
    A[User Query] --> B{Semantic Cache}
    B -->|Hit| C[Instant Response - < 1ms]
    B -->|Miss| D{Complexity Router}
    D -->|Simple| E[Tier 2: Fast Model]
    D -->|Medium| F[Tier 3: Balanced]
    D -->|Complex| G[Tier 4-5: Premium]
    E --> H[Response + Cache]
    F --> H
    G --> H
    H --> I[User]
    C --> I

Key Innovations

Innovation	Description	Impact
Hybrid Cache Matching	Lexical O(1) + Semantic O(log n) lookup	74% hit rate
Query Preprocessing	Normalization, contraction expansion, stopword removal	+15% hit rate
FAISS Vector Index	Facebook AI Similarity Search for embeddings	Sub-ms lookups
Persistent Storage	SQLite + FAISS index serialization	Cross-session persistence
Adaptive Eviction	Recency + Frequency + Confidence weighted eviction	Optimal cache utilization
Complexity Classification	Feature-based query complexity scoring	86% routing accuracy
Auto-Escalation	Confidence-based tier escalation	Quality assurance

System Architecture

High-Level Architecture

flowchart TB
    subgraph Input Layer
        A[User Query]
    end

    subgraph Preprocessing Layer
        B[Query Normalizer]
        C[Lexical Hasher]
        D[Embedding Encoder]
    end

    subgraph Caching Layer
        E[(Lexical Index - HashMap)]
        F[(FAISS Index - Vector Store)]
        G[(SQLite - Persistent Storage)]
    end

    subgraph Routing Layer
        H[Feature Extractor]
        I[Complexity Classifier]
        J[Tier Selector]
    end

    subgraph Inference Layer
        K[Tier 1: Cache]
        L[Tier 2: Llama 8B - Groq API]
        M[Tier 3: Llama 70B - Groq API]
        N[Tier 4: GPT-4o-mini - OpenAI]
        O[Tier 5: GPT-4o - OpenAI]
    end

    subgraph Validation Layer
        P[Confidence Scorer]
        Q[Escalation Manager]
    end

    subgraph Output Layer
        R[Response]
        S[Cache Writer]
    end

    A --> B
    B --> C
    B --> D
    C --> E
    D --> F
    E & F --> |Hit| K
    E & F --> |Miss| H
    H --> I
    I --> J
    J --> L & M & N & O
    K --> R
    L & M & N & O --> P
    P --> |Low Confidence| Q
    Q --> |Escalate| J
    P --> |High Confidence| R
    R --> S
    S --> E & F & G

    style K fill:#22c55e
    style L fill:#3b82f6
    style M fill:#8b5cf6
    style N fill:#f59e0b
    style O fill:#ef4444

Request Flow Sequence

sequenceDiagram
    participant U as User
    participant P as Preprocessor
    participant C as Cache
    participant R as Router
    participant I as Inference
    participant V as Validator

    U->>P: Query
    P->>P: Normalize & Hash
    P->>C: Lookup (Lexical)
    
    alt Cache Hit (Lexical)
        C-->>U: Response (< 1ms)
    else Cache Miss (Lexical)
        P->>C: Lookup (Semantic/FAISS)
        alt Cache Hit (Semantic)
            C-->>U: Response (< 5ms)
        else Cache Miss
            C->>R: Route Query
            R->>R: Extract Features
            R->>R: Classify Complexity
            R->>I: Select Tier & Generate
            I->>V: Validate Response
            alt High Confidence
                V->>C: Store in Cache
                V-->>U: Response
            else Low Confidence
                V->>R: Escalate to Higher Tier
                R->>I: Generate with Better Model
                I->>V: Validate
                V->>C: Store in Cache
                V-->>U: Response
            end
        end
    end

Cache Architecture

flowchart LR
    subgraph Query Processing
        A[Raw Query] --> B[Lowercase]
        B --> C[Expand Contractions]
        C --> D[Normalize Whitespace]
        D --> E[Normalized Query]
    end

    subgraph Dual Index System
        E --> F[Extract Key Terms]
        E --> G[Generate Embedding]
        F --> H[Lexical Index]
        G --> I[FAISS Index]
    end

    subgraph Storage Backend
        J[(SQLite DB)]
        K[embeddings.npy]
        L[faiss_index.bin]
        M[lexical_index.pkl]
    end

    H --> J
    I --> K
    I --> L
    H --> M

    style H fill:#22c55e
    style I fill:#3b82f6
    style J fill:#64748b

Performance Benchmarks

Latency Comparison

xychart-beta
    title "Latency by Query Type (ms, log scale)"
    x-axis ["Cache Hit", "Tier 2 (8B)", "Tier 3 (70B)", "Tier 4 (GPT-4o-mini)", "Tier 5 (GPT-4o)"]
    y-axis "Latency (ms)" 0 --> 2500
    bar [0.5, 300, 800, 1200, 2000]

Performance Metrics

Metric	Value	Benchmark
Cache Hit Latency	0.5ms	vs 1500ms API
Speedup Factor	3000x	On cache hits
Cache Hit Rate	74.3%	After warmup
Lexical Hits	54%	Exact/normalized matches
Semantic Hits	20%	Paraphrase matches
Routing Accuracy	86%	Tier classification
P99 Latency	1989ms	Worst case
Avg Cost/Query	$0.000007	Blended average

Cache Efficiency

pie title Cache Hit Distribution
    "Lexical Hits (O(1))" : 54
    "Semantic Hits (FAISS)" : 20
    "Cache Misses" : 26

Tier Distribution

pie title Query Tier Distribution
    "Tier 1 (Cache)" : 74
    "Tier 2 (Llama 8B)" : 4
    "Tier 3 (Llama 70B)" : 17
    "Tier 4 (GPT-4o-mini)" : 4
    "Tier 5 (GPT-4o)" : 1

Technology Stack

Core Frameworks

Category	Technology	Purpose
Deep Learning	PyTorch 2.0+	Tensor operations, model inference
Embeddings	Sentence-Transformers	Query vectorization (`all-MiniLM-L6-v2`)
Vector Search	FAISS (Facebook AI)	Approximate nearest neighbor search
Database	SQLite	Persistent cache storage
API Clients	Groq SDK, OpenAI SDK	Cloud inference
Serialization	NumPy, Pickle	Efficient array storage

Model Stack

graph TB
    subgraph Embedding Models
        A[all-MiniLM-L6-v2 - 384-dim embeddings - 22M params]
    end

    subgraph Inference Models
        B[Llama 3.1 8B - Groq API - ~300ms latency]
        C[Llama 3.1 70B - Groq API - ~800ms latency]
        D[GPT-4o-mini - OpenAI API - ~1.2s latency]
        E[GPT-4o - OpenAI API - ~2s latency]
    end

    subgraph Vector Index
        F[FAISS IndexFlatIP - Inner Product Search - Normalized Cosine Similarity]
    end

    A --> F
    F --> B & C & D & E

    style A fill:#3b82f6
    style F fill:#22c55e

Architecture Patterns

Pattern	Implementation
Strategy Pattern	Eviction policies (LRU, Adaptive)
Factory Pattern	Model registry, inference backend creation
Observer Pattern	Metrics logging, cache statistics
Pipeline Pattern	Query -> Route -> Infer -> Validate -> Cache
Singleton Pattern	Configuration management

Dependencies

# Core ML Stack
torch>=2.0.0
sentence-transformers>=2.2.0
faiss-cpu>=1.7.0  # or faiss-gpu for CUDA
numpy>=1.24.0

# API Integration
groq>=0.4.0
openai>=1.0.0

# Data & Storage
sqlite3  # Built-in
python-dotenv>=1.0.0

# Utilities
dataclasses  # Built-in
pathlib  # Built-in

Installation

Prerequisites

Python 3.9+
pip or conda
4GB+ RAM (for embedding model)
macOS/Linux/Windows

Quick Install

# Clone repository
git clone https://github.com/yourusername/swift-llm.git
cd swift-llm

# Create virtual environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env with your API keys

Environment Variables

# Required (at least one)
GROQ_API_KEY=gsk_xxxxxxxxxxxx
OPENAI_API_KEY=sk-xxxxxxxxxxxx

# Optional tuning
SWIFT_CACHE_THRESHOLD=0.70
SWIFT_DEFAULT_TIER=2
SWIFT_LOG_LEVEL=INFO

Usage

Basic Usage

from swift_llm import SwiftLLM

# Initialize
swift = SwiftLLM()

# Query
response = swift.query("What is machine learning?")

print(f"Response: {response.text}")
print(f"Latency: {response.total_latency_ms:.1f}ms")
print(f"Cache Hit: {response.cache_hit}")
print(f"Model: {response.model_used} (Tier {response.model_tier})")

With Cache Warming

from swift_llm import SwiftLLM
from swift_llm.cache import warm_cache, get_warmup_queries

swift = SwiftLLM()

# Pre-populate cache with common queries
warm_cache(swift.cache, get_warmup_queries())

# Now common queries are instant
response = swift.query("What is the capital of France?")
# Latency: 0.5ms (cache hit)

Conversation Mode

# Multi-turn conversation
messages = [
    {"role": "user", "content": "What is Python?"},
    {"role": "assistant", "content": "Python is a programming language..."},
    {"role": "user", "content": "What are its main features?"},
]

response = swift.chat(messages)

Custom Configuration

from swift_llm import SwiftLLM
from swift_llm.core.config import Config, CacheConfig

config = Config()
config.cache.similarity_threshold = 0.75  # Stricter matching
config.cache.max_entries = 50000          # Larger cache
config.inference.temperature = 0.5        # More deterministic

swift = SwiftLLM(config=config)

Component Deep Dive

1. Semantic Cache

The cache uses a hybrid dual-index architecture:

# Lexical Index: O(1) exact match
lexical_hash = normalize(query)  # "what is france capital"
if lexical_hash in lexical_index:
    return cached_response  # < 0.1ms

# Semantic Index: O(log n) approximate match
embedding = encode(query)  # 384-dim vector
similarities, indices = faiss_index.search(embedding, k=5)
if max(similarities) > threshold:
    return cached_response  # < 5ms

Query Preprocessing Pipeline:

flowchart LR
    A["What's the capital of France?"] 
    --> B["what is the capital of france"]
    --> C["capital france"]
    --> D[Hash: 0x7a3f...]
    
    A --> E[Embedding - 384-dim]
    
    style D fill:#22c55e
    style E fill:#3b82f6

2. Complexity Classifier

Feature extraction for tier routing:

Feature	Weight	Example
`word_count`	0.15	Long queries = complex
`has_code_request`	0.40	"Write a function..."
`has_comparison`	0.35	"Compare X vs Y"
`has_reasoning`	0.35	"Explain why..."
`technical_terms`	0.15	ML, API, algorithm
`multiple_questions`	0.25	"What is X? And Y?"

flowchart TD
    A[Query] --> B{Simple Pattern?}
    B -->|Yes| C[Tier 1-2]
    B -->|No| D[Extract Features]
    D --> E[Calculate Score]
    E --> F{Score < 0.25?}
    F -->|Yes| G[Tier 1]
    F -->|No| H{Score < 0.50?}
    H -->|Yes| I[Tier 2]
    H -->|No| J{Score < 0.75?}
    J -->|Yes| K[Tier 3]
    J -->|No| L[Tier 4-5]

3. Response Validation

Confidence scoring and auto-escalation:

# Confidence factors
confidence = (
    0.3 * length_score +      # Response length adequacy
    0.3 * relevance_score +   # Query-response relevance
    0.2 * coherence_score +   # Internal consistency
    0.2 * specificity_score   # Concrete vs vague
)

if confidence < threshold:
    escalate_to_higher_tier()

API Reference

SwiftLLM

class SwiftLLM:
    def __init__(
        self,
        config: Optional[Config] = None,
        enable_cache: bool = True,
        enable_metrics: bool = True,
    ) -> None: ...
    
    def query(
        self,
        query: str,
        system_prompt: Optional[str] = None,
        skip_cache: bool = False,
        max_tokens: int = 512,
        temperature: float = 0.7,
    ) -> SwiftResponse: ...
    
    def chat(
        self,
        messages: List[Dict[str, str]],
        **kwargs,
    ) -> SwiftResponse: ...
    
    def save_cache(self) -> None: ...
    def clear_cache(self) -> None: ...
    def print_stats(self) -> None: ...

SwiftResponse

@dataclass
class SwiftResponse:
    text: str                    # Generated response
    cache_hit: bool              # Whether from cache
    model_used: str              # Model name
    model_tier: int              # 1-5
    total_latency_ms: float      # End-to-end latency
    confidence_score: float      # 0-1 quality score
    was_escalated: bool          # Tier escalation occurred
    estimated_cost: float        # USD cost estimate
    tokens_used: int             # Total tokens

Configuration

Full Configuration Reference

@dataclass
class Config:
    cache: CacheConfig
    router: RouterConfig
    inference: InferenceConfig
    validation: ValidationConfig

@dataclass
class CacheConfig:
    similarity_threshold: float = 0.70  # 0-1, lower = more hits
    max_entries: int = 10000
    ttl_seconds: int = 3600
    embedding_model: str = "all-MiniLM-L6-v2"
    cache_dir: Path = "./cache_data"
    enabled: bool = True

@dataclass
class RouterConfig:
    tier_thresholds: List[float] = [0.25, 0.5, 0.75]
    default_tier: int = 2

@dataclass
class InferenceConfig:
    tier_models: Dict[int, str] = {
        1: "cache",
        2: "groq/llama-3.1-8b-instant",
        3: "groq/llama-3.1-70b-versatile",
        4: "openai/gpt-4o-mini",
    }
    max_tokens: int = 512
    temperature: float = 0.7
    timeout: int = 30

@dataclass
class ValidationConfig:
    confidence_threshold: float = 0.7
    enable_auto_escalation: bool = True
    max_escalations: int = 2

Project Structure

swift-llm/
├── swift_llm/
│   ├── __init__.py
│   ├── cache/
│   │   ├── __init__.py
│   │   ├── semantic_cache.py    # Core cache + FAISS
│   │   ├── cache_store.py       # SQLite backend
│   │   ├── eviction.py          # Eviction policies
│   │   └── warmup.py            # Cache warming
│   ├── router/
│   │   ├── __init__.py
│   │   ├── complexity_classifier.py  # Query classification
│   │   ├── routing_policy.py         # Tier selection
│   │   └── model_registry.py         # Model catalog
│   ├── inference/
│   │   ├── __init__.py
│   │   ├── base.py              # Abstract interfaces
│   │   └── api_inference.py     # Groq/OpenAI clients
│   ├── validation/
│   │   ├── __init__.py
│   │   └── validator.py         # Response validation
│   └── core/
│       ├── __init__.py
│       ├── pipeline.py          # Main orchestration
│       ├── config.py            # Configuration
│       └── metrics.py           # Performance logging
├── benchmarks/
│   └── run_benchmarks.py
├── cache_data/                  # Persistent storage
│   ├── cache.db                 # SQLite database
│   ├── embeddings.npy           # Embedding vectors
│   ├── faiss_index.bin          # FAISS index
│   └── lexical_index.pkl        # Lexical hash map
├── demo.py
├── chat.py
├── requirements.txt
├── .env
├── LICENSE
└── README.md

Resume Bullet Points

Built SWIFT-LLM, a production-grade LLM optimization framework:
- Achieved 3000x latency reduction (0.5ms cache hits vs 1.5s+ API calls)
- Implemented hybrid semantic cache with FAISS achieving 74% hit rate
- Designed query complexity router with 86% classification accuracy
- Created persistent storage layer with auto-save across sessions
- Tech: Python, PyTorch, FAISS, Sentence-Transformers, Groq/OpenAI APIs, SQLite

Technical Report

A comprehensive NeurIPS-style research paper documenting the methodology and results:

Read the Full Paper (PDF) | LaTeX Source

The paper includes:

Formal methodology with algorithms and equations
System architecture diagrams (TikZ)
Comprehensive experimental results with ablation studies
17 citations to relevant literature (GPT-4, LLaMA, FAISS, FlashAttention, etc.)
NeurIPS 2024 conference formatting

License

This software and associated documentation files (the "Software") are proprietary and confidential. Unauthorized copying, modification, distribution, or use of this Software, via any medium, is strictly prohibited without explicit written permission from the copyright holder.

To request permission for use, please contact the author.

See LICENSE for full terms.

Author

Jyotishman Das

_{Built with dedication for innovation in LLM infrastructure}

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
benchmarks		benchmarks
paper		paper
swift_llm		swift_llm
.gitignore		.gitignore
LICENSE		LICENSE
PERFORMANCE_REPORT.md		PERFORMANCE_REPORT.md
README.md		README.md
chat.py		chat.py
demo.py		demo.py
env.example.txt		env.example.txt
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation