Skip to content

CLAUDE.md

Latest commit

 

History

History
82 lines (60 loc) · 4.06 KB

CLAUDE.md

File metadata and controls

82 lines (60 loc) · 4.06 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

OParl Index is a FastAPI backend that indexes data from OParl APIs (a German standard for parliamentary information systems, v1.1). It scrapes OParl endpoints, stores parliamentary data in PostgreSQL, and exposes it via a read-only REST API.

Tech Stack

  • Python 3.13 (managed with uv)
  • FastAPI with Uvicorn
  • SQLModel (SQLAlchemy + Pydantic hybrid) for models/ORM
  • PostgreSQL 18 via Docker
  • Alembic for database migrations
  • Docker Compose for local development

Commands

# Start the full stack (DB + app)
docker-compose up --build

# Run migrations
docker-compose exec app alembic upgrade head

# Run tests
docker-compose exec app pytest

# Run the OParl scraper (from project root)
uv run python -m backend.scripts.oparl_scraper https://oparl.stadt-muenster.de/system
uv run python -m backend.scripts.oparl_scraper https://oparl.stadt-muenster.de/system --since 2026-01-01

# Print DB statistics (row counts, oldest/newest per table)
uv run python -m backend.scripts.db_stats

# Check data integrity (orphaned refs, missing fields, duplicates)
uv run python -m backend.scripts.db_check

# Install dependencies locally
uv sync

Architecture

Entry Points

  • backend/fastapi_main.py — FastAPI app setup, mounts all routers. Serves the frontend at /, Swagger docs at /docs.
  • backend/scripts/oparl_scraper.py — CLI scraper that walks System → Bodies → all entity lists with pagination. Supports --since YYYY-MM-DD to stop at older items.
  • backend/scripts/db_stats.py — Prints per-table row counts and oldest/newest created timestamps.
  • backend/scripts/db_check.py — Data integrity checks: orphaned foreign keys, broken JSON refs, missing fields, duplicates.

Core Modules

  • backend/models.py — SQLModel table definitions for all OParl types: Organization, Person, Membership, Meeting, AgendaItem, Paper, Consultation, File. Each model has a from_oparl(data) classmethod that maps OParl JSON to model fields.
  • backend/database.py — Engine creation, get_db() session dependency. Reads DATABASE_URL from .env or environment.
  • backend/*_router.py — Read-only routers for each entity type (GET list + GET by ID). Detail endpoints use {id:path} because OParl IDs are URLs containing slashes.

Frontend

  • backend/static/index.html — Single-page vanilla JS app using Pico CSS (via CDN). Sidebar lists entity types; clicking one loads a paginated table; clicking a row shows all fields. No build step required.

Scraper Design

  • Fetches paginated lists from each Body's endpoints (organization, person, meeting, paper).
  • Extracts embedded objects: Memberships from People, AgendaItems + invitation Files from Meetings, Consultations + Files from Papers.
  • Uses session.merge() for upsert (insert or update by primary key).
  • --since flag stops pagination early since lists are sorted newest-first.

Database

  • Alembic config in alembic.ini, migrations in alembic/versions/.
  • alembic/env.py imports backend.models to register all table metadata.
  • Column naming uses camelCase in the DB (e.g., shortName, startDate, paperType) mapped to snake_case Python attributes via sa_column.
  • JSON columns for array fields (e.g., Meeting.organization, Paper.underDirectionOf).
  • All IDs are OParl URLs (strings), not integers.

OParl Spec

  • specs/oparl_specs.md — OParl v1.1 technical summary.
  • specs/muenster_api_findings.md — field-level findings from the live Munster API, including quirks and actual data shapes.

Key Conventions

  • The Docker app service mounts the project root at /app and runs uvicorn with --reload.
  • All modules use proper package imports (e.g., from backend.models import Organization). Run from the project root.
  • The API is read-only (GET). The scraper writes to the DB directly via SQLModel sessions.
  • Default DB credentials: postgres:postgres@db:5432/oparl_db (in docker-compose) or via DATABASE_URL env var / .env file.