Link Guardian

Русская версия

Link Guardian — локальный анализатор риска HTTP(S) URL, который формирует объяснимые risk_score, verdict и причины результата. Проект объединяет детерминированные лексические признаки, защищённую от SSRF сетевую проверку и опциональную интеграцию с URLhaus в backend-приложении на FastAPI.

Link Guardian показывает индикаторы риска и не гарантирует, что URL безопасен или вредоносен. Вердикт low_risk означает только отсутствие сильных сигналов в рамках выполненных проверок, а не доказательство безопасности.

Возможности

Единая нормализация URL с поддержкой IDNA, IPv4, IPv6, сохранением query и стабильным SHA-256.
Детерминированный лексический анализ с объяснимыми findings.
SSRF-aware сетевая проверка с валидацией DNS и закреплением TCP-соединения за проверенным IP-адресом.
Сохранение исходных Host, TLS SNI и имени для проверки HTTPS-сертификата.
Ручная обработка redirect chain с повторной проверкой каждого target.
Конфигурируемый scoring с итоговыми risk_score, verdict и упорядоченными reasons.
Режимы reputation provider: mock, disabled и опциональный urlhaus.
История анализов через асинхронный SQLAlchemy repository.
SQLite и PostgreSQL со схемой, управляемой только через Alembic.
REST API, OpenAPI/Swagger и небольшой локальный web-интерфейс.
Docker Compose с PostgreSQL и непривилегированным контейнером приложения.
JSON/console logs, X-Request-ID и Prometheus metrics.
Автоматические тесты, Ruff, mypy strict и GitHub Actions.

Архитектура

Запрос
  -> нормализация URL
  -> лексический анализ
  -> сетевая проверка
  -> reputation check
  -> scoring
  -> сохранение
  -> API response

Ответственность разделена по слоям:

api — FastAPI routes и dependencies;
services — оркестрация анализа;
analyzers — детерминированные лексические findings без сетевого доступа;
network — DNS/IP validation, pinned connections и redirects;
reputation — Protocol и реализации mock, disabled и URLhaus;
repositories и db — асинхронное хранение и проверка Alembic revision;
config — настройки окружения через Pydantic Settings;
observability — request context, структурированные логи и метрики.

Технологический стек

Python 3.12+, FastAPI, Pydantic v2 и Pydantic Settings
SQLAlchemy 2 AsyncIO и Alembic
SQLite/aiosqlite и PostgreSQL/asyncpg
HTTPX, httpcore, dnspython и tldextract
URLhaus Community API
prometheus-client
Docker и Docker Compose
pytest, pytest-asyncio и respx
Ruff и mypy strict
HTML, CSS и vanilla JavaScript
GitHub Actions

Быстрый запуск через Docker

Потребуются Git, Docker Engine или Docker Desktop и Docker Compose v2.

git clone git@github.com:akiamuradev/link-guardian.git
cd link-guardian
cp .env.example .env

В Windows PowerShell:

Copy-Item .env.example .env

Перед запуском замените placeholder-пароль PostgreSQL в .env и включите Compose URL с hostname db:

POSTGRES_PASSWORD=replace-with-a-local-password
DATABASE_URL=postgresql+asyncpg://link_guardian:replace-with-a-local-password@db:5432/link_guardian

Оставьте reputation provider в режиме mock, если URLhaus не настраивается намеренно.

docker compose up -d --build
docker compose ps

Доступные адреса:

docker compose logs -f app
docker compose down

docker compose down сохраняет PostgreSQL volume. Команда docker compose down -v удаляет volume и всю сохранённую историю анализов.

Локальная разработка

Linux с Fish:

python -m venv .venv
source .venv/bin/activate.fish
python -m pip install -e ".[dev]"
cp .env.example .env
alembic upgrade head
uvicorn link_guardian.main:app --reload

Windows PowerShell:

py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
Copy-Item .env.example .env
alembic upgrade head
uvicorn link_guardian.main:app --reload

По умолчанию .env.example использует SQLite и offline mock provider. HTTP-приложение не создаёт таблицы и не запускает миграции при startup: оно только проверяет текущую Alembic revision и сообщает о необходимости выполнить alembic upgrade head.

Конфигурация

Настройки читаются из environment variables и локального .env, который игнорируется Git.

Переменная	Назначение	Безопасный default/example	Обязательность
`DATABASE_URL`	Async SQLAlchemy URL для приложения и Alembic	`sqlite+aiosqlite:///./link_guardian.db`	Нет
`LINK_GUARDIAN_REPUTATION_PROVIDER`	`mock`, `urlhaus` или `disabled`	`mock`	Нет
`LINK_GUARDIAN_URLHAUS_AUTH_KEY`	Секретный URLhaus `Auth-Key`	пусто	Только для `urlhaus`
`LINK_GUARDIAN_URLHAUS_API_URL`	Официальный endpoint URLhaus	`https://urlhaus-api.abuse.ch/v1/url/`	Нет
`LINK_GUARDIAN_URLHAUS_TIMEOUT_SECONDS`	Ограниченный timeout URLhaus	`3.0`	Нет
`LINK_GUARDIAN_SCORING_TRUSTED_REPUTATION_PROVIDERS`	Providers для confirmed-threat verdict	`mock`	Нет
`LINK_GUARDIAN_LOG_LEVEL`	Уровень логирования	`INFO`	Нет
`LINK_GUARDIAN_LOG_FORMAT`	`json` или `console`	`json`	Нет
`POSTGRES_DB`	Имя Compose database	`link_guardian`	Для Compose
`POSTGRES_USER`	Пользователь Compose PostgreSQL	`link_guardian`	Для Compose
`POSTGRES_PASSWORD`	Пароль Compose PostgreSQL	только placeholder	Для Compose

Режим mock детерминирован и не требует сети. disabled возвращает нейтральный результат. В режиме urlhaus нормализованный URL передаётся внешнему сервису и требуется собственный Auth-Key. Ответ URLhaus no_results не доказывает безопасность URL.

API

Метод	Endpoint	Назначение
`GET`	`/`	Локальный web-интерфейс
`GET`	`/health`	Health contract
`GET`	`/api/v1/health`	Обратно совместимый health alias
`GET`	`/metrics`	Prometheus exposition endpoint
`POST`	`/api/v1/analyses`	Анализ одного HTTP(S) URL
`GET`	`/api/v1/analyses`	Последние анализы, `limit` от 1 до 100
`GET`	`/api/v1/analyses/{analysis_id}`	Один сохранённый анализ

curl -X POST http://127.0.0.1:8000/api/v1/analyses \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/"}'

Полный актуальный контракт доступен в /docs.

Модель риска

Scoring engine объединяет лексические, сетевые, redirect и reputation signals. Вклады ограничиваются caps, итог находится в диапазоне 0..100 и преобразуется в low_risk, caution, suspicious или high_risk. Подтверждённый сигнал явно доверенного provider может сформировать known_malicious.

Ошибки сети и внешнего provider не считаются доказательством вредоносности. Невыполненные обязательные проверки дают inconclusive. Результат остаётся оценкой риска на момент проверки, а не окончательным доказательством.

Безопасность

Принимаются только HTTP/HTTPS URL; credentials и некорректные порты отклоняются.
Ограничены длина URL, redirects, timeout и разрешённые порты.
Все DNS-ответы IPv4/IPv6 классифицируются до сетевого доступа.
Блокируются private, loopback, link-local, multicast, reserved, unspecified и опасные IPv4-mapped IPv6 адреса.
TCP destination закрепляется за проверенным IP для уменьшения DNS rebinding/TOCTOU окна.
HTTPS сохраняет исходные Host, TLS SNI и certificate hostname verification.
Каждый redirect повторно нормализуется, разрешается через DNS и проверяется.
System proxy, cookies, Authorization и пользовательские credentials не пересылаются.
Тела целевых страниц не потребляются; файлы не скачиваются, JavaScript не выполняется.
Секреты принимаются через environment и исключаются из application logs.
Docker-контейнер приложения работает под UID/GID 10001 без root.

Эти меры уменьшают SSRF-риск, но не являются заявлением о полном устранении всех сетевых атак.

Наблюдаемость

Поддерживаются JSON и console logs. Middleware принимает или создаёт X-Request-ID и изолирует его через ContextVar. /metrics публикует шесть process-local metric families для HTTP, analysis и reputation requests с количеством и duration. Labels не содержат полные URL, hostname, request ID или exception messages. В production доступ к /metrics следует ограничить reverse proxy или внутренней сетью.

Тесты и CI

pytest
ruff check .
ruff format --check src tests
mypy --strict src

PostgreSQL integration tests требуют URL отдельной тестовой базы:

pytest -m postgres

GitHub Actions выполняет четыре jobs: Quality, SQLite tests, PostgreSQL integration и Docker smoke test. Обычный test suite использует изолированные SQLite databases и mock transports и не обращается к реальному URLhaus API.

Статус и ограничения

Стабильный портфолио-релиз — v1.0.0. Основной функциональный scope завершён.

Эвристики могут давать false positives и false negatives.
URLhaus содержит только известные ему записи; no_results не означает safe.
Содержимое страницы, rendered state и JavaScript не анализируются.
Файлы и payloads не скачиваются и не проверяются.
DNS, redirects и содержимое URL могут измениться после анализа.
Внешние DNS, target servers и URLhaus могут быть недоступны.
Метрики process-local и не агрегируются между несколькими workers.
Compose — локальный deployment baseline, а не production orchestration platform или SLA.

Проект распространяется по лицензии MIT.

English version

Link Guardian is a local HTTP(S) URL risk analyzer that produces an explainable risk score and verdict. It combines deterministic local signals, SSRF-aware network inspection, and an optional URLhaus reputation lookup in a security-focused FastAPI backend.

Link Guardian provides risk indicators and does not guarantee that a URL is safe or malicious. A low_risk verdict means that the performed checks found no strong signal; it is not proof of safety.

Features

Canonical URL normalization with IDNA, IPv4, IPv6, query preservation, and stable hashing.
Deterministic lexical risk analysis with explainable findings.
SSRF-aware HTTP inspection with DNS validation and a pinned TCP destination.
Host header, TLS SNI, and certificate hostname verification preservation.
Manual redirect-chain validation and bounded HEAD/streaming GET behavior.
Configurable scoring that returns risk_score, verdict, and ordered reasons.
mock, disabled, and optional URLhaus reputation modes.
Analysis history through an async SQLAlchemy repository.
SQLite and PostgreSQL support with Alembic-managed schemas.
REST API, OpenAPI documentation, and a small local web interface.
Docker Compose startup with PostgreSQL and a non-root application container.
JSON or console logging, X-Request-ID, and Prometheus metrics.
Automated tests, Ruff, strict mypy, and GitHub Actions configuration.

Architecture

Request
  -> URL normalization
  -> lexical analysis
  -> network inspection
  -> reputation check
  -> scoring
  -> persistence
  -> API response

The project keeps responsibilities separated:

api defines FastAPI routes and dependencies;
services orchestrates the analysis workflow;
analyzers emits deterministic lexical findings without network access;
network resolves and validates targets, pins connections, and processes redirects;
reputation exposes a provider Protocol and the mock, disabled, and URLhaus adapters;
repositories and db provide async persistence and schema validation;
config validates environment-based settings with Pydantic Settings;
observability owns request context, structured logs, and metrics.

Technology stack

Python 3.12+
FastAPI and Pydantic v2
Pydantic Settings
SQLAlchemy 2 AsyncIO
Alembic
SQLite with aiosqlite
PostgreSQL with asyncpg
HTTPX, httpcore, dnspython, and tldextract
URLhaus Community API integration
prometheus-client
Docker and Docker Compose
pytest, pytest-asyncio, and respx
Ruff and mypy strict
GitHub Actions

Quick start with Docker

Requirements: Git, Docker Engine or Docker Desktop, and Docker Compose v2.

git clone git@github.com:akiamuradev/link-guardian.git
cd link-guardian
cp .env.example .env

On Windows PowerShell, copy the configuration with:

Copy-Item .env.example .env

Before startup, replace the placeholder PostgreSQL password in .env and enable the internal Compose database URL. The database URL must use db as its hostname and the same password as POSTGRES_PASSWORD:

POSTGRES_PASSWORD=replace-with-a-local-password
DATABASE_URL=postgresql+asyncpg://link_guardian:replace-with-a-local-password@db:5432/link_guardian

Keep the default reputation mode set to mock unless you intentionally configure URLhaus. Then build and start the services:

docker compose up -d --build
docker compose ps

The Compose stack contains:

app: the non-root FastAPI container exposed on host port 8000;
db: PostgreSQL 17 with pg_isready and the postgres_data named volume.

The PostgreSQL port is not published to the host. The app waits for a healthy database, runs alembic upgrade head, and then starts Uvicorn.

Available endpoints:

Logs and shutdown:

docker compose logs -f app
docker compose down

docker compose down preserves the PostgreSQL volume. The following command deletes the volume and all saved analysis history:

docker compose down -v

Local development

Linux with Fish

python -m venv .venv
source .venv/bin/activate.fish
python -m pip install -e ".[dev]"
cp .env.example .env
alembic upgrade head
uvicorn link_guardian.main:app --reload

The default .env.example uses SQLite and the offline mock provider, so no API key is required for local development.

Windows PowerShell

py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
Copy-Item .env.example .env
alembic upgrade head
uvicorn link_guardian.main:app --reload

The HTTP application does not create tables or run migrations during normal startup. It performs a read-only Alembic revision check and reports alembic upgrade head when the schema is missing or outdated.

Configuration

Settings are read from environment variables and the ignored local .env file.

Variable	Purpose	Safe default/example	Required
`DATABASE_URL`	Async SQLAlchemy database URL used by both the app and Alembic	`sqlite+aiosqlite:///./link_guardian.db`	No
`LINK_GUARDIAN_REPUTATION_PROVIDER`	Selects `mock`, `urlhaus`, or `disabled`	`mock`	No
`LINK_GUARDIAN_URLHAUS_AUTH_KEY`	URLhaus `Auth-Key`; stored as `SecretStr`	empty	Only in `urlhaus` mode
`LINK_GUARDIAN_URLHAUS_API_URL`	Fixed official URLhaus lookup endpoint	`https://urlhaus-api.abuse.ch/v1/url/`	No
`LINK_GUARDIAN_URLHAUS_TIMEOUT_SECONDS`	Bounded URLhaus request timeout	`3.0`	No
`LINK_GUARDIAN_SCORING_TRUSTED_REPUTATION_PROVIDERS`	Providers allowed to produce a confirmed-threat verdict	`mock`	No
`LINK_GUARDIAN_LOG_LEVEL`	`DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`	`INFO`	No
`LINK_GUARDIAN_LOG_FORMAT`	`json` or `console`	`json`	No
`POSTGRES_DB`	Compose PostgreSQL database name	`link_guardian`	For Compose
`POSTGRES_USER`	Compose PostgreSQL user	`link_guardian`	For Compose
`POSTGRES_PASSWORD`	Compose PostgreSQL password	placeholder only	For Compose

Reputation modes:

mock is deterministic, offline, and suitable for development and tests;
disabled returns a neutral disabled result and performs no reputation lookup;
urlhaus sends the normalized URL to the official URLhaus endpoint and requires a user-provided Auth-Key.

URLhaus is not trusted by the default scoring policy. To allow an available, flagged URLhaus result to produce known_malicious, configure it explicitly:

LINK_GUARDIAN_SCORING_TRUSTED_REPUTATION_PROVIDERS=urlhaus

A URLhaus no_results response means only that the URL was absent from the available URLhaus data. It does not prove that the URL is safe. In urlhaus mode, the normalized URL, including its query values, is sent to an external service; do not submit confidential internal URLs without assessing the privacy impact.

API

Method	Endpoint	Purpose
`GET`	`/`	Local web interface
`GET`	`/health`	Process health contract
`GET`	`/api/v1/health`	Backward-compatible health alias
`GET`	`/metrics`	Prometheus exposition endpoint
`POST`	`/api/v1/analyses`	Analyze one HTTP(S) URL
`GET`	`/api/v1/analyses`	List recent analyses; `limit` is bounded to `1..100`
`GET`	`/api/v1/analyses/{analysis_id}`	Read one saved analysis

Analyze a public example URL:

curl -X POST http://127.0.0.1:8000/api/v1/analyses \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/"}'

Abbreviated response shape:

{
  "id": 1,
  "submitted_url": "https://example.com/",
  "final_url": "https://example.com/",
  "risk_score": 0,
  "verdict": "low_risk",
  "reasons": [
    {
      "code": "reputation_clear",
      "message": "Reputation provider mock reported no threat.",
      "score_delta": 0
    }
  ]
}

The public response exposes normalized URLs, verdict, score, reasons, redirect metadata, timestamp, and a disclaimer. Internal lexical Finding objects and provider status are converted into explainable reasons; they are not separate top-level response fields. See /docs for the complete current schema.

Risk model

The scoring engine combines configured contributions from lexical findings, network inspection, redirects, and reputation results. Contributions are capped and the final score is bounded to 0..100, then mapped to low_risk, caution, suspicious, or high_risk.

An available and flagged result from an explicitly trusted provider can produce known_malicious. Network or provider failures do not add malicious evidence; incomplete required checks produce an inconclusive result. URLhaus errors are normalized into an unavailable result so the optional integration fails open rather than taking down the API.

The score is an explainable estimate based on the checks performed at that moment, not a final proof about the URL.

Security

Only HTTP and HTTPS URLs are accepted; credentials and malformed ports are rejected.
URL length, redirect count, request timeouts, and allowed ports are bounded.
Every target is resolved before access and all returned IPv4/IPv6 addresses are checked.
Private, loopback, link-local, multicast, reserved, unspecified, and unsafe IPv4-mapped IPv6 destinations are blocked.
The TCP destination is pinned to a validated address to reduce DNS rebinding/TOCTOU risk.
HTTPS keeps the original hostname for Host, TLS SNI, and certificate verification.
Every redirect target is normalized, resolved, classified, and pinned independently.
System proxies, user credentials, cookies, and Authorization headers are not forwarded.
Target page bodies are not consumed; a streaming GET is used only when HEAD is unsupported.
Files and malware payloads are neither downloaded nor executed; JavaScript is not run.
Secrets are accepted only through environment configuration and are excluded from logs.
Application logs omit the full analyzed URL, query string, credentials, and response body.
The Docker application runs as non-root UID/GID 10001.

These controls reduce SSRF exposure but do not claim to eliminate every SSRF or network attack technique.

Observability

Application-owned events support JSON and console formats. Request middleware accepts or generates X-Request-ID, stores it in a ContextVar, returns it to the caller, and prevents request context from leaking across concurrent tasks.

GET /metrics exposes six process-local metric families:

link_guardian_http_requests_total
link_guardian_http_request_duration_seconds
link_guardian_analyses_total
link_guardian_analysis_duration_seconds
link_guardian_reputation_requests_total
link_guardian_reputation_request_duration_seconds

Metric labels use route templates and bounded values. They do not contain raw URL paths, hostnames, request IDs, analysis IDs, or exception messages. Restrict /metrics with an internal network or reverse proxy in production deployments.

Tests and quality

pytest
ruff check .
ruff format --check src tests
mypy --strict src

PostgreSQL integration tests are opt-in and require a URL for a dedicated test database:

pytest -m postgres

Set LINK_GUARDIAN_TEST_POSTGRES_DATABASE_URL in the shell that runs this command. Never point it at the Compose development database or another database containing user data.

The regular suite uses isolated SQLite databases and mocked network transports. It does not call the real URLhaus API.

CI

.github/workflows/ci.yml defines four jobs on GitHub-hosted Linux runners:

Quality: Ruff lint, format check, and strict mypy;
SQLite tests: the regular mock-provider test suite;
PostgreSQL integration: PostgreSQL 17, Alembic, and pytest -m postgres;
Docker smoke test: image build, non-root user check, Compose startup, health, OpenAPI, and Alembic checks.

The workflow runs for pushes and pull requests targeting main, plus manual dispatch. It uses contents: read, does not publish images, and does not require a real URLhaus key.

Project status

Stable portfolio release — v1.0.0.

The primary functional scope is complete. Further work is limited to fixes, documentation, and small improvements within the current architecture; the project is not intended to become a universal security platform.

Limitations

Lexical rules and scoring heuristics can produce false positives and false negatives.
URLhaus covers known records in its dataset, not every malicious or phishing URL.
URLhaus no_results is not evidence that a URL is safe.
Page content, rendered state, and JavaScript behavior are not analyzed.
Files, payloads, and malware samples are not downloaded or inspected.
A URL, its DNS records, redirects, or served content can change after analysis.
External DNS, target servers, and URLhaus can be unavailable or rate-limited.
The first deterministically selected valid IP is used; connection fallback is not attempted.
Metrics are process-local and are not aggregated across multiple worker processes.
Compose is a local deployment baseline, not a production orchestration platform or SLA.

License

Link Guardian is available under the MIT License.

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
.github		.github
docker		docker
migrations		migrations
src/link_guardian		src/link_guardian
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
ROADMAP.md		ROADMAP.md
alembic.ini		alembic.ini
compose.yaml		compose.yaml
pyproject.toml		pyproject.toml

Navigation Menu

Repository files navigation

Link Guardian

Русская версия

Возможности

Архитектура

Технологический стек

Быстрый запуск через Docker

Локальная разработка

Конфигурация

API

Модель риска

Безопасность

Наблюдаемость

Тесты и CI

Статус и ограничения

English version

Features

Architecture

Technology stack

Quick start with Docker

Local development

Linux with Fish

Windows PowerShell

Configuration

API

Risk model

Security

Observability

Tests and quality

CI

Project status

Limitations

License

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

Link Guardian

Русская версия

Возможности

Архитектура

Технологический стек

Быстрый запуск через Docker

Локальная разработка

Конфигурация

API

Модель риска

Безопасность

Наблюдаемость

Тесты и CI

Статус и ограничения

English version

Features

Architecture

Technology stack

Quick start with Docker

Local development

Linux with Fish

Windows PowerShell

Configuration

API

Risk model

Security

Observability

Tests and quality

CI

Project status

Limitations

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages