feat: server-side telemetry collector infrastructure (issue #559) by shekharprateek · Pull Request #674 · agentic-community/mcp-gateway-registry

shekharprateek · 2026-03-19T22:21:42Z

Summary

Implements the server-side telemetry collector infrastructure for anonymous usage tracking. Pairs with #659 (client-side telemetry, #558).

⚠️ Depends on #659 — please merge that first. The #558 client commits are included in this branch as the base.

Infrastructure (`terraform/telemetry-collector/`)

API Gateway HTTP API — HTTPS /v1/collect endpoint
Lambda (Python 3.12, 256MB, VPC-enabled) — validates events with Pydantic, stores to DocumentDB
DocumentDB — MongoDB-compatible storage with 365-day TTL
DynamoDB — privacy-preserving rate limiting (hashed IPs, 10 req/min, window resets correctly)
Secrets Manager — credential management
CloudWatch — logs and alarms (production stage)
Bastion host (optional, off by default) — t2.micro EC2 in VPC for DocumentDB access via mongosh

Lambda Collector

Pydantic validation for startup and heartbeat events
Sync pymongo (avoids event loop conflicts in Lambda)
Always returns 204 (no information leakage)
Extra/unknown fields stripped before storage
Rate limiter resets per time window (not reliant on DynamoDB TTL lazy deletion)
Lazy-init AWS clients for CI compatibility

Tests

20 unit tests for collector Lambda (tests/unit/lambda/test_collector.py)
All pass without AWS credentials (CI compatible)

End-to-End Verified

Tested against deployed AWS infrastructure:

Registry startup ping → API Gateway → Lambda → DocumentDB ✅
Validation, rate limiting, field stripping ✅
Warm Lambda latency: ~31ms ✅

Deployment

cd terraform/telemetry-collector
cp terraform.tfvars.example terraform.tfvars
terraform init && terraform apply

Closes #559

codecov-commenter · 2026-03-19T23:01:05Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 54.97835% with 208 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
registry/repositories/stats_repository.py	31.13%	71 Missing and 2 partials ⚠️
registry/core/telemetry.py	73.30%	53 Missing and 10 partials ⚠️
registry/api/registry_routes.py	9.09%	10 Missing ⚠️
registry/api/wellknown_routes.py	0.00%	7 Missing ⚠️
registry/services/ans_client.py	0.00%	7 Missing ⚠️
registry/api/agent_routes.py	0.00%	6 Missing ⚠️
auth_server/providers/okta.py	73.68%	1 Missing and 4 partials ⚠️
registry/services/ans_service.py	0.00%	5 Missing ⚠️
registry/services/federation_reconciliation.py	0.00%	5 Missing ⚠️
registry/main.py	20.00%	4 Missing ⚠️
... and 14 more

📢 Thoughts on this report? Let us know!

…ity#558) - Implement privacy-first anonymous telemetry with conspicuous disclosure - Support opt-out (default ON) via MCP_TELEMETRY_DISABLED=1 - Support opt-in for richer data (default OFF) via MCP_TELEMETRY_OPT_IN=1 - Multi-replica safe with MongoDB distributed locks - Never impacts registry operation (fail-silent, 5s timeout) - Comprehensive unit tests (28/28 passing) - Complete user documentation Changes: - Add registry/core/telemetry.py (536 lines) - Full implementation - Add tests/unit/core/test_telemetry.py (608 lines) - Comprehensive tests - Add docs/TELEMETRY.md (136 lines) - User-facing documentation - Update registry/core/config.py - Add telemetry settings and data_dir property - Update registry/core/metrics.py - Add telemetry_sends_total metric - Update registry/main.py - Integrate into startup/shutdown lifecycle - Update registry/api/system_routes.py - Add get_server_start_time() getter - Update .gitignore - Ignore registry/data/ directory Closes agentic-community#558

…ic-community#559) Deploy AWS infrastructure for collecting anonymous telemetry events from MCP Gateway Registry instances. Pairs with client-side telemetry (agentic-community#558). Infrastructure (terraform/telemetry-collector/): - API Gateway HTTP API with /v1/collect endpoint - Lambda function (Python 3.12, 256MB) with VPC access - DocumentDB cluster for event storage with 365-day TTL - DynamoDB-based rate limiting with privacy-preserving IP hashing - Secrets Manager for credential management - CloudWatch logging and alarms (production stage) - Custom domain support via Route53/ACM (optional) Lambda collector: - Pydantic validation for startup and heartbeat events - Sync pymongo for reliable DocumentDB writes - Rate limit with automatic window reset (not reliant on DynamoDB TTL) - Always returns 204 (no information leakage) - Extra/unknown fields stripped before storage Also includes: - Unit tests for collector Lambda (20 tests) - Updated docs/TELEMETRY.md with self-hosted deployment guide - Operational scripts (deploy.sh, destroy.sh, check-status.sh) - DocumentDB index creation script - .gitignore updates for build artifacts and state files Tested end-to-end against deployed AWS infrastructure: - Registry telemetry -> API Gateway -> Lambda -> DocumentDB ✓ - Validation, rate limiting, warm performance (31ms) ✓ Closes agentic-community#559

…ntic-community#558) asyncio.create_task() gets garbage collected in uvicorn lifespan context before the HTTP request completes. Awaiting directly ensures the startup ping is actually sent.

…gentic-community#559) boto3.resource('dynamodb') at module level fails in CI where no AWS credentials are configured. Lazy-initialize on first use instead.

…cker-compose - Fix heartbeat payload builder: list_servers/list_agents/list_skills → list_all() to match the actual ServerRepositoryBase/AgentRepositoryBase/SkillRepositoryBase interface method names (fixes silent warnings in opt-in detailed mode) - Add telemetry env vars to registry service in docker-compose.prebuilt.yml so customers can control opt-in/opt-out via .env without editing the compose file: MCP_TELEMETRY_DISABLED, MCP_TELEMETRY_OPT_IN, TELEMETRY_ENDPOINT, MCP_TELEMETRY_DEBUG - Add integration tests for opt-in/opt-out and detailed information mode (17 tests) covering all three customer scenarios: default, opt-out, opt-in/heartbeat, debug mode - Register 'live' pytest marker for AWS infrastructure tests (run manually with -m live) Closes agentic-community#558

…agentic-community#558) - Remove undocumented schema_version field from startup and heartbeat payloads to match issue agentic-community#558 spec exactly - Fix unit test mocks: list_servers/list_agents/list_skills -> list_all (tests were passing accidentally via exception handling fallback) - Redeploy Lambda collector with updated schemas.py

- Add stats_repository.py with hourly/daily/forever semantic search counters - Wire increment_search_counter into search_routes and agent_routes - Add search_queries_total field to both startup and heartbeat payloads - Update Lambda collector schemas to accept search_queries_total - Convert ts string to BSON ISODate in Lambda collector - Fix timezone-naive vs timezone-aware datetime comparison in stats repo - Update unit tests for async _build_startup_payload and new field

…oad (agentic-community#558) Add search_queries_daily_7d_moving_avg and search_queries_hourly_moving_avg fields to the heartbeat (Tier 2) telemetry event, set to None for now. Updates Lambda collector schema to accept them as optional float|None.

…rsion (agentic-community#558) - _build_startup_payload is now async, add await - Remove schema_version assertions (removed in 574b5b2) - Add get_search_count mock to all heartbeat payload tests

t2.micro EC2 in telemetry VPC public subnet with: - IAM role to read credentials from Secrets Manager - Security group allowing SSH from configurable CIDRs - DocumentDB SG rule allowing bastion access on 27017 - mongosh pre-installed with connect.sh and query.sh helpers - Disabled by default (bastion_enabled=false)

The $default stage invoke_url already has a trailing slash. Using trimsuffix() to strip it before appending /v1/collect prevents the //v1/collect double-slash in the output URL.

…umentation Security fixes: - Restrict CORS allow_origins from wildcard to configurable variable - Remove hardcoded fallback env vars in Lambda (fail-fast on missing config) - Remove DynamoDB DeleteItem from IAM policy (not needed by collector) - Restrict Lambda SG egress to DocumentDB (27017) and HTTPS (443) only - Default bastion SSH CIDR to empty list (require explicit configuration) - Fix bastion password exposure: use env var instead of CLI argument Reliability fixes: - Fix 2 failing integration tests (async/await bug in payload builders) - Add schema_version field to telemetry payloads for forward compatibility - Group Prometheus metric labels into categories (2xx/4xx/5xx) to prevent cardinality explosion - Add fcntl file locking to stats_repository to prevent concurrent write race condition - Break security group cycle with standalone aws_security_group_rule resources - Add Lambda test conftest to set required env vars before module import Infrastructure improvements: - Replace cloud-init user_data with post-deploy setup-bastion.sh script - Add bastion-scripts/ with connect.sh, query.sh, and setup-bastion.sh - Add aws_region to terraform outputs Documentation: - Merge QUICKSTART.md into README.md (single file) - Add Lambda packaging instructions (missing step) - Add ASCII architecture diagram - Add bastion SSH key generation and setup instructions - Add common DocumentDB query examples - Add telemetry env vars to .env.example - Set default telemetry endpoint to deployed collector URL

…iability, and documentation Security fixes: - Restrict CORS allow_origins from wildcard to configurable variable - Remove hardcoded fallback env vars in Lambda (fail-fast on missing config) - Remove DynamoDB DeleteItem from IAM policy (not needed by collector) - Restrict Lambda SG egress to DocumentDB (27017) and HTTPS (443) only - Default bastion SSH CIDR to empty list (require explicit configuration) - Fix bastion password exposure: use env var instead of CLI argument Reliability fixes: - Fix 2 failing integration tests (async/await bug in payload builders) - Add schema_version field to telemetry payloads for forward compatibility - Group Prometheus metric labels into categories (2xx/4xx/5xx) to prevent cardinality explosion - Add fcntl file locking to stats_repository to prevent concurrent write race condition - Break security group cycle with standalone aws_security_group_rule resources - Add Lambda test conftest to set required env vars before module import Infrastructure improvements: - Replace cloud-init user_data with post-deploy setup-bastion.sh script - Add bastion-scripts/ with connect.sh, query.sh, and setup-bastion.sh - Add aws_region to terraform outputs Documentation: - Merge QUICKSTART.md into README.md (single file) - Add Lambda packaging instructions (missing step) - Add ASCII architecture diagram - Add bastion SSH key generation and setup instructions - Add common DocumentDB query examples - Add telemetry env vars to .env.example - Set default telemetry endpoint to deployed collector URL

Added MCP_TELEMETRY_DISABLED, MCP_TELEMETRY_OPT_IN, TELEMETRY_ENDPOINT, and TELEMETRY_DEBUG to docker-compose.yml and docker-compose.podman.yml. Fixed docker-compose.prebuilt.yml: renamed MCP_TELEMETRY_DEBUG to TELEMETRY_DEBUG to match the Pydantic field name in config.py.

shekharprateek force-pushed the feature/issue-559-server-telemetry branch 4 times, most recently from a9d53b8 to f1d8246 Compare March 26, 2026 23:58

prsinp added 8 commits March 26, 2026 20:00

fix: await telemetry send instead of fire-and-forget create_task (age…

99e772f

…ntic-community#558) asyncio.create_task() gets garbage collected in uvicorn lifespan context before the HTTP request completes. Awaiting directly ensures the startup ping is actually sent.

fix: lazy-init AWS clients in collector Lambda for CI compatibility (a…

6c10c34

…gentic-community#559) boto3.resource('dynamodb') at module level fails in CI where no AWS credentials are configured. Lazy-initialize on first use instead.

shekharprateek force-pushed the feature/issue-559-server-telemetry branch from f1d8246 to c1ce92d Compare March 27, 2026 00:02

prsinp added 3 commits March 26, 2026 20:46

fix: update e2e tests for async startup payload and removed schema_ve…

9ab1135

…rsion (agentic-community#558) - _build_startup_payload is now async, add await - Remove schema_version assertions (removed in 574b5b2) - Add get_search_count mock to all heartbeat payload tests

fix: remove double slash in collector_url output

ca6db50

The $default stage invoke_url already has a trailing slash. Using trimsuffix() to strip it before appending /v1/collect prevents the //v1/collect double-slash in the output URL.

shekharprateek force-pushed the feature/issue-559-server-telemetry branch from c1ce92d to c13c6e8 Compare March 27, 2026 00:46

aarora79 force-pushed the feature/issue-559-server-telemetry branch from 155db53 to ba0541d Compare March 27, 2026 04:04

aarora79 merged commit c471e4c into agentic-community:main Mar 27, 2026
16 checks passed

shekharprateek mentioned this pull request Mar 27, 2026

feat: add client-side telemetry instrumentation (issue #558) #659

Closed

8 tasks

aarora79 mentioned this pull request Mar 28, 2026

Add lightweight anonymous telemetry to track registry adoption #557

Closed

2 tasks

Navigation Menu

Uh oh!

feat: server-side telemetry collector infrastructure (issue #559) - #674

feat: server-side telemetry collector infrastructure (issue #559)#674
aarora79 merged 13 commits into
agentic-community:mainfrom
shekharprateek:feature/issue-559-server-telemetry

shekharprateek commented Mar 19, 2026

Uh oh!

codecov-commenter commented Mar 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

shekharprateek commented Mar 19, 2026

Summary

Infrastructure (terraform/telemetry-collector/)

Lambda Collector

Tests

End-to-End Verified

Deployment

Uh oh!

codecov-commenter commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Infrastructure (`terraform/telemetry-collector/`)

codecov-commenter commented Mar 19, 2026 •

edited

Loading