Skip to content

API_SPECIFICATION.md

Latest commit

 

History

History
861 lines (697 loc) · 20.2 KB

API_SPECIFICATION.md

File metadata and controls

861 lines (697 loc) · 20.2 KB
Raw

Common Knowledge Base - API Specification

Overview

The Common Knowledge Base provides multiple API interfaces for different purposes:

  • External API (Ruuter): Public endpoints with authentication
  • Internal API (Ruuter Internal): Service-to-service communication
  • Microservice APIs: Direct service endpoints for specialized operations

API Access

External API (Port 8080)

Base URL: http://localhost:8080/ckb Authentication: JWT Bearer token required Documentation: Available at /docs (OpenAPI/Swagger)

Service APIs

Each microservice exposes its own API with auto-generated documentation:

Service Port Docs URL Purpose
File Processing 8888 http://localhost:8888/docs File operations and storage
Scrapper 8000 http://localhost:8000/docs Web scraping tasks
Cleaning 8123 http://localhost:8123/docs Content cleaning
Scheduler 8003 http://localhost:8003/docs Task scheduling
Data Export 8002 http://localhost:8002/docs Data export operations
Search 3000 - Content search and indexing

Authentication

Login

POST /ckb/auth/login
Content-Type: application/json

{
  "username": "string",
  "password": "string"
}

Response:

{
  "access_token": "eyJ...",
  "token_type": "bearer",
  "expires_in": 3600
}

Using JWT Token

Authorization: Bearer eyJ...

User Info

GET /ckb/auth/jwt/userinfo
Authorization: Bearer eyJ...

External API Endpoints

Agency Management

GET /ckb/agency/all

List all agencies.

Response:

[
  {
    "id": "uuid",
    "base_id": "uuid", 
    "name": "Agency Name",
    "sector": "Healthcare",
    "created_at": "2025-01-01T00:00:00Z",
    "updated_at": "2025-01-01T00:00:00Z",
    "type": "client",
    "data_hash": "sha1_hash"
  }
]

GET /ckb/agency/get

Get specific agency details.

Query Parameters:

  • base_id (UUID): Agency base ID

POST /ckb/agency/add

Create new agency.

Constraint: CKB supports only one agency per deployment. This endpoint returns 409 Conflict if an agency already exists. The Add Agency button in the GUI is hidden once an agency has been created.

Request Body:

{
  "name": "string",
  "sector": "string"
}

Responses:

  • 200 OK — Agency created successfully.
  • 409 Conflict — An agency already exists. Body: "An agency already exists. CKB supports only one agency per deployment."

POST /ckb/agency/edit

Update agency information.

Request Body:

{
  "base_id": "uuid",
  "name": "string",
  "sector": "string"
}

POST /ckb/agency/remove

Blocked: Agency deletion is not permitted. This endpoint always returns 405 Method Not Allowed. An agency can only be updated via POST /ckb/agency/edit.

Response:

  • 405 Method Not Allowed — Body: "Agency deletion is not permitted. An agency can only be updated."

Source Management

GET /ckb/source/all

List all sources.

Query Parameters:

  • agency_base_id (UUID, optional): Filter by agency

Response:

[
  {
    "id": "uuid",
    "base_id": "uuid",
    "agency_base_id": "uuid", 
    "url": "https://example.com",
    "subsector": "Public Health",
    "type": "url_to_scrape",
    "status": "running",
    "update_automatically": true,
    "cron_schedule": "0 */6 * * *",
    "last_scraped_at": "2025-01-01T12:00:00Z",
    "next_scrapping_at": "2025-01-01T18:00:00Z"
  }
]

GET /ckb/source/api/all

List API sources specifically.

GET /ckb/source/get

Get specific source details.

Query Parameters:

  • base_id (UUID): Source base ID

POST /ckb/source/add

Create new data source.

Behavior: Triggers first-time scraping via sitemap collect (is_initial_scrape: true), then pauses before cleaning, sets source and source_files state to in_review, and waits for user action from Start Cleaning.

Request Body:

{
  "agencyBaseId": "uuid",
  "url": "https://example.com",
  "subsector": "string",
  "type": "url_to_scrape",
  "qualityControl": "basic",
  "extractImages": true
}

qualityControl values:

  • "basic" - enable LLM extraction checks
  • "comprehensive" - enable LLM extraction + correction
  • null or omitted - no LLM quality control

Narrowed scraping rule (single URL): For single-URL narrowed scraping (for example via source-file refresh), scraping is limited to the given URL scope and its subdomains/subpages under it.

POST /ckb/source/add-with-url-list

Create a source from a pre-selected URL list and trigger scraping for the explicitly specified URLs.

Selected URL list scope rule: Only the URLs provided in urls are scraped.

Request Body:

{
  "agencyBaseId": "uuid",
  "url": "https://example.com",
  "subsector": "string",
  "type": "specified",
  "qualityControl": "basic",
  "extractImages": true,
  "urls": [
    { "url": "https://page-1" },
    { "url": "https://page-2" }
  ]
}

Response:

{
  "source": {
    "baseId": "uuid"
  },
  "urls_count": 2,
  "scraping_triggered": true
}

POST /ckb/source/edit-scrape-interval

Update scraping frequency.

Request Body:

{
  "baseId": "uuid",
  "cronSchedule": "0 */12 * * *",
  "updateAutomatically": true,
  "qualityControl": "basic",
  "extractImages": true
}

POST /ckb/source/edit-subsector

Update source subsector.

Request Body:

{
  "base_id": "uuid", 
  "subsector": "New Subsector"
}

POST /ckb/source/refresh

Trigger source scraping.

Request Body:

{
  "base_id": "uuid"
}

Behavior by source type:

  • pre_selected_urls -> refreshes eligible source files and triggers specified-pages pipeline
  • api -> triggers API scraping pipeline
  • other web sources -> triggers entire-source pipeline

POST /ckb/source/stop

Stop source processing.

Request Body:

{
  "base_id": "uuid"
}

POST /ckb/source/remove

Delete source.

Request Body:

{
  "base_id": "uuid"
}

File Operations

GET /ckb/source-file/all

List source files.

Query Parameters:

  • agency_base_id (UUID, optional): Filter by agency
  • source_base_id (UUID, optional): Filter by source
  • type (string, optional): Filter by file type

Response:

[
  {
    "id": "uuid",
    "base_id": "uuid",
    "source_base_id": "uuid",
    "agency_base_id": "uuid",
    "url": "https://example.com/page",
    "page_title": "Document Title",
    "file_name": "document.pdf",
    "type": "scraped_file",
    "status": "finished",
    "subsector": "Healthcare",
    "is_excluded": false,
    "original_data_url": "s3://bucket/raw/file",
    "cleaned_data_url": "s3://bucket/cleaned/file",
    "created_at": "2025-01-01T00:00:00Z",
    "last_scraped_at": "2025-01-01T12:00:00Z"
  }
]

POST /ckb/source-file/add-uploaded-files

Add uploaded files to a source.

Request Headers:

  • Cookie: Contains JWT with user information for tracking uploader

Request Body:

{
  "agencyId": "uuid",
  "sourceId": "uuid",
  "files": [
    {
      "base_id": "uuid",
      "file_name": "document.pdf",
      "original_data_url": "s3://bucket/uploads/file",
      "subsector": "Legal",
      "file_size": 13264
    }
  ]
}

Response:

[
  {
    "id": "uuid",
    "url": null,
    "hash": "",
    "original_data_url": "s3://bucket/uploads/file",
    "path": "s3://bucket/uploads/file"
  }
]

Note: The uploaded_by field is automatically populated from the JWT cookie (user's idCode).

POST /ckb/source-file/refresh

Refresh (re-scrape) one source file by baseId.

Request Body:

{
  "baseId": "uuid"
}

POST /ckb/source-file/refresh-multiple

Refresh (re-scrape) multiple source files in one request.

Request Body:

{
  "baseIds": ["uuid1", "uuid2", "uuid3"]
}

Behavior:

  • If files are API-backed, triggers the API specified-files pipeline
  • Otherwise, triggers specified-pages pipeline

POST /ckb/source-file/bulk-exclude

Bulk include/exclude source files.

Request Body:

{
  "baseIds": ["uuid1", "uuid2"],
  "excluded": true
}

Behavior:

  • Updates exclusion status in DB
  • Moves corresponding files between include/exclude S3 paths

POST /ckb/source-file/bulk-remove

Bulk delete source files.

Request Body:

{
  "baseIds": ["uuid1", "uuid2"]
}

Behavior:

  • is_deleted value updates to true in DB as a new row.
  • Removes indexed documents from search in bulk
  • Triggers async S3 deletion task

POST /ckb/source-file/get-upload-urls

Get presigned upload URLs.

Request Body:

{
  "source_base_id": "uuid",
  "file_names": ["file1.pdf", "file2.docx"]
}

Response:

{
  "upload_urls": [
    {
      "file_name": "file1.pdf", 
      "upload_url": "https://s3.amazonaws.com/presigned-url",
      "expires_at": "2025-01-01T01:00:00Z"
    }
  ]
}

POST /ckb/source-file/exclude

Exclude files from processing.

Request Body:

{
  "base_ids": ["uuid1", "uuid2"]
}

POST /ckb/source-file/edit-file

Edit file metadata.

Request Body:

{
  "base_id": "uuid",
  "page_title": "New Title",
  "subsector": "New Subsector"
}

POST /ckb/source-file/refresh

Trigger file reprocessing.

Request Body:

{
  "base_ids": ["uuid1", "uuid2"]
}

POST /ckb/source-file/remove

Delete files.

Request Body:

{
  "base_ids": ["uuid1", "uuid2"]
}

Reports and Monitoring

GET /ckb/reports/all

List processing reports.

Query Parameters:

  • agency_base_id (UUID, optional): Filter by agency
  • source_base_id (UUID, optional): Filter by source

Response:

[
  {
    "id": "uuid",
    "base_id": "uuid",
    "agency_base_id": "uuid",
    "source_base_id": "uuid", 
    "agency_name": "Agency Name",
    "url": "https://example.com",
    "scraping_started_at": "2025-01-01T12:00:00Z",
    "scraping_finished_at": "2025-01-01T12:30:00Z",
    "errors": 2,
    "scraping_log_url": "s3://bucket/logs/scraping.log",
    "cleaning_log_url": "s3://bucket/logs/cleaning.log"
  }
]

GET /ckb/reports/logs/all

Get detailed processing logs.

POST /ckb/reports/remove

Delete processing reports.

Request Body:

{
  "base_ids": ["uuid1", "uuid2"] 
}

File Download

GET /ckb/get-download-url

Generate download URL for a file.

Query Parameters:

  • blob_storage_path (string): Path to file in blob storage

Response:

{
  "download_url": "https://s3.amazonaws.com/presigned-url",
  "expires_at": "2025-01-01T01:00:00Z"
}

Microservice APIs

File Processing API (Port 8888)

OpenAPI Docs: http://localhost:8888/docs

Upload Operations

  • POST /upload-urls - Generate presigned upload URLs
  • POST /upload - Async file upload with task tracking
  • POST /upload-sync - Synchronous file upload
  • POST /upload-file-content - Direct content upload
  • GET /upload/{task_id} - Upload task status

Download Operations

  • POST /download-urls - Generate download URLs
  • POST /download-files-to-volume - Download to local storage
  • POST /download-files-to-volume-async - Async download
  • GET /download-task/{task_id} - Download task status

File Management

  • POST /delete-files - Delete from blob storage
  • POST /delete-files-async - Async deletion
  • POST /move-files - Move files in storage
  • POST /move-files-async - Async file move
  • POST /zip-and-upload-folders - Create archives

Scrapper API (Port 8000)

OpenAPI Docs: http://localhost:8000/docs

Scraping Tasks

  • POST /specified-pages-scrapper-task - Scrape specific URLs
  • POST /entire-source-scrapper-task - Comprehensive site scraping
  • POST /sitemap-collect-scrapper-task - Sitemap-based scraping
  • POST /eesti-scrapper-task - Estonian government sites
  • POST /specified-api-files-scrapper-task - Process API files
  • POST /uploaded-file - Process uploaded files
  • POST /generate-edited-metadata - Generate file metadata

Cleaning API (Port 8123)

OpenAPI Docs: http://localhost:8123/docs

Content Processing

  • POST /clean_file - Clean and extract text from a single file (synchronous, 10-minute timeout)
  • POST /clean_source_async - Clean a batch of files (asynchronous, returns immediately)

Scheduler API (Port 8003)

OpenAPI Docs: http://localhost:8003/docs

Schedule Management

  • POST /render_next_run - Calculate next execution time

Data Export API (Port 8002)

OpenAPI Docs: http://localhost:8002/docs

Export Operations

  • GET /exports - List available export tasks
  • POST /exports - Trigger all exports
  • GET /exports/{task_name} - Get specific export task
  • POST /exports/{task_name} - Trigger specific export

Search API (Port 3000)

Search Operations

  • POST /index/{sourceId}/bulk - Bulk index documents
  • GET /search/{sourceId} - Search within source
  • GET /document/{sourceId}/{documentId} - Get full document
  • DELETE /index/{sourceId} - Delete source index
  • DELETE /documents/{sourceId}/{sourceFileId} - Delete documents
  • GET /stats/{sourceId} - Get source statistics
  • GET /health - Service health check

Error Responses

Standard Error Format

{
  "error": "Error description",
  "code": "ERROR_CODE", 
  "details": "Additional error details"
}

Common HTTP Status Codes

  • 200: Success
  • 400: Bad Request - Invalid parameters
  • 401: Unauthorized - Missing or invalid authentication
  • 403: Forbidden - Insufficient permissions
  • 404: Not Found - Resource does not exist
  • 500: Internal Server Error - Server processing error

Data Models

Agency

{
  "id": "uuid",
  "base_id": "uuid",
  "name": "string",
  "sector": "string",
  "type": "client | api",
  "zip_dirty": "boolean",
  "is_zipping": "boolean",
  "data_hash": "string",
  "created_at": "timestamp",
  "updated_at": "timestamp"
}

Source

{
  "id": "uuid",
  "base_id": "uuid", 
  "agency_base_id": "uuid",
  "url": "string",
  "subsector": "string",
  "type": "url_to_scrape | file | api",
  "status": "new | running | finished | failed",
  "update_automatically": "boolean",
  "cron_schedule": "string",
  "last_scraped_at": "timestamp",
  "next_scrapping_at": "timestamp",
  "is_stopping": "boolean",
  "created_at": "timestamp",
  "updated_at": "timestamp"
}

Source File

{
  "id": "uuid",
  "base_id": "uuid",
  "source_base_id": "uuid",
  "agency_base_id": "uuid",
  "url": "string",
  "page_title": "string",
  "file_name": "string",
  "type": "scraped_file | uploaded_file | api_file",
  "status": "scraping | cleaning | finished | not_found | failed",
  "subsector": "string",
  "is_excluded": "boolean",
  "original_data_url": "string",
  "cleaned_data_url": "string", 
  "edited_data_url": "string",
  "original_metadata_url": "string",
  "cleaned_metadata_url": "string",
  "edited_metadata_url": "string",
  "original_data_hash": "string",
  "external_id": "string",
  "created_at": "timestamp",
  "updated_at": "timestamp",
  "last_scraped_at": "timestamp",
  "originally_scraped": "timestamp"
}

Source Run Report

{
  "id": "uuid",
  "base_id": "uuid",
  "agency_base_id": "uuid",
  "source_base_id": "uuid",
  "agency_name": "string",
  "url": "string",
  "scraping_started_at": "timestamp",
  "scraping_finished_at": "timestamp", 
  "errors": "integer",
  "scraping_log_url": "string",
  "cleaning_log_url": "string",
  "created_at": "timestamp",
  "updated_at": "timestamp"
}

API Usage Examples

Complete Workflow Example

# 1. Login and get token
TOKEN=$(curl -X POST http://localhost:8080/ckb/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "password"}' \
  | jq -r '.access_token')

# 2. Create agency
AGENCY_RESPONSE=$(curl -X POST http://localhost:8080/ckb/agency/add \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "Test Agency", "sector": "Healthcare"}')

AGENCY_ID=$(echo $AGENCY_RESPONSE | jq -r '.base_id')

# 3. Add data source
SOURCE_RESPONSE=$(curl -X POST http://localhost:8080/ckb/source/add \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"agency_base_id\": \"$AGENCY_ID\", \"url\": \"https://example.com\", \"type\": \"url_to_scrape\"}")

SOURCE_ID=$(echo $SOURCE_RESPONSE | jq -r '.base_id')

# 4. Trigger scraping
curl -X POST http://localhost:8080/ckb/source/refresh \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"base_id\": \"$SOURCE_ID\"}"

# 5. Monitor progress
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8080/ckb/reports/all?source_base_id=$SOURCE_ID"

# 6. List collected files
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8080/ckb/source-file/all?source_base_id=$SOURCE_ID"

# 7. Search content (if search service is running)
curl "http://localhost:3000/search/$SOURCE_ID?q=healthcare&page=1&size=10"

File Upload Example

# 1. Get upload URLs
UPLOAD_RESPONSE=$(curl -X POST http://localhost:8080/ckb/source-file/get-upload-urls \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"source_base_id\": \"$SOURCE_ID\", \"file_names\": [\"document.pdf\"]}")

UPLOAD_URL=$(echo $UPLOAD_RESPONSE | jq -r '.upload_urls[0].upload_url')

# 2. Upload file to S3
curl -X PUT "$UPLOAD_URL" \
  -H "Content-Type: application/pdf" \
  --data-binary @document.pdf

# 3. Add file to source
curl -X POST http://localhost:8080/ckb/source-file/add-uploaded-files \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"source_base_id\": \"$SOURCE_ID\", \"files\": [{\"file_name\": \"document.pdf\", \"original_data_url\": \"uploads/path/document.pdf\"}]}"

Rate Limiting

External API

  • Authentication: 100 requests/minute per user
  • Data operations: 1000 requests/hour per user
  • File uploads: 50 uploads/hour per user

Internal APIs

  • Service calls: No rate limiting (internal network)
  • Resource limits: Based on infrastructure capacity

Versioning

API Versioning Strategy

  • External API: Versioned via URL path (/v1/, /v2/)
  • Internal APIs: Backward compatibility maintained
  • Database Schema: Liquibase migration versioning
  • Service APIs: Semantic versioning in OpenAPI specs

Breaking Changes

  • External API: New version for breaking changes
  • Internal APIs: Gradual migration approach
  • Database: Liquibase rollback support
  • Client Libraries: Version compatibility matrix

Security

Authentication

  • JWT Tokens: RS256 signature algorithm
  • Token Expiry: 1 hour default, refresh tokens available
  • Scopes: Role-based access control

Authorization

  • User Roles: Admin, operator, viewer
  • Resource Access: Agency-based data isolation
  • API Permissions: Endpoint-level access control

Data Protection

  • HTTPS Only: All external communications encrypted
  • Parameter Validation: Input sanitization and validation
  • SQL Injection Prevention: Parameterized queries via Resql
  • File Security: Presigned URLs with expiration

OpenAPI Documentation

Accessing API Documentation

Each FastAPI service automatically generates interactive API documentation:

# File Processing API
open http://localhost:8888/docs

# Scrapper API  
open http://localhost:8000/docs

# Cleaning API
open http://localhost:8123/docs

# Scheduler API
open http://localhost:8003/docs

# Data Export API
open http://localhost:8002/docs

Features

  • Interactive Testing: Test endpoints directly in browser
  • Request/Response Examples: Complete API examples
  • Schema Validation: Parameter and response validation
  • Authentication Testing: JWT token integration

Integration Patterns

Service Communication

  • External → Internal: User operations trigger internal workflows
  • Internal → Services: Pipeline coordination via service APIs
  • Service → Service: Direct API calls for processing workflows
  • Database: All data access via Resql query engine

Error Handling

  • Graceful Degradation: Continue processing when non-critical services fail
  • Retry Logic: Automatic retry for transient failures
  • Circuit Breakers: Prevent cascade failures
  • Error Propagation: Proper error context preservation