query-orchestration/docs/ai.generated/03-api-documentation.md

API Documentation

API Overview

The DoczyAI Query Orchestration Platform exposes a comprehensive REST API following OpenAPI 3.0.3 specification. The API is automatically generated from the specification file serviceAPIs/queryAPI.yaml using oapi-codegen.

Base Configuration

• Base URL: https://doczy.com/v1 (production), http://localhost:8080 (development)
• API Version: 0.0.1
• Content Type: application/json
• Documentation: Auto-generated Swagger UI at /swagger/index.html

Authentication and Authorization

Security Schemes

JWT Authentication

• Type: Bearer token authentication
• Header: Authorization: Bearer <token>
• Token Source: JWT cookie set via OAuth flow

OAuth2 Flow

• Type: OAuth2 Authorization Code Flow
• Provider: AWS Cognito
• Scopes: Group-based permissions

Authentication Endpoints

GET /login

Initiates OAuth2 authentication flow with AWS Cognito.

Response: Redirect to Cognito login page

GET /login-callback

OAuth2 callback handler that processes authentication response.

Parameters:

• code (query): Authorization code from Cognito
• state (query): CSRF protection parameter

Response: Sets JWT cookie and redirects to dashboard

GET /logout

Terminates user session and clears authentication cookies.

Response: Redirect to login page

GET /home

Protected dashboard endpoint showing user information.

Rate Limiting

The queryAPI service implements dual-layer rate limiting to protect against DDoS attacks and ensure fair resource allocation.

Rate Limit Architecture

Dual-Layer Protection:

• Layer 1: Global/Aggregate Rate Limiting - Protects total system capacity across all IP addresses
• Layer 2: Per-IP Rate Limiting - Prevents individual client abuse

Both layers must pass for a request to proceed. The global limiter checks first for faster failure paths under DDoS conditions.

Default Rate Limits

By default, all endpoints are limited to:

• Global: 500 requests per second total (1000 burst)
• Per-IP: 5 requests per second per IP (10 burst)
• Cleanup Interval: 3 minutes for inactive IP limiters

Important: No endpoints are exempt from rate limiting. This is a security requirement to prevent DDoS attacks.

Configuration

Rate limits are configured via environment variables:

# Global/Aggregate rate limiting (protects total system capacity)
RATE_LIMIT_GLOBAL_RATE=500         # total req/s across ALL IPs
RATE_LIMIT_GLOBAL_BURST=1000        # total burst capacity across ALL IPs

# Per-IP rate limiting (protects against individual abusers)
RATE_LIMIT_DEFAULT_RATE=5          # req/s per IP
RATE_LIMIT_DEFAULT_BURST=10         # burst per IP
RATE_LIMIT_EXPIRES_IN=180           # seconds (3 minutes)

# Per-endpoint overrides (JSON format)
RATE_LIMIT_ENDPOINT_OVERRIDES={}

Configuring Endpoint-Specific Overrides

Endpoints can have custom rate limits based on their resource requirements. Configure using the RATE_LIMIT_ENDPOINT_OVERRIDES environment variable.

Endpoint Route Format

Routes use the format: {HTTP_METHOD} {PATH}

Examples:

• GET /health
• POST /documents
• GET /documents/{id} (use route template, not actual IDs)
• POST /client/{id}/document/batch

Override Configuration Structure

Each endpoint requires 4 values:

{
  "ENDPOINT_ROUTE": {
    "global_rate": 100,    // Total req/s across ALL IPs
    "global_burst": 200,   // Total burst across ALL IPs
    "rate": 5,             // Req/s per individual IP
    "burst": 10            // Burst per individual IP
  }
}

Setting Environment Variable

Single endpoint:

export RATE_LIMIT_ENDPOINT_OVERRIDES='{"POST /documents": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10}}'

Multiple endpoints:

  export RATE_LIMIT_ENDPOINT_OVERRIDES='{
    "GET /health": {"global_rate": 500, "global_burst": 1000, "rate": 100, "burst": 200},
    "GET /metrics": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
    "GET /swagger/*": {"global_rate": 200, "global_burst": 400, "rate": 50, "burst": 100},
    "GET /auth/login": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
    "GET /auth/callback": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
    "GET /auth/logout": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
    "GET /auth/home": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
    "POST /documents": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
    "POST /documents/batch": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2},
    "GET /documents/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
    "GET /client/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
    "POST /query/test": {"global_rate": 20, "global_burst": 40, "rate": 2, "burst": 5},
    "POST /export": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2}
  }'

In .env file (single line, valid JSON):

RATE_LIMIT_ENDPOINT_OVERRIDES={"GET /health":{"global_rate":500,"global_burst":1000,"rate":100,"burst":200},"POST /documents":{"global_rate":50,"global_burst":100,"rate":5,"burst":10}}

Recommended Endpoint Tiers

Tier 1: Monitoring/Infrastructure (High Limits)

{
  "GET /health": {"global_rate": 500, "global_burst": 1000, "rate": 100, "burst": 200},
  "GET /metrics": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
  "GET /swagger/*": {"global_rate": 200, "global_burst": 400, "rate": 50, "burst": 100}
}

Tier 2: Authentication (Moderate - Prevent Brute Force)

{
  "GET /auth/login": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
  "GET /auth/callback": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
  "GET /auth/logout": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20}
}

Tier 3: Read Operations (Moderate)

{
  "GET /client/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
  "GET /documents/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
  "GET /collector/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40}
}

Tier 4: Write/Processing (Low - Resource Intensive)

{
  "POST /documents": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
  "POST /documents/batch": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2},
  "POST /query/test": {"global_rate": 20, "global_burst": 40, "rate": 2, "burst": 5},
  "POST /export": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2}
}

Verification

After restarting the service, check logs for confirmation:

level=INFO msg="Rate limiting enabled" global_rate=1000 global_burst=2000
  default_rate=10 default_burst=20 overrides_count=12

The overrides_count should match your configured endpoints.

Troubleshooting

Invalid JSON: Validate with echo $RATE_LIMIT_ENDPOINT_OVERRIDES | jq .

Override not applied:

• Verify HTTP method matches exactly (e.g., POST not post)
• Use route template for paths with parameters (/documents/{id} not /documents/123)
• Check for extra/missing slashes in path
• Enable debug logging: DEBUG=true to see route matching

Rate Limit Headers

All API responses include a RateLimit header indicating the current limit:

RateLimit: 20;window=2

Format: {burst};window={seconds}

• burst: Maximum requests allowed in a burst
• window: Time window in seconds (calculated as burst/rate)

Rate Limit Exceeded Response

When the rate limit is exceeded, the API returns:

HTTP Status: 429 Too Many Requests

Headers:

• Retry-After: {seconds} - Number of seconds before retrying
• RateLimit: {burst};window={seconds} - Current rate limit

Response Body:

{
  "message": "rate limit exceeded"
}

Testing Rate Limits

Test rate limiting behavior with curl:

# Exceed rate limit by making rapid requests
for i in {1..25}; do
  curl -i http://localhost:8080/client/AAA \
    -H "Authorization: Bearer $TOKEN"
done

# Observe 429 responses after burst is exhausted

Technical Implementation

• Algorithm: Dual-layer token bucket (golang.org/x/time/rate)
• Layer 1: Global rate limiter (shared across all IPs)
• Layer 2: Per-IP rate limiter (separate bucket per IP via c.RealIP())
• Storage: In-memory with automatic cleanup
• Middleware Position: Applied BEFORE authentication to protect auth endpoints
• Protection: Defends against both distributed DDoS and single-source attacks

Security: Requires authentication Response: HTML dashboard page

Authorization Model

Permission Groups

• exporters: Can access export operations
• uploaders: Can upload documents and manage clients
• querybuilders: Can create and modify queries

Route Permissions

/client/*     → exporters, uploaders, querybuilders
/document/*   → exporters, querybuilders  
/query/*      → exporters, uploaders, querybuilders
/export/*     → exporters only
/health       → public (no authentication required)

Core API Endpoints

Client Management

POST /client

Creates a new client in the system.

Security: Requires uploaders or querybuilders permission

Request Body:

{
  "clientId": "string (max 256 chars)",
  "name": "string (max 256 chars)"
}

Responses:

• 201: Client created successfully
• 400: Invalid request data
• 401: Authentication required
• 403: Insufficient permissions

GET /client/{id}

Retrieves client details by ID.

Path Parameters:

• id: Client identifier (string, max 256 chars)

Response Example:

{
  "clientId": "acme-corp",
  "name": "ACME Corporation"
}

PATCH /client/{id}

Updates client information.

Request Body:

{
  "name": "string (max 256 chars)"
}

GET /client/{id}/status

Returns client synchronization status.

Response Example:

{
  "status": "IN_SYNC" | "NOT_SYNCED" | "NOT_SYNCING"
}

Document Operations

POST /client/{id}/document

Uploads a document for a specific client.

Content Type: multipart/form-data Max File Size: 100GB Supported Types: PDF (extensible)

Form Parameters:

• file: Document file (required)

Response:

{
  "documentId": "uuid",
  "clientId": "string",
  "filename": "string",
  "uploadedAt": "2024-01-01T00:00:00Z"
}

GET /client/{id}/document

Lists documents for a client with optional filtering.

Query Parameters:

• limit: Maximum results (integer)
• offset: Pagination offset (integer)

Response:

{
  "documents": [
    {
      "documentId": "uuid",
      "clientId": "string",
      "filename": "string",
      "status": "string",
      "uploadedAt": "2024-01-01T00:00:00Z"
    }
  ],
  "total": "integer"
}

GET /document/{id}

Retrieves document metadata by ID.

Path Parameters:

• id: Document UUID

Batch Document Operations

POST /client/{id}/document/batch

Uploads a ZIP archive containing multiple PDF documents for asynchronous batch processing.

Security: Requires uploaders or querybuilders permission Content Type: multipart/form-data Max Archive Size: 1GB Supported Archive Type: ZIP only

Form Parameters:

• archive: ZIP file containing PDF documents (required)

Response (202 Accepted):

{
  "batch_id": "uuid",
  "status": "processing",
  "status_url": "/client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a"
}

GET /client/{id}/document/batch

Lists batch uploads for a client with pagination support.

Query Parameters:

• limit: Maximum results (integer, default: 20)
• offset: Pagination offset (integer, default: 0)

Response:

{
  "batches": [
    {
      "batch_id": "uuid",
      "status": "completed" | "processing" | "failed" | "cancelled",
      "total_documents": 15,
      "processed_documents": 15,
      "failed_documents": 0,
      "created_at": "2024-01-01T00:00:00Z",
      "completed_at": "2024-01-01T00:05:00Z"
    }
  ],
  "total": 42
}

GET /client/{id}/document/batch/{batch_id}

Retrieves detailed status information for a specific batch upload.

Path Parameters:

• batch_id: Batch upload identifier (UUID)

Response:

{
  "batch_id": "uuid",
  "status": "completed",
  "total_documents": 15,
  "processed_documents": 15,
  "failed_documents": 0,
  "failed_filenames": [],
  "created_at": "2024-01-01T00:00:00Z",
  "completed_at": "2024-01-01T00:05:00Z"
}

DELETE /client/{id}/document/batch/{batch_id}

Cancels a batch upload that is currently processing.

Path Parameters:

• batch_id: Batch upload identifier (UUID)

Response:

• 204: Batch upload cancelled successfully
• 409: Cannot cancel completed or already cancelled batch

Query Management

GET /query

Lists all available queries in the system.

Response:

{
  "queries": [
    {
      "queryId": "uuid",
      "queryType": "JSON_EXTRACTOR" | "CONTEXT_FULL",
      "name": "string",
      "description": "string",
      "version": "integer"
    }
  ]
}

POST /query

Creates a new query definition.

Request Body:

{
  "queryType": "JSON_EXTRACTOR" | "CONTEXT_FULL",
  "name": "string (max 256 chars)",
  "description": "string (max 512 chars)",
  "config": "object (JSON configuration)"
}

GET /query/{id}

Retrieves query details including configuration.

PATCH /query/{id}

Updates query definition and increments version.

POST /query/{id}/test

Tests query execution against sample data.

Request Body:

{
  "testData": "string",
  "documentId": "uuid (optional)"
}

Collector Configuration

GET /client/{id}/collector

Retrieves collector configuration for a client.

Response:

{
  "clientId": "string",
  "queries": [
    {
      "name": "string",
      "queryId": "uuid"
    }
  ],
  "minCleanVersion": "integer",
  "minTextVersion": "integer"
}

PATCH /client/{id}/collector

Updates collector configuration.

Export Operations

POST /client/{id}/export

Triggers data export for a client.

Request Body:

{
  "format": "CSV" | "JSON" | "EXCEL",
  "filters": {
    "queryIds": ["uuid"],
    "dateRange": {
      "start": "2024-01-01T00:00:00Z",
      "end": "2024-12-31T23:59:59Z"
    }
  }
}

GET /export/{id}

Checks export status and retrieves download URL.

Response:

{
  "exportId": "uuid",
  "status": "PENDING" | "PROCESSING" | "COMPLETED" | "FAILED",
  "downloadUrl": "string (when completed)",
  "createdAt": "2024-01-01T00:00:00Z",
  "completedAt": "2024-01-01T00:05:00Z"
}

Request/Response Schemas

Standard Response Headers

All API responses include:

• RateLimit: Current rate limit status (format: 100;window=60)
• Content-Type: application/json (except file downloads)

Error Response Format

{
  "message": "Error description (max 256 chars)"
}

Common HTTP Status Codes

• 200: Success
• 201: Created
• 400: Bad Request (validation errors)
• 401: Unauthorized (authentication required)
• 403: Forbidden (insufficient permissions)
• 404: Not Found
• 429: Too Many Requests (rate limited)
• 500: Internal Server Error

Rate Limiting

Current Limits

• Rate: 100 requests per 60-second window
• Scope: Per authenticated user
• Headers: RateLimit header shows current status
• Exceeded: Returns 429 with Retry-After header

API Versioning

Current Strategy

• URL-based versioning: /v1 prefix
• Backward Compatibility: Maintained within major versions
• OpenAPI Version: 0.0.1 (development phase)

Future Considerations

• Semantic versioning for production releases
• Deprecation notices for breaking changes
• Version sunset policies

Code Generation

OpenAPI Integration

The API is generated from the OpenAPI specification using:

oapi-codegen -generate "types,server" -o api.gen.go serviceAPIs/queryAPI.yaml

Generated Artifacts

• Type-safe Go structs for requests/responses
• Echo framework route handlers
• OpenAPI validation middleware
• Swagger documentation

Validation

• Automatic request validation against OpenAPI schema
• Response validation in development mode
• Parameter binding with type safety