query-orchestration/docs/ai.generated/03-api-documentation.md

# API Documentation

## API Overview

The DoczyAI Query Orchestration Platform exposes a comprehensive REST API following OpenAPI 3.0.3 specification. The API is automatically generated from the specification file `serviceAPIs/queryAPI.yaml` using oapi-codegen.

### Base Configuration
- **Base URL**: `https://doczy.com/v1` (production), `http://localhost:8080` (development)
- **API Version**: 0.0.1
- **Content Type**: `application/json`
- **Documentation**: Auto-generated Swagger UI at `/swagger/index.html`

## Authentication and Authorization

### Security Schemes

#### JWT Authentication
- **Type**: Bearer token authentication
- **Header**: `Authorization: Bearer <token>`
- **Token Source**: JWT cookie set via OAuth flow

#### OAuth2 Flow
- **Type**: OAuth2 Authorization Code Flow
- **Provider**: AWS Cognito
- **Scopes**: Group-based permissions

### Authentication Endpoints

#### `GET /login`
Initiates OAuth2 authentication flow with AWS Cognito.

**Response**: Redirect to Cognito login page

#### `GET /login-callback`
OAuth2 callback handler that processes authentication response.

**Parameters**:
- `code` (query): Authorization code from Cognito
- `state` (query): CSRF protection parameter

**Response**: Sets JWT cookie and redirects to dashboard

#### `GET /logout`
Terminates user session and clears authentication cookies.

**Response**: Redirect to login page

#### `GET /home`
Protected dashboard endpoint showing user information.

## Rate Limiting

The queryAPI service implements dual-layer rate limiting to protect against DDoS attacks and ensure fair resource allocation.

### Rate Limit Architecture

**Dual-Layer Protection:**
- **Layer 1: Global/Aggregate Rate Limiting** - Protects total system capacity across all IP addresses
- **Layer 2: Per-IP Rate Limiting** - Prevents individual client abuse

Both layers must pass for a request to proceed. The global limiter checks first for faster failure paths under DDoS conditions.

### Default Rate Limits

By default, all endpoints are limited to:
- **Global**: 500 requests per second total (1000 burst)
- **Per-IP**: 5 requests per second per IP (10 burst)
- **Cleanup Interval**: 3 minutes for inactive IP limiters

**Important:** No endpoints are exempt from rate limiting. This is a security requirement to prevent DDoS attacks.

### Configuration

Rate limits are configured via environment variables:

```bash
# Global/Aggregate rate limiting (protects total system capacity)
RATE_LIMIT_GLOBAL_RATE=500         # total req/s across ALL IPs
RATE_LIMIT_GLOBAL_BURST=1000        # total burst capacity across ALL IPs

# Per-IP rate limiting (protects against individual abusers)
RATE_LIMIT_DEFAULT_RATE=5          # req/s per IP
RATE_LIMIT_DEFAULT_BURST=10         # burst per IP
RATE_LIMIT_EXPIRES_IN=180           # seconds (3 minutes)

# Per-endpoint overrides (JSON format)
RATE_LIMIT_ENDPOINT_OVERRIDES={}
```

### Configuring Endpoint-Specific Overrides

Endpoints can have custom rate limits based on their resource requirements. Configure using the `RATE_LIMIT_ENDPOINT_OVERRIDES` environment variable.

#### Endpoint Route Format

Routes use the format: `{HTTP_METHOD} {PATH}`

**Examples:**
- `GET /health`
- `POST /documents`
- `GET /documents/{id}` (use route template, not actual IDs)
- `POST /client/{id}/document/batch`

#### Override Configuration Structure

Each endpoint requires 4 values:

```json
{
  "ENDPOINT_ROUTE": {
    "global_rate": 100,    // Total req/s across ALL IPs
    "global_burst": 200,   // Total burst across ALL IPs
    "rate": 5,             // Req/s per individual IP
    "burst": 10            // Burst per individual IP
  }
}
```

#### Setting Environment Variable

**Single endpoint:**
```bash
export RATE_LIMIT_ENDPOINT_OVERRIDES='{"POST /documents": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10}}'
```

**Multiple endpoints:**
```bash
  export RATE_LIMIT_ENDPOINT_OVERRIDES='{
    "GET /health": {"global_rate": 500, "global_burst": 1000, "rate": 100, "burst": 200},
    "GET /metrics": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
    "GET /swagger/*": {"global_rate": 200, "global_burst": 400, "rate": 50, "burst": 100},
    "GET /auth/login": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
    "GET /auth/callback": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
    "GET /auth/logout": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
    "GET /auth/home": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
    "POST /documents": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
    "POST /documents/batch": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2},
    "GET /documents/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
    "GET /client/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
    "POST /query/test": {"global_rate": 20, "global_burst": 40, "rate": 2, "burst": 5},
    "POST /export": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2}
  }'
```

**In .env file (single line, valid JSON):**
```bash
RATE_LIMIT_ENDPOINT_OVERRIDES={"GET /health":{"global_rate":500,"global_burst":1000,"rate":100,"burst":200},"POST /documents":{"global_rate":50,"global_burst":100,"rate":5,"burst":10}}
```

#### Recommended Endpoint Tiers

**Tier 1: Monitoring/Infrastructure (High Limits)**
```json
{
  "GET /health": {"global_rate": 500, "global_burst": 1000, "rate": 100, "burst": 200},
  "GET /metrics": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
  "GET /swagger/*": {"global_rate": 200, "global_burst": 400, "rate": 50, "burst": 100}
}
```

**Tier 2: Authentication (Moderate - Prevent Brute Force)**
```json
{
  "GET /auth/login": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
  "GET /auth/callback": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
  "GET /auth/logout": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20}
}
```

**Tier 3: Read Operations (Moderate)**
```json
{
  "GET /client/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
  "GET /documents/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
  "GET /collector/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40}
}
```

**Tier 4: Write/Processing (Low - Resource Intensive)**
```json
{
  "POST /documents": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
  "POST /documents/batch": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2},
  "POST /query/test": {"global_rate": 20, "global_burst": 40, "rate": 2, "burst": 5},
  "POST /export": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2}
}
```

#### Verification

After restarting the service, check logs for confirmation:
```
level=INFO msg="Rate limiting enabled" global_rate=1000 global_burst=2000
  default_rate=10 default_burst=20 overrides_count=12
```

The `overrides_count` should match your configured endpoints.

#### Troubleshooting

**Invalid JSON:** Validate with `echo $RATE_LIMIT_ENDPOINT_OVERRIDES | jq .`

**Override not applied:**
- Verify HTTP method matches exactly (e.g., `POST` not `post`)
- Use route template for paths with parameters (`/documents/{id}` not `/documents/123`)
- Check for extra/missing slashes in path
- Enable debug logging: `DEBUG=true` to see route matching

### Rate Limit Headers

All API responses include a `RateLimit` header indicating the current limit:

```
RateLimit: 20;window=2
```

**Format:** `{burst};window={seconds}`
- `burst`: Maximum requests allowed in a burst
- `window`: Time window in seconds (calculated as burst/rate)

### Rate Limit Exceeded Response

When the rate limit is exceeded, the API returns:

**HTTP Status:** `429 Too Many Requests`

**Headers:**
- `Retry-After: {seconds}` - Number of seconds before retrying
- `RateLimit: {burst};window={seconds}` - Current rate limit

**Response Body:**
```json
{
  "message": "rate limit exceeded"
}
```

### Testing Rate Limits

Test rate limiting behavior with curl:

```bash
# Exceed rate limit by making rapid requests
for i in {1..25}; do
  curl -i http://localhost:8080/client/AAA \
    -H "Authorization: Bearer $TOKEN"
done

# Observe 429 responses after burst is exhausted
```

### Technical Implementation

- **Algorithm**: Dual-layer token bucket (golang.org/x/time/rate)
- **Layer 1**: Global rate limiter (shared across all IPs)
- **Layer 2**: Per-IP rate limiter (separate bucket per IP via c.RealIP())
- **Storage**: In-memory with automatic cleanup
- **Middleware Position**: Applied BEFORE authentication to protect auth endpoints
- **Protection**: Defends against both distributed DDoS and single-source attacks

**Security**: Requires authentication
**Response**: HTML dashboard page

### Authorization Model

#### Permission Groups
- **exporters**: Can access export operations
- **uploaders**: Can upload documents and manage clients  
- **querybuilders**: Can create and modify queries

#### Route Permissions
```
/client/*     → exporters, uploaders, querybuilders
/document/*   → exporters, querybuilders  
/query/*      → exporters, uploaders, querybuilders
/export/*     → exporters only
/health       → public (no authentication required)
```

## Core API Endpoints

### Client Management

#### `POST /client`
Creates a new client in the system.

**Security**: Requires `uploaders` or `querybuilders` permission

**Request Body**:
```json
{
  "clientId": "string (max 256 chars)",
  "name": "string (max 256 chars)"
}
```

**Responses**:
- `201`: Client created successfully
- `400`: Invalid request data
- `401`: Authentication required
- `403`: Insufficient permissions

#### `GET /client/{id}`
Retrieves client details by ID.

**Path Parameters**:
- `id`: Client identifier (string, max 256 chars)

**Response Example**:
```json
{
  "clientId": "acme-corp",
  "name": "ACME Corporation"
}
```

#### `PATCH /client/{id}`
Updates client information.

**Request Body**:
```json
{
  "name": "string (max 256 chars)"
}
```

#### `GET /client/{id}/status`
Returns client synchronization status.

**Response Example**:
```json
{
  "status": "IN_SYNC" | "NOT_SYNCED" | "NOT_SYNCING"
}
```

### Document Operations

#### `POST /client/{id}/document`
Uploads a document for a specific client.

**Content Type**: `multipart/form-data`
**Max File Size**: 100GB
**Supported Types**: PDF (extensible)

**Form Parameters**:
- `file`: Document file (required)

**Response**:
```json
{
  "documentId": "uuid",
  "clientId": "string",
  "filename": "string",
  "uploadedAt": "2024-01-01T00:00:00Z"
}
```

#### `GET /client/{id}/document`
Lists documents for a client with optional filtering.

**Query Parameters**:
- `limit`: Maximum results (integer)
- `offset`: Pagination offset (integer)

**Response**:
```json
{
  "documents": [
    {
      "documentId": "uuid",
      "clientId": "string",
      "filename": "string",
      "status": "string",
      "uploadedAt": "2024-01-01T00:00:00Z"
    }
  ],
  "total": "integer"
}
```

#### `GET /document/{id}`
Retrieves document metadata by ID.

**Path Parameters**:
- `id`: Document UUID

### Batch Document Operations

#### `POST /client/{id}/document/batch`
Uploads a ZIP archive containing multiple PDF documents for asynchronous batch processing.

**Security**: Requires `uploaders` or `querybuilders` permission
**Content Type**: `multipart/form-data`
**Max Archive Size**: 1GB
**Supported Archive Type**: ZIP only

**Form Parameters**:
- `archive`: ZIP file containing PDF documents (required)

**Response** (202 Accepted):
```json
{
  "batch_id": "uuid",
  "status": "processing",
  "status_url": "/client/AAA/document/batch/019580df-ef65-7676-8de9-94435a93337a"
}
```

#### `GET /client/{id}/document/batch`
Lists batch uploads for a client with pagination support.

**Query Parameters**:
- `limit`: Maximum results (integer, default: 20)
- `offset`: Pagination offset (integer, default: 0)

**Response**:
```json
{
  "batches": [
    {
      "batch_id": "uuid",
      "status": "completed" | "processing" | "failed" | "cancelled",
      "total_documents": 15,
      "processed_documents": 15,
      "failed_documents": 0,
      "created_at": "2024-01-01T00:00:00Z",
      "completed_at": "2024-01-01T00:05:00Z"
    }
  ],
  "total": 42
}
```

#### `GET /client/{id}/document/batch/{batch_id}`
Retrieves detailed status information for a specific batch upload.

**Path Parameters**:
- `batch_id`: Batch upload identifier (UUID)

**Response**:
```json
{
  "batch_id": "uuid",
  "status": "completed",
  "total_documents": 15,
  "processed_documents": 15,
  "failed_documents": 0,
  "failed_filenames": [],
  "created_at": "2024-01-01T00:00:00Z",
  "completed_at": "2024-01-01T00:05:00Z"
}
```

#### `DELETE /client/{id}/document/batch/{batch_id}`
Cancels a batch upload that is currently processing.

**Path Parameters**:
- `batch_id`: Batch upload identifier (UUID)

**Response**:
- `204`: Batch upload cancelled successfully
- `409`: Cannot cancel completed or already cancelled batch

### Query Management

#### `GET /query`
Lists all available queries in the system.

**Response**:
```json
{
  "queries": [
    {
      "queryId": "uuid",
      "queryType": "JSON_EXTRACTOR" | "CONTEXT_FULL",
      "name": "string",
      "description": "string",
      "version": "integer"
    }
  ]
}
```

#### `POST /query`
Creates a new query definition.

**Request Body**:
```json
{
  "queryType": "JSON_EXTRACTOR" | "CONTEXT_FULL",
  "name": "string (max 256 chars)",
  "description": "string (max 512 chars)",
  "config": "object (JSON configuration)"
}
```

#### `GET /query/{id}`
Retrieves query details including configuration.

#### `PATCH /query/{id}`
Updates query definition and increments version.

#### `POST /query/{id}/test`
Tests query execution against sample data.

**Request Body**:
```json
{
  "testData": "string",
  "documentId": "uuid (optional)"
}
```

### Collector Configuration

#### `GET /client/{id}/collector`
Retrieves collector configuration for a client.

**Response**:
```json
{
  "clientId": "string",
  "queries": [
    {
      "name": "string",
      "queryId": "uuid"
    }
  ],
  "minCleanVersion": "integer",
  "minTextVersion": "integer"
}
```

#### `PATCH /client/{id}/collector`
Updates collector configuration.

### Export Operations

#### `POST /client/{id}/export`
Triggers data export for a client.

**Request Body**:
```json
{
  "format": "CSV" | "JSON" | "EXCEL",
  "filters": {
    "queryIds": ["uuid"],
    "dateRange": {
      "start": "2024-01-01T00:00:00Z",
      "end": "2024-12-31T23:59:59Z"
    }
  }
}
```

#### `GET /export/{id}`
Checks export status and retrieves download URL.

**Response**:
```json
{
  "exportId": "uuid",
  "status": "PENDING" | "PROCESSING" | "COMPLETED" | "FAILED",
  "downloadUrl": "string (when completed)",
  "createdAt": "2024-01-01T00:00:00Z",
  "completedAt": "2024-01-01T00:05:00Z"
}
```

## Request/Response Schemas

### Standard Response Headers
All API responses include:
- `RateLimit`: Current rate limit status (format: `100;window=60`)
- `Content-Type`: `application/json` (except file downloads)

### Error Response Format
```json
{
  "message": "Error description (max 256 chars)"
}
```

### Common HTTP Status Codes
- `200`: Success
- `201`: Created
- `400`: Bad Request (validation errors)
- `401`: Unauthorized (authentication required)
- `403`: Forbidden (insufficient permissions)
- `404`: Not Found
- `429`: Too Many Requests (rate limited)
- `500`: Internal Server Error

## Rate Limiting

### Current Limits
- **Rate**: 100 requests per 60-second window
- **Scope**: Per authenticated user
- **Headers**: `RateLimit` header shows current status
- **Exceeded**: Returns `429` with `Retry-After` header

## API Versioning

### Current Strategy
- **URL-based versioning**: `/v1` prefix
- **Backward Compatibility**: Maintained within major versions
- **OpenAPI Version**: 0.0.1 (development phase)

### Future Considerations
- Semantic versioning for production releases
- Deprecation notices for breaking changes
- Version sunset policies

## Code Generation

### OpenAPI Integration
The API is generated from the OpenAPI specification using:
```bash
oapi-codegen -generate "types,server" -o api.gen.go serviceAPIs/queryAPI.yaml
```

### Generated Artifacts
- Type-safe Go structs for requests/responses
- Echo framework route handlers
- OpenAPI validation middleware
- Swagger documentation

### Validation
- Automatic request validation against OpenAPI schema
- Response validation in development mode
- Parameter binding with type safety