query-orchestration/plans/mutable.metadata.plan.claude.md

# Mutable Metadata & Per-Client Custom Schemas - Design Plan

## Status: DRAFT - Updated with Q's design decisions (2026-03-23)

---

## 1. Problem Statement

Today, all field extraction data is stored in a rigid 3-table schema with ~130 statically-defined columns (19 single-value fields + 112 array fields). Every client shares the same field definitions. This means:

- Clients cannot define domain-specific fields (e.g., "aircraft engineering" vs. "auto engineering" document classes)
- Adding new fields requires database migrations and code changes across the full stack
- The system cannot accommodate varying data shapes per document class within a single client

**Goal**: Allow each client to curate their own collections of document schemas, where each schema defines the custom fields (with types and constraints) that a document's metadata must conform to.

---

## 2. Current Architecture Summary

### 2.1 Existing Tables

```
documentFieldExtractions          -- 19 single-value fields (1:1 per document)
documentFieldExtractionVersions   -- Version tracking with row-level locking
documentFieldExtractionArrayFields -- 112 array fields (1:N per extraction)
```

### 2.2 Existing API Endpoints

| Method | Path | Description |
|--------|------|-------------|
| POST | `/field-extractions` | Create versioned extraction (singleFields + arrayFields) |
| GET | `/field-extractions` | Get current (latest version) extraction for a document |
| GET | `/field-extractions/version` | Get extraction by specific version |
| GET | `/field-extractions/history` | List all versions for a document |

### 2.3 Key Relationships

- `documents.id` -> `documentFieldExtractions.documentId` (1:N via versions)
- `documentFieldExtractions.id` -> `documentFieldExtractionArrayFields.fieldExtractionId` (1:N)
- `documentFieldExtractions.id` -> `documentFieldExtractionVersions.fieldExtractionId` (1:N)
- Documents belong to clients via `documents.clientId`
- No per-client customization of field definitions exists today

---

## 3. Design Principles

1. **Additive, not destructive** - The existing static field extraction system continues to work unchanged. Custom schemas are an orthogonal capability.
2. **Schema-as-data** - Schemas are stored as JSON Schema documents in the database, not as DDL. No migrations needed when clients add/modify schemas.
3. **Immutable schema versions** - When a schema is updated, a new version (with new ID) is created. Old versions remain for historical reference. No backward compatibility checking is required between versions since each version gets a distinct ID.
4. **Schema binding is permanent** - Once custom metadata is written to a document with a schema ID, that schema ID cannot be changed on the document. This guarantees data integrity.
5. **JSON blob storage** - Custom metadata is stored as JSONB in PostgreSQL, validated against the schema at write time. This avoids dynamic column creation.
6. **Client isolation** - Schemas are scoped to clients. A client cannot reference another client's schema.
7. **Schema size limit** - Schema definitions are capped at 64KB (`MaxSchemaDefinitionBytes = 65536`). This is a tunable constant that can be increased later if needed.
8. **No JSONB search/filter** - The system does NOT support querying or filtering documents by custom metadata values. Custom metadata is opaque storage validated on write. If search is needed in the future, it would require GIN indexes on the JSONB columns and a dedicated query API (separate phase).
9. **Admin-scoped schema management** - Schema CRUD operations live under `/admin` and require `client_admin` role. Custom metadata read/write on documents is available to regular users.
10. **No migration from static fields** - There is no migration path from the existing static field extraction data into the custom schema system. They are independent systems. Out of scope.

---

## 4. Database Schema Design

### 4.1 New Table: `client_metadata_schemas`

Stores client-defined JSON Schema documents. Each row is an immutable version of a schema definition.

```sql
CREATE TABLE client_metadata_schemas (
    id              uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    client_id       varchar(255) NOT NULL REFERENCES clients(clientId),
    name            varchar(255) NOT NULL,
    description     text,
    schema_def      jsonb NOT NULL,          -- The JSON Schema document (max 64KB enforced at app layer)
    version         int NOT NULL DEFAULT 1,   -- Auto-incremented per (client_id, name)
    is_active       boolean NOT NULL DEFAULT true,
    created_at      timestamptz NOT NULL DEFAULT NOW(),
    created_by      varchar(255) NOT NULL,

    CONSTRAINT uq_client_schema_name_version UNIQUE (client_id, name, version),
    CONSTRAINT chk_schema_def_size CHECK (pg_column_size(schema_def) <= 65536)
);

CREATE INDEX idx_cms_client_id ON client_metadata_schemas(client_id);
CREATE INDEX idx_cms_client_name ON client_metadata_schemas(client_id, name);
CREATE INDEX idx_cms_active ON client_metadata_schemas(client_id, is_active);
```

> **Note**: The 64KB limit is enforced at both the application layer (Go constant `MaxSchemaDefinitionBytes = 65536`) and the database layer (CHECK constraint). The app-layer check provides a better error message; the DB constraint is a safety net.

**Design decisions:**

- **`name` + `version`** scoped to client: Allows human-friendly naming ("aircraft-engineering-v3") with automatic versioning. When a schema is "updated," a new row is inserted with `version = max(version) + 1` for that `(client_id, name)`.
- **`schema_def` is JSONB**: Stores the full JSON Schema document. PostgreSQL can index into it if needed later.
- **`is_active`**: Soft-delete flag. When a schema is "deleted," it is marked inactive. Schemas referenced by existing documents cannot be hard-deleted.
- **`id` is the schema version ID**: This is what documents reference. Each update creates a new `id`. The `(client_id, name, version)` tuple provides the human-readable lineage.

### 4.2 New Table: `document_custom_metadata`

Stores the actual custom metadata for documents, validated against the referenced schema.

```sql
CREATE TABLE document_custom_metadata (
    id              uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    document_id     uuid NOT NULL REFERENCES documents(id),
    schema_id       uuid NOT NULL REFERENCES client_metadata_schemas(id),
    metadata        jsonb NOT NULL,           -- The actual custom field values
    version         int NOT NULL DEFAULT 1,   -- Versioned like field extractions
    created_at      timestamptz NOT NULL DEFAULT NOW(),
    created_by      varchar(255) NOT NULL,

    CONSTRAINT uq_doc_custom_metadata_version UNIQUE (document_id, version)
);

CREATE INDEX idx_dcm_document_id ON document_custom_metadata(document_id);
CREATE INDEX idx_dcm_schema_id ON document_custom_metadata(schema_id);
```

**Design decisions:**

- **Versioned**: Like field extractions, custom metadata supports version history. Each write creates a new version.
- **`schema_id` is immutable per document**: Enforced at the application layer (see Rule 4.3). All versions of custom metadata for a document must reference the same schema ID.
- **`metadata` is JSONB**: Validated against `client_metadata_schemas.schema_def` at write time. Stored as a single blob for simplicity and query flexibility.

### 4.3 Alteration: `documents` Table

Add an optional schema binding column:

```sql
ALTER TABLE documents
    ADD COLUMN custom_schema_id uuid NULL REFERENCES client_metadata_schemas(id);

CREATE INDEX idx_documents_custom_schema_id ON documents(custom_schema_id);
```

**Behavior:**

- `NULL` = document uses only the legacy static field extraction system (default, backward-compatible)
- Once set AND custom metadata has been written, this field becomes immutable (enforced in application code)
- Can be set before metadata is written (pre-binding) and changed freely until first metadata write
- Must reference a schema owned by the same client as the document (cross-client reference check)

### 4.4 New View: `current_document_custom_metadata`

```sql
CREATE VIEW current_document_custom_metadata AS
SELECT DISTINCT ON (document_id)
    id,
    document_id,
    schema_id,
    metadata,
    version,
    created_at,
    created_by
FROM document_custom_metadata
ORDER BY document_id, version DESC;
```

### 4.5 Entity Relationship Diagram

```
clients
  |
  |-- 1:N --> client_metadata_schemas (per-client schema definitions)
  |              |
  |              |-- referenced by --> documents.custom_schema_id
  |              |-- referenced by --> document_custom_metadata.schema_id
  |
  |-- 1:N --> documents
                 |
                 |-- 1:N --> document_custom_metadata (versioned JSONB blobs)
                 |-- 1:N --> documentFieldExtractions (existing static fields)
```

---

## 5. API Design

### 5.1 Schema Management Endpoints (Admin)

All under the existing `AdminService` tag. These endpoints require the `client_admin` role (see Section 5.5 for auth details). Endpoints are prefixed with `/admin/custom-schemas`.

#### `POST /admin/custom-schemas`

Create a new schema for a client.

**Request:**
```json
{
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "Custom fields for aircraft engineering documents",
    "schema": {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties": {
            "aircraft_model": {
                "type": "string",
                "maxLength": 100
            },
            "certification_date": {
                "type": "string",
                "format": "date"
            },
            "max_altitude_ft": {
                "type": "number",
                "minimum": 0,
                "maximum": 100000
            },
            "engine_count": {
                "type": "integer",
                "minimum": 1,
                "maximum": 12
            }
        },
        "required": ["aircraft_model"],
        "additionalProperties": false
    },
    "createdBy": "admin@acme.com"
}
```

**Response (201):**
```json
{
    "id": "019577a3-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "Custom fields for aircraft engineering documents",
    "schema": { ... },
    "version": 1,
    "isActive": true,
    "createdAt": "2026-03-23T12:00:00Z",
    "createdBy": "admin@acme.com"
}
```

**Validation:**
- The `schema` field MUST be a valid JSON Schema document (validated server-side using a JSON Schema meta-validator)
- The `schema` field must be <= 64KB in size (returns 400 with message referencing the limit)
- `name` must be unique within the client (for active schemas at the same version)
- `additionalProperties: false` is recommended but not required

#### `PUT /admin/custom-schemas/{schemaId}`

Update an existing schema. Creates a new version with a new ID. The old version remains intact. No backward compatibility check is performed -- the new version is treated as an independent schema that happens to share a name lineage.

**Request:**
```json
{
    "description": "Updated aircraft engineering fields (added wingspan)",
    "schema": {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties": {
            "aircraft_model": { "type": "string", "maxLength": 100 },
            "certification_date": { "type": "string", "format": "date" },
            "max_altitude_ft": { "type": "number", "minimum": 0, "maximum": 100000 },
            "engine_count": { "type": "integer", "minimum": 1, "maximum": 12 },
            "wingspan_meters": { "type": "number", "minimum": 0 }
        },
        "required": ["aircraft_model"],
        "additionalProperties": false
    },
    "createdBy": "admin@acme.com"
}
```

**Response (201):**
```json
{
    "id": "019577b4-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "version": 2,
    "previousVersionId": "019577a3-...",
    ...
}
```

**Behavior:**
- The `schemaId` in the path identifies the schema to "update" (really: create a new version of)
- The new version inherits the `name` from the previous version
- The old version remains active and referenced by existing documents
- Returns the newly created schema version

#### `GET /admin/custom-schemas`

List all schemas for a client.

**Query Parameters:**
- `clientId` (required): The client whose schemas to list
- `name` (optional): Filter by schema name
- `activeOnly` (optional, default: true): Only return active schemas
- `includeAllVersions` (optional, default: false): Return all versions or just latest per name

**Response (200):**
```json
{
    "schemas": [
        {
            "id": "019577b4-...",
            "clientId": "acme-corp",
            "name": "aircraft-engineering",
            "description": "...",
            "version": 2,
            "isActive": true,
            "documentCount": 45,
            "createdAt": "...",
            "createdBy": "..."
        },
        {
            "id": "019577c1-...",
            "clientId": "acme-corp",
            "name": "auto-engineering",
            "description": "...",
            "version": 1,
            "isActive": true,
            "documentCount": 120,
            "createdAt": "...",
            "createdBy": "..."
        }
    ]
}
```

**Note:** `documentCount` shows how many documents reference each schema version. This is computed via a subquery or join.

#### `GET /admin/custom-schemas/{schemaId}`

Get a specific schema by ID (returns full schema definition).

**Response (200):**
```json
{
    "id": "019577a3-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "...",
    "schema": { ... },
    "version": 1,
    "isActive": true,
    "documentCount": 45,
    "createdAt": "...",
    "createdBy": "..."
}
```

#### `DELETE /admin/custom-schemas/{schemaId}`

Soft-delete a schema. Only succeeds if no documents reference it.

**Response (204):** No content on success.

**Response (409 Conflict):**
```json
{
    "error": "Schema is in use by 45 documents and cannot be deleted"
}
```

**Behavior:**
- Sets `is_active = false`
- Fails with 409 if any documents reference this schema ID
- Inactive schemas still serve validation for existing documents but cannot be assigned to new documents

---

### 5.2 Document Schema Assignment Endpoints

#### `PATCH /document/{id}/schema`

Assign or change a custom schema on a single document.

**Request:**
```json
{
    "customSchemaId": "019577a3-..."
}
```

**Response (200):**
```json
{
    "documentId": "...",
    "customSchemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "schemaVersion": 1
}
```

**Validation rules:**
- Schema must belong to the same client as the document
- Schema must be active
- If the document already has custom metadata written, the schema cannot be changed (returns 409)
- Setting `customSchemaId` to `null` removes the binding (only if no metadata exists)

#### `POST /admin/folders/{folderId}/assign-schema` (Bulk Folder Assignment)

Assign a custom schema to ALL documents in a folder and all its subfolders (recursive).

**Request:**
```json
{
    "customSchemaId": "019577a3-...",
    "createdBy": "admin@acme.com"
}
```

**Response (200):**
```json
{
    "folderId": "...",
    "customSchemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "documentsUpdated": 87,
    "documentsSkipped": 3,
    "skippedReasons": [
        {"documentId": "...", "reason": "Document already has metadata with a different schema"},
        {"documentId": "...", "reason": "Document already has metadata with a different schema"},
        {"documentId": "...", "reason": "Document already assigned to same schema (no-op)"}
    ]
}
```

**Behavior:**
- Uses `WITH RECURSIVE` CTE to walk the entire folder subtree (same pattern as folder deletion)
- For each document in the tree:
  - If the document has no schema and no custom metadata -> assign the schema
  - If the document already has the same schema -> skip (no-op, counted in `documentsSkipped`)
  - If the document has a different schema but no metadata yet -> reassign to new schema
  - If the document has a different schema AND has metadata -> skip with reason (cannot change)
- Runs in a single transaction
- Returns summary of what was updated and what was skipped (with reasons)
- Requires `client_admin` role (admin endpoint)

#### `GET /document/{id}` (Modified)

Add `customSchemaId` and `customMetadata` fields to `DocumentEnriched`:

```json
{
    "id": "...",
    "clientId": "acme-corp",
    "hash": "...",
    "customSchemaId": "019577a3-...",
    "customSchemaName": "aircraft-engineering",
    "hasCustomMetadata": true,
    "customMetadata": { ... },
    ...existing fields...
}
```

**Query parameter:** `customMetadata` (boolean, default: false) - when true, includes the full custom metadata in the response (like the existing `textRecord` parameter).

---

### 5.5 Authorization & Permit.io Changes

#### New Role: `client_admin`

A new Permit.io role needs to be created between `user_admin` and `client_user`:

| Role | Schema CRUD | Schema Assignment | Metadata Read | Metadata Write | Existing Admin |
|------|------------|-------------------|---------------|----------------|----------------|
| `super_admin` | Yes | Yes | Yes | Yes | Yes (full) |
| `client_admin` | Yes | Yes | Yes | Yes | No |
| `user_admin` | No | No | Yes | Yes | Yes (full) |
| `auditor` | Read-only | No | Read-only | No | Read-only |
| `client_user` | No | No | Yes | Yes | No |

**Permit.io policy additions** (in `permit_policies.yaml`):

```yaml
client_admin:
  admin:
    - get
    - post
    - patch
    - delete
    - head
  # Plus all permissions that client_user has:
  client:
    - get
    - post
    - patch
  documents:
    - get
    - post
  document:
    - get
    - post
    - patch
    - delete
  folders:
    - get
    - post
    - patch
  field-extractions:
    - get
    - post
  custom-metadata:
    - get
    - post
  export:
    - get
    - post
```

**Key distinction**: `client_admin` gets `admin:*` permissions (for schema management under `/admin/custom-schemas`) plus all document/metadata operations. Unlike `super_admin`, it does NOT get system-wide admin capabilities (user management, EULA management, etc.).

> **Open question**: The current Permit.io setup maps `/admin/*` to the `admin` resource uniformly. If `client_admin` should access `/admin/custom-schemas` but NOT `/admin/users`, the resource mapping may need to become more granular (e.g., separate `admin-schemas` resource). Alternative: use a single `admin` resource and control via tenant-scoped checks. This is an implementation detail to resolve during Phase 6.

---

### 5.3 Custom Metadata CRUD Endpoints (New)

All under a new `CustomMetadataService` tag.

#### `POST /custom-metadata`

Create or update custom metadata for a document. Creates a new version.

**Request:**
```json
{
    "documentId": "...",
    "metadata": {
        "aircraft_model": "Boeing 737-800",
        "certification_date": "2024-06-15",
        "max_altitude_ft": 41000,
        "engine_count": 2
    },
    "createdBy": "analyst@acme.com"
}
```

**Response (201):**
```json
{
    "id": "...",
    "documentId": "...",
    "schemaId": "019577a3-...",
    "metadata": { ... },
    "version": 1,
    "createdAt": "...",
    "createdBy": "analyst@acme.com"
}
```

**Validation:**
- Document must have a `custom_schema_id` assigned (400 if not)
- The `metadata` payload is validated against the JSON Schema stored in `client_metadata_schemas.schema_def`
- If validation fails, returns 400 with detailed validation errors:
  ```json
  {
      "error": "Metadata does not conform to schema",
      "validationErrors": [
          {"field": "engine_count", "message": "must be >= 1, got 0"},
          {"field": "aircraft_model", "message": "required field missing"}
      ]
  }
  ```
- On first write, locks the document's `custom_schema_id` (makes it immutable)
- Subsequent writes create new versions but must use the same schema

#### `GET /custom-metadata`

Get current (latest version) custom metadata for a document.

**Query Parameters:**
- `documentId` (required)

**Response (200):**
```json
{
    "id": "...",
    "documentId": "...",
    "schemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "metadata": { ... },
    "version": 3,
    "createdAt": "...",
    "createdBy": "..."
}
```

#### `GET /custom-metadata/version`

Get a specific version of custom metadata.

**Query Parameters:**
- `documentId` (required)
- `version` (required, >= 1)

#### `GET /custom-metadata/history`

Get version history for a document's custom metadata.

**Query Parameters:**
- `documentId` (required)

**Response (200):**
```json
{
    "versions": [
        {"id": "...", "documentId": "...", "version": 3, "createdAt": "...", "createdBy": "..."},
        {"id": "...", "documentId": "...", "version": 2, "createdAt": "...", "createdBy": "..."},
        {"id": "...", "documentId": "...", "version": 1, "createdAt": "...", "createdBy": "..."}
    ]
}
```

---

## 6. Validation Strategy

### 6.1 JSON Schema Validation

Use a Go JSON Schema validation library for both:

1. **Schema validation** (meta-validation): When a client submits a new schema, validate it against the JSON Schema meta-schema to ensure it is a well-formed JSON Schema document.
2. **Metadata validation**: When custom metadata is submitted, validate the payload against the document's assigned schema.

**Recommended library**: `github.com/santhosh-tekuri/jsonschema/v6`

Reasons:
- Supports JSON Schema draft 2020-12
- Active maintenance
- Good error reporting
- Supports custom formats

### 6.2 Supported JSON Schema Features

The system will support the following JSON Schema keywords for custom field definitions:

| Feature | JSON Schema Keyword | Example |
|---------|-------------------|---------|
| Data type | `type` | `"string"`, `"number"`, `"integer"`, `"boolean"`, `"array"`, `"object"` |
| Required fields | `required` | `["field1", "field2"]` |
| String length | `minLength`, `maxLength` | `"maxLength": 100` |
| Numeric range | `minimum`, `maximum` | `"minimum": 0` |
| Pattern matching | `pattern` | `"pattern": "^[A-Z]{2}$"` |
| Enum values | `enum` | `"enum": ["active", "inactive"]` |
| Date/time format | `format` | `"format": "date"` |
| Array items | `items`, `minItems`, `maxItems` | Nested array support |
| Nested objects | `properties` | Hierarchical field groups |
| No extra fields | `additionalProperties` | `false` to enforce strict schemas |

### 6.3 Validation Error Reporting

Validation errors will be returned as structured JSON with:
- The JSON pointer to the failing field (`/aircraft_model`)
- The constraint that was violated (`maxLength`)
- A human-readable message

---

## 7. Service Layer Design

### 7.1 New Service: `internal/customschema/`

```
internal/customschema/
    service.go       -- CustomSchemaService (CRUD + validation)
    models.go        -- Input/output types
    validator.go     -- JSON Schema meta-validation + payload validation
    constants.go     -- MaxSchemaDefinitionBytes and other limits
    service_test.go  -- Integration tests
```

**Constants (`constants.go`):**

```go
const (
    // MaxSchemaDefinitionBytes is the maximum size of a JSON Schema definition.
    // This limit can be increased if clients need larger schemas.
    MaxSchemaDefinitionBytes = 65536 // 64KB
)
```

**Key methods:**

```go
type Service struct {
    cfg       serviceconfig.ConfigProvider
    validator *SchemaValidator
}

// Schema management (admin operations)
func (s *Service) CreateSchema(ctx, input CreateSchemaInput) (*Schema, error)
func (s *Service) UpdateSchema(ctx, schemaID uuid.UUID, input UpdateSchemaInput) (*Schema, error)
func (s *Service) GetSchema(ctx, schemaID uuid.UUID) (*Schema, error)
func (s *Service) ListSchemas(ctx, clientID string, filters ListFilters) ([]SchemaSummary, error)
func (s *Service) DeleteSchema(ctx, schemaID uuid.UUID) error

// Metadata operations (regular user operations)
func (s *Service) SetDocumentMetadata(ctx, input SetMetadataInput) (*CustomMetadata, error)
func (s *Service) GetCurrentMetadata(ctx, documentID uuid.UUID) (*CustomMetadata, error)
func (s *Service) GetMetadataByVersion(ctx, documentID uuid.UUID, version int) (*CustomMetadata, error)
func (s *Service) GetMetadataHistory(ctx, documentID uuid.UUID) ([]MetadataVersion, error)

// Document schema binding
func (s *Service) AssignSchema(ctx, documentID uuid.UUID, schemaID uuid.UUID) error
func (s *Service) AssignSchemaToFolder(ctx, folderID uuid.UUID, schemaID uuid.UUID, createdBy string) (*BulkAssignResult, error)
```

**BulkAssignResult:**

```go
type BulkAssignResult struct {
    FolderID         uuid.UUID
    SchemaID         uuid.UUID
    DocumentsUpdated int
    DocumentsSkipped int
    SkippedReasons   []SkippedDocument
}

type SkippedDocument struct {
    DocumentID uuid.UUID
    Reason     string
}
```

### 7.2 Validator Component

```go
type SchemaValidator struct{}

// ValidateSchemaDefinition checks that a schema_def is valid JSON Schema
func (v *SchemaValidator) ValidateSchemaDefinition(schemaDef json.RawMessage) error

// ValidateMetadata checks that metadata conforms to a compiled schema
func (v *SchemaValidator) ValidateMetadata(schemaDef json.RawMessage, metadata json.RawMessage) ([]ValidationError, error)
```

### 7.3 API Controller: `api/queryAPI/customschemas.go`

Follows the same pattern as `fieldextractions.go`:
- Parse and validate request
- Call service layer
- Convert between API and service types
- Return appropriate HTTP status codes

---

## 8. Migration Plan

### 8.1 Migration Files

Three new migrations:

**Migration 120: Create `client_metadata_schemas` table**
```
00000000000120_create_client_metadata_schemas.up.sql
00000000000120_create_client_metadata_schemas.down.sql
```

**Migration 121: Create `document_custom_metadata` table + view**
```
00000000000121_create_document_custom_metadata.up.sql
00000000000121_create_document_custom_metadata.down.sql
```

**Migration 122: Add `custom_schema_id` to `documents` table**
```
00000000000122_add_custom_schema_id_to_documents.up.sql
00000000000122_add_custom_schema_id_to_documents.down.sql
```

### 8.2 SQLC Queries

New query file: `internal/database/queries/customschemas.sql`

**Schema operations:**
- `CreateClientMetadataSchema` - Insert new schema
- `GetClientMetadataSchema` - Get by ID
- `ListClientMetadataSchemas` - List by client with filters
- `GetLatestSchemaByName` - Get latest version of a named schema
- `GetSchemaDocumentCount` - Count documents using a schema
- `DeactivateClientMetadataSchema` - Soft-delete
- `GetMaxSchemaVersion` - For auto-incrementing version

**Metadata operations:**
- `CreateDocumentCustomMetadata` - Insert new metadata version
- `GetCurrentDocumentCustomMetadata` - Latest version (via view)
- `GetDocumentCustomMetadataByVersion` - Specific version
- `GetDocumentCustomMetadataHistory` - All versions
- `LockDocumentCustomMetadataForVersion` - Row-level locking

**Document modifications:**
- `SetDocumentCustomSchemaId` - Update documents.custom_schema_id
- `GetDocumentCustomSchemaId` - Read documents.custom_schema_id
- `BulkSetDocumentCustomSchemaIdInFolderTree` - Set schema for all docs in folder subtree (WITH RECURSIVE)
- `GetDocumentsWithMetadataInFolderTree` - Find docs that already have metadata (for skip reporting)

### 8.3 OpenAPI Spec Changes

Add to `serviceAPIs/queryAPI.yaml`:
- New admin paths: `/admin/custom-schemas`, `/admin/custom-schemas/{schemaId}`, `/admin/folders/{folderId}/assign-schema`
- New user paths: `/custom-metadata`, `/custom-metadata/version`, `/custom-metadata/history`, `/document/{id}/schema`
- New schemas: `CustomSchemaRequest`, `CustomSchemaResponse`, `CustomSchemaListResponse`, `CustomMetadataRequest`, `CustomMetadataResponse`, `CustomMetadataHistoryResponse`, `ValidationErrorResponse`, `BulkSchemaAssignRequest`, `BulkSchemaAssignResponse`
- Modified schemas: `DocumentEnriched` (add `customSchemaId`, `customSchemaName`, `hasCustomMetadata`, `customMetadata` fields)

---

## 9. Invariants & Business Rules

| # | Rule | Enforcement |
|---|------|-------------|
| 1 | Schema must be valid JSON Schema | Application: meta-validation on create/update |
| 2 | Schema `name` + `version` unique per client | Database: unique constraint |
| 3 | Updated schema gets new ID and version | Application: INSERT, not UPDATE |
| 4 | Schema deletion blocked if documents reference it | Application: count check before deactivation |
| 5 | Document's `custom_schema_id` is immutable once metadata exists | Application: check in AssignSchema and SetMetadata |
| 6 | Custom metadata must conform to assigned schema | Application: JSON Schema validation on write |
| 7 | All custom metadata versions for a document use same schema ID | Application: consistency check on write |
| 8 | Schema must belong to same client as document | Application: cross-reference check |
| 9 | Only active schemas can be assigned to new documents | Application: is_active check |
| 10 | Custom metadata is versioned (no in-place updates) | Database: version column + unique constraint |
| 11 | Schema definition must be <= 64KB | Application: size check + Database: CHECK constraint |
| 12 | Schema management requires `client_admin` role | Permit.io: role-based access on `/admin` resource |
| 13 | No query/filter on custom metadata values | By design: not implemented (see Principle 8) |

---

## 10. Relationship to Existing Field Extraction System

The existing static field extraction system (`/field-extractions`) and the new custom schema system (`/custom-schemas` + `/custom-metadata`) are **independent and complementary**:

- A document can have BOTH static field extractions AND custom metadata
- A document can have EITHER one, or neither
- They are stored in separate tables with separate versioning
- The existing 130 static fields continue to serve the current healthcare contract analysis use case
- Custom schemas serve new, client-specific use cases that don't fit the static model

**No breaking changes** to the existing API or database schema.

---

## 11. Implementation Order

| Phase | Work | Dependencies |
|-------|------|--------------|
| 1 | Database migrations (tables, view, document column) | None |
| 2 | SQLC queries + `task generate` | Phase 1 |
| 3 | Schema validator component (`internal/customschema/validator.go`) | JSON Schema library added to go.mod |
| 4 | Service layer (`internal/customschema/service.go`) incl. bulk folder assignment | Phase 2, 3 |
| 5 | OpenAPI spec updates (admin + user endpoints) | None (can parallel with 1-3) |
| 6 | Permit.io: Create `client_admin` role + policy updates | None (can parallel with 1-5) |
| 7 | API controllers + code generation (admin schema CRUD, metadata CRUD) | Phase 4, 5 |
| 8 | Document endpoint modifications (PATCH schema, GET enrichment) | Phase 4, 7 |
| 9 | Bulk folder schema assignment endpoint | Phase 4, 7 |
| 10 | Integration tests | Phase 9 |
| 11 | Documentation updates (`docs/ai.generated/`) incl. sample schemas | Phase 10 |

---

## 12. Testing Strategy

- **Schema validation tests**: Valid and invalid JSON Schema documents, edge cases (empty schema, deeply nested, self-referential, oversized >64KB)
- **Metadata validation tests**: Conforming and non-conforming payloads, all supported JSON Schema keywords
- **Service integration tests**: Full CRUD lifecycle using testcontainers (real PostgreSQL)
- **API integration tests**: HTTP-level tests for all new endpoints (admin + user)
- **Invariant tests**: Schema immutability, cross-client isolation, deletion guards, size limits
- **Bulk folder assignment tests**: Recursive assignment, skip behavior for locked docs, empty folders, deeply nested folder trees
- **Authorization tests**: `client_admin` can manage schemas, `client_user` cannot, `super_admin` can
- **Backward compatibility tests**: Existing field extraction endpoints continue to work unchanged

---

## 13. Resolved Design Decisions

| # | Question | Decision |
|---|----------|----------|
| 1 | Schema size limits | 64KB cap via `MaxSchemaDefinitionBytes` constant (tunable) |
| 2 | Schema backward compatibility | Not required. New version = new ID. Old docs keep old schema. |
| 3 | Bulk operations | Yes. `POST /admin/folders/{folderId}/assign-schema` assigns recursively through folder tree. |
| 4 | Custom metadata search/filter | **Not supported.** No GIN indexes, no query API. Metadata is opaque storage. |
| 5 | Schema templates | No system templates. Documentation will include sample schemas covering all supported types. |
| 6 | Existing field migration path | **Out of scope.** Static fields and custom schemas are independent systems. |
| 7 | Endpoint naming/placement | Schema CRUD under `/admin/custom-schemas` (requires `client_admin` role, not `super_admin`). Metadata CRUD under `/custom-metadata` (regular user access). |

---

## Appendix A: Sample JSON Schemas

These examples demonstrate all supported JSON Schema features and data types. They should be included in the API documentation.

### A.1 Comprehensive Schema (All Supported Types)

```json
{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "document_title": {
            "type": "string",
            "minLength": 1,
            "maxLength": 500,
            "description": "Title of the document"
        },
        "document_code": {
            "type": "string",
            "pattern": "^[A-Z]{3}-[0-9]{4}$",
            "description": "Document code in format XXX-0000"
        },
        "status": {
            "type": "string",
            "enum": ["draft", "review", "approved", "rejected", "archived"],
            "description": "Current document status"
        },
        "review_date": {
            "type": "string",
            "format": "date",
            "description": "Date of last review (YYYY-MM-DD)"
        },
        "created_timestamp": {
            "type": "string",
            "format": "date-time",
            "description": "Creation timestamp (ISO 8601)"
        },
        "page_count": {
            "type": "integer",
            "minimum": 1,
            "maximum": 10000,
            "description": "Total number of pages"
        },
        "confidence_score": {
            "type": "number",
            "minimum": 0.0,
            "maximum": 1.0,
            "description": "Extraction confidence (0.0 to 1.0)"
        },
        "is_verified": {
            "type": "boolean",
            "description": "Whether the extraction has been human-verified"
        },
        "tags": {
            "type": "array",
            "items": { "type": "string", "maxLength": 50 },
            "minItems": 0,
            "maxItems": 20,
            "description": "Classification tags"
        },
        "numeric_values": {
            "type": "array",
            "items": { "type": "number" },
            "maxItems": 100,
            "description": "Extracted numeric values"
        },
        "contact_info": {
            "type": "object",
            "properties": {
                "name": { "type": "string", "maxLength": 200 },
                "email": { "type": "string", "format": "email" },
                "phone": { "type": "string", "pattern": "^\\+?[0-9\\-\\s()]+$" }
            },
            "required": ["name"],
            "additionalProperties": false,
            "description": "Primary contact information"
        }
    },
    "required": ["document_title", "status"],
    "additionalProperties": false
}
```

### A.2 Aircraft Engineering Schema (Domain Example)

```json
{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "aircraft_model": {
            "type": "string",
            "maxLength": 100
        },
        "manufacturer": {
            "type": "string",
            "enum": ["Boeing", "Airbus", "Embraer", "Bombardier", "Other"]
        },
        "certification_date": {
            "type": "string",
            "format": "date"
        },
        "max_altitude_ft": {
            "type": "integer",
            "minimum": 0,
            "maximum": 100000
        },
        "max_range_nm": {
            "type": "number",
            "minimum": 0
        },
        "engine_count": {
            "type": "integer",
            "minimum": 1,
            "maximum": 12
        },
        "engine_type": {
            "type": "string",
            "enum": ["turbofan", "turboprop", "turbojet", "piston", "electric"]
        },
        "wingspan_meters": {
            "type": "number",
            "minimum": 0,
            "maximum": 100
        },
        "is_pressurized": {
            "type": "boolean"
        },
        "applicable_regulations": {
            "type": "array",
            "items": {
                "type": "string",
                "pattern": "^(FAR|EASA|CAAC)-[0-9A-Z\\.]+$"
            },
            "maxItems": 50
        }
    },
    "required": ["aircraft_model", "manufacturer"],
    "additionalProperties": false
}
```

### A.3 Auto Engineering Schema (Domain Example)

```json
{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "vehicle_make": {
            "type": "string",
            "maxLength": 100
        },
        "vehicle_model": {
            "type": "string",
            "maxLength": 100
        },
        "model_year": {
            "type": "integer",
            "minimum": 1900,
            "maximum": 2100
        },
        "vin": {
            "type": "string",
            "pattern": "^[A-HJ-NPR-Z0-9]{17}$",
            "description": "17-character Vehicle Identification Number"
        },
        "engine_displacement_liters": {
            "type": "number",
            "minimum": 0,
            "maximum": 20
        },
        "horsepower": {
            "type": "integer",
            "minimum": 0
        },
        "transmission_type": {
            "type": "string",
            "enum": ["manual", "automatic", "cvt", "dct", "ev"]
        },
        "is_electric": {
            "type": "boolean"
        },
        "safety_ratings": {
            "type": "object",
            "properties": {
                "nhtsa_overall": { "type": "integer", "minimum": 1, "maximum": 5 },
                "iihs_rating": { "type": "string", "enum": ["Poor", "Marginal", "Acceptable", "Good", "Good+"] }
            },
            "additionalProperties": false
        },
        "recall_ids": {
            "type": "array",
            "items": { "type": "string", "maxLength": 20 },
            "maxItems": 100
        }
    },
    "required": ["vehicle_make", "vehicle_model", "model_year"],
    "additionalProperties": false
}
```

### Supported JSON Schema Data Types Summary

| JSON Schema Type | Go Equivalent | Example Constraint Keywords |
|-----------------|---------------|---------------------------|
| `string` | `string` | `minLength`, `maxLength`, `pattern`, `format`, `enum` |
| `integer` | `int64` | `minimum`, `maximum` |
| `number` | `float64` | `minimum`, `maximum` |
| `boolean` | `bool` | (none) |
| `array` | `[]T` | `items`, `minItems`, `maxItems` |
| `object` | `map/struct` | `properties`, `required`, `additionalProperties` |

**Supported `format` values**: `date`, `date-time`, `email`, `uri`, `uuid`