Files

T

Jay Brown 17fc813823 Merged in feature/mutable-metadata1 (pull request #221 )

M1, M2 and M3 complete

* M1, M2 and M3 complete

* review changes

* docs

* docs

2026-04-16 23:11:26 +00:00

39 KiB

Raw Blame History

Mutable Metadata & Per-Client Custom Schemas - Design Plan

Status: DRAFT - Updated with Q's design decisions (2026-03-23)

1. Problem Statement

Today, all field extraction data is stored in a rigid 3-table schema with ~130 statically-defined columns (19 single-value fields + 112 array fields). Every client shares the same field definitions. This means:

Clients cannot define domain-specific fields (e.g., "aircraft engineering" vs. "auto engineering" document classes)
Adding new fields requires database migrations and code changes across the full stack
The system cannot accommodate varying data shapes per document class within a single client

Goal: Allow each client to curate their own collections of document schemas, where each schema defines the custom fields (with types and constraints) that a document's metadata must conform to.

2. Current Architecture Summary

2.1 Existing Tables

documentFieldExtractions          -- 19 single-value fields (1:1 per document)
documentFieldExtractionVersions   -- Version tracking with row-level locking
documentFieldExtractionArrayFields -- 112 array fields (1:N per extraction)

2.2 Existing API Endpoints

Method	Path	Description
POST	`/field-extractions`	Create versioned extraction (singleFields + arrayFields)
GET	`/field-extractions`	Get current (latest version) extraction for a document
GET	`/field-extractions/version`	Get extraction by specific version
GET	`/field-extractions/history`	List all versions for a document

2.3 Key Relationships

documents.id -> documentFieldExtractions.documentId (1:N via versions)
documentFieldExtractions.id -> documentFieldExtractionArrayFields.fieldExtractionId (1:N)
documentFieldExtractions.id -> documentFieldExtractionVersions.fieldExtractionId (1:N)
Documents belong to clients via documents.clientId
No per-client customization of field definitions exists today

3. Design Principles

Additive, not destructive - The existing static field extraction system continues to work unchanged. Custom schemas are an orthogonal capability.
Schema-as-data - Schemas are stored as JSON Schema documents in the database, not as DDL. No migrations needed when clients add/modify schemas.
Immutable schema versions - When a schema is updated, a new version (with new ID) is created. Old versions remain for historical reference. No backward compatibility checking is required between versions since each version gets a distinct ID.
Schema binding is permanent - Once custom metadata is written to a document with a schema ID, that schema ID cannot be changed on the document. This guarantees data integrity.
JSON blob storage - Custom metadata is stored as JSONB in PostgreSQL, validated against the schema at write time. This avoids dynamic column creation.
Client isolation - Schemas are scoped to clients. A client cannot reference another client's schema.
Schema size limit - Schema definitions are capped at 64KB (MaxSchemaDefinitionBytes = 65536). This is a tunable constant that can be increased later if needed.
No JSONB search/filter - The system does NOT support querying or filtering documents by custom metadata values. Custom metadata is opaque storage validated on write. If search is needed in the future, it would require GIN indexes on the JSONB columns and a dedicated query API (separate phase).
Admin-scoped schema management - Schema CRUD operations live under /admin and require client_admin role. Custom metadata read/write on documents is available to regular users.
No migration from static fields - There is no migration path from the existing static field extraction data into the custom schema system. They are independent systems. Out of scope.

4. Database Schema Design

4.1 New Table: `client_metadata_schemas`

Stores client-defined JSON Schema documents. Each row is an immutable version of a schema definition.

CREATE TABLE client_metadata_schemas (
    id              uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    client_id       varchar(255) NOT NULL REFERENCES clients(clientId),
    name            varchar(255) NOT NULL,
    description     text,
    schema_def      jsonb NOT NULL,          -- The JSON Schema document (max 64KB enforced at app layer)
    version         int NOT NULL DEFAULT 1,   -- Auto-incremented per (client_id, name)
    is_active       boolean NOT NULL DEFAULT true,
    created_at      timestamptz NOT NULL DEFAULT NOW(),
    created_by      varchar(255) NOT NULL,

    CONSTRAINT uq_client_schema_name_version UNIQUE (client_id, name, version),
    CONSTRAINT chk_schema_def_size CHECK (pg_column_size(schema_def) <= 65536)
);

CREATE INDEX idx_cms_client_id ON client_metadata_schemas(client_id);
CREATE INDEX idx_cms_client_name ON client_metadata_schemas(client_id, name);
CREATE INDEX idx_cms_active ON client_metadata_schemas(client_id, is_active);

Note

: The 64KB limit is enforced at both the application layer (Go constant MaxSchemaDefinitionBytes = 65536) and the database layer (CHECK constraint). The app-layer check provides a better error message; the DB constraint is a safety net.

Design decisions:

name + version scoped to client: Allows human-friendly naming ("aircraft-engineering-v3") with automatic versioning. When a schema is "updated," a new row is inserted with version = max(version) + 1 for that (client_id, name).
schema_def is JSONB: Stores the full JSON Schema document. PostgreSQL can index into it if needed later.
is_active: Soft-delete flag. When a schema is "deleted," it is marked inactive. Schemas referenced by existing documents cannot be hard-deleted.
id is the schema version ID: This is what documents reference. Each update creates a new id. The (client_id, name, version) tuple provides the human-readable lineage.

4.2 New Table: `document_custom_metadata`

Stores the actual custom metadata for documents, validated against the referenced schema.

CREATE TABLE document_custom_metadata (
    id              uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    document_id     uuid NOT NULL REFERENCES documents(id),
    schema_id       uuid NOT NULL REFERENCES client_metadata_schemas(id),
    metadata        jsonb NOT NULL,           -- The actual custom field values
    version         int NOT NULL DEFAULT 1,   -- Versioned like field extractions
    created_at      timestamptz NOT NULL DEFAULT NOW(),
    created_by      varchar(255) NOT NULL,

    CONSTRAINT uq_doc_custom_metadata_version UNIQUE (document_id, version)
);

CREATE INDEX idx_dcm_document_id ON document_custom_metadata(document_id);
CREATE INDEX idx_dcm_schema_id ON document_custom_metadata(schema_id);

Design decisions:

Versioned: Like field extractions, custom metadata supports version history. Each write creates a new version.
schema_id is immutable per document: Enforced at the application layer (see Rule 4.3). All versions of custom metadata for a document must reference the same schema ID.
metadata is JSONB: Validated against client_metadata_schemas.schema_def at write time. Stored as a single blob for simplicity and query flexibility.

4.3 Alteration: `documents` Table

Add an optional schema binding column:

ALTER TABLE documents
    ADD COLUMN custom_schema_id uuid NULL REFERENCES client_metadata_schemas(id);

CREATE INDEX idx_documents_custom_schema_id ON documents(custom_schema_id);

Behavior:

NULL = document uses only the legacy static field extraction system (default, backward-compatible)
Once set AND custom metadata has been written, this field becomes immutable (enforced in application code)
Can be set before metadata is written (pre-binding) and changed freely until first metadata write
Must reference a schema owned by the same client as the document (cross-client reference check)

4.4 New View: `current_document_custom_metadata`

CREATE VIEW current_document_custom_metadata AS
SELECT DISTINCT ON (document_id)
    id,
    document_id,
    schema_id,
    metadata,
    version,
    created_at,
    created_by
FROM document_custom_metadata
ORDER BY document_id, version DESC;

4.5 Entity Relationship Diagram

clients
  |
  |-- 1:N --> client_metadata_schemas (per-client schema definitions)
  |              |
  |              |-- referenced by --> documents.custom_schema_id
  |              |-- referenced by --> document_custom_metadata.schema_id
  |
  |-- 1:N --> documents
                 |
                 |-- 1:N --> document_custom_metadata (versioned JSONB blobs)
                 |-- 1:N --> documentFieldExtractions (existing static fields)

5. API Design

5.1 Schema Management Endpoints (Admin)

All under the existing AdminService tag. These endpoints require the client_admin role (see Section 5.5 for auth details). Endpoints are prefixed with /admin/custom-schemas.

`POST /admin/custom-schemas`

Create a new schema for a client.

Request:

{
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "Custom fields for aircraft engineering documents",
    "schema": {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties": {
            "aircraft_model": {
                "type": "string",
                "maxLength": 100
            },
            "certification_date": {
                "type": "string",
                "format": "date"
            },
            "max_altitude_ft": {
                "type": "number",
                "minimum": 0,
                "maximum": 100000
            },
            "engine_count": {
                "type": "integer",
                "minimum": 1,
                "maximum": 12
            }
        },
        "required": ["aircraft_model"],
        "additionalProperties": false
    },
    "createdBy": "admin@acme.com"
}

Response (201):

{
    "id": "019577a3-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "Custom fields for aircraft engineering documents",
    "schema": { ... },
    "version": 1,
    "isActive": true,
    "createdAt": "2026-03-23T12:00:00Z",
    "createdBy": "admin@acme.com"
}

Validation:

The schema field MUST be a valid JSON Schema document (validated server-side using a JSON Schema meta-validator)
The schema field must be <= 64KB in size (returns 400 with message referencing the limit)
name must be unique within the client (for active schemas at the same version)
additionalProperties: false is recommended but not required

`PUT /admin/custom-schemas/{schemaId}`

Update an existing schema. Creates a new version with a new ID. The old version remains intact. No backward compatibility check is performed -- the new version is treated as an independent schema that happens to share a name lineage.

Request:

{
    "description": "Updated aircraft engineering fields (added wingspan)",
    "schema": {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties": {
            "aircraft_model": { "type": "string", "maxLength": 100 },
            "certification_date": { "type": "string", "format": "date" },
            "max_altitude_ft": { "type": "number", "minimum": 0, "maximum": 100000 },
            "engine_count": { "type": "integer", "minimum": 1, "maximum": 12 },
            "wingspan_meters": { "type": "number", "minimum": 0 }
        },
        "required": ["aircraft_model"],
        "additionalProperties": false
    },
    "createdBy": "admin@acme.com"
}

Response (201):

{
    "id": "019577b4-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "version": 2,
    "previousVersionId": "019577a3-...",
    ...
}

Behavior:

The schemaId in the path identifies the schema to "update" (really: create a new version of)
The new version inherits the name from the previous version
The old version remains active and referenced by existing documents
Returns the newly created schema version

`GET /admin/custom-schemas`

List all schemas for a client.

Query Parameters:

clientId (required): The client whose schemas to list
name (optional): Filter by schema name
activeOnly (optional, default: true): Only return active schemas
includeAllVersions (optional, default: false): Return all versions or just latest per name

Response (200):

{
    "schemas": [
        {
            "id": "019577b4-...",
            "clientId": "acme-corp",
            "name": "aircraft-engineering",
            "description": "...",
            "version": 2,
            "isActive": true,
            "documentCount": 45,
            "createdAt": "...",
            "createdBy": "..."
        },
        {
            "id": "019577c1-...",
            "clientId": "acme-corp",
            "name": "auto-engineering",
            "description": "...",
            "version": 1,
            "isActive": true,
            "documentCount": 120,
            "createdAt": "...",
            "createdBy": "..."
        }
    ]
}

Note: documentCount shows how many documents reference each schema version. This is computed via a subquery or join.

`GET /admin/custom-schemas/{schemaId}`

Get a specific schema by ID (returns full schema definition).

Response (200):

{
    "id": "019577a3-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "...",
    "schema": { ... },
    "version": 1,
    "isActive": true,
    "documentCount": 45,
    "createdAt": "...",
    "createdBy": "..."
}

`DELETE /admin/custom-schemas/{schemaId}`

Soft-delete a schema. Only succeeds if no documents reference it.

Response (204): No content on success.

Response (409 Conflict):

{
    "error": "Schema is in use by 45 documents and cannot be deleted"
}

Behavior:

Sets is_active = false
Fails with 409 if any documents reference this schema ID
Inactive schemas still serve validation for existing documents but cannot be assigned to new documents

5.2 Document Schema Assignment Endpoints

`PATCH /document/{id}/schema`

Assign or change a custom schema on a single document.

Request:

{
    "customSchemaId": "019577a3-..."
}

Response (200):

{
    "documentId": "...",
    "customSchemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "schemaVersion": 1
}

Validation rules:

Schema must belong to the same client as the document
Schema must be active
If the document already has custom metadata written, the schema cannot be changed (returns 409)
Setting customSchemaId to null removes the binding (only if no metadata exists)

`POST /admin/folders/{folderId}/assign-schema` (Bulk Folder Assignment)

Assign a custom schema to ALL documents in a folder and all its subfolders (recursive).

Request:

{
    "customSchemaId": "019577a3-...",
    "createdBy": "admin@acme.com"
}

Response (200):

{
    "folderId": "...",
    "customSchemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "documentsUpdated": 87,
    "documentsSkipped": 3,
    "skippedReasons": [
        {"documentId": "...", "reason": "Document already has metadata with a different schema"},
        {"documentId": "...", "reason": "Document already has metadata with a different schema"},
        {"documentId": "...", "reason": "Document already assigned to same schema (no-op)"}
    ]
}

Behavior:

Uses WITH RECURSIVE CTE to walk the entire folder subtree (same pattern as folder deletion)
For each document in the tree:
- If the document has no schema and no custom metadata -> assign the schema
- If the document already has the same schema -> skip (no-op, counted in documentsSkipped)
- If the document has a different schema but no metadata yet -> reassign to new schema
- If the document has a different schema AND has metadata -> skip with reason (cannot change)
Runs in a single transaction
Returns summary of what was updated and what was skipped (with reasons)
Requires client_admin role (admin endpoint)

`GET /document/{id}` (Modified)

Add customSchemaId and customMetadata fields to DocumentEnriched:

{
    "id": "...",
    "clientId": "acme-corp",
    "hash": "...",
    "customSchemaId": "019577a3-...",
    "customSchemaName": "aircraft-engineering",
    "hasCustomMetadata": true,
    "customMetadata": { ... },
    ...existing fields...
}

Query parameter: customMetadata (boolean, default: false) - when true, includes the full custom metadata in the response (like the existing textRecord parameter).

5.5 Authorization & Permit.io Changes

New Role: `client_admin`

A new Permit.io role needs to be created between user_admin and client_user:

Role	Schema CRUD	Schema Assignment	Metadata Read	Metadata Write	Existing Admin
`super_admin`	Yes	Yes	Yes	Yes	Yes (full)
`client_admin`	Yes	Yes	Yes	Yes	No
`user_admin`	No	No	Yes	Yes	Yes (full)
`auditor`	Read-only	No	Read-only	No	Read-only
`client_user`	No	No	Yes	Yes	No

Permit.io policy additions (in permit_policies.yaml):

client_admin:
  admin:
    - get
    - post
    - patch
    - delete
    - head
  # Plus all permissions that client_user has:
  client:
    - get
    - post
    - patch
  documents:
    - get
    - post
  document:
    - get
    - post
    - patch
    - delete
  folders:
    - get
    - post
    - patch
  field-extractions:
    - get
    - post
  custom-metadata:
    - get
    - post
  export:
    - get
    - post

Key distinction: client_admin gets admin:* permissions (for schema management under /admin/custom-schemas) plus all document/metadata operations. Unlike super_admin, it does NOT get system-wide admin capabilities (user management, EULA management, etc.).

Open question: The current Permit.io setup maps /admin/* to the admin resource uniformly. If client_admin should access /admin/custom-schemas but NOT /admin/users, the resource mapping may need to become more granular (e.g., separate admin-schemas resource). Alternative: use a single admin resource and control via tenant-scoped checks. This is an implementation detail to resolve during Phase 6.

5.3 Custom Metadata CRUD Endpoints (New)

All under a new CustomMetadataService tag.

`POST /custom-metadata`

Create or update custom metadata for a document. Creates a new version.

Request:

{
    "documentId": "...",
    "metadata": {
        "aircraft_model": "Boeing 737-800",
        "certification_date": "2024-06-15",
        "max_altitude_ft": 41000,
        "engine_count": 2
    },
    "createdBy": "analyst@acme.com"
}

Response (201):

{
    "id": "...",
    "documentId": "...",
    "schemaId": "019577a3-...",
    "metadata": { ... },
    "version": 1,
    "createdAt": "...",
    "createdBy": "analyst@acme.com"
}

Validation:

Document must have a custom_schema_id assigned (400 if not)
The metadata payload is validated against the JSON Schema stored in client_metadata_schemas.schema_def

If validation fails, returns 400 with detailed validation errors:

{
    "error": "Metadata does not conform to schema",
    "validationErrors": [
        {"field": "engine_count", "message": "must be >= 1, got 0"},
        {"field": "aircraft_model", "message": "required field missing"}
    ]
}

On first write, locks the document's custom_schema_id (makes it immutable)
Subsequent writes create new versions but must use the same schema

`GET /custom-metadata`

Get current (latest version) custom metadata for a document.

Query Parameters:

documentId (required)

Response (200):

{
    "id": "...",
    "documentId": "...",
    "schemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "metadata": { ... },
    "version": 3,
    "createdAt": "...",
    "createdBy": "..."
}

`GET /custom-metadata/version`

Get a specific version of custom metadata.

Query Parameters:

documentId (required)
version (required, >= 1)

`GET /custom-metadata/history`

Get version history for a document's custom metadata.

Query Parameters:

documentId (required)

Response (200):

{
    "versions": [
        {"id": "...", "documentId": "...", "version": 3, "createdAt": "...", "createdBy": "..."},
        {"id": "...", "documentId": "...", "version": 2, "createdAt": "...", "createdBy": "..."},
        {"id": "...", "documentId": "...", "version": 1, "createdAt": "...", "createdBy": "..."}
    ]
}

6. Validation Strategy

6.1 JSON Schema Validation

Use a Go JSON Schema validation library for both:

Schema validation (meta-validation): When a client submits a new schema, validate it against the JSON Schema meta-schema to ensure it is a well-formed JSON Schema document.
Metadata validation: When custom metadata is submitted, validate the payload against the document's assigned schema.

Recommended library: github.com/santhosh-tekuri/jsonschema/v6

Reasons:

Supports JSON Schema draft 2020-12
Active maintenance
Good error reporting
Supports custom formats

6.2 Supported JSON Schema Features

The system will support the following JSON Schema keywords for custom field definitions:

Feature	JSON Schema Keyword	Example
Data type	`type`	`"string"`, `"number"`, `"integer"`, `"boolean"`, `"array"`, `"object"`
Required fields	`required`	`["field1", "field2"]`
String length	`minLength`, `maxLength`	`"maxLength": 100`
Numeric range	`minimum`, `maximum`	`"minimum": 0`
Pattern matching	`pattern`	`"pattern": "^[A-Z]{2}$"`
Enum values	`enum`	`"enum": ["active", "inactive"]`
Date/time format	`format`	`"format": "date"`
Array items	`items`, `minItems`, `maxItems`	Nested array support
Nested objects	`properties`	Hierarchical field groups
No extra fields	`additionalProperties`	`false` to enforce strict schemas

6.3 Validation Error Reporting

Validation errors will be returned as structured JSON with:

The JSON pointer to the failing field (/aircraft_model)
The constraint that was violated (maxLength)
A human-readable message

7. Service Layer Design

7.1 New Service: `internal/customschema/`

internal/customschema/
    service.go       -- CustomSchemaService (CRUD + validation)
    models.go        -- Input/output types
    validator.go     -- JSON Schema meta-validation + payload validation
    constants.go     -- MaxSchemaDefinitionBytes and other limits
    service_test.go  -- Integration tests

Constants (constants.go):

const (
    // MaxSchemaDefinitionBytes is the maximum size of a JSON Schema definition.
    // This limit can be increased if clients need larger schemas.
    MaxSchemaDefinitionBytes = 65536 // 64KB
)

Key methods:

type Service struct {
    cfg       serviceconfig.ConfigProvider
    validator *SchemaValidator
}

// Schema management (admin operations)
func (s *Service) CreateSchema(ctx, input CreateSchemaInput) (*Schema, error)
func (s *Service) UpdateSchema(ctx, schemaID uuid.UUID, input UpdateSchemaInput) (*Schema, error)
func (s *Service) GetSchema(ctx, schemaID uuid.UUID) (*Schema, error)
func (s *Service) ListSchemas(ctx, clientID string, filters ListFilters) ([]SchemaSummary, error)
func (s *Service) DeleteSchema(ctx, schemaID uuid.UUID) error

// Metadata operations (regular user operations)
func (s *Service) SetDocumentMetadata(ctx, input SetMetadataInput) (*CustomMetadata, error)
func (s *Service) GetCurrentMetadata(ctx, documentID uuid.UUID) (*CustomMetadata, error)
func (s *Service) GetMetadataByVersion(ctx, documentID uuid.UUID, version int) (*CustomMetadata, error)
func (s *Service) GetMetadataHistory(ctx, documentID uuid.UUID) ([]MetadataVersion, error)

// Document schema binding
func (s *Service) AssignSchema(ctx, documentID uuid.UUID, schemaID uuid.UUID) error
func (s *Service) AssignSchemaToFolder(ctx, folderID uuid.UUID, schemaID uuid.UUID, createdBy string) (*BulkAssignResult, error)

BulkAssignResult:

type BulkAssignResult struct {
    FolderID         uuid.UUID
    SchemaID         uuid.UUID
    DocumentsUpdated int
    DocumentsSkipped int
    SkippedReasons   []SkippedDocument
}

type SkippedDocument struct {
    DocumentID uuid.UUID
    Reason     string
}

7.2 Validator Component

type SchemaValidator struct{}

// ValidateSchemaDefinition checks that a schema_def is valid JSON Schema
func (v *SchemaValidator) ValidateSchemaDefinition(schemaDef json.RawMessage) error

// ValidateMetadata checks that metadata conforms to a compiled schema
func (v *SchemaValidator) ValidateMetadata(schemaDef json.RawMessage, metadata json.RawMessage) ([]ValidationError, error)

7.3 API Controller: `api/queryAPI/customschemas.go`

Follows the same pattern as fieldextractions.go:

Parse and validate request
Call service layer
Convert between API and service types
Return appropriate HTTP status codes

8. Migration Plan

8.1 Migration Files

Three new migrations:

Migration 120: Create client_metadata_schemas table

00000000000120_create_client_metadata_schemas.up.sql
00000000000120_create_client_metadata_schemas.down.sql

Migration 121: Create document_custom_metadata table + view

00000000000121_create_document_custom_metadata.up.sql
00000000000121_create_document_custom_metadata.down.sql

Migration 122: Add custom_schema_id to documents table

00000000000122_add_custom_schema_id_to_documents.up.sql
00000000000122_add_custom_schema_id_to_documents.down.sql

8.2 SQLC Queries

New query file: internal/database/queries/customschemas.sql

Schema operations:

CreateClientMetadataSchema - Insert new schema
GetClientMetadataSchema - Get by ID
ListClientMetadataSchemas - List by client with filters
GetLatestSchemaByName - Get latest version of a named schema
GetSchemaDocumentCount - Count documents using a schema
DeactivateClientMetadataSchema - Soft-delete
GetMaxSchemaVersion - For auto-incrementing version

Metadata operations:

CreateDocumentCustomMetadata - Insert new metadata version
GetCurrentDocumentCustomMetadata - Latest version (via view)
GetDocumentCustomMetadataByVersion - Specific version
GetDocumentCustomMetadataHistory - All versions
LockDocumentCustomMetadataForVersion - Row-level locking

Document modifications:

SetDocumentCustomSchemaId - Update documents.custom_schema_id
GetDocumentCustomSchemaId - Read documents.custom_schema_id
BulkSetDocumentCustomSchemaIdInFolderTree - Set schema for all docs in folder subtree (WITH RECURSIVE)
GetDocumentsWithMetadataInFolderTree - Find docs that already have metadata (for skip reporting)

8.3 OpenAPI Spec Changes

Add to serviceAPIs/queryAPI.yaml:

New admin paths: /admin/custom-schemas, /admin/custom-schemas/{schemaId}, /admin/folders/{folderId}/assign-schema
New user paths: /custom-metadata, /custom-metadata/version, /custom-metadata/history, /document/{id}/schema
New schemas: CustomSchemaRequest, CustomSchemaResponse, CustomSchemaListResponse, CustomMetadataRequest, CustomMetadataResponse, CustomMetadataHistoryResponse, ValidationErrorResponse, BulkSchemaAssignRequest, BulkSchemaAssignResponse
Modified schemas: DocumentEnriched (add customSchemaId, customSchemaName, hasCustomMetadata, customMetadata fields)

9. Invariants & Business Rules

#	Rule	Enforcement
1	Schema must be valid JSON Schema	Application: meta-validation on create/update
2	Schema `name` + `version` unique per client	Database: unique constraint
3	Updated schema gets new ID and version	Application: INSERT, not UPDATE
4	Schema deletion blocked if documents reference it	Application: count check before deactivation
5	Document's `custom_schema_id` is immutable once metadata exists	Application: check in AssignSchema and SetMetadata
6	Custom metadata must conform to assigned schema	Application: JSON Schema validation on write
7	All custom metadata versions for a document use same schema ID	Application: consistency check on write
8	Schema must belong to same client as document	Application: cross-reference check
9	Only active schemas can be assigned to new documents	Application: is_active check
10	Custom metadata is versioned (no in-place updates)	Database: version column + unique constraint
11	Schema definition must be <= 64KB	Application: size check + Database: CHECK constraint
12	Schema management requires `client_admin` role	Permit.io: role-based access on `/admin` resource
13	No query/filter on custom metadata values	By design: not implemented (see Principle 8)

10. Relationship to Existing Field Extraction System

The existing static field extraction system (/field-extractions) and the new custom schema system (/custom-schemas + /custom-metadata) are independent and complementary:

A document can have BOTH static field extractions AND custom metadata
A document can have EITHER one, or neither
They are stored in separate tables with separate versioning
The existing 130 static fields continue to serve the current healthcare contract analysis use case
Custom schemas serve new, client-specific use cases that don't fit the static model

No breaking changes to the existing API or database schema.

11. Implementation Order

Phase	Work	Dependencies
1	Database migrations (tables, view, document column)	None
2	SQLC queries + `task generate`	Phase 1
3	Schema validator component (`internal/customschema/validator.go`)	JSON Schema library added to go.mod
4	Service layer (`internal/customschema/service.go`) incl. bulk folder assignment	Phase 2, 3
5	OpenAPI spec updates (admin + user endpoints)	None (can parallel with 1-3)
6	Permit.io: Create `client_admin` role + policy updates	None (can parallel with 1-5)
7	API controllers + code generation (admin schema CRUD, metadata CRUD)	Phase 4, 5
8	Document endpoint modifications (PATCH schema, GET enrichment)	Phase 4, 7
9	Bulk folder schema assignment endpoint	Phase 4, 7
10	Integration tests	Phase 9
11	Documentation updates (`docs/ai.generated/`) incl. sample schemas	Phase 10

12. Testing Strategy

Schema validation tests: Valid and invalid JSON Schema documents, edge cases (empty schema, deeply nested, self-referential, oversized >64KB)
Metadata validation tests: Conforming and non-conforming payloads, all supported JSON Schema keywords
Service integration tests: Full CRUD lifecycle using testcontainers (real PostgreSQL)
API integration tests: HTTP-level tests for all new endpoints (admin + user)
Invariant tests: Schema immutability, cross-client isolation, deletion guards, size limits
Bulk folder assignment tests: Recursive assignment, skip behavior for locked docs, empty folders, deeply nested folder trees
Authorization tests: client_admin can manage schemas, client_user cannot, super_admin can
Backward compatibility tests: Existing field extraction endpoints continue to work unchanged

13. Resolved Design Decisions

#	Question	Decision
1	Schema size limits	64KB cap via `MaxSchemaDefinitionBytes` constant (tunable)
2	Schema backward compatibility	Not required. New version = new ID. Old docs keep old schema.
3	Bulk operations	Yes. `POST /admin/folders/{folderId}/assign-schema` assigns recursively through folder tree.
4	Custom metadata search/filter	Not supported. No GIN indexes, no query API. Metadata is opaque storage.
5	Schema templates	No system templates. Documentation will include sample schemas covering all supported types.
6	Existing field migration path	Out of scope. Static fields and custom schemas are independent systems.
7	Endpoint naming/placement	Schema CRUD under `/admin/custom-schemas` (requires `client_admin` role, not `super_admin`). Metadata CRUD under `/custom-metadata` (regular user access).

Appendix A: Sample JSON Schemas

These examples demonstrate all supported JSON Schema features and data types. They should be included in the API documentation.

A.1 Comprehensive Schema (All Supported Types)

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "document_title": {
            "type": "string",
            "minLength": 1,
            "maxLength": 500,
            "description": "Title of the document"
        },
        "document_code": {
            "type": "string",
            "pattern": "^[A-Z]{3}-[0-9]{4}$",
            "description": "Document code in format XXX-0000"
        },
        "status": {
            "type": "string",
            "enum": ["draft", "review", "approved", "rejected", "archived"],
            "description": "Current document status"
        },
        "review_date": {
            "type": "string",
            "format": "date",
            "description": "Date of last review (YYYY-MM-DD)"
        },
        "created_timestamp": {
            "type": "string",
            "format": "date-time",
            "description": "Creation timestamp (ISO 8601)"
        },
        "page_count": {
            "type": "integer",
            "minimum": 1,
            "maximum": 10000,
            "description": "Total number of pages"
        },
        "confidence_score": {
            "type": "number",
            "minimum": 0.0,
            "maximum": 1.0,
            "description": "Extraction confidence (0.0 to 1.0)"
        },
        "is_verified": {
            "type": "boolean",
            "description": "Whether the extraction has been human-verified"
        },
        "tags": {
            "type": "array",
            "items": { "type": "string", "maxLength": 50 },
            "minItems": 0,
            "maxItems": 20,
            "description": "Classification tags"
        },
        "numeric_values": {
            "type": "array",
            "items": { "type": "number" },
            "maxItems": 100,
            "description": "Extracted numeric values"
        },
        "contact_info": {
            "type": "object",
            "properties": {
                "name": { "type": "string", "maxLength": 200 },
                "email": { "type": "string", "format": "email" },
                "phone": { "type": "string", "pattern": "^\\+?[0-9\\-\\s()]+$" }
            },
            "required": ["name"],
            "additionalProperties": false,
            "description": "Primary contact information"
        }
    },
    "required": ["document_title", "status"],
    "additionalProperties": false
}

A.2 Aircraft Engineering Schema (Domain Example)

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "aircraft_model": {
            "type": "string",
            "maxLength": 100
        },
        "manufacturer": {
            "type": "string",
            "enum": ["Boeing", "Airbus", "Embraer", "Bombardier", "Other"]
        },
        "certification_date": {
            "type": "string",
            "format": "date"
        },
        "max_altitude_ft": {
            "type": "integer",
            "minimum": 0,
            "maximum": 100000
        },
        "max_range_nm": {
            "type": "number",
            "minimum": 0
        },
        "engine_count": {
            "type": "integer",
            "minimum": 1,
            "maximum": 12
        },
        "engine_type": {
            "type": "string",
            "enum": ["turbofan", "turboprop", "turbojet", "piston", "electric"]
        },
        "wingspan_meters": {
            "type": "number",
            "minimum": 0,
            "maximum": 100
        },
        "is_pressurized": {
            "type": "boolean"
        },
        "applicable_regulations": {
            "type": "array",
            "items": {
                "type": "string",
                "pattern": "^(FAR|EASA|CAAC)-[0-9A-Z\\.]+$"
            },
            "maxItems": 50
        }
    },
    "required": ["aircraft_model", "manufacturer"],
    "additionalProperties": false
}

A.3 Auto Engineering Schema (Domain Example)

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "vehicle_make": {
            "type": "string",
            "maxLength": 100
        },
        "vehicle_model": {
            "type": "string",
            "maxLength": 100
        },
        "model_year": {
            "type": "integer",
            "minimum": 1900,
            "maximum": 2100
        },
        "vin": {
            "type": "string",
            "pattern": "^[A-HJ-NPR-Z0-9]{17}$",
            "description": "17-character Vehicle Identification Number"
        },
        "engine_displacement_liters": {
            "type": "number",
            "minimum": 0,
            "maximum": 20
        },
        "horsepower": {
            "type": "integer",
            "minimum": 0
        },
        "transmission_type": {
            "type": "string",
            "enum": ["manual", "automatic", "cvt", "dct", "ev"]
        },
        "is_electric": {
            "type": "boolean"
        },
        "safety_ratings": {
            "type": "object",
            "properties": {
                "nhtsa_overall": { "type": "integer", "minimum": 1, "maximum": 5 },
                "iihs_rating": { "type": "string", "enum": ["Poor", "Marginal", "Acceptable", "Good", "Good+"] }
            },
            "additionalProperties": false
        },
        "recall_ids": {
            "type": "array",
            "items": { "type": "string", "maxLength": 20 },
            "maxItems": 100
        }
    },
    "required": ["vehicle_make", "vehicle_model", "model_year"],
    "additionalProperties": false
}

Supported JSON Schema Data Types Summary

JSON Schema Type	Go Equivalent	Example Constraint Keywords
`string`	`string`	`minLength`, `maxLength`, `pattern`, `format`, `enum`
`integer`	`int64`	`minimum`, `maximum`
`number`	`float64`	`minimum`, `maximum`
`boolean`	`bool`	(none)
`array`	`[]T`	`items`, `minItems`, `maxItems`
`object`	`map/struct`	`properties`, `required`, `additionalProperties`

Supported format values: date, date-time, email, uri, uuid

39 KiB Raw Blame History

Mutable Metadata & Per-Client Custom Schemas - Design Plan

Status: DRAFT - Updated with Q's design decisions (2026-03-23)

1. Problem Statement

2. Current Architecture Summary

2.1 Existing Tables

2.2 Existing API Endpoints

2.3 Key Relationships

3. Design Principles

4. Database Schema Design

4.1 New Table: client_metadata_schemas

4.2 New Table: document_custom_metadata

4.3 Alteration: documents Table

4.4 New View: current_document_custom_metadata

4.5 Entity Relationship Diagram

5. API Design

5.1 Schema Management Endpoints (Admin)

POST /admin/custom-schemas

PUT /admin/custom-schemas/{schemaId}

GET /admin/custom-schemas

GET /admin/custom-schemas/{schemaId}

DELETE /admin/custom-schemas/{schemaId}

5.2 Document Schema Assignment Endpoints

PATCH /document/{id}/schema

POST /admin/folders/{folderId}/assign-schema (Bulk Folder Assignment)

GET /document/{id} (Modified)

5.5 Authorization & Permit.io Changes

New Role: client_admin

5.3 Custom Metadata CRUD Endpoints (New)

POST /custom-metadata

GET /custom-metadata

GET /custom-metadata/version

GET /custom-metadata/history

6. Validation Strategy

6.1 JSON Schema Validation

6.2 Supported JSON Schema Features

6.3 Validation Error Reporting

7. Service Layer Design

7.1 New Service: internal/customschema/

7.2 Validator Component

7.3 API Controller: api/queryAPI/customschemas.go

8. Migration Plan

8.1 Migration Files

8.2 SQLC Queries

8.3 OpenAPI Spec Changes

9. Invariants & Business Rules

10. Relationship to Existing Field Extraction System

11. Implementation Order

12. Testing Strategy

13. Resolved Design Decisions

Appendix A: Sample JSON Schemas

A.1 Comprehensive Schema (All Supported Types)

A.2 Aircraft Engineering Schema (Domain Example)

A.3 Auto Engineering Schema (Domain Example)

Supported JSON Schema Data Types Summary

39 KiB

Raw Blame History

4.1 New Table: `client_metadata_schemas`

4.2 New Table: `document_custom_metadata`

4.3 Alteration: `documents` Table

4.4 New View: `current_document_custom_metadata`

`POST /admin/custom-schemas`

`PUT /admin/custom-schemas/{schemaId}`

`GET /admin/custom-schemas`

`GET /admin/custom-schemas/{schemaId}`

`DELETE /admin/custom-schemas/{schemaId}`

`PATCH /document/{id}/schema`

`POST /admin/folders/{folderId}/assign-schema` (Bulk Folder Assignment)

`GET /document/{id}` (Modified)

New Role: `client_admin`

`POST /custom-metadata`

`GET /custom-metadata`

`GET /custom-metadata/version`

`GET /custom-metadata/history`

7.1 New Service: `internal/customschema/`

7.3 API Controller: `api/queryAPI/customschemas.go`