Files

T

Jay Brown 17fc813823 Merged in feature/mutable-metadata1 (pull request #221 )

M1, M2 and M3 complete

* M1, M2 and M3 complete

* review changes

* docs

* docs

2026-04-16 23:11:26 +00:00

76 KiB

Raw Blame History

Mutable Metadata & Per-Client Custom Schemas - Combined Design Plan (v2)

Status: DRAFT - Combined Claude + Codex + Jay's recommendations (2026-03-24, rev 2026-03-31 post-review, rev 2026-04-10: schema administration restricted to `super_admin` role only)

Lineage: This plan was informed by two original detailed specs done adversarily. One by claude Opus 4.6 and one by Codex 5.4 (xhigh) based on human written requirements. Each was human reviewed for common items and conflicting items. Where there were conflicts the human was the final decision maker. This resulting hybrid document represents an improved design over each of the source documents.

v2 change summary (2026-04-10): In v1 all mutable-metadata administrative endpoints lived under /admin/*. Because the Permit.io middleware maps resources from the first path segment and both user_admin and super_admin hold full permissions on the admin resource, v1 exposed schema CRUD and schema assignment to the lesser (client-scoped) user_admin role. v2 moves every schema administration endpoint onto a new /super-admin/* path prefix backed by a new Permit.io resource named super-admin that only the super_admin role can access. User-facing custom metadata read/write endpoints (/custom-metadata/*) are unchanged. The underlying service layer, database schema, triggers, and validation strategy are unchanged -- this is strictly an authorization and routing change.

1. Problem Statement

Today, all field extraction data is stored in a rigid 3-table schema with ~130 statically-defined columns (19 single-value fields + 112 array fields). Every client shares the same field definitions. This means:

Clients cannot define domain-specific fields (e.g., "aircraft engineering" vs. "auto engineering" document classes)
Adding new fields requires database migrations and code changes across the full stack
The system cannot accommodate varying data shapes per document class within a single client

Goal: Allow each client to curate their own collections of document schemas, where each schema defines the custom fields (with types and constraints) that a document's metadata must conform to. Administration of these schemas (create, update, retire, list, assign to documents/folders) is a platform-level responsibility and must be restricted to the super_admin role, not delegated to the lower-privileged user_admin role.

2. Current Architecture Summary

2.1 Existing Tables

documentFieldExtractions          -- 19 single-value fields (1:1 per document)
documentFieldExtractionVersions   -- Version tracking with row-level locking
documentFieldExtractionArrayFields -- 112 array fields (1:N per extraction)

2.2 Existing API Endpoints

Method	Path	Description
POST	`/field-extractions`	Create versioned extraction (singleFields + arrayFields)
GET	`/field-extractions`	Get current (latest version) extraction for a document
GET	`/field-extractions/version`	Get extraction by specific version
GET	`/field-extractions/history`	List all versions for a document

2.3 Key Relationships

documents.id -> documentFieldExtractions.documentId (1:N via versions)
documentFieldExtractions.id -> documentFieldExtractionArrayFields.fieldExtractionId (1:N)
documentFieldExtractions.id -> documentFieldExtractionVersions.fieldExtractionId (1:N)
Documents belong to clients via documents.clientId
No per-client customization of field definitions exists today

2.4 Current Role Model (Relevant to This Plan)

Two admin-class roles exist today, both configured in cmd/auth_related/permit.setup/permit_policies.yaml:

super_admin: Platform administrator. Holds full permissions across every resource (admin, client, clients, document, documents, folders, etc.). Intended for AArete-internal operators.
user_admin: Client-scoped administrator. Primarily used for creating new client users via the admin user tool. Holds full permissions on the admin resource but intentionally does not hold permissions on client, clients, or document at the resource level.

Because the middleware extracts the resource from the first path segment (/admin/foo -> resource admin), any endpoint placed under /admin/* is accessible to both super_admin and user_admin. v1 of this plan placed schema administration under /admin/* and therefore exposed schema CRUD and assignment to user_admin. v2 corrects this.

3. Design Principles

Additive, not destructive - The existing static field extraction system continues to work unchanged for documents that do not opt into custom schemas. Custom schemas are an additive capability.
Mutual exclusivity - A document uses EITHER the legacy static field extraction system OR the custom schema system, never both. This prevents ambiguity about which system is authoritative for a document's metadata. Once a document is assigned a custom schema, the legacy field extraction endpoints reject writes for that document, and vice versa.
Schema-as-data - Schemas are stored as JSON Schema documents in the database, not as DDL. No migrations needed when clients add/modify schemas.
Immutable schema versions - When a schema is updated, a new version (with new ID) is created. Old versions remain for historical reference. No backward compatibility checking is required between versions since each version gets a distinct ID.
Schema binding is permanent - Once custom metadata is written to a document with a schema ID, that schema ID cannot be changed on the document. This guarantees data integrity.
JSON blob storage - Custom metadata is stored as JSONB in PostgreSQL, validated against the schema at write time. This avoids dynamic column creation.
Client isolation - Schemas are scoped to clients. A client cannot reference another client's schema.
Size limits - Schema definitions are capped at 64KB (MaxSchemaDefinitionBytes = 65536). Metadata payloads are capped at 1MB (MaxMetadataPayloadBytes = 1048576). Bulk folder assignments are capped at 10,000 documents (MaxBulkAssignDocuments = 10000). All are tunable constants.
No JSONB search/filter - The system does NOT support querying or filtering documents by custom metadata values. Custom metadata is opaque storage validated on write. If search is needed in the future, it would require GIN indexes on the JSONB columns and a dedicated query API (separate phase).
super_admin-scoped schema management - Schema CRUD operations, single-document schema assignment, and bulk folder-level schema assignment live under /super-admin/* and require the super_admin role. The lesser user_admin role does NOT have access to schema administration. Custom metadata read/write on documents (/custom-metadata/*) is available to regular client_user users and to both admin roles.
Defense-in-depth - Critical invariants (schema lock after first extraction, client/schema ownership match, schema_id consistency across metadata versions) are enforced at BOTH the application layer AND the database layer via triggers. The app-layer checks provide better error messages; the DB triggers are safety nets against bypass.
No migration from static fields - There is no migration path from the existing static field extraction data into the custom schema system. They are mutually exclusive per document. Out of scope.

4. Database Schema Design

4.1 New Type: `schema_status_type`

A three-state enum that distinguishes between different reasons a schema is no longer assignable:

CREATE TYPE schema_status_type AS ENUM ('active', 'superseded', 'retired');

active: The schema version is current and can be assigned to new documents.
superseded: A newer version of this schema (same name) was created. The schema is still valid for documents already using it, but should not be assigned to new documents. Set automatically when a new version is created via PUT.
retired: An admin explicitly disabled this schema version. Functionally similar to superseded but indicates a deliberate administrative action rather than an automatic version succession.

4.2 New Table: `client_metadata_schemas`

Stores client-defined JSON Schema documents. Each row is an immutable version of a schema definition.

CREATE TABLE client_metadata_schemas (
    id              uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    client_id       varchar(255) NOT NULL REFERENCES clients(clientId),
    name            varchar(255) NOT NULL,
    description     text,
    schema_def      jsonb NOT NULL,          -- The JSON Schema document (max 64KB enforced at app layer)
    version         int NOT NULL DEFAULT 1,   -- Auto-incremented per (client_id, name)
    status          schema_status_type NOT NULL DEFAULT 'active',
    created_at      timestamptz NOT NULL DEFAULT NOW(),
    created_by      varchar(255) NOT NULL,

    CONSTRAINT uq_client_schema_name_version UNIQUE (client_id, name, version),
    CONSTRAINT chk_schema_def_size CHECK (pg_column_size(schema_def) <= 70000)  -- ~68KB to account for JSONB overhead; app layer enforces 64KB on raw JSON
);

CREATE INDEX idx_cms_client_id ON client_metadata_schemas(client_id);
CREATE INDEX idx_cms_client_name ON client_metadata_schemas(client_id, name);
CREATE INDEX idx_cms_client_status ON client_metadata_schemas(client_id, status);

Note

: The 64KB limit is enforced at the application layer (Go constant MaxSchemaDefinitionBytes = 65536) by checking len(rawJSON). The database CHECK constraint uses pg_column_size() which measures on-disk JSONB storage (includes header overhead), so it is set slightly higher (~68KB) as a safety net. The app-layer check provides the user-facing 64KB limit and a better error message.

Design decisions:

name + version scoped to client: Allows human-friendly naming ("aircraft-engineering-v3") with automatic versioning. When a schema is "updated," a new row is inserted with version = max(version) + 1 for that (client_id, name).
schema_def is JSONB: Stores the full JSON Schema document. PostgreSQL can index into it if needed later.
status enum: Three-state lifecycle. superseded is set automatically when a newer version is created. retired is set explicitly by admin action (soft-delete). Both states prevent assignment to new documents but allow existing documents to continue using the schema for validation. This is richer than a boolean is_active because it distinguishes "replaced by v2" from "admin decided this is bad."
id is the schema version ID: This is what documents reference. Each update creates a new id. The (client_id, name, version) tuple provides the human-readable lineage.

4.3 New Table: `document_custom_metadata`

Stores the actual custom metadata for documents, validated against the referenced schema.

CREATE TABLE document_custom_metadata (
    id              uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    document_id     uuid NOT NULL REFERENCES documents(id),
    schema_id       uuid NOT NULL REFERENCES client_metadata_schemas(id),
    metadata        jsonb NOT NULL,           -- The actual custom field values
    version         int NOT NULL DEFAULT 1,   -- Versioned like field extractions
    created_at      timestamptz NOT NULL DEFAULT NOW(),
    created_by      varchar(255) NOT NULL,

    CONSTRAINT uq_doc_custom_metadata_version UNIQUE (document_id, version)
);

CREATE INDEX idx_dcm_document_id ON document_custom_metadata(document_id);
CREATE INDEX idx_dcm_schema_id ON document_custom_metadata(schema_id);
CREATE INDEX idx_dcm_doc_version_desc ON document_custom_metadata(document_id, version DESC);  -- Enables efficient DISTINCT ON in current_document_custom_metadata view

Design decisions:

Single table with version column: We use one table (rather than separate extraction + version tables like the legacy pattern) because the simpler design is sufficient. The legacy two-table pattern exists to support row-level locking on the version counter independently of the data row. For custom metadata, the UNIQUE (document_id, version) constraint plus SELECT ... FOR UPDATE on the max-version row provides the same concurrency safety with less complexity. If this proves insufficient under load, splitting into two tables is a backward-compatible change.
schema_id is immutable per document: Enforced at both the application layer and the database trigger (see Section 4.7). All versions of custom metadata for a document must reference the same schema ID.
metadata is JSONB: Validated against client_metadata_schemas.schema_def at write time. Stored as a single blob for simplicity and query flexibility.

4.4 Alteration: `documents` Table

Add an optional schema binding column:

ALTER TABLE documents
    ADD COLUMN custom_schema_id uuid NULL REFERENCES client_metadata_schemas(id);

CREATE INDEX idx_documents_custom_schema_id ON documents(custom_schema_id);

Behavior:

NULL = document uses the legacy static field extraction system (default, backward-compatible)
NOT NULL = document is opted into the custom schema system; legacy field extraction endpoints will reject writes for this document
Once set AND custom metadata has been written, this field becomes immutable (enforced at app layer AND DB trigger)
Can be set before metadata is written (pre-binding) and changed freely until first metadata write
Must reference a schema owned by the same client as the document (cross-client reference check enforced at app layer AND DB trigger)
Cannot be set on a document that already has legacy field extractions (returns 409 Conflict)

4.5 New View: `current_document_custom_metadata`

CREATE VIEW current_document_custom_metadata AS
SELECT DISTINCT ON (document_id)
    id,
    document_id,
    schema_id,
    metadata,
    version,
    created_at,
    created_by
FROM document_custom_metadata
ORDER BY document_id, version DESC;

4.6 Entity Relationship Diagram

clients
  |
  |-- 1:N --> client_metadata_schemas (per-client schema definitions)
  |              |
  |              |-- referenced by --> documents.custom_schema_id
  |              |-- referenced by --> document_custom_metadata.schema_id
  |
  |-- 1:N --> documents
                 |
                 |-- 1:N --> document_custom_metadata (versioned JSONB blobs)
                 |           [MUTUALLY EXCLUSIVE with legacy extractions]
                 |-- 1:N --> documentFieldExtractions (existing static fields)
                             [MUTUALLY EXCLUSIVE with custom metadata]

4.7 Database Triggers (Defense-in-Depth)

Three triggers enforce critical invariants at the database level, providing a safety net regardless of which code path modifies the data.

Trigger 1: Prevent schema reassignment after first extraction

Fires on documents BEFORE UPDATE of custom_schema_id. Prevents changes if the document has any extraction data (legacy or custom).

CREATE OR REPLACE FUNCTION trg_prevent_schema_reassignment()
RETURNS TRIGGER AS $$
BEGIN
    -- Only fire if custom_schema_id is actually changing
    IF OLD.custom_schema_id IS DISTINCT FROM NEW.custom_schema_id THEN
        -- Check for existing custom metadata
        IF EXISTS (
            SELECT 1 FROM document_custom_metadata
            WHERE document_id = NEW.id
            LIMIT 1
        ) THEN
            RAISE EXCEPTION 'Cannot change custom_schema_id on document % because custom metadata already exists', NEW.id;
        END IF;

        -- Check for existing legacy field extractions
        IF EXISTS (
            SELECT 1 FROM "documentFieldExtractionVersions"
            WHERE "documentId" = NEW.id
            LIMIT 1
        ) THEN
            RAISE EXCEPTION 'Cannot change custom_schema_id on document % because legacy field extractions already exist', NEW.id;
        END IF;
    END IF;

    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_documents_prevent_schema_reassignment
    BEFORE UPDATE OF custom_schema_id ON documents
    FOR EACH ROW
    EXECUTE FUNCTION trg_prevent_schema_reassignment();

Trigger 2: Validate client/schema ownership match

Fires on documents BEFORE INSERT OR UPDATE of custom_schema_id. Ensures the referenced schema belongs to the same client as the document.

CREATE OR REPLACE FUNCTION trg_validate_schema_client_match()
RETURNS TRIGGER AS $$
DECLARE
    schema_client_id varchar(255);
BEGIN
    -- Only check when custom_schema_id is non-null
    IF NEW.custom_schema_id IS NOT NULL THEN
        SELECT client_id INTO schema_client_id
        FROM client_metadata_schemas
        WHERE id = NEW.custom_schema_id;

        IF schema_client_id IS NULL THEN
            RAISE EXCEPTION 'Schema % does not exist', NEW.custom_schema_id;
        END IF;

        IF schema_client_id != NEW."clientId" THEN
            RAISE EXCEPTION 'Schema % belongs to client %, but document % belongs to client %',
                NEW.custom_schema_id, schema_client_id, NEW.id, NEW."clientId";
        END IF;
    END IF;

    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_documents_validate_schema_client
    BEFORE INSERT OR UPDATE OF custom_schema_id ON documents
    FOR EACH ROW
    EXECUTE FUNCTION trg_validate_schema_client_match();

Trigger 3: Enforce consistent schema_id across metadata versions

Fires on document_custom_metadata BEFORE INSERT. Ensures all metadata versions for a document reference the same schema ID. This is the database-level safety net for Invariant 7.

CREATE OR REPLACE FUNCTION trg_enforce_consistent_schema_id()
RETURNS TRIGGER AS $$
DECLARE
    existing_schema_id uuid;
BEGIN
    SELECT schema_id INTO existing_schema_id
    FROM document_custom_metadata
    WHERE document_id = NEW.document_id
    LIMIT 1;

    IF existing_schema_id IS NOT NULL AND existing_schema_id != NEW.schema_id THEN
        RAISE EXCEPTION 'All custom metadata for document % must use schema %, but got %',
            NEW.document_id, existing_schema_id, NEW.schema_id;
    END IF;

    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_dcm_enforce_consistent_schema
    BEFORE INSERT ON document_custom_metadata
    FOR EACH ROW
    EXECUTE FUNCTION trg_enforce_consistent_schema_id();

Note

: These triggers duplicate checks that also exist in the application service layer. This is intentional. The app-layer checks return structured HTTP error responses (400/409). The triggers are a last-resort safety net that fires even if a future code path, migration script, or admin query bypasses the service layer.

5. API Design

5.1 Schema Management Endpoints (`super_admin` only)

All schema CRUD endpoints live under the /super-admin/custom-schemas path prefix. The first path segment super-admin maps (via the existing first-segment resource extraction rule in the Permit.io middleware) to a new Permit.io resource named super-admin. Only the super_admin role holds permissions on this resource. The user_admin role does NOT have access.

These endpoints are grouped under a new OpenAPI tag SuperAdminSchemaService (distinct from the existing AdminService) to make the authorization boundary visible in the generated client and in Swagger.

`POST /super-admin/custom-schemas`

Create a new schema for a client.

Request:

{
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "Custom fields for aircraft engineering documents",
    "schema": {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties": {
            "aircraft_model": {
                "type": "string",
                "maxLength": 100
            },
            "certification_date": {
                "type": "string",
                "format": "date"
            },
            "max_altitude_ft": {
                "type": "number",
                "minimum": 0,
                "maximum": 100000
            },
            "engine_count": {
                "type": "integer",
                "minimum": 1,
                "maximum": 12
            }
        },
        "required": ["aircraft_model"],
        "additionalProperties": false
    },
    "createdBy": "admin@acme.com"
}

Response (201):

{
    "id": "019577a3-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "Custom fields for aircraft engineering documents",
    "schema": { ... },
    "version": 1,
    "status": "active",
    "createdAt": "2026-03-23T12:00:00Z",
    "createdBy": "admin@acme.com"
}

Validation:

The schema field MUST be a valid JSON Schema document (validated server-side using a JSON Schema meta-validator)
The schema field must be <= 64KB in size (returns 400 with message referencing the limit)
Root type must be "object"
Root additionalProperties MUST be explicitly present (either true or false). If omitted, the server returns 400 with a message explaining that additionalProperties must be explicitly declared. This prevents the common mistake of omitting it and inadvertently accepting arbitrary extra fields.
name must be unique within the client (for active schemas at the same version)

`PUT /super-admin/custom-schemas/{schemaId}`

Update an existing schema. Creates a new version with a new ID. The old version remains intact and its status is set to superseded. No backward compatibility check is performed -- the new version is treated as an independent schema that happens to share a name lineage.

Request:

{
    "description": "Updated aircraft engineering fields (added wingspan)",
    "schema": {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties": {
            "aircraft_model": { "type": "string", "maxLength": 100 },
            "certification_date": { "type": "string", "format": "date" },
            "max_altitude_ft": { "type": "number", "minimum": 0, "maximum": 100000 },
            "engine_count": { "type": "integer", "minimum": 1, "maximum": 12 },
            "wingspan_meters": { "type": "number", "minimum": 0 }
        },
        "required": ["aircraft_model"],
        "additionalProperties": false
    },
    "createdBy": "admin@acme.com"
}

Response (201):

{
    "id": "019577b4-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "version": 2,
    "status": "active",
    "previousVersionId": "019577a3-...",
    ...
}

Behavior:

The schemaId in the path identifies the schema to "update" (really: create a new version of)
The new version inherits the name from the previous version
The old version's status is set to superseded (it remains valid for existing documents but should not be assigned to new ones)
Returns the newly created schema version
The schema field in the request body follows the same validation rules as POST (including the additionalProperties requirement)
Concurrency: Uses SELECT ... FOR UPDATE on client_metadata_schemas WHERE client_id = $1 AND name = $2 to lock existing versions before reading max(version). This prevents two concurrent PUTs from computing the same next version number. The unique constraint uq_client_schema_name_version is the final safety net.

`GET /super-admin/custom-schemas`

List all schemas for a client.

Query Parameters:

clientId (required): The client whose schemas to list
name (optional): Filter by schema name
status (optional): Filter by status (active, superseded, retired). Default: return only active.
includeAllVersions (optional, default: false): Return all versions or just latest per name
limit (optional, default: 50, max: 200): Maximum number of results to return
offset (optional, default: 0): Number of results to skip for pagination

Response (200):

{
    "schemas": [
        {
            "id": "019577b4-...",
            "clientId": "acme-corp",
            "name": "aircraft-engineering",
            "description": "...",
            "version": 2,
            "status": "active",
            "documentCount": 45,
            "canDelete": false,
            "createdAt": "...",
            "createdBy": "..."
        },
        {
            "id": "019577c1-...",
            "clientId": "acme-corp",
            "name": "auto-engineering",
            "description": "...",
            "version": 1,
            "status": "active",
            "documentCount": 0,
            "canDelete": true,
            "createdAt": "...",
            "createdBy": "..."
        }
    ]
}

Notes:

documentCount shows how many documents reference each schema version. Computed via a subquery or join.
canDelete is true when documentCount == 0 AND no document_custom_metadata rows reference the schema. This saves the client a round-trip to check before attempting delete.

`GET /super-admin/custom-schemas/{schemaId}`

Get a specific schema by ID (returns full schema definition).

Response (200):

{
    "id": "019577a3-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "...",
    "schema": { ... },
    "version": 1,
    "status": "superseded",
    "documentCount": 45,
    "canDelete": false,
    "createdAt": "...",
    "createdBy": "..."
}

`DELETE /super-admin/custom-schemas/{schemaId}`

Retire a schema. Only succeeds if no documents reference it.

Response (204): No content on success.

Response (409 Conflict):

{
    "error": "Schema is in use by 45 documents and cannot be deleted"
}

Behavior:

Sets status = 'retired'
Fails with 409 if any documents reference this schema ID (via documents.custom_schema_id or document_custom_metadata.schema_id)
Retired schemas still serve validation for existing documents but cannot be assigned to new documents

5.2 Document Schema Assignment Endpoints (`super_admin` only)

Schema assignment endpoints also live under /super-admin/* so that the same first-segment resource extraction rule confines them to the super-admin Permit.io resource. Placing them under /admin/* (v1) or under /document/* would expose them to user_admin or client_user respectively, which we explicitly do not want.

`PATCH /super-admin/documents/{id}/schema`

Assign or change a custom schema on a single document.

Path note: The sub-segment uses plural documents/{id} (not document/{id} as in v1) to avoid any possible collision with the existing top-level /document/{id} resource path during code review or manual URL inspection. This has no authorization impact because the first segment (super-admin) is what the middleware uses to resolve the resource, but it makes the namespacing visually unambiguous.

Request:

{
    "customSchemaId": "019577a3-..."
}

Response (200):

{
    "documentId": "...",
    "customSchemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "schemaVersion": 1
}

Validation rules:

Schema must belong to the same client as the document (enforced at app layer + DB trigger)
Schema must have status active
If the document already has custom metadata written, the schema cannot be changed (returns 409, also enforced by DB trigger)
If the document already has legacy field extractions, the schema cannot be assigned (returns 409 -- mutual exclusivity)
Setting customSchemaId to null removes the binding (only if no custom metadata exists)

`POST /super-admin/folders/{folderId}/assign-schema` (Bulk Folder Assignment)

Assign a custom schema to ALL documents in a folder and all its subfolders (recursive).

Request:

{
    "customSchemaId": "019577a3-...",
    "createdBy": "admin@acme.com"
}

Response (200):

{
    "folderId": "...",
    "customSchemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "documentsUpdated": 87,
    "documentsSkipped": 5,
    "skippedReasons": [
        {"documentId": "...", "reason": "Document already has custom metadata with a different schema"},
        {"documentId": "...", "reason": "Document already has custom metadata with a different schema"},
        {"documentId": "...", "reason": "Document already assigned to same schema (no-op)"},
        {"documentId": "...", "reason": "Document has legacy field extractions"},
        {"documentId": "...", "reason": "Document has legacy field extractions"}
    ]
}

Behavior:

Uses WITH RECURSIVE CTE to walk the entire folder subtree (same pattern as folder deletion)
For each document in the tree:
- If the document has no schema, no custom metadata, and no legacy extractions -> assign the schema
- If the document already has the same schema -> skip (no-op, counted in documentsSkipped)
- If the document has a different schema but no custom metadata yet -> reassign to new schema
- If the document has a different schema AND has custom metadata -> skip with reason (cannot change)
- If the document has legacy field extractions -> skip with reason (mutual exclusivity)
Runs in a single transaction
Document count limit: If the folder subtree contains more than MaxBulkAssignDocuments (10,000) documents, returns 422 Unprocessable Entity with a message explaining the limit. This prevents unbounded transaction duration, excessive lock contention, and replication lag.
Returns summary of what was updated and what was skipped (with reasons)
Requires super_admin role (the super-admin resource grants neither user_admin nor client_user access)

`GET /document/{id}` (Modified)

Add customSchemaId and customMetadata fields to DocumentEnriched:

{
    "id": "...",
    "clientId": "acme-corp",
    "hash": "...",
    "customSchemaId": "019577a3-...",
    "customSchemaName": "aircraft-engineering",
    "hasCustomMetadata": true,
    "customMetadata": { ... },
    ...existing fields...
}

Query parameter: customMetadata (boolean, default: false) - when true, includes the full custom metadata in the response (like the existing textRecord parameter).

Authorization note: The GET /document/{id} endpoint stays on the existing document resource. Both admin roles, client_user, and auditor can read it today. Returning customSchemaId / customSchemaName / customMetadata on this endpoint is read-only and does not grant any administrative capability. The super-admin gating applies only to the write/assign/manage paths.

5.3 Authorization & Permit.io Changes

v2 splits admin-class authorization for mutable metadata into two tiers:

Role	Schema CRUD (`/super-admin/custom-schemas/*`)	Schema Assignment (`/super-admin/documents/{id}/schema`, `/super-admin/folders/{folderId}/assign-schema`)	Metadata Read (`GET /custom-metadata/*`, `GET /document/{id}` enrichment)	Metadata Write (`POST /custom-metadata`)
`super_admin`	Yes	Yes	Yes	Yes
`user_admin`	No	No	Yes	Yes
`auditor`	Read-only (`GET` only)	No	Read-only	No
`client_user`	No	No	Yes	Yes

Key change from v1: user_admin has NO schema CRUD and NO schema assignment rights. The role continues to function exactly as it does today for everything else (user creation, labels, folders, etc.).

New Permit.io resource: `super-admin`

Add a new resource to cmd/auth_related/permit.setup/permit_policies.yaml:

resources:
  # ...existing resources unchanged...

  - name: super-admin
    description: Platform-level administrative functions restricted to super_admin role (custom metadata schema management, schema assignment)
    actions:
      - get
      - post
      - patch
      - delete

Role permission updates

roles:
  - name: super_admin
    description: Full system access for platform administrators
    permissions:
      # ...existing permissions unchanged...
      super-admin:
        - get
        - post
        - patch
        - delete
      custom-metadata:
        - get
        - post

  - name: user_admin
    description: User management - primarily for creating new client users
    permissions:
      # ...existing permissions unchanged (admin, documents, field-extractions, folders)...
      # NOTE: user_admin is intentionally NOT granted super-admin:* permissions.
      custom-metadata:
        - get
        - post

  - name: auditor
    description: Full read-only access across the entire system for auditing and compliance
    permissions:
      # ...existing permissions unchanged...
      super-admin:
        - get
      custom-metadata:
        - get

  - name: client_user
    description: Client-specific access for external users (requires tenant filtering)
    permissions:
      # ...existing permissions unchanged...
      custom-metadata:
        - get
        - post

New custom-metadata resource

The user-facing custom metadata endpoints (/custom-metadata/*) use a separate custom-metadata resource (first path segment). This is unchanged from v1 except that it now also needs to be granted to super_admin explicitly (v1 relied on admin:* coverage which no longer covers the metadata path).

resources:
  - name: custom-metadata
    description: Per-document custom metadata read/write (validated against client schemas)
    actions:
      - get
      - post

Policy mapping documentation additions

Add to the documentation-only policy_mappings section at the bottom of permit_policies.yaml:

super_admin_endpoints:
  - path: /super-admin/custom-schemas
    methods:
      GET: super-admin:get
      POST: super-admin:post
  - path: /super-admin/custom-schemas/{schemaId}
    methods:
      GET: super-admin:get
      PUT: super-admin:patch      # PUT maps to patch action (create-new-version semantics)
      DELETE: super-admin:delete
  - path: /super-admin/documents/{id}/schema
    methods:
      PATCH: super-admin:patch
  - path: /super-admin/folders/{folderId}/assign-schema
    methods:
      POST: super-admin:post

custom_metadata_endpoints:
  - path: /custom-metadata
    methods:
      GET: custom-metadata:get
      POST: custom-metadata:post
  - path: /custom-metadata/version
    methods:
      GET: custom-metadata:get
  - path: /custom-metadata/history
    methods:
      GET: custom-metadata:get

PUT-to-patch action note: The Permit.io middleware lowercases the HTTP method. Permit.io's default action vocabulary does not include put; the existing middleware either maps PUT to patch or requires the action to exist. Confirm during implementation whether the middleware needs a put action added to each resource or whether mapping PUT -> patch is acceptable. If a new action is needed, add put to both super-admin.actions and the super_admin role permissions. The same consideration applies to any other resources that currently have no put action.

Middleware impact

The Permit.io resource-extraction middleware uses the first path segment. Introducing super-admin as a resource requires no middleware code changes -- it is purely a configuration addition via the permit setup tool. The setup tool (cmd/auth_related/permit.setup/) must be re-run in each environment (dev, uat, prod) to provision the new resource and role permissions before these endpoints can pass authorization.

Deployment ordering

Because the authorization changes must land in Permit.io before the API routes are live, the deployment sequence is:

Update permit_policies.yaml with the new resources and role grants.
Run the permit setup tool against dev/uat/prod to provision the new resource and permissions.
Deploy the API code containing the new endpoints.

Step 2 must complete before step 3 in each environment, otherwise the new endpoints would either be rejected by Permit.io (if fail-closed) or accidentally allowed (if fail-open) during the window between code deploy and permit sync.

5.4 Custom Metadata CRUD Endpoints (User-Facing)

All under a new CustomMetadataService tag. These endpoints are NOT moved under /super-admin/* because they are intended for regular users (including client_user) to read and write the custom metadata values on documents. Schema administration and schema assignment are separated from value read/write precisely so the day-to-day data entry workload does not require super_admin.

`POST /custom-metadata`

Create or update custom metadata for a document. Creates a new version.

Request:

{
    "documentId": "...",
    "metadata": {
        "aircraft_model": "Boeing 737-800",
        "certification_date": "2024-06-15",
        "max_altitude_ft": 41000,
        "engine_count": 2
    },
    "createdBy": "analyst@acme.com"
}

Response (201):

{
    "id": "...",
    "documentId": "...",
    "schemaId": "019577a3-...",
    "metadata": { ... },
    "version": 1,
    "createdAt": "...",
    "createdBy": "analyst@acme.com"
}

Validation:

Document must have a custom_schema_id assigned (400 if not -- a super_admin must have assigned the schema first)
Document must NOT have any legacy field extractions (409 if it does -- mutual exclusivity, also enforced by DB trigger on schema assignment)
The metadata payload must be <= 1MB in size (MaxMetadataPayloadBytes = 1048576). Returns 400 if exceeded.
The metadata payload is validated against the JSON Schema stored in client_metadata_schemas.schema_def
The request body does NOT include schemaId -- the server derives it from the document's custom_schema_id. This prevents client/server drift.

If validation fails, returns 400 with detailed validation errors:

{
    "error": "Metadata does not conform to schema",
    "validationErrors": [
        {"field": "engine_count", "message": "must be >= 1, got 0"},
        {"field": "aircraft_model", "message": "required field missing"}
    ]
}

On first write, locks the document's custom_schema_id (makes it immutable via app-layer flag + DB trigger)
Subsequent writes create new versions but must use the same schema

`GET /custom-metadata`

Get current (latest version) custom metadata for a document.

Query Parameters:

documentId (required)

Response (200):

{
    "id": "...",
    "documentId": "...",
    "schemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "metadata": { ... },
    "version": 3,
    "createdAt": "...",
    "createdBy": "..."
}

`GET /custom-metadata/version`

Get a specific version of custom metadata.

Query Parameters:

documentId (required)
version (required, >= 1)

`GET /custom-metadata/history`

Get version history for a document's custom metadata.

Query Parameters:

documentId (required)
limit (optional, default: 50, max: 200): Maximum number of versions to return
offset (optional, default: 0): Number of versions to skip for pagination

Response (200):

{
    "versions": [
        {"id": "...", "documentId": "...", "version": 3, "createdAt": "...", "createdBy": "..."},
        {"id": "...", "documentId": "...", "version": 2, "createdAt": "...", "createdBy": "..."},
        {"id": "...", "documentId": "...", "version": 1, "createdAt": "...", "createdBy": "..."}
    ]
}

6. Validation Strategy

6.1 JSON Schema Validation

Use a Go JSON Schema validation library for both:

Schema validation (meta-validation): When a super_admin submits a new schema, validate it against the JSON Schema meta-schema to ensure it is a well-formed JSON Schema document.
Metadata validation: When custom metadata is submitted, validate the payload against the document's assigned schema.

Recommended library: github.com/santhosh-tekuri/jsonschema/v6

Reasons:

Supports JSON Schema draft 2020-12
Active maintenance
Good error reporting
Supports custom formats

6.2 Schema Definition Requirements

When a schema definition is submitted (via POST or PUT), the server validates:

It is a valid JSON object
It compiles successfully as a JSON Schema document (meta-validation)
Root type is "object"
Root additionalProperties is explicitly present (either true or false). This is required because JSON Schema defaults additionalProperties to true when omitted, which silently accepts any extra fields. Requiring explicit declaration forces a conscious choice.
Size is <= 64KB

6.3 Supported JSON Schema Features

The system will support the following JSON Schema keywords for custom field definitions:

Feature	JSON Schema Keyword	Example
Data type	`type`	`"string"`, `"number"`, `"integer"`, `"boolean"`, `"array"`, `"object"`
Required fields	`required`	`["field1", "field2"]`
String length	`minLength`, `maxLength`	`"maxLength": 100`
Numeric range	`minimum`, `maximum`	`"minimum": 0`
Pattern matching	`pattern`	`"pattern": "^[A-Z]{2}$"`
Enum values	`enum`	`"enum": ["active", "inactive"]`
Date/time format	`format`	`"format": "date"`
Array items	`items`, `minItems`, `maxItems`	Nested array support
Nested objects	`properties`	Hierarchical field groups
No extra fields	`additionalProperties`	`false` to enforce strict schemas

6.4 Validation Error Reporting

Validation errors will be returned as structured JSON with:

The JSON pointer to the failing field (/aircraft_model)
The constraint that was violated (maxLength)
A human-readable message

7. Service Layer Design

7.1 New Service: `internal/customschema/`

internal/customschema/
    service.go       -- CustomSchemaService (CRUD + validation)
    models.go        -- Input/output types
    validator.go     -- JSON Schema meta-validation + payload validation
    constants.go     -- MaxSchemaDefinitionBytes and other limits
    service_test.go  -- Integration tests

Constants (constants.go):

const (
    // MaxSchemaDefinitionBytes is the maximum size of a JSON Schema definition.
    // This limit can be increased if clients need larger schemas.
    MaxSchemaDefinitionBytes = 65536 // 64KB

    // MaxMetadataPayloadBytes is the maximum size of a custom metadata JSON payload.
    // Prevents unbounded JSONB blobs when schemas use additionalProperties: true.
    MaxMetadataPayloadBytes = 1048576 // 1MB

    // MaxBulkAssignDocuments is the maximum number of documents that can be updated
    // in a single bulk folder schema assignment. Prevents unbounded transactions.
    MaxBulkAssignDocuments = 10000
)

Key methods:

type Service struct {
    cfg       serviceconfig.ConfigProvider
    validator *SchemaValidator
}

// Schema management (super_admin operations)
func (s *Service) CreateSchema(ctx, input CreateSchemaInput) (*Schema, error)
func (s *Service) UpdateSchema(ctx, schemaID uuid.UUID, input UpdateSchemaInput) (*Schema, error)
func (s *Service) GetSchema(ctx, schemaID uuid.UUID) (*Schema, error)
func (s *Service) ListSchemas(ctx, clientID string, filters ListFilters) ([]SchemaSummary, error)
func (s *Service) DeleteSchema(ctx, schemaID uuid.UUID) error

// Metadata operations (client_user / admin operations)
func (s *Service) SetDocumentMetadata(ctx, input SetMetadataInput) (*CustomMetadata, error)
func (s *Service) GetCurrentMetadata(ctx, documentID uuid.UUID) (*CustomMetadata, error)
func (s *Service) GetMetadataByVersion(ctx, documentID uuid.UUID, version int) (*CustomMetadata, error)
func (s *Service) GetMetadataHistory(ctx, documentID uuid.UUID) ([]MetadataVersion, error)

// Document schema binding (super_admin operations)
func (s *Service) AssignSchema(ctx, documentID uuid.UUID, schemaID uuid.UUID) error
func (s *Service) AssignSchemaToFolder(ctx, folderID uuid.UUID, schemaID uuid.UUID, createdBy string) (*BulkAssignResult, error)

BulkAssignResult:

type BulkAssignResult struct {
    FolderID         uuid.UUID
    SchemaID         uuid.UUID
    DocumentsUpdated int
    DocumentsSkipped int
    SkippedReasons   []SkippedDocument
}

type SkippedDocument struct {
    DocumentID uuid.UUID
    Reason     string
}

Authorization is enforced at the HTTP middleware layer via Permit.io, not inside these service methods. The service methods are agnostic to the caller's role. The routing layer for /super-admin/* endpoints is what ensures only super_admin can invoke CreateSchema, UpdateSchema, DeleteSchema, AssignSchema, and AssignSchemaToFolder. Keeping the service layer role-unaware preserves testability and makes integration tests simpler.

7.2 Validator Component

type SchemaValidator struct{}

// ValidateSchemaDefinition checks that a schema_def is valid JSON Schema.
// Returns error if: not a valid JSON object, fails meta-validation,
// root type is not "object", or additionalProperties is not explicitly present.
func (v *SchemaValidator) ValidateSchemaDefinition(schemaDef json.RawMessage) error

// ValidateMetadata checks that metadata conforms to a compiled schema
func (v *SchemaValidator) ValidateMetadata(schemaDef json.RawMessage, metadata json.RawMessage) ([]ValidationError, error)

7.3 Mutual Exclusivity Guard

The service layer enforces mutual exclusivity at two points:

Schema assignment (AssignSchema): Before setting custom_schema_id on a document, check that no rows exist in documentFieldExtractionVersions for that document. If they do, return 409 Conflict.
Legacy field extraction write (modification to existing FieldExtractionService): Before creating a new field extraction, check that documents.custom_schema_id IS NULL for the target document. If it is not null, return 409 Conflict with a message explaining that the document uses the custom schema system.

Note

: The DB trigger on documents.custom_schema_id (Section 4.7, Trigger 1) also prevents schema assignment if legacy extractions exist. This is the defense-in-depth pattern: the service layer returns a clean 409; the trigger catches anything that slips through.

7.4 API Controller: `api/queryAPI/customschemas.go`

Follows the same pattern as fieldextractions.go:

Parse and validate request
Call service layer
Convert between API and service types
Return appropriate HTTP status codes

The controller file holds all /super-admin/custom-schemas/*, /super-admin/documents/{id}/schema, and /super-admin/folders/{folderId}/assign-schema handlers. A separate controller file api/queryAPI/custommetadata.go holds the user-facing /custom-metadata/* handlers. Separating them by file mirrors the separation of authorization tiers and makes it obvious at code-review time which handlers are super_admin-gated.

8. Migration Plan

8.1 Migration Files

Four new migrations:

Migration 120: Create schema_status_type enum + client_metadata_schemas table

00000000000120_create_client_metadata_schemas.up.sql
00000000000120_create_client_metadata_schemas.down.sql

Migration 121: Create document_custom_metadata table + view

00000000000121_create_document_custom_metadata.up.sql
00000000000121_create_document_custom_metadata.down.sql

Migration 122: Add custom_schema_id to documents table

00000000000122_add_custom_schema_id_to_documents.up.sql
00000000000122_add_custom_schema_id_to_documents.down.sql

Migration 123: Add database triggers for invariant enforcement

00000000000123_add_schema_invariant_triggers.up.sql
00000000000123_add_schema_invariant_triggers.down.sql

Contents:

trg_prevent_schema_reassignment function + trigger (Section 4.7, Trigger 1)
trg_validate_schema_client_match function + trigger (Section 4.7, Trigger 2)
trg_enforce_consistent_schema_id function + trigger (Section 4.7, Trigger 3)
Down migration drops triggers and functions

8.2 SQLC Queries

New query file: internal/database/queries/customschemas.sql

Schema operations:

CreateClientMetadataSchema - Insert new schema
GetClientMetadataSchema - Get by ID
ListClientMetadataSchemas - List by client with filters (status filter)
GetLatestSchemaByName - Get latest version of a named schema
GetSchemaDocumentCount - Count documents using a schema
GetSchemaMetadataRecordCount - Count metadata records referencing a schema (for canDelete)
SetSchemaStatus - Update status (for supersede/retire)
GetMaxSchemaVersion - For auto-incrementing version
LockSchemaVersionsForName - SELECT ... FOR UPDATE on all versions of a (client_id, name) pair to prevent concurrent version creation race

Metadata operations:

CreateDocumentCustomMetadata - Insert new metadata version
GetCurrentDocumentCustomMetadata - Latest version (via view)
GetDocumentCustomMetadataByVersion - Specific version
GetDocumentCustomMetadataHistory - All versions
LockDocumentCustomMetadataForVersion - Row-level locking (SELECT FOR UPDATE on max version)

Document modifications:

SetDocumentCustomSchemaId - Update documents.custom_schema_id
GetDocumentCustomSchemaId - Read documents.custom_schema_id
BulkSetDocumentCustomSchemaIdInFolderTree - Set schema for all docs in folder subtree (WITH RECURSIVE)
GetDocumentsWithMetadataInFolderTree - Find docs that already have metadata (for skip reporting)
GetDocumentsWithLegacyExtractionsInFolderTree - Find docs that have legacy extractions (for skip reporting)
DocumentHasLegacyExtractions - Check if a single document has any legacy field extraction versions
CountDocumentsInFolderTree - Count total documents in a folder subtree (for MaxBulkAssignDocuments limit check)

Delete cascade operations (additions to document.sql and client.sql):

DeleteDocumentCustomMetadata - Delete all custom metadata versions for a document
NullifyDocumentCustomSchemaId - Set custom_schema_id = NULL on a document (before hard delete)
DeleteClientMetadataSchemas - Delete all schemas for a client (after all documents deleted)

8.3 OpenAPI Spec Changes

Add to serviceAPIs/queryAPI.yaml:

New SuperAdminSchemaService tag for super_admin-only endpoints:

POST /super-admin/custom-schemas
GET /super-admin/custom-schemas
GET /super-admin/custom-schemas/{schemaId}
PUT /super-admin/custom-schemas/{schemaId}
DELETE /super-admin/custom-schemas/{schemaId}
PATCH /super-admin/documents/{id}/schema
POST /super-admin/folders/{folderId}/assign-schema

New CustomMetadataService tag for end-user endpoints:

GET /custom-metadata
POST /custom-metadata
GET /custom-metadata/version
GET /custom-metadata/history

New schemas: CustomSchemaRequest, CustomSchemaResponse, CustomSchemaListResponse, CustomMetadataRequest, CustomMetadataResponse, CustomMetadataHistoryResponse, ValidationErrorResponse, BulkSchemaAssignRequest, BulkSchemaAssignResponse, DocumentSchemaAssignRequest, DocumentSchemaAssignResponse

New enum: SchemaStatus with values active, superseded, retired

Modified schemas: DocumentEnriched (add customSchemaId, customSchemaName, hasCustomMetadata, customMetadata fields)

Tag descriptions should explicitly call out the authorization boundary:

SuperAdminSchemaService description: "Custom metadata schema management and assignment. Restricted to the super_admin role. Not accessible to user_admin."
CustomMetadataService description: "Per-document custom metadata read/write. Accessible to client_user, user_admin, and super_admin."

8.4 Legacy Field Extraction Guard

Add a check to the existing field extraction service (internal/fieldextraction/service.go):

Before creating a new field extraction, verify that documents.custom_schema_id IS NULL for the target document. If the document has a custom schema assigned, return a 409 Conflict error explaining that the document uses the custom schema system and legacy field extractions are not allowed.

This is a small modification to the existing code path, not a new endpoint.

8.5 Delete Path Updates (Cascade Cleanup)

The new tables and FKs require updates to existing delete paths. Without these changes, hard-deleting a document or client will fail with FK violations.

Document Delete Cascade

The existing DeleteDocumentCascade function (in internal/document/) and the SQLC queries in internal/database/queries/document.sql must be updated to delete custom metadata before deleting the document:

New SQLC queries to add to document.sql:

DeleteDocumentCustomMetadata - DELETE FROM document_custom_metadata WHERE document_id = @document_id
NullifyDocumentCustomSchemaId - UPDATE documents SET custom_schema_id = NULL WHERE id = @document_id

Updated delete order in DeleteDocumentCascade:

Delete documentFieldExtractionArrayFields (existing)
Delete documentFieldExtractionVersions (existing)
Delete documentFieldExtractions (existing)
Delete document_custom_metadata (new -- must come before document row delete)
Nullify documents.custom_schema_id (new -- removes FK to client_metadata_schemas before document delete)
Delete documentCleanEntries / documentCleans / documentEntries (existing)
Delete documents row (existing)

Note

: Step 5 (nullify custom_schema_id) must happen before step 7 because the FK from documents.custom_schema_id to client_metadata_schemas would block deletion otherwise. The nullify is safe because step 4 already removed all custom metadata, and Trigger 1 only fires on changes when metadata exists.

Client Delete Cascade

The existing client.HardDelete and deleteClientDependencies functions must be updated to clean up schema data after all documents are deleted:

New SQLC queries:

DeleteClientMetadataSchemas - DELETE FROM client_metadata_schemas WHERE client_id = @client_id

Updated delete order in client.HardDelete:

Delete all documents with full cascade (existing, now includes custom metadata cleanup per above)
Delete documentUploads (existing)
Delete all folders (existing)
Delete client_metadata_schemas (new -- safe after all documents are gone since no FKs reference these rows)
Delete collector data, batch uploads, sync records (existing)
Delete client row (existing)

8.6 Permit.io Setup Tool Run

After updating permit_policies.yaml per Section 5.3, run the permit setup tool in each environment:

cmd/auth_related/permit.setup/run.tool.dev.sh
cmd/auth_related/permit.setup/run.tool.uat.sh
cmd/auth_related/permit.setup/run.tool.prod.sh

The tool is idempotent and will provision the new super-admin and custom-metadata resources and update role grants without disturbing existing resources. Run this before deploying the API code so the new endpoints are authorized the moment they go live.

9. Invariants & Business Rules

#	Rule	Enforcement
1	Schema must be valid JSON Schema with explicit `additionalProperties`	Application: meta-validation on create/update
2	Schema `name` + `version` unique per client	Database: unique constraint
3	Updated schema gets new ID and version; old version becomes `superseded`	Application: INSERT new row, UPDATE old status
4	Schema deletion blocked if documents reference it	Application: count check before retirement
5	Document's `custom_schema_id` is immutable once any extraction exists	Application: check in AssignSchema + DB trigger: `trg_prevent_schema_reassignment`
6	Custom metadata must conform to assigned schema	Application: JSON Schema validation on write
7	All custom metadata versions for a document use same schema ID	Application: consistency check on write + DB trigger: `trg_enforce_consistent_schema_id`
8	Schema must belong to same client as document	Application: cross-reference check + DB trigger: `trg_validate_schema_client_match`
9	Only `active` schemas can be assigned to new documents	Application: status check
10	Custom metadata is versioned (no in-place updates)	Database: version column + unique constraint
11	Schema definition must be <= 64KB	Application: size check + Database: CHECK constraint
12	Schema management requires `super_admin` role (NOT `user_admin`)	Permit.io: role-based access on new `super-admin` resource; `user_admin` is explicitly NOT granted this resource
13	No query/filter on custom metadata values	By design: not implemented (see Principle 9)
14	A document uses EITHER legacy extractions OR custom metadata, never both	Application: mutual exclusivity checks in both services + DB trigger: `trg_prevent_schema_reassignment`
15	Legacy field extraction writes rejected if document has `custom_schema_id` set	Application: guard in `FieldExtractionService`
16	Custom schema assignment rejected if document has legacy extractions	Application: guard in `AssignSchema` + DB trigger
17	Metadata payload must be <= 1MB	Application: size check (`MaxMetadataPayloadBytes`)
18	Bulk folder assignment capped at 10,000 documents	Application: count check (`MaxBulkAssignDocuments`), returns 422 if exceeded
19	Schema assignment (single doc + bulk folder) requires `super_admin` role	Permit.io: endpoints under `/super-admin/` prefix map to `super-admin` resource; `user_admin` is NOT granted
20	Delete cascade must clean up custom metadata and schema bindings	Application: updated `DeleteDocumentCascade` + `deleteClientDependencies` (Section 8.5)
21	Metadata read/write is available to `client_user`, `user_admin`, `super_admin` (not `auditor` for writes)	Permit.io: `custom-metadata` resource grants on each role per Section 5.3

10. Relationship to Existing Field Extraction System

The existing static field extraction system (/field-extractions) and the new custom schema system (/super-admin/custom-schemas + /custom-metadata) are mutually exclusive per document:

A document uses EITHER the legacy static field extraction system OR the custom schema system
A document CANNOT have both legacy field extractions AND custom metadata
Assigning a custom schema to a document that already has legacy extractions returns 409 Conflict
Writing a legacy field extraction to a document that has a custom schema assigned returns 409 Conflict
Documents without a custom schema (the default) continue to use the legacy system unchanged
There is no migration path from legacy to custom for individual documents. If a document needs custom metadata, it should not have legacy extractions written to it first.

Rationale: This prevents ambiguity about which system is "authoritative" for a document's metadata. With mutual exclusivity, there is exactly one source of truth per document. This is simpler to reason about, simpler to query, and avoids the data governance problem of conflicting data in two stores.

No breaking changes to the existing API or database schema for documents that do not opt into custom schemas. The existing /field-extractions endpoints remain on the field-extractions resource and are unaffected by the new super-admin resource.

11. Implementation Order

Phase	Work	Dependencies
1	Database migrations (enum, tables, view, document column, triggers)	None
2	SQLC queries + `task generate`	Phase 1
3	Schema validator component (`internal/customschema/validator.go`)	JSON Schema library added to go.mod
4	Service layer (`internal/customschema/service.go`) incl. bulk folder assignment + mutual exclusivity guards	Phase 2, 3
5a	Legacy field extraction guard (add check to existing `FieldExtractionService`)	Phase 2
5b	Delete cascade updates (`DeleteDocumentCascade`, `deleteClientDependencies`, new SQLC queries)	Phase 2
6	OpenAPI spec updates (`SuperAdminSchemaService` + `CustomMetadataService` tags)	None (can parallel with 1-3)
7a	Permit.io policy file updates (`permit_policies.yaml`: new `super-admin` + `custom-metadata` resources, role grants)	None (can parallel with 1-6)
7b	Run permit setup tool against dev/uat/prod to provision new resources and role permissions	Phase 7a
8	API controllers + code generation (super_admin schema CRUD in `customschemas.go`, metadata CRUD in `custommetadata.go`)	Phase 4, 6
9	Document endpoint modifications (`PATCH /super-admin/documents/{id}/schema`, `GET /document/{id}` enrichment)	Phase 4, 8
10	Bulk folder schema assignment endpoint (`POST /super-admin/folders/{folderId}/assign-schema`)	Phase 4, 8
11	Integration tests (including explicit role-based authorization tests per Section 12)	Phase 10, 7b
12	Documentation updates (`docs/ai.generated/`) incl. sample schemas, authorization matrix	Phase 11

Ordering constraint: Phase 7b (permit setup run) must complete in each environment before Phase 8 code is deployed to that environment. Otherwise the new endpoints will be rejected by Permit.io (fail-closed) until the setup runs.

12. Testing Strategy

Schema validation tests: Valid and invalid JSON Schema documents, edge cases (empty schema, deeply nested, self-referential, oversized >64KB, missing additionalProperties)
Metadata validation tests: Conforming and non-conforming payloads, all supported JSON Schema keywords
Service integration tests: Full CRUD lifecycle using testcontainers (real PostgreSQL)
API integration tests: HTTP-level tests for all new endpoints under both /super-admin/* and /custom-metadata/*
Invariant tests: Schema immutability, cross-client isolation, deletion guards, size limits
DB trigger tests: Verify triggers fire correctly when bypassing the service layer (direct SQL inserts/updates via testcontainers)
Schema status lifecycle tests: active -> superseded on version create, active -> retired on delete, assignment rejected for non-active schemas
Mutual exclusivity tests: Legacy extraction blocked when schema assigned, schema assignment blocked when legacy extractions exist, bulk folder assignment skips documents with legacy extractions
Bulk folder assignment tests: Recursive assignment, skip behavior for locked docs and legacy docs, empty folders, deeply nested folder trees
Authorization tests (expanded for v2):
- super_admin CAN create, list, get, update, delete schemas, assign schemas to documents, assign schemas to folders
- user_admin CANNOT access any /super-admin/* endpoint (expect 403 on every method + path combination)
- user_admin CAN read and write custom metadata via /custom-metadata (no regression)
- client_user CANNOT access any /super-admin/* endpoint (expect 403)
- client_user CAN read and write custom metadata via /custom-metadata
- auditor CAN read (GET) /super-admin/custom-schemas and /super-admin/custom-schemas/{schemaId} but CANNOT POST/PUT/PATCH/DELETE
- auditor CAN read but CANNOT write /custom-metadata
- Each test should assert both the HTTP status code AND that the database state is unchanged on denial
Backward compatibility tests: Existing field extraction endpoints continue to work unchanged for documents without custom schemas; existing /admin/* endpoints are unaffected by the new super-admin resource
Delete cascade tests: Document hard-delete succeeds when document has custom metadata, client hard-delete succeeds when client has schemas and metadata, verify no orphaned rows remain
Metadata size limit tests: Payloads at/above 1MB boundary, verify 400 response with clear message
Bulk assignment limit tests: Folder trees exceeding MaxBulkAssignDocuments, verify 422 response
Schema version concurrency tests: Concurrent PUT requests for same schema name produce distinct version numbers
Permit.io setup tool tests: After running the setup tool against a disposable dev tenant, verify that the super-admin and custom-metadata resources exist with the expected actions and that each role has exactly the grants defined in Section 5.3

13. Resolved Design Decisions

#	Question	Decision
1	Schema size limits	64KB cap via `MaxSchemaDefinitionBytes` constant (tunable)
2	Schema backward compatibility	Not required. New version = new ID. Old docs keep old schema.
3	Bulk operations	Yes. `POST /super-admin/folders/{folderId}/assign-schema` assigns recursively through folder tree.
4	Custom metadata search/filter	Not supported. No GIN indexes, no query API. Metadata is opaque storage.
5	Schema templates	No system templates. Documentation will include sample schemas covering all supported types.
6	Existing field migration path	Out of scope. Static fields and custom schemas are mutually exclusive per document.
7	Endpoint naming/placement	Schema CRUD under `/super-admin/custom-schemas` (requires the `super_admin` role on the new `super-admin` Permit.io resource). Metadata CRUD under `/custom-metadata` (regular `client_user` access; also granted to `user_admin` and `super_admin`).
8	Legacy/custom coexistence	Mutually exclusive. A document uses EITHER legacy OR custom, never both. Simplifies data governance and prevents source-of-truth ambiguity.
9	Schema status model	Three-state enum (`active`/`superseded`/`retired`) instead of boolean. Distinguishes automatic version succession from explicit admin retirement.
10	DB-level invariant enforcement	Yes. Three database triggers enforce schema lock, client ownership match, and consistent schema_id as defense-in-depth alongside application-layer checks.
11	`additionalProperties` in schemas	Required to be explicitly present (not necessarily `false`). Prevents the common mistake of omitting it and silently accepting arbitrary extra fields.
12	Metadata table design	Single table with version column (simpler than the legacy two-table pattern). Sufficient for our concurrency needs. Can split later if needed.
13	Schema assignment endpoint path	Under `/super-admin/documents/{id}/schema` (NOT `/admin/document/{id}/schema` as in v1 and NOT `/document/{id}/schema`). The `/super-admin` first segment maps to the new `super-admin` Permit.io resource which only `super_admin` holds. Placing it under `/admin` would also grant `user_admin`; placing it under `/document` would also grant `client_user`.
14	Metadata payload size limit	1MB cap via `MaxMetadataPayloadBytes` constant. Prevents unbounded JSONB blobs when schemas allow `additionalProperties: true`.
15	Bulk assignment document limit	10,000 documents via `MaxBulkAssignDocuments` constant. Prevents unbounded transactions and lock contention.
16	Schema_id consistency trigger	DB trigger `trg_enforce_consistent_schema_id` on `document_custom_metadata` INSERT enforces same schema across all metadata versions for a document (defense-in-depth for Invariant 7).
17	v2: Who can administer schemas?	Only `super_admin`. v1 unintentionally granted `user_admin` access by placing endpoints under `/admin/`. v2 moves them to `/super-admin/` and introduces a new `super-admin` Permit.io resource that only `super_admin` holds. `user_admin` retains its existing capabilities (user creation, field extractions, documents/folders, admin endpoints) but has no schema administration rights.
18	v2: Why a new resource instead of path-specific rules?	The Permit.io middleware resolves resources from the first path segment. Adding path-specific overrides would require middleware code changes. Adding a new first-segment resource (`super-admin`) is a pure configuration change and keeps the middleware simple.
19	v2: Does the service layer enforce role checks?	No. Authorization lives in the HTTP/Permit.io layer. The service layer (`internal/customschema/`) is role-agnostic and remains easy to test without fake identity contexts.

14. v1 -> v2 Change Log

For reviewers comparing to plans/mutable.metadata.plan.combo.md:

Area	v1	v2
Schema CRUD path	`/admin/custom-schemas`	`/super-admin/custom-schemas`
Single-doc schema assignment path	`/admin/document/{id}/schema`	`/super-admin/documents/{id}/schema`
Bulk folder assignment path	`/admin/folders/{folderId}/assign-schema`	`/super-admin/folders/{folderId}/assign-schema`
Permit.io resource for schema admin	`admin` (shared by `user_admin` and `super_admin`)	NEW `super-admin` (held only by `super_admin`)
`user_admin` schema CRUD	Allowed (unintentional)	Denied
`user_admin` schema assignment	Allowed (unintentional)	Denied
`user_admin` custom metadata read/write	Allowed	Allowed (unchanged)
`client_user` custom metadata read/write	Allowed	Allowed (unchanged)
`auditor` read schemas / metadata	Allowed (read-only)	Allowed (read-only, now via `super-admin:get` grant)
OpenAPI tag for schema admin	`AdminService`	NEW `SuperAdminSchemaService`
Service layer	Role-agnostic	Role-agnostic (unchanged)
Database schema, triggers, migrations	Unchanged	Unchanged
Validation strategy, size limits, constants	Unchanged	Unchanged

No database migration or data change is required to move from v1 to v2 -- this is strictly a routing and authorization change. If v1 were already deployed, the migration would be: (1) add the new Permit.io resource and grants, (2) add the new endpoints alongside the old ones, (3) point clients at the new paths, (4) remove the old paths once all clients have migrated. Since v1 is not deployed, v2 simply supersedes it.

Appendix A: Sample JSON Schemas

These examples demonstrate all supported JSON Schema features and data types. They should be included in the API documentation.

A.1 Comprehensive Schema (All Supported Types)

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "document_title": {
            "type": "string",
            "minLength": 1,
            "maxLength": 500,
            "description": "Title of the document"
        },
        "document_code": {
            "type": "string",
            "pattern": "^[A-Z]{3}-[0-9]{4}$",
            "description": "Document code in format XXX-0000"
        },
        "status": {
            "type": "string",
            "enum": ["draft", "review", "approved", "rejected", "archived"],
            "description": "Current document status"
        },
        "review_date": {
            "type": "string",
            "format": "date",
            "description": "Date of last review (YYYY-MM-DD)"
        },
        "created_timestamp": {
            "type": "string",
            "format": "date-time",
            "description": "Creation timestamp (ISO 8601)"
        },
        "page_count": {
            "type": "integer",
            "minimum": 1,
            "maximum": 10000,
            "description": "Total number of pages"
        },
        "confidence_score": {
            "type": "number",
            "minimum": 0.0,
            "maximum": 1.0,
            "description": "Extraction confidence (0.0 to 1.0)"
        },
        "is_verified": {
            "type": "boolean",
            "description": "Whether the extraction has been human-verified"
        },
        "tags": {
            "type": "array",
            "items": { "type": "string", "maxLength": 50 },
            "minItems": 0,
            "maxItems": 20,
            "description": "Classification tags"
        },
        "numeric_values": {
            "type": "array",
            "items": { "type": "number" },
            "maxItems": 100,
            "description": "Extracted numeric values"
        },
        "contact_info": {
            "type": "object",
            "properties": {
                "name": { "type": "string", "maxLength": 200 },
                "email": { "type": "string", "format": "email" },
                "phone": { "type": "string", "pattern": "^\\+?[0-9\\-\\s()]+$" }
            },
            "required": ["name"],
            "additionalProperties": false,
            "description": "Primary contact information"
        }
    },
    "required": ["document_title", "status"],
    "additionalProperties": false
}

A.2 Aircraft Engineering Schema (Domain Example)

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "aircraft_model": {
            "type": "string",
            "maxLength": 100
        },
        "manufacturer": {
            "type": "string",
            "enum": ["Boeing", "Airbus", "Embraer", "Bombardier", "Other"]
        },
        "certification_date": {
            "type": "string",
            "format": "date"
        },
        "max_altitude_ft": {
            "type": "integer",
            "minimum": 0,
            "maximum": 100000
        },
        "max_range_nm": {
            "type": "number",
            "minimum": 0
        },
        "engine_count": {
            "type": "integer",
            "minimum": 1,
            "maximum": 12
        },
        "engine_type": {
            "type": "string",
            "enum": ["turbofan", "turboprop", "turbojet", "piston", "electric"]
        },
        "wingspan_meters": {
            "type": "number",
            "minimum": 0,
            "maximum": 100
        },
        "is_pressurized": {
            "type": "boolean"
        },
        "applicable_regulations": {
            "type": "array",
            "items": {
                "type": "string",
                "pattern": "^(FAR|EASA|CAAC)-[0-9A-Z\\.]+$"
            },
            "maxItems": 50
        }
    },
    "required": ["aircraft_model", "manufacturer"],
    "additionalProperties": false
}

A.3 Auto Engineering Schema (Domain Example)

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "vehicle_make": {
            "type": "string",
            "maxLength": 100
        },
        "vehicle_model": {
            "type": "string",
            "maxLength": 100
        },
        "model_year": {
            "type": "integer",
            "minimum": 1900,
            "maximum": 2100
        },
        "vin": {
            "type": "string",
            "pattern": "^[A-HJ-NPR-Z0-9]{17}$",
            "description": "17-character Vehicle Identification Number"
        },
        "engine_displacement_liters": {
            "type": "number",
            "minimum": 0,
            "maximum": 20
        },
        "horsepower": {
            "type": "integer",
            "minimum": 0
        },
        "transmission_type": {
            "type": "string",
            "enum": ["manual", "automatic", "cvt", "dct", "ev"]
        },
        "is_electric": {
            "type": "boolean"
        },
        "safety_ratings": {
            "type": "object",
            "properties": {
                "nhtsa_overall": { "type": "integer", "minimum": 1, "maximum": 5 },
                "iihs_rating": { "type": "string", "enum": ["Poor", "Marginal", "Acceptable", "Good", "Good+"] }
            },
            "additionalProperties": false
        },
        "recall_ids": {
            "type": "array",
            "items": { "type": "string", "maxLength": 20 },
            "maxItems": 100
        }
    },
    "required": ["vehicle_make", "vehicle_model", "model_year"],
    "additionalProperties": false
}

Supported JSON Schema Data Types Summary

JSON Schema Type	Go Equivalent	Example Constraint Keywords
`string`	`string`	`minLength`, `maxLength`, `pattern`, `format`, `enum`
`integer`	`int64`	`minimum`, `maximum`
`number`	`float64`	`minimum`, `maximum`
`boolean`	`bool`	(none)
`array`	`[]T`	`items`, `minItems`, `maxItems`
`object`	`map/struct`	`properties`, `required`, `additionalProperties`

Supported format values: date, date-time, email, uri, uuid

Implemnentation journal

In order to be able to have another agent pick up the work in progress, all completed work and milestones must be journaled to the ./journals/implement_mutableMetadata.md

Each time a milestone or signficant part of a milestone is completed there needs to be a date/time stamped journal entry showing the item is done. So that a new agent taking over will know exactly where we are in the process and can resume the work.

At any give time I should be able to clear your context and have you resume your work based on this document, the code (git diffs) and the journal of what is completed.

If we run into any issues where Q has to be consulted about an issue that needs a decision , then Qs answer should also be written to the journal along with the question.

76 KiB Raw Blame History

Mutable Metadata & Per-Client Custom Schemas - Combined Design Plan (v2)

Status: DRAFT - Combined Claude + Codex + Jay's recommendations (2026-03-24, rev 2026-03-31 post-review, rev 2026-04-10: schema administration restricted to super_admin role only)

1. Problem Statement

2. Current Architecture Summary

2.1 Existing Tables

2.2 Existing API Endpoints

2.3 Key Relationships

2.4 Current Role Model (Relevant to This Plan)

3. Design Principles

4. Database Schema Design

4.1 New Type: schema_status_type

4.2 New Table: client_metadata_schemas

4.3 New Table: document_custom_metadata

4.4 Alteration: documents Table

4.5 New View: current_document_custom_metadata

4.6 Entity Relationship Diagram

4.7 Database Triggers (Defense-in-Depth)

Trigger 1: Prevent schema reassignment after first extraction

Trigger 2: Validate client/schema ownership match

Trigger 3: Enforce consistent schema_id across metadata versions

5. API Design

5.1 Schema Management Endpoints (super_admin only)

POST /super-admin/custom-schemas

PUT /super-admin/custom-schemas/{schemaId}

GET /super-admin/custom-schemas

GET /super-admin/custom-schemas/{schemaId}

DELETE /super-admin/custom-schemas/{schemaId}

5.2 Document Schema Assignment Endpoints (super_admin only)

PATCH /super-admin/documents/{id}/schema

POST /super-admin/folders/{folderId}/assign-schema (Bulk Folder Assignment)

GET /document/{id} (Modified)

5.3 Authorization & Permit.io Changes

New Permit.io resource: super-admin

Role permission updates

New custom-metadata resource

Policy mapping documentation additions

Middleware impact

Deployment ordering

5.4 Custom Metadata CRUD Endpoints (User-Facing)

POST /custom-metadata

GET /custom-metadata

GET /custom-metadata/version

GET /custom-metadata/history

6. Validation Strategy

6.1 JSON Schema Validation

6.2 Schema Definition Requirements

6.3 Supported JSON Schema Features

6.4 Validation Error Reporting

7. Service Layer Design

7.1 New Service: internal/customschema/

7.2 Validator Component

7.3 Mutual Exclusivity Guard

7.4 API Controller: api/queryAPI/customschemas.go

8. Migration Plan

8.1 Migration Files

8.2 SQLC Queries

8.3 OpenAPI Spec Changes

8.4 Legacy Field Extraction Guard

8.5 Delete Path Updates (Cascade Cleanup)

Document Delete Cascade

Client Delete Cascade

8.6 Permit.io Setup Tool Run

9. Invariants & Business Rules

10. Relationship to Existing Field Extraction System

11. Implementation Order

12. Testing Strategy

13. Resolved Design Decisions

14. v1 -> v2 Change Log

Appendix A: Sample JSON Schemas

A.1 Comprehensive Schema (All Supported Types)

A.2 Aircraft Engineering Schema (Domain Example)

A.3 Auto Engineering Schema (Domain Example)

Supported JSON Schema Data Types Summary

Implemnentation journal

76 KiB

Raw Blame History

Status: DRAFT - Combined Claude + Codex + Jay's recommendations (2026-03-24, rev 2026-03-31 post-review, rev 2026-04-10: schema administration restricted to `super_admin` role only)

4.1 New Type: `schema_status_type`

4.2 New Table: `client_metadata_schemas`

4.3 New Table: `document_custom_metadata`

4.4 Alteration: `documents` Table

4.5 New View: `current_document_custom_metadata`

5.1 Schema Management Endpoints (`super_admin` only)

`POST /super-admin/custom-schemas`

`PUT /super-admin/custom-schemas/{schemaId}`

`GET /super-admin/custom-schemas`

`GET /super-admin/custom-schemas/{schemaId}`

`DELETE /super-admin/custom-schemas/{schemaId}`

5.2 Document Schema Assignment Endpoints (`super_admin` only)

`PATCH /super-admin/documents/{id}/schema`

`POST /super-admin/folders/{folderId}/assign-schema` (Bulk Folder Assignment)

`GET /document/{id}` (Modified)

New Permit.io resource: `super-admin`

`POST /custom-metadata`

`GET /custom-metadata`

`GET /custom-metadata/version`

`GET /custom-metadata/history`

7.1 New Service: `internal/customschema/`

7.4 API Controller: `api/queryAPI/customschemas.go`