query-orchestration/plans/mutable.metadata.plan.combo.v4.md

# Mutable Metadata & Per-Client Custom Schemas - Combined Design Plan (v4)

## Status: DRAFT - Combined Claude + Codex + Jay's recommendations (2026-03-24, rev 2026-03-31 post-review, rev 2026-04-10: schema administration restricted to `super_admin` role only, rev 2026-04-13: add reset-metadata escape hatch, rev 2026-04-14: v4 simplification pass)

**Lineage**: This plan was informed by two original detailed specs done adversarily. One by claude Opus 4.6 and one by Codex 5.4 (xhigh) based on human written requirements. Each was human reviewed for common items and conflicting items. Where there were conflicts the human was the final decision maker. This resulting hybrid document represents an improved design over each of the source documents.

**v2 change summary (2026-04-10)**: In v1 all mutable-metadata administrative endpoints lived under `/admin/*`. Because the Permit.io middleware maps resources from the first path segment and both `user_admin` and `super_admin` hold full permissions on the `admin` resource, v1 exposed schema CRUD and schema assignment to the lesser (client-scoped) `user_admin` role. v2 moves every schema administration endpoint onto a new `/super-admin/*` path prefix backed by a new Permit.io resource named `super-admin` that only the `super_admin` role can access. User-facing custom metadata read/write endpoints (`/custom-metadata/*`) are unchanged. The underlying service layer, database schema, triggers, and validation strategy are unchanged -- this is strictly an authorization and routing change.

**v3 change summary (2026-04-13)**: Adds a single new super_admin endpoint, `POST /super-admin/documents/{id}/reset-metadata`, to support upgrading a document to a newer schema version when the document already has custom metadata written. Once custom metadata exists, Trigger 1 (`trg_prevent_schema_reassignment`) freezes `documents.custom_schema_id` permanently. The practical effect is that documents cannot move to a revised schema even when the new schema is a strict additive superset (e.g., one new optional field). v3 adds an explicit opt-in path: reset-metadata wipes all `document_custom_metadata` rows for the document and nulls `documents.custom_schema_id` in a single transaction, after which the existing `PATCH /super-admin/documents/{id}/schema` and `POST /custom-metadata` endpoints can bind a new schema and re-enter values. The escape hatch is deliberately destructive (metadata version history for the document is lost) so the invariant "schema is frozen once metadata exists" remains true -- you are explicitly opting out of "metadata exists" first.

**v4 change summary (2026-04-14)**: Applies a simplification pass that removes redundant state and replaces imperative trigger code with declarative constraints while preserving every feature designed in v3 (schema CRUD, single-doc assign, bulk folder assign, reset-metadata, mutual exclusivity, schema versioning). The full v3 -> v4 change log is in Section 16. Core simplifications:

1. **`document_custom_metadata.schema_id` removed.** The document row (`documents.custom_schema_id`) is already the single source of truth for schema binding (Trigger 1 freezes it once metadata exists; reset-metadata wipes rows before rebinding). Keeping a second schema pointer on every metadata row was redundant. Dropping it removes one index (`idx_dcm_schema_id`), one trigger (Trigger 3 `trg_enforce_consistent_schema_id`), one SQLC query (`GetSchemaMetadataRecordCount`), and an entire class of app-layer consistency checks. All invariants still hold.
2. **v3's Trigger 2 replaced by a composite foreign key.** The "schema must belong to same client as document" invariant is expressed as `FOREIGN KEY (custom_schema_id, "clientId") REFERENCES client_metadata_schemas(id, client_id)` on the `documents` table. Declarative integrity replaces PL/pgSQL. v4 keeps **two** triggers: Trigger 1 (`trg_prevent_schema_reassignment`) blocks schema reassignment once metadata or legacy extractions exist, and a new Trigger 2 (`trg_prevent_legacy_extraction_on_custom_document`) blocks legacy field extraction inserts on documents already bound to a custom schema. Together they close the concurrent-writer race between `AssignSchema` and `CreateFieldExtraction` (see Section 7.3).
3. **`current_document_custom_metadata` view removed.** The new metadata table is simple enough that `ORDER BY version DESC LIMIT 1` using `idx_dcm_doc_version_desc` is clearer and has zero overhead compared to a `DISTINCT ON` view.
4. **FK cascades handle delete cleanup.** `document_custom_metadata.document_id` uses `ON DELETE CASCADE`. `client_metadata_schemas.client_id` uses `ON DELETE CASCADE`. Hard-deleting a document automatically removes its custom metadata rows; hard-deleting a client automatically removes its schemas after its documents are gone. **The existing `DeleteDocumentCascade` and `client.HardDelete` code paths are not modified by v4.** The v3 claim that `custom_schema_id` must be nulled before a document delete was incorrect -- deleting the referencing row removes the reference without needing to null it first. The only surviving "manual delete" code is inside reset-metadata (which explicitly wipes metadata while keeping the document row alive) -- see Section 5.2.
5. **`POST /super-admin/custom-schemas/{schemaId}/versions`** replaces `PUT /super-admin/custom-schemas/{schemaId}`. The operation creates a new schema row with a new ID and supersedes the old one -- that is creation semantics, not update semantics. The new route matches the actual behavior, makes code review clearer, and sidesteps the unresolved Permit.io "`PUT` -> `patch` action mapping" question that the v3 plan deferred to implementation.
6. **`createdBy` removed from new request bodies; response `createdBy` is the Cognito subject ID, not an email.** v3 already said the server must derive actor identity from the JWT and ignore any body-supplied `createdBy`. v4 drops the misleading field from the OpenAPI schemas of all new endpoints so the contract matches the behavior. The value used server-side is the JWT `sub` claim (an opaque Cognito subject UUID), sourced via `cognitoauth.GetUserSubject(c)`. Email is **not** guaranteed present in the JWT on this service and is deliberately not used. Response `createdBy` / `resetBy` fields on all new endpoints are therefore declared as plain strings in OpenAPI (no `format: email`) — see Section 8.3 "Actor fields". This is a deliberate contract divergence from existing endpoints elsewhere in `queryAPI.yaml` that use `format: email`; those are not touched by v4.
7. **`GET /document/{id}` enrichment trimmed.** v3 proposed adding `customSchemaId`, `customSchemaName`, `hasCustomMetadata`, and an optional `customMetadata` blob (gated by a `?customMetadata=true` query param) to the existing read path. v4 adds only `customSchemaId` and `hasCustomMetadata` and does **not** inline the metadata payload. Clients who need the full metadata call `GET /custom-metadata` (which already exists). This keeps a stable cross-cutting read path narrow.
8. **`documentCount` / `canDelete` kept on the schema list response.** These two fields are cheap to compute in a single subquery, prevent an extra round-trip for the delete-guard UX, and survived the simplification review. `previousVersionId` on the version-creation response and `customSchemaName` decorations on unrelated responses are dropped.
9. **Vertical slice milestones replace 12 horizontal phases.** Each milestone is testable end-to-end, so the feature is usable much earlier in the rollout. The full milestone shape is in Section 11 and drives the structure of `plans/mutable.metadata.plan.combo.v4.tracking.md`.

No functional requirements were cut. Bulk folder assignment, reset-metadata, schema versioning, mutual exclusivity, and the `super_admin`-only authorization boundary are all preserved exactly as in v3.

---

## 1. Problem Statement

Today, all field extraction data is stored in a rigid 3-table schema with ~130 statically-defined columns (19 single-value fields + 112 array fields). Every client shares the same field definitions. This means:

- Clients cannot define domain-specific fields (e.g., "aircraft engineering" vs. "auto engineering" document classes)
- Adding new fields requires database migrations and code changes across the full stack
- The system cannot accommodate varying data shapes per document class within a single client

**Goal**: Allow each client to curate their own collections of document schemas, where each schema defines the custom fields (with types and constraints) that a document's metadata must conform to. Administration of these schemas (create, update, retire, list, assign to documents/folders) is a platform-level responsibility and must be restricted to the `super_admin` role, not delegated to the lower-privileged `user_admin` role.

---

## 2. Current Architecture Summary

### 2.1 Existing Tables

```
documentFieldExtractions          -- 19 single-value fields (1:1 per document)
documentFieldExtractionVersions   -- Version tracking with row-level locking
documentFieldExtractionArrayFields -- 112 array fields (1:N per extraction)
```

### 2.2 Existing API Endpoints

| Method | Path | Description |
|--------|------|-------------|
| POST | `/field-extractions` | Create versioned extraction (singleFields + arrayFields) |
| GET | `/field-extractions` | Get current (latest version) extraction for a document |
| GET | `/field-extractions/version` | Get extraction by specific version |
| GET | `/field-extractions/history` | List all versions for a document |

### 2.3 Key Relationships

- `documents.id` -> `documentFieldExtractions.documentId` (1:N via versions)
- `documentFieldExtractions.id` -> `documentFieldExtractionArrayFields.fieldExtractionId` (1:N)
- `documentFieldExtractions.id` -> `documentFieldExtractionVersions.fieldExtractionId` (1:N)
- Documents belong to clients via `documents.clientId`
- No per-client customization of field definitions exists today

### 2.4 Current Role Model (Relevant to This Plan)

Two admin-class roles exist today, both configured in `cmd/auth_related/permit.setup/permit_policies.yaml`:

- **`super_admin`**: Platform administrator. Holds full permissions across every resource (`admin`, `client`, `clients`, `document`, `documents`, `folders`, etc.). Intended for AArete-internal operators.
- **`user_admin`**: Client-scoped administrator. Primarily used for creating new client users via the admin user tool. Holds full permissions on the `admin` resource but intentionally does **not** hold permissions on `client`, `clients`, or `document` at the resource level.

Because the middleware extracts the resource from the first path segment (`/admin/foo` -> resource `admin`), any endpoint placed under `/admin/*` is accessible to both `super_admin` and `user_admin`. v1 of this plan placed schema administration under `/admin/*` and therefore exposed schema CRUD and assignment to `user_admin`. v2 corrects this.

---

## 3. Design Principles

1. **Additive, not destructive** - The existing static field extraction system continues to work unchanged for documents that do not opt into custom schemas. Custom schemas are an additive capability.
2. **Mutual exclusivity** - A document uses EITHER the legacy static field extraction system OR the custom schema system, never both. This prevents ambiguity about which system is authoritative for a document's metadata. Once a document is assigned a custom schema, the legacy field extraction endpoints reject writes for that document, and vice versa.
3. **Schema-as-data** - Schemas are stored as JSON Schema documents in the database, not as DDL. No migrations needed when clients add/modify schemas.
4. **Immutable schema versions** - When a schema is updated, a new version (with new ID) is created. Old versions remain for historical reference. No backward compatibility checking is required between versions since each version gets a distinct ID.
5. **Schema binding is permanent** - Once custom metadata is written to a document with a schema ID, that schema ID cannot be changed on the document. This guarantees data integrity.
6. **JSON blob storage** - Custom metadata is stored as JSONB in PostgreSQL, validated against the schema at write time. This avoids dynamic column creation.
7. **Client isolation** - Schemas are scoped to clients. A client cannot reference another client's schema.
8. **Size limits** - Schema definitions are capped at 64KB (`MaxSchemaDefinitionBytes = 65536`). Metadata payloads are capped at 1MB (`MaxMetadataPayloadBytes = 1048576`). Bulk folder assignments are capped at 10,000 documents (`MaxBulkAssignDocuments = 10000`). All are tunable constants.
9. **No JSONB search/filter** - The system does NOT support querying or filtering documents by custom metadata values. Custom metadata is opaque storage validated on write. If search is needed in the future, it would require GIN indexes on the JSONB columns and a dedicated query API (separate phase).
10. **`super_admin`-scoped schema management** - Schema CRUD operations, single-document schema assignment, and bulk folder-level schema assignment live under `/super-admin/*` and require the `super_admin` role. The lesser `user_admin` role does NOT have access to schema administration. Custom metadata read/write on documents (`/custom-metadata/*`) is available to regular `client_user` users and to both admin roles.
11. **Defense-in-depth** - Critical invariants are enforced at BOTH the application layer AND the database layer, but v4 prefers declarative database constraints over imperative triggers wherever possible. The app-layer checks provide better error messages; the DB constraints are safety nets against bypass. v4 retains two triggers: `trg_prevent_schema_reassignment` on `documents` UPDATE ("schema is frozen once metadata or legacy extraction exists") and `trg_prevent_legacy_extraction_on_custom_document` on `documentFieldExtractions` INSERT (closes the concurrent-writer race on the mutual-exclusivity invariant; see Section 7.3). Both invariants are stateful and cannot be expressed as constraints. Client/schema ownership is enforced by a composite foreign key. Schema-id consistency across metadata versions is no longer a separate invariant because `document_custom_metadata` no longer carries a `schema_id` column -- the document row is the single source of truth.
12. **No migration from static fields** - There is no migration path from the existing static field extraction data into the custom schema system. They are mutually exclusive per document. Out of scope.

---

## 4. Database Schema Design

### 4.1 New Type: `schema_status_type`

A three-state enum that distinguishes between different reasons a schema is no longer assignable:

```sql
CREATE TYPE schema_status_type AS ENUM ('active', 'superseded', 'retired');
```

- **`active`**: The schema version is current and can be assigned to new documents.
- **`superseded`**: A newer version of this schema (same `name`) was created. The schema is still valid for documents already using it, but should not be assigned to new documents. Set automatically when a new version is created via `POST /super-admin/custom-schemas/{schemaId}/versions`.
- **`retired`**: An admin explicitly disabled this schema version. Functionally similar to `superseded` but indicates a deliberate administrative action rather than an automatic version succession.

### 4.2 New Table: `client_metadata_schemas`

Stores client-defined JSON Schema documents. Each row is an immutable version of a schema definition.

```sql
CREATE TABLE client_metadata_schemas (
    id              uuid NOT NULL DEFAULT uuid_generate_v7(),
    client_id       varchar(255) NOT NULL REFERENCES clients(clientId) ON DELETE CASCADE,
    name            varchar(255) NOT NULL,
    description     text,
    schema_def      jsonb NOT NULL,          -- The JSON Schema document (max 64KB enforced at app layer)
    version         int NOT NULL DEFAULT 1,   -- Auto-incremented per (client_id, name)
    status          schema_status_type NOT NULL DEFAULT 'active',
    created_at      timestamptz NOT NULL DEFAULT NOW(),
    created_by      varchar(255) NOT NULL,

    CONSTRAINT pk_client_metadata_schemas PRIMARY KEY (id),
    CONSTRAINT uq_client_schema_name_version UNIQUE (client_id, name, version),
    -- Composite uniqueness required as the target of the same-client foreign key on `documents` (Section 4.4).
    -- Because `id` is already the primary key, `(id, client_id)` is trivially unique -- this constraint exists
    -- purely to make the tuple referenceable by a foreign key.
    CONSTRAINT uq_cms_id_client UNIQUE (id, client_id),
    CONSTRAINT chk_schema_def_size CHECK (pg_column_size(schema_def) <= 70000)  -- ~68KB to account for JSONB overhead; app layer enforces 64KB on raw JSON
);

CREATE INDEX idx_cms_client_id ON client_metadata_schemas(client_id);
CREATE INDEX idx_cms_client_name ON client_metadata_schemas(client_id, name);
CREATE INDEX idx_cms_client_status ON client_metadata_schemas(client_id, status);
```

> **Note**: The 64KB limit is enforced at the application layer (Go constant `MaxSchemaDefinitionBytes = 65536`) by checking `len(rawJSON)`. The database CHECK constraint uses `pg_column_size()` which measures on-disk JSONB storage (includes header overhead), so it is set slightly higher (~68KB) as a safety net. The app-layer check provides the user-facing 64KB limit and a better error message.

**Design decisions:**

- **`name` + `version`** scoped to client: Allows human-friendly naming ("aircraft-engineering-v3") with automatic versioning. When a new version is created, a new row is inserted with `version = max(version) + 1` for that `(client_id, name)`.
- **`schema_def` is JSONB**: Stores the full JSON Schema document. PostgreSQL can index into it if needed later.
- **`status` enum**: Three-state lifecycle. `superseded` is set automatically when a newer version is created. `retired` is set explicitly by admin action (soft-delete). Both states prevent assignment to new documents but allow existing documents to continue using the schema for validation. This is richer than a boolean `is_active` because it distinguishes "replaced by v2" from "admin decided this is bad."
- **Retirement permanently reserves `(client_id, name)` (v4)**: Once every row in a `(client_id, name)` lineage is non-`active` (i.e. the lineage has been fully retired), the `name` cannot be reused for that client. Three independent rules interact to produce this: (a) `POST /super-admin/custom-schemas` always inserts at `version = 1`, which collides with the retired v1 row on `uq_client_schema_name_version`; (b) `POST /super-admin/custom-schemas/{schemaId}/versions` rejects non-`active` parents (Section 5.1); (c) the UNIQUE constraint prevents any lineage branching. This is **intentional**: it keeps the audit history unambiguous (a name in the history always refers to exactly one lineage) and keeps the invariants from Section 4 stated as simple structural rules. Callers that need a "similar" schema after retirement must pick a new name (e.g. `aircraft-engineering-v2`, `aircraft-engineering-replacement`). See also the retirement note on `DELETE /super-admin/custom-schemas/{schemaId}` in Section 5.1.
- **`id` is the schema version ID**: This is what documents reference. Each version creates a new `id`. The `(client_id, name, version)` tuple provides the human-readable lineage.
- **`ON DELETE CASCADE` on `client_id`**: Hard-deleting a client automatically removes all its schema rows. Because the FK on `documents.custom_schema_id` (Section 4.4) points here, and because documents are deleted before the client row in `client.HardDelete`, the cascade fires only after every document referencing a schema has already been deleted. This eliminates the need for a feature-specific `DeleteClientMetadataSchemas` step in the client delete path.
- **`uq_cms_id_client`**: Required as the target of the composite FK on `documents` (Section 4.4). It adds no data-model meaning beyond the primary key but is necessary because PostgreSQL only allows FKs to reference columns backed by a unique constraint.

### 4.3 New Table: `document_custom_metadata`

Stores the actual custom metadata for documents. Each row is a versioned JSON blob bound to a single document. The schema used to validate the blob is derived from `documents.custom_schema_id` at write time -- it is not stored on the row.

```sql
CREATE TABLE document_custom_metadata (
    id              uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    document_id     uuid NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
    metadata        jsonb NOT NULL,           -- The actual custom field values
    version         int NOT NULL DEFAULT 1,   -- Versioned like field extractions
    created_at      timestamptz NOT NULL DEFAULT NOW(),
    created_by      varchar(255) NOT NULL,

    CONSTRAINT uq_doc_custom_metadata_version UNIQUE (document_id, version)
);

CREATE INDEX idx_dcm_doc_version_desc ON document_custom_metadata(document_id, version DESC);
```

**Design decisions:**

- **No `schema_id` column (v4 simplification)**: The document row (`documents.custom_schema_id`) is the single source of truth for which schema a document's metadata conforms to. Writing a second schema pointer onto every metadata row was redundant because (1) `SetDocumentMetadata` already derives the schema from `documents.custom_schema_id` at write time, (2) Trigger 1 freezes `documents.custom_schema_id` as soon as any metadata row exists, and (3) `ResetDocumentMetadata` deletes all metadata rows before any rebinding can occur. Removing the column also removes `idx_dcm_schema_id`, Trigger 3 (`trg_enforce_consistent_schema_id`), the `GetSchemaMetadataRecordCount` SQLC query, and an entire class of "make sure these two pointers agree" app-layer checks. If a future release ever needs non-destructive in-place schema migration (relaxing the "frozen schema" invariant), the column can be added back with a dedicated migration.
- **`ON DELETE CASCADE` on `document_id`**: Hard-deleting a document automatically removes its metadata rows. The existing `DeleteDocumentCascade` code path is unchanged -- no feature-specific delete step is needed. Reset-metadata still wipes rows explicitly (Section 5.2) because the document row itself survives that operation.
- **Single table with version column**: We use one table (rather than separate extraction + version tables like the legacy pattern) because the simpler design is sufficient. The legacy two-table pattern exists to support row-level locking on the version counter independently of the data row.
- **Concurrency strategy for version assignment**: `SELECT ... FOR UPDATE` on the max-version row correctly serializes the *second and subsequent* writes for a document, but it does **not** serialize the *first* write -- there is no row to lock yet, so two concurrent first-writers both see `max(version) = 0`, both try to INSERT `version = 1`, and one fails with a `UNIQUE (document_id, version)` violation. v4 handles this by locking the parent `documents` row with `SELECT id FROM documents WHERE id = $1 FOR UPDATE` *before* reading the max version. The document row is guaranteed to exist (enforced by the service layer's existence check and by the FK) so this lock serializes the first write as well as every subsequent write. As a belt-and-suspenders measure, the service layer also retries once on a `UNIQUE (document_id, version)` violation; if the retry still conflicts the request returns `500`. If this proves insufficient under load, splitting into two tables is a backward-compatible change.
- **`metadata` is JSONB**: Validated against `client_metadata_schemas.schema_def` (looked up via the document's `custom_schema_id`) at write time. Stored as a single blob for simplicity and query flexibility.

### 4.4 Alteration: `documents` Table

Add an optional schema binding column and a composite foreign key that enforces the same-client invariant declaratively:

```sql
ALTER TABLE documents
    ADD COLUMN custom_schema_id uuid NULL;

-- Composite FK: the (custom_schema_id, clientId) tuple on a document must match the
-- (id, client_id) tuple on a schema. This is what guarantees a document can only be
-- bound to a schema owned by its own client -- no trigger needed.
ALTER TABLE documents
    ADD CONSTRAINT fk_documents_custom_schema_same_client
    FOREIGN KEY (custom_schema_id, "clientId")
    REFERENCES client_metadata_schemas(id, client_id);

CREATE INDEX idx_documents_custom_schema_id ON documents(custom_schema_id);
```

**Behavior:**

- `NULL` = document uses the legacy static field extraction system (default, backward-compatible)
- `NOT NULL` = document is opted into the custom schema system; legacy field extraction endpoints will reject writes for this document
- Once set AND custom metadata has been written, this field becomes immutable (enforced at app layer AND DB trigger: `trg_prevent_schema_reassignment`)
- Can be set before metadata is written (pre-binding) and changed freely until first metadata write
- Must reference a schema owned by the same client as the document. **v4: this is enforced declaratively by the composite FK `fk_documents_custom_schema_same_client`**, not by a trigger. Any attempt to set `custom_schema_id` to a schema owned by a different client fails with a foreign-key-violation error, which the service layer maps to `409 Conflict` with a human-readable message. The app-layer check remains as well, so the user-facing error message is clean on the common path; the FK is the defense-in-depth safety net.
- Cannot be set on a document that already has legacy field extractions (returns 409 Conflict)

> **Why a composite FK instead of a trigger (v4)**: In v3, Trigger 2 (`trg_validate_schema_client_match`) read the referenced schema's `client_id` in PL/pgSQL and raised if it did not match the document's `clientId`. That approach worked but required a function definition, per-operation execution cost, and a dedicated unit test. The composite FK expresses the same invariant as a single DDL constraint that PostgreSQL enforces natively. `uq_cms_id_client` on `client_metadata_schemas` makes the `(id, client_id)` tuple referenceable (Section 4.2). No `ON UPDATE CASCADE` is needed because neither `documents.clientId` nor `client_metadata_schemas.client_id` is ever updated in place.

> **Note on the deleted plain FK**: The v3 plan wrote `ADD COLUMN custom_schema_id uuid NULL REFERENCES client_metadata_schemas(id)`, creating an implicit single-column FK alongside Trigger 2. v4 deletes that line entirely. The composite FK above already references `client_metadata_schemas(id, client_id)`, which transitively guarantees that `custom_schema_id` points at a real schema row -- a second, single-column FK would duplicate the reference and add a redundant constraint entry in `pg_constraint`.

### 4.5 Reading the latest metadata version (no view, v4 simplification)

v3 defined a `current_document_custom_metadata` view using `DISTINCT ON (document_id) ... ORDER BY document_id, version DESC`. v4 removes the view. The equivalent query is short, uses the `idx_dcm_doc_version_desc` index directly, and produces identical results:

```sql
-- name: GetCurrentDocumentCustomMetadata :one
SELECT id, document_id, metadata, version, created_at, created_by
FROM document_custom_metadata
WHERE document_id = @document_id
ORDER BY version DESC
LIMIT 1;
```

**Why remove the view**: The legacy field-extraction system benefits from a view because its state is spread across multiple tables. Custom metadata is a single table, so the indirection added no clarity. Removing it trims one migration artifact, one drift risk between query and view definitions, and one thing for the test matrix to verify.

### 4.6 Entity Relationship Diagram

```
clients
  |
  |-- 1:N --> client_metadata_schemas (per-client schema definitions)
  |              |                                     [ON DELETE CASCADE via client_id]
  |              |
  |              |-- referenced by --> documents.custom_schema_id
  |                                    (composite FK on (custom_schema_id, clientId))
  |
  |-- 1:N --> documents
                 |
                 |-- 1:N --> document_custom_metadata (versioned JSONB blobs)
                 |           [ON DELETE CASCADE via document_id]
                 |           [MUTUALLY EXCLUSIVE with legacy extractions]
                 |           [no schema_id column -- schema is derived from documents.custom_schema_id]
                 |
                 |-- 1:N --> documentFieldExtractions (existing static fields)
                             [MUTUALLY EXCLUSIVE with custom metadata]
```

### 4.7 Database Triggers (Defense-in-Depth)

**v4 keeps two triggers.** The v3 plan defined three triggers; v4 removes v3's former Trigger 2 for client/schema ownership (replaced by the composite FK in Section 4.4) and v3's former Trigger 3 for schema_id consistency (no longer needed because `document_custom_metadata` has no `schema_id` column -- schema consistency is now a tautology, not an invariant to enforce). v4 reintroduces a different Trigger 2 that closes the mutual-exclusivity race between `AssignSchema` and legacy `CreateFieldExtraction` (see Section 7.3).

#### Trigger 1: Prevent schema reassignment after first extraction

Fires on `documents` BEFORE UPDATE of `custom_schema_id`. Prevents changes if the document has any extraction data (legacy or custom). This invariant is stateful -- it depends on the existence of rows in other tables -- and cannot be expressed as a constraint, so it stays as a trigger.

```sql
CREATE OR REPLACE FUNCTION trg_prevent_schema_reassignment()
RETURNS TRIGGER AS $$
BEGIN
    -- Only fire if custom_schema_id is actually changing
    IF OLD.custom_schema_id IS DISTINCT FROM NEW.custom_schema_id THEN
        -- Check for existing custom metadata
        IF EXISTS (
            SELECT 1 FROM document_custom_metadata
            WHERE document_id = NEW.id
            LIMIT 1
        ) THEN
            RAISE EXCEPTION 'Cannot change custom_schema_id on document % because custom metadata already exists', NEW.id;
        END IF;

        -- Check for existing legacy field extractions
        IF EXISTS (
            SELECT 1 FROM "documentFieldExtractionVersions"
            WHERE "documentId" = NEW.id
            LIMIT 1
        ) THEN
            RAISE EXCEPTION 'Cannot change custom_schema_id on document % because legacy field extractions already exist', NEW.id;
        END IF;
    END IF;

    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_documents_prevent_schema_reassignment
    BEFORE UPDATE OF custom_schema_id ON documents
    FOR EACH ROW
    EXECUTE FUNCTION trg_prevent_schema_reassignment();
```

> **Note on reset-metadata interaction**: `ResetDocumentMetadata` (Section 5.2) deletes all `document_custom_metadata` rows for the document in the same transaction *before* setting `custom_schema_id = NULL`. Because the trigger's `EXISTS` check runs at `UPDATE` time and reads the already-deleted rows as absent, the UPDATE succeeds without any relaxation of the trigger. v4 does not weaken this trigger.

#### Trigger 2: Prevent legacy field extraction on documents with a custom schema

Fires on `documentFieldExtractions` BEFORE INSERT. Rejects the insert if the target document already has `custom_schema_id IS NOT NULL`. This is the symmetric counterpart to Trigger 1: Trigger 1 blocks schema assignment when legacy rows exist; Trigger 2 blocks legacy writes when a schema is bound. Together they eliminate the last-writer-wins race that would otherwise be possible if two concurrent transactions each passed their application-layer pre-check against a stale snapshot and then committed in opposite orders (see Section 7.3 for the full race analysis).

```sql
CREATE OR REPLACE FUNCTION trg_prevent_legacy_extraction_on_custom_document()
RETURNS TRIGGER AS $$
DECLARE
    v_custom_schema_id uuid;
BEGIN
    SELECT custom_schema_id INTO v_custom_schema_id
    FROM documents
    WHERE id = NEW."documentId"
    FOR SHARE;

    IF v_custom_schema_id IS NOT NULL THEN
        RAISE EXCEPTION 'Cannot create legacy field extraction on document % because it is bound to custom schema %', NEW."documentId", v_custom_schema_id
            USING ERRCODE = 'check_violation';
    END IF;

    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_dfe_prevent_legacy_extraction_on_custom_document
    BEFORE INSERT ON "documentFieldExtractions"
    FOR EACH ROW
    EXECUTE FUNCTION trg_prevent_legacy_extraction_on_custom_document();
```

The `FOR SHARE` lock on the `documents` row makes the trigger conflict with any concurrent `AssignSchema` that is holding `FOR UPDATE` on the same row: the trigger waits for the assign transaction to commit or roll back, then reads the final value. Together with the application-layer parent-row lock described in Section 7.3, this closes both sides of the race at the DB layer. The `check_violation` SQLSTATE is what `internal/fieldextraction/service.go` will detect and convert to the sentinel `ErrMutualExclusivityViolation` returned to the controller layer.

#### Removed in v4: v3's client/schema ownership match trigger

v3 used `trg_validate_schema_client_match` to reject `INSERT` or `UPDATE` on `documents` when the referenced schema belonged to a different client. v4 expresses the same invariant declaratively using the composite foreign key `fk_documents_custom_schema_same_client` defined in Section 4.4. The FK refuses any `(custom_schema_id, clientId)` tuple that is not present in `client_metadata_schemas(id, client_id)`, which is exactly the same guarantee with no function code to maintain.

#### Removed in v4: v3's consistent schema_id trigger

v3 used `trg_enforce_consistent_schema_id` on `document_custom_metadata INSERT` to ensure every metadata version for a document referenced the same schema. v4 removes the `schema_id` column from the table entirely, so every row for a given `document_id` trivially "uses" whatever schema `documents.custom_schema_id` currently points at -- there is no way for them to disagree. The invariant is preserved by making it structurally impossible rather than by enforcing it at runtime.

> **Overall v4 trigger surface**: two triggers (`trg_prevent_schema_reassignment` and `trg_prevent_legacy_extraction_on_custom_document`), one composite FK (`fk_documents_custom_schema_same_client`), one `ON DELETE CASCADE` on each new table. Both triggers duplicate checks that also exist in the application service layer. This is intentional: the app-layer check returns a structured 400/409 on the common path; the triggers are a last-resort safety net that fires even if a future code path, migration script, or admin query bypasses the service layer.

---

## 5. API Design

### 5.1 Schema Management Endpoints (`super_admin` only)

All schema CRUD endpoints live under the `/super-admin/custom-schemas` path prefix. The first path segment `super-admin` maps (via the existing first-segment resource extraction rule in the Permit.io middleware) to a new Permit.io resource named `super-admin`. Only the `super_admin` role holds permissions on this resource. The `user_admin` role does NOT have access.

These endpoints are grouped under a new OpenAPI tag `SuperAdminSchemaService` (distinct from the existing `AdminService`) to make the authorization boundary visible in the generated client and in Swagger.

#### `POST /super-admin/custom-schemas`

Create a new schema for a client.

**Request:**
```json
{
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "Custom fields for aircraft engineering documents",
    "schema": {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties": {
            "aircraft_model": {
                "type": "string",
                "maxLength": 100
            },
            "certification_date": {
                "type": "string",
                "format": "date"
            },
            "max_altitude_ft": {
                "type": "number",
                "minimum": 0,
                "maximum": 100000
            },
            "engine_count": {
                "type": "integer",
                "minimum": 1,
                "maximum": 12
            }
        },
        "required": ["aircraft_model"],
        "additionalProperties": false
    }
}
```

> **Actor identity (v4)**: The request body no longer contains a `createdBy` field. The handler derives the actor from the authenticated JWT via `cognitoauth.GetUserSubject(c)` and passes it to the service layer as a separate argument. The value stored and echoed back is the **Cognito subject ID** (the JWT `sub` claim — an opaque UUID like `b7a4c1d2-8e3f-4a5b-9c6d-0e1f2a3b4c5d`), **not** an email address. Email is not guaranteed to be present in the JWT and is explicitly rejected as a source of actor identity for these endpoints. `createdBy` / `resetBy` are therefore declared in OpenAPI as plain strings (no `format: email`) — see Section 5.5. Clients that send an unexpected `createdBy` key will have it silently ignored by the generated OpenAPI parser; the same applies to every new endpoint in Sections 5.1, 5.2, and 5.4.

**Response (201):**
```json
{
    "id": "019577a3-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "Custom fields for aircraft engineering documents",
    "schema": { ... },
    "version": 1,
    "status": "active",
    "createdAt": "2026-03-23T12:00:00Z",
    "createdBy": "b7a4c1d2-8e3f-4a5b-9c6d-0e1f2a3b4c5d"
}
```

**Validation:**
- The `schema` field MUST be a valid JSON Schema document (validated server-side using a JSON Schema meta-validator)
- The `schema` field must be <= 64KB in size (returns 400 with message referencing the limit)
- Root `type` must be `"object"`
- Root `additionalProperties` MUST be explicitly present (either `true` or `false`). If omitted, the server returns 400 with a message explaining that `additionalProperties` must be explicitly declared. This prevents the common mistake of omitting it and inadvertently accepting arbitrary extra fields.
- `name` must be unique within the client (for active schemas at the same version)

#### `POST /super-admin/custom-schemas/{schemaId}/versions`

Create a new version of an existing schema. This operation creates a new schema row with a **new ID** and supersedes the old one. The old version remains intact and its status is automatically set to `superseded`. No backward compatibility check is performed -- the new version is treated as an independent schema that happens to share a name lineage.

> **Why `POST /versions` instead of `PUT` (v4)**: The v3 plan used `PUT /super-admin/custom-schemas/{schemaId}` for this operation, but `PUT` in HTTP semantics means "replace the resource at this URL." That is not what this operation does -- it returns a *new* resource at a *new* URL and leaves the original resource in place (with its status changed). That is creation semantics, not replacement. `POST /super-admin/custom-schemas/{schemaId}/versions` describes the behavior accurately and also avoids the unresolved Permit.io action-mapping question (Permit.io's default action vocabulary does not include `put`; v3 Section 5.3 flagged this as an implementation decision). `POST` maps unambiguously to the `super-admin:post` grant.

**Request:**
```json
{
    "description": "Updated aircraft engineering fields (added wingspan)",
    "schema": {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties": {
            "aircraft_model": { "type": "string", "maxLength": 100 },
            "certification_date": { "type": "string", "format": "date" },
            "max_altitude_ft": { "type": "number", "minimum": 0, "maximum": 100000 },
            "engine_count": { "type": "integer", "minimum": 1, "maximum": 12 },
            "wingspan_meters": { "type": "number", "minimum": 0 }
        },
        "required": ["aircraft_model"],
        "additionalProperties": false
    }
}
```

**Response (201):**
```json
{
    "id": "019577b4-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "version": 2,
    "status": "active",
    ...
}
```

**Behavior:**
- The `schemaId` in the path identifies the schema whose lineage is being extended (the parent version whose `name` + `client_id` tuple is inherited)
- The new version inherits the `name` and `client_id` from the parent version
- **Parent must be the currently active version for its `(client_id, name)` lineage.** If the parent's status is `superseded` or `retired`, the request is rejected with `409 Conflict` and message `"Cannot version from a non-active parent schema. Latest active version for this lineage is {activeId}."`. This prevents forking lineages and guarantees there is never more than one `active` row per `(client_id, name)` (see also the transaction sketch below).
- On success, the parent version's status is set to `superseded` in the **same** transaction that inserts the new row — no other version in the lineage can be active by construction.
- Returns the newly created schema version (HTTP 201 with `Location: /super-admin/custom-schemas/{newId}`)
- The `schema` field in the request body follows the same validation rules as the root `POST` endpoint (including the `additionalProperties` requirement)
- **Concurrency**: The transaction runs `SELECT id, version, status FROM client_metadata_schemas WHERE client_id = $1 AND name = $2 ORDER BY version FOR UPDATE` so every existing row in the lineage is locked before the active-parent check runs. The handler then verifies the path `schemaId` matches the single row with `status = 'active'` (rejecting with 409 if not), computes `max(version) + 1`, inserts the new row with `status = 'active'`, and marks the old active row `superseded`. The unique constraint `uq_client_schema_name_version` is the final safety net against a concurrent duplicate.
- **`previousVersionId` not returned (v4)**: v3's response decorated this endpoint with a `previousVersionId` field. v4 drops the decoration because the caller already knows the parent ID (it is in the request URL). Callers that want the lineage explicitly can call `GET /super-admin/custom-schemas?name=...&includeAllVersions=true`.

#### `GET /super-admin/custom-schemas`

List all schemas for a client.

**Query Parameters:**
- `clientId` (required): The client whose schemas to list
- `name` (optional): Filter by schema name
- `status` (optional): Filter by status. Accepted values: `active`, `superseded`, `retired`, or the sentinel `any` to return all statuses. Default: `active` (only active schemas are returned when the parameter is omitted).
- `includeAllVersions` (optional, default: `false`): When `false`, return only the latest version per `(client_id, name)` lineage; when `true`, return every row the other filters match (one row per version). The default is deliberately narrow so the common admin UX does not have to paginate through historical versions.
- `limit` (optional, default: 50, max: 200): Maximum number of results to return
- `offset` (optional, default: 0): Number of results to skip for pagination

> **Filter interaction note**: To see a full lineage with both the currently active version and its superseded predecessors in one response, callers must pass **both** `includeAllVersions=true` **and** `status=any` (the default `status=active` would strip out superseded rows even when `includeAllVersions=true`). The integration tests in tracking Milestone 1 rely on this exact combination; see tracking Milestone 1 exit criteria.

**Response (200):**
```json
{
    "schemas": [
        {
            "id": "019577b4-...",
            "clientId": "acme-corp",
            "name": "aircraft-engineering",
            "description": "...",
            "version": 2,
            "status": "active",
            "documentCount": 45,
            "canDelete": false,
            "createdAt": "...",
            "createdBy": "..."
        },
        {
            "id": "019577c1-...",
            "clientId": "acme-corp",
            "name": "auto-engineering",
            "description": "...",
            "version": 1,
            "status": "active",
            "documentCount": 0,
            "canDelete": true,
            "createdAt": "...",
            "createdBy": "..."
        }
    ]
}
```

**Notes:**
- `documentCount` shows how many documents reference each schema version via `documents.custom_schema_id`. Computed via a single subquery in the list query.
- `canDelete` is `true` when `documentCount == 0`. Because `document_custom_metadata` no longer carries a `schema_id` column (Section 4.3), schema-usage is knowable from `documents.custom_schema_id` alone -- there is no second table to consult. This saves the client a round-trip to check before attempting delete.

#### `GET /super-admin/custom-schemas/{schemaId}`

Get a specific schema by ID (returns full schema definition).

**Response (200):**
```json
{
    "id": "019577a3-...",
    "clientId": "acme-corp",
    "name": "aircraft-engineering",
    "description": "...",
    "schema": { ... },
    "version": 1,
    "status": "superseded",
    "documentCount": 45,
    "canDelete": false,
    "createdAt": "...",
    "createdBy": "..."
}
```

#### `DELETE /super-admin/custom-schemas/{schemaId}`

Retire a schema. Only succeeds if no documents reference it.

**Response (204):** No content on success.

**Response (409 Conflict):**
```json
{
    "error": "Schema is in use by 45 documents and cannot be deleted"
}
```

**Behavior:**
- Sets `status = 'retired'`
- Fails with 409 if any documents reference this schema ID via `documents.custom_schema_id`. v4 no longer needs to consult `document_custom_metadata` (the `schema_id` column was removed in Section 4.3), so the check is a single count against `documents`.
- Retired schemas still serve validation for existing documents but cannot be assigned to new documents
- **Retirement permanently reserves the name (v4)**: if the retired row was the only active version of its `(client_id, name)` lineage, the lineage is now fully retired and **the name cannot be reused** for that client. `POST /super-admin/custom-schemas` with the same `name` will collide on `uq_client_schema_name_version` (version 1 already exists as retired), and `POST /super-admin/custom-schemas/{schemaId}/versions` against the retired row is rejected by the non-`active` parent check above. This is intentional (see Section 4.2 design decisions); callers that need a similar schema must choose a new `name` such as `{originalName}-v2` or `{originalName}-replacement`. Operators who want to retire a lineage without losing the ability to reuse the name should instead supersede it by creating a new version and then retiring the old version — superseding is reversible in principle, full retirement is not.

---

### 5.2 Document Schema Assignment Endpoints (`super_admin` only)

Schema assignment endpoints also live under `/super-admin/*` so that the same first-segment resource extraction rule confines them to the `super-admin` Permit.io resource. Placing them under `/admin/*` (v1) or under `/document/*` would expose them to `user_admin` or `client_user` respectively, which we explicitly do not want.

#### `PATCH /super-admin/documents/{id}/schema`

Assign or change a custom schema on a single document.

> **Path note**: The sub-segment uses plural `documents/{id}` (not `document/{id}` as in v1) to avoid any possible collision with the existing top-level `/document/{id}` resource path during code review or manual URL inspection. This has no authorization impact because the first segment (`super-admin`) is what the middleware uses to resolve the resource, but it makes the namespacing visually unambiguous.

**Request:**
```json
{
    "customSchemaId": "019577a3-..."
}
```

**Response (200):**
```json
{
    "documentId": "...",
    "customSchemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "schemaVersion": 1
}
```

**Validation rules:**
- Schema must belong to the same client as the document (enforced at app layer + **composite FK `fk_documents_custom_schema_same_client`** -- v4 replaces Trigger 2 with the FK)
- Schema must have status `active`
- If the document already has custom metadata written, the schema cannot be changed (returns 409, also enforced by DB Trigger 1 `trg_prevent_schema_reassignment`)
- If the document already has legacy field extractions, the schema cannot be assigned (returns 409 -- mutual exclusivity)
- Setting `customSchemaId` to `null` removes the binding (only if no custom metadata exists)

#### `POST /super-admin/folders/{folderId}/assign-schema` (Bulk Folder Assignment)

Assign a custom schema to ALL documents in a folder and all its subfolders (recursive).

**Request:**
```json
{
    "customSchemaId": "019577a3-..."
}
```

**Response (200):**
```json
{
    "folderId": "...",
    "customSchemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "documentsUpdated": 87,
    "documentsSkipped": 5,
    "skippedReasons": [
        {"documentId": "...", "reason": "Document already has custom metadata with a different schema"},
        {"documentId": "...", "reason": "Document already has custom metadata with a different schema"},
        {"documentId": "...", "reason": "Document already assigned to same schema (no-op)"},
        {"documentId": "...", "reason": "Document has legacy field extractions"},
        {"documentId": "...", "reason": "Document has legacy field extractions"}
    ]
}
```

**Behavior:**
- Uses `WITH RECURSIVE` CTE to walk the entire folder subtree (same pattern as folder deletion)
- For each document in the tree:
  - If the document has no schema, no custom metadata, and no legacy extractions -> assign the schema
  - If the document already has the same schema -> skip (no-op, counted in `documentsSkipped`)
  - If the document has a different schema but no custom metadata yet -> reassign to new schema
  - If the document has a different schema AND has custom metadata -> skip with reason (cannot change)
  - If the document has legacy field extractions -> skip with reason (mutual exclusivity)
- Runs in a single transaction
- **Document count limit**: If the folder subtree contains more than `MaxBulkAssignDocuments` (10,000) documents, returns `422 Unprocessable Entity` with a message explaining the limit. This prevents unbounded transaction duration, excessive lock contention, and replication lag.
- Returns summary of what was updated and what was skipped (with reasons)
- Requires `super_admin` role (the `super-admin` resource grants neither `user_admin` nor `client_user` access)

#### `POST /super-admin/documents/{id}/reset-metadata`

Wipe all custom metadata and schema binding from a single document so the document can be re-bound to a different schema (typically a newer version of the same schema, e.g., after an additive field was added via `POST /super-admin/custom-schemas/{schemaId}/versions`).

This is the explicit, audited escape hatch for the "schema is frozen once metadata exists" invariant. Under normal circumstances, once `document_custom_metadata` has any rows for a document, Trigger 1 (`trg_prevent_schema_reassignment`) blocks `documents.custom_schema_id` from ever changing. Reset removes the blocking state (the metadata rows themselves and the schema binding) in a single transaction so the existing assign-schema + write-metadata endpoints can operate from a clean slate. **The one surviving trigger (`trg_prevent_schema_reassignment`) is not added, modified, or bypassed** -- after the metadata rows are deleted inside the transaction, the trigger's `EXISTS` check simply has nothing to object to.

**Request:** empty body (path parameter `id` is the document ID).

**Response (200):**
```json
{
    "documentId": "019577de-...",
    "previousSchemaId": "019577a3-...",
    "previousSchemaName": "aircraft-engineering",
    "previousSchemaVersion": 1,
    "metadataVersionsDeleted": 3,
    "resetAt": "2026-04-13T12:00:00Z",
    "resetBy": "b7a4c1d2-8e3f-4a5b-9c6d-0e1f2a3b4c5d"
}
```

The `previous*` fields describe the state *before* the reset so the caller can log or display what was wiped. The post-reset state (`custom_schema_id = NULL`, no metadata rows) is implicit — callers that want to confirm it can hit `GET /document/{id}` next. We deliberately do **not** echo a `customSchemaId: null` / `hasCustomMetadata: false` pair back on this response; they are invariant for every successful reset and the `ResetMetadataResult` struct (Section 7.1) does not carry them.

**Transaction sequence** (single transaction, serializable-acceptable at read committed with the row locks below):

1. `SELECT id, "clientId", custom_schema_id FROM documents WHERE id = $1 FOR UPDATE` -- lock the document row so concurrent assigns/writes serialize behind the reset.
2. If the document does not exist, return `404 Not Found`.
3. Capture `previousSchemaId` (may be `NULL`) and join to `client_metadata_schemas` to capture `previousSchemaName` / `previousSchemaVersion` for the response (null if no schema was bound).
4. `SELECT COUNT(*) FROM document_custom_metadata WHERE document_id = $1` -- captured as `metadataVersionsDeleted` for the response.
5. `DELETE FROM document_custom_metadata WHERE document_id = $1`.
6. `UPDATE documents SET custom_schema_id = NULL WHERE id = $1`. Trigger 1 fires on this UPDATE; because step 5 already removed all metadata rows and the document is guaranteed (by Invariant 14 and the existing mutual-exclusivity guards) to have no legacy field extractions, the trigger's `EXISTS` checks both return false and the UPDATE succeeds.
7. Insert an audit log entry (same log sink used by `AssignSchema` / `DeleteSchema`) recording `actor`, `documentId`, `previousSchemaId`, `metadataVersionsDeleted`, and `resetAt`.
8. Commit.

**Behavior and guarantees:**

- **Atomic**: If any step fails, the transaction rolls back and the document is unchanged.
- **Idempotent on already-clean documents**: If the document has no custom metadata and `custom_schema_id IS NULL`, the endpoint succeeds and returns `metadataVersionsDeleted: 0`, `previousSchemaId: null`. This keeps retries safe.
- **Does NOT touch legacy field extractions**: The mutual-exclusivity invariant (Invariant 14) still holds. A document that has any legacy field extraction rows is **rejected with `409 Conflict` before any DELETE runs** -- see validation below. Reset-metadata is deliberately scoped to documents using the custom schema system; it is not a "force wipe everything" endpoint and will not silently no-op on legacy documents.
- **Does NOT touch the schema definitions**: `client_metadata_schemas` rows are untouched. Retiring the document's binding does not affect other documents using the same schema.
- **Does NOT re-assign the new schema**: This endpoint is deliberately scoped to "wipe" only. The caller must follow up with `PATCH /super-admin/documents/{id}/schema` to bind the new schema and `POST /custom-metadata` to re-enter values. Keeping the two operations separate preserves the existing endpoints' semantics and makes the audit trail explicit (one log entry for the reset, one for the re-assignment, one per metadata write).
- **History is lost**: All prior metadata versions for this document are deleted. If the caller needs to preserve values across the reset, they must `GET /custom-metadata` beforehand and re-POST after the new schema is bound. Consider this the v3 "release valve," not the normal path. The `metadataVersionsDeleted` count in the response is a safety check for callers who want to confirm how much history they gave up.

**Validation and error responses:**

- `404 Not Found` if the document does not exist in this stack. Per Section 5.3 the stack itself is the client scope, so the handler only needs to check existence — no additional client-ownership check is performed.
- `409 Conflict` if the document has legacy field extraction rows (`documentFieldExtractionVersions`). Message: "Document uses the legacy field extraction system; reset-metadata only applies to documents using the custom schema system." This mirrors the mutual-exclusivity guard used elsewhere.
- `200 OK` on successful wipe (even if the document was already clean).
- `500` on unexpected DB errors. The transaction ensures partial state is impossible.

**Authorization:**

- Path lives under `/super-admin/*`, so the first-segment resource extraction rule maps the request to the `super-admin` Permit.io resource. Only the `super_admin` role holds `super-admin:post`, so `user_admin`, `client_user`, and `auditor` all receive `403`. No new Permit.io resource or action is required -- the existing `super-admin:post` grant already covers this route.
- The endpoint is also reflected in the documentation-only `policy_mappings` block in `permit_policies.yaml` (see Section 5.3).

**Why wipe-and-reassign instead of allowing a compatible schema swap in place?**

An earlier design considered relaxing Trigger 1 to permit reassignment when the new schema is a strict superset of the old one (additional optional fields, no type changes, no newly-required fields). That approach was rejected because:

1. **Invariant 5 stays clean.** "`custom_schema_id` is immutable once metadata exists" is a strong, trivially auditable rule. Introducing "immutable *except* when the new schema is compatible" replaces a one-line check with a structural JSON Schema comparator that must agree exactly across the app layer and the DB trigger. Keeping the trigger unchanged is worth the cost of a destructive reset for the edge case of in-place upgrades.
2. **No silent reinterpretation of stored data.** Deleting the metadata and requiring the caller to re-submit guarantees that the new schema is actually applied to the values being stored, with full validation. An in-place schema swap would leave the JSONB blobs untouched and implicitly promote them to the new schema -- which means a later schema tightening (e.g., a pattern constraint added to an existing field) would not catch violations in legacy rows.
3. **Audit clarity.** The reset leaves an explicit log entry indicating that a super_admin chose to discard prior metadata history for this document. An in-place swap would silently change the schema linkage with no indication that the on-disk data predates the new schema.
4. **Simpler to implement and test.** One new endpoint, zero migrations, zero trigger changes, and the reset path owns its own focused queries (`DeleteDocumentCustomMetadataForReset`, `NullifyDocumentCustomSchemaIdForReset`, see Section 8.2). No JSON Schema compatibility comparator to build, maintain, or debug. (v4 note: v3 anticipated sharing these with a delete-cascade workstream; v4 replaced that workstream with `ON DELETE CASCADE` FKs, so the reset path is the sole caller.)

If a non-destructive upgrade path is needed in the future, it can be added as a separate post-v4 endpoint (`POST /super-admin/documents/{id}/upgrade-schema`) that performs a JSON Schema compatibility check and a targeted UPDATE of `documents.custom_schema_id` guarded by a narrowly scoped relaxation of Trigger 1. v4 does not commit to that design.

#### `GET /document/{id}` (Modified)

Add exactly two fields to `DocumentEnriched`:

```json
{
    "id": "...",
    "clientId": "acme-corp",
    "hash": "...",
    "customSchemaId": "019577a3-...",
    "hasCustomMetadata": true,
    ...existing fields...
}
```

**v4 scope trim**: v3 proposed also adding `customSchemaName` (decoration) and an optional inlined `customMetadata` JSONB blob gated by a new `?customMetadata=true` query parameter. v4 drops both:

- `customSchemaName`: callers that need the name can resolve it via `GET /super-admin/custom-schemas/{schemaId}` (or a schema-list endpoint already on the same client). Duplicating it on every document read response is a denormalization cost on a cross-cutting read path.
- Inline `customMetadata` blob: the feature already ships a dedicated `GET /custom-metadata` endpoint that returns the current metadata version in full. Inlining the same payload on `GET /document/{id}` would add controller, service, query, OpenAPI, and test work to a widely used read path without giving callers any new capability.

The remaining two fields (`customSchemaId`, `hasCustomMetadata`) are deliberately minimal: `customSchemaId` is already a column on the `documents` row (no extra query), and `hasCustomMetadata` is a single `EXISTS` subquery that piggybacks on the existing document read.

> **Authorization note**: The `GET /document/{id}` endpoint stays on the existing `document` resource. Both admin roles, `client_user`, and `auditor` can read it today. Returning `customSchemaId` / `hasCustomMetadata` on this endpoint is read-only and does not grant any administrative capability. The `super-admin` gating applies only to the write/assign/manage paths.

---

### 5.3 Authorization & Permit.io Changes

> **Isolation model (v4).** Production is deployed **one client per stack**: each client runs in its own isolated environment with a separate database, separate services, and a separate Cognito pool. There is no shared production tenant, so the first-path-segment Permit.io check performed by `internal/cognitoauth/middleware.go` and `internal/cognitoauth/permitio.go` is sufficient for production — a caller's JWT can only reach their own client's data because the stack itself only contains that client's data. The new `/custom-metadata/*` and `/super-admin/custom-schemas/*` endpoints inherit this model intentionally; role gating (the table below) is the whole authorization story for these routes.
>
> **Shared non-production caution.** Shared dev/uat stacks may host fixtures for more than one client in the same database. Do not load customer-sensitive fixtures into a shared stack — treat those environments as test data only. The role gating below still applies, but "client" is not an enforced security boundary in shared stacks by design.
>
> **If the deployment model ever changes.** If any future environment hosts more than one real client in a single stack, the `custom-metadata` and `super-admin` resources must be retrofitted to receive document and client objects in the Permit.io check, and the service layer must refuse cross-stack reads/writes with `404 Not Found` (never `403`, to avoid leaking existence). That is a separate initiative that would also need to retrofit `documents`, `field-extractions`, and `folders`, and is explicitly **out of scope** for this plan.

v2 splits admin-class authorization for mutable metadata into two tiers; v3 keeps the same split and simply adds the reset-metadata endpoint to the `super_admin`-only column:

| Role | Schema CRUD (`/super-admin/custom-schemas/*`) | Schema Assignment (`/super-admin/documents/{id}/schema`, `/super-admin/folders/{folderId}/assign-schema`) | Reset Metadata (`/super-admin/documents/{id}/reset-metadata`) | Metadata Read (`GET /custom-metadata/*`, `GET /document/{id}` enrichment) | Metadata Write (`POST /custom-metadata`) |
|------|-----------------------------------------------|-----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------|----------------------------------------------------------------------------|-------------------------------------------|
| `super_admin` | Yes | Yes | Yes | Yes | Yes |
| `user_admin`  | **No** | **No** | **No** | Yes | Yes |
| `auditor`     | Read-only (`GET` only) | No | No | Read-only | No |
| `client_user` | No | No | No | Yes | Yes |

Key change from v1: `user_admin` has NO schema CRUD and NO schema assignment rights. The role continues to function exactly as it does today for everything else (user creation, labels, folders, etc.). Reset-metadata (new in v3) follows the same confinement: only `super_admin` can call it, because it is destructive and structurally equivalent to a schema reassignment.

#### New Permit.io resource: `super-admin`

Add a new resource to `cmd/auth_related/permit.setup/permit_policies.yaml`:

```yaml
resources:
  # ...existing resources unchanged...

  - name: super-admin
    description: Platform-level administrative functions restricted to super_admin role (custom metadata schema management, schema assignment)
    actions:
      - get
      - post
      - patch
      - delete
```

#### Role permission updates

```yaml
roles:
  - name: super_admin
    description: Full system access for platform administrators
    permissions:
      # ...existing permissions unchanged...
      super-admin:
        - get
        - post
        - patch
        - delete
      custom-metadata:
        - get
        - post

  - name: user_admin
    description: User management - primarily for creating new client users
    permissions:
      # ...existing permissions unchanged (admin, documents, field-extractions, folders)...
      # NOTE: user_admin is intentionally NOT granted super-admin:* permissions.
      custom-metadata:
        - get
        - post

  - name: auditor
    description: Full read-only access across the entire system for auditing and compliance
    permissions:
      # ...existing permissions unchanged...
      super-admin:
        - get
      custom-metadata:
        - get

  - name: client_user
    description: Client-specific access for external users (requires tenant filtering)
    permissions:
      # ...existing permissions unchanged...
      custom-metadata:
        - get
        - post
```

#### New custom-metadata resource

The user-facing custom metadata endpoints (`/custom-metadata/*`) use a separate `custom-metadata` resource (first path segment). This is unchanged from v1 except that it now also needs to be granted to `super_admin` explicitly (v1 relied on `admin:*` coverage which no longer covers the metadata path).

```yaml
resources:
  - name: custom-metadata
    description: Per-document custom metadata read/write (validated against client schemas)
    actions:
      - get
      - post
```

#### Policy mapping documentation additions

Add to the documentation-only `policy_mappings` section at the bottom of `permit_policies.yaml`:

```yaml
super_admin_endpoints:
  - path: /super-admin/custom-schemas
    methods:
      GET: super-admin:get
      POST: super-admin:post
  - path: /super-admin/custom-schemas/{schemaId}
    methods:
      GET: super-admin:get
      DELETE: super-admin:delete
  - path: /super-admin/custom-schemas/{schemaId}/versions
    methods:
      POST: super-admin:post       # v4: create-new-version, replaces v3's PUT on the parent path
  - path: /super-admin/documents/{id}/schema
    methods:
      PATCH: super-admin:patch
  - path: /super-admin/folders/{folderId}/assign-schema
    methods:
      POST: super-admin:post
  - path: /super-admin/documents/{id}/reset-metadata
    methods:
      POST: super-admin:post

custom_metadata_endpoints:
  - path: /custom-metadata
    methods:
      GET: custom-metadata:get
      POST: custom-metadata:post
  - path: /custom-metadata/version
    methods:
      GET: custom-metadata:get
  - path: /custom-metadata/history
    methods:
      GET: custom-metadata:get
```

> **v4 note on HTTP methods**: v4 avoids the `PUT`-to-`patch` action-mapping ambiguity entirely by replacing the v3 `PUT /super-admin/custom-schemas/{schemaId}` with `POST /super-admin/custom-schemas/{schemaId}/versions`. No `put` action needs to be added to the `super-admin` resource. The `super_admin` role's existing `super-admin:post` grant already covers the new route.

#### Middleware impact

The Permit.io resource-extraction middleware uses the first path segment. Introducing `super-admin` as a resource requires no middleware code changes -- it is purely a configuration addition via the permit setup tool. The setup tool (`cmd/auth_related/permit.setup/`) must be re-run in each environment (dev, uat, prod) to provision the new resource and role permissions before these endpoints can pass authorization.

#### Deployment ordering

Because the authorization changes must land in Permit.io before the API routes are live, the deployment sequence is:

1. Update `permit_policies.yaml` with the new resources and role grants.
2. Run the permit setup tool against dev/uat/prod to provision the new resource and permissions.
3. Deploy the API code containing the new endpoints.

Step 2 must complete before step 3 in each environment, otherwise the new endpoints would either be rejected by Permit.io (if fail-closed) or accidentally allowed (if fail-open) during the window between code deploy and permit sync.

---

### 5.4 Custom Metadata CRUD Endpoints (User-Facing)

All under a new `CustomMetadataService` tag. These endpoints are NOT moved under `/super-admin/*` because they are intended for regular users (including `client_user`) to read and write the custom metadata values on documents. Schema administration and schema assignment are separated from value read/write precisely so the day-to-day data entry workload does not require super_admin.

> **Actor identity is authoritative, not client-supplied (v4).** v4 removes `createdBy` from every new-endpoint request body schema. The handler derives the actor identity from the authenticated JWT / Cognito subject via `cognitoauth.GetUserSubject(c)` and passes it to the service layer as a separate function argument. The resulting value is the JWT `sub` claim (an opaque Cognito subject UUID), **not** an email — email is not guaranteed present in the JWT on this service and is deliberately not used. Response payloads still include `createdBy` / `resetBy` for audit visibility (those values come from the server, not from the caller) and are therefore typed as plain strings, not `format: email`.
>
> Applies to all new endpoints: `POST /super-admin/custom-schemas`, `POST /super-admin/custom-schemas/{schemaId}/versions`, `DELETE /super-admin/custom-schemas/{schemaId}`, `PATCH /super-admin/documents/{id}/schema`, `POST /super-admin/folders/{folderId}/assign-schema`, `POST /super-admin/documents/{id}/reset-metadata`, and `POST /custom-metadata`.
>
> **Audit sink**: Sections 5.2 and 8 reference "the same log sink used by `AssignSchema` / `DeleteSchema`." If that sink does not already exist as a first-class abstraction in `internal/customschema/` or an `audit` package, Milestone 1 must define it (signature, where records are written, whether it is synchronous or queued) before Reset-metadata, Schema CRUD, and Schema assignment can log through it. The audit record must include at minimum: `actor` (derived from auth context), `action`, `resourceId`, `timestamp`, and an action-specific detail payload.

#### `POST /custom-metadata`

Create or update custom metadata for a document. Creates a new version.

**Request:**
```json
{
    "documentId": "...",
    "metadata": {
        "aircraft_model": "Boeing 737-800",
        "certification_date": "2024-06-15",
        "max_altitude_ft": 41000,
        "engine_count": 2
    }
}
```

**Response (201):**
```json
{
    "id": "...",
    "documentId": "...",
    "schemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "schemaVersion": 1,
    "metadata": { ... },
    "version": 1,
    "createdAt": "...",
    "createdBy": "b7a4c1d2-8e3f-4a5b-9c6d-0e1f2a3b4c5d"
}
```

> The response `schemaId`, `schemaName`, and `schemaVersion` are returned as a convenience (resolved server-side from `documents.custom_schema_id -> client_metadata_schemas` via the joined query in Section 8.2) even though the `document_custom_metadata` table no longer stores them. `createdBy` is also server-derived; see the "Actor identity" note in Section 5 — it is the caller's Cognito subject ID, not an email.

**Validation:**
- Document must have a `custom_schema_id` assigned (400 if not -- a super_admin must have assigned the schema first)
- Document must NOT have any legacy field extractions (409 if it does -- mutual exclusivity, also enforced by DB trigger on schema assignment)
- The `metadata` payload must be <= 1MB in size (`MaxMetadataPayloadBytes = 1048576`). Returns 400 if exceeded.
- The `metadata` payload is validated against the JSON Schema stored in `client_metadata_schemas.schema_def`
- The request body does NOT include `schemaId` -- the server derives it from the document's `custom_schema_id`. This prevents client/server drift.
- If validation fails, returns 400 with detailed validation errors:
  ```json
  {
      "error": "Metadata does not conform to schema",
      "validationErrors": [
          {"field": "engine_count", "message": "must be >= 1, got 0"},
          {"field": "aircraft_model", "message": "required field missing"}
      ]
  }
  ```
- On first write, locks the document's `custom_schema_id` (makes it immutable via app-layer flag + DB trigger)
- Subsequent writes create new versions but must use the same schema

#### `GET /custom-metadata`

Get current (latest version) custom metadata for a document.

**Query Parameters:**
- `documentId` (required)

**Response (200):**
```json
{
    "id": "...",
    "documentId": "...",
    "schemaId": "019577a3-...",
    "schemaName": "aircraft-engineering",
    "schemaVersion": 1,
    "metadata": { ... },
    "version": 3,
    "createdAt": "...",
    "createdBy": "b7a4c1d2-8e3f-4a5b-9c6d-0e1f2a3b4c5d"
}
```

> `schemaId`, `schemaName`, and `schemaVersion` are resolved server-side from the document's current binding via the joined `GetCurrentDocumentCustomMetadata` query (Section 8.2). `createdBy` is the Cognito subject ID (see Section 5 "Actor identity"), not an email.

#### `GET /custom-metadata/version`

Get a specific version of custom metadata.

**Query Parameters:**
- `documentId` (required)
- `version` (required, >= 1)

#### `GET /custom-metadata/history`

Get version history for a document's custom metadata.

**Query Parameters:**
- `documentId` (required)
- `limit` (optional, default: 50, max: 200): Maximum number of versions to return
- `offset` (optional, default: 0): Number of versions to skip for pagination

**Response (200):**
```json
{
    "versions": [
        {"id": "...", "documentId": "...", "version": 3, "createdAt": "...", "createdBy": "..."},
        {"id": "...", "documentId": "...", "version": 2, "createdAt": "...", "createdBy": "..."},
        {"id": "...", "documentId": "...", "version": 1, "createdAt": "...", "createdBy": "..."}
    ]
}
```

**Edge-case behavior (v4 — committed, not implementer's choice):**

| Input | Result | Rationale |
|-------|--------|-----------|
| `limit=0` | `400 Bad Request` with message `"limit must be between 1 and 200"` | Zero-limit is a client bug (the caller already decided not to fetch anything); rejecting is louder than returning an empty page. |
| `limit > 200` | Clamp to `200` and return `200 OK` | Matches how the existing list endpoints on this service clamp rather than reject, and keeps callers that pass a large "give me everything" sentinel from failing outright. The clamped value is the one used for the DB query; no warning header is added. |
| `limit < 0` | `400 Bad Request` with message `"limit must be between 1 and 200"` | Same shape as `limit=0` — obvious client bug. |
| `offset < 0` | `400 Bad Request` with message `"offset must be >= 0"` | Same reasoning. |
| `documentId` refers to a document that does not exist (or belongs to another client in a shared stack) | `404 Not Found` with message `"document not found"` | Matches the existing `GET /document/{id}` shape. Using `200 + empty array` would make the endpoint unable to distinguish "document doesn't exist" from "document has never had metadata." |
| `documentId` exists but has zero metadata rows | `200 OK` with `{"versions": []}` | The resource (document) exists; its history is legitimately empty. Returning `404` here would conflate "no history" with "no document" and force callers to double-check. |

These are hard declarations, not guidance — handlers must implement exactly this shape and the integration tests in tracking Milestone 2 cover every row above. Any deviation is a test failure.

---

## 6. Validation Strategy

### 6.1 JSON Schema Validation

Use a Go JSON Schema validation library for both:

1. **Schema validation** (meta-validation): When a super_admin submits a new schema, validate it against the JSON Schema meta-schema to ensure it is a well-formed JSON Schema document.
2. **Metadata validation**: When custom metadata is submitted, validate the payload against the document's assigned schema.

**Recommended library**: `github.com/santhosh-tekuri/jsonschema/v6`

Reasons:
- Supports JSON Schema draft 2020-12
- Active maintenance
- Good error reporting
- Supports custom formats

### 6.2 Schema Definition Requirements

When a schema definition is submitted (via `POST /super-admin/custom-schemas` or `POST /super-admin/custom-schemas/{schemaId}/versions`), the server validates:

1. It is a valid JSON object
2. It compiles successfully as a JSON Schema document (meta-validation)
3. Root `type` is `"object"`
4. Root `additionalProperties` is explicitly present (either `true` or `false`). This is required because JSON Schema defaults `additionalProperties` to `true` when omitted, which silently accepts any extra fields. Requiring explicit declaration forces a conscious choice.
5. Size is <= 64KB

### 6.3 Supported JSON Schema Features

The system will support the following JSON Schema keywords for custom field definitions:

| Feature | JSON Schema Keyword | Example |
|---------|-------------------|---------|
| Data type | `type` | `"string"`, `"number"`, `"integer"`, `"boolean"`, `"array"`, `"object"` |
| Required fields | `required` | `["field1", "field2"]` |
| String length | `minLength`, `maxLength` | `"maxLength": 100` |
| Numeric range | `minimum`, `maximum` | `"minimum": 0` |
| Pattern matching | `pattern` | `"pattern": "^[A-Z]{2}$"` |
| Enum values | `enum` | `"enum": ["active", "inactive"]` |
| Date/time format | `format` | `"format": "date"` |
| Array items | `items`, `minItems`, `maxItems` | Nested array support |
| Nested objects | `properties` | Hierarchical field groups |
| No extra fields | `additionalProperties` | `false` to enforce strict schemas |

### 6.4 Validation Error Reporting

Validation errors will be returned as structured JSON with:
- The JSON pointer to the failing field (`/aircraft_model`)
- The constraint that was violated (`maxLength`)
- A human-readable message

---

## 7. Service Layer Design

### 7.1 New Service: `internal/customschema/`

```
internal/customschema/
    service.go       -- CustomSchemaService (CRUD + validation)
    models.go        -- Input/output types
    validator.go     -- JSON Schema meta-validation + payload validation
    constants.go     -- MaxSchemaDefinitionBytes and other limits
    service_test.go  -- Integration tests
```

**Constants (`constants.go`):**

```go
const (
    // MaxSchemaDefinitionBytes is the maximum size of a JSON Schema definition.
    // This limit can be increased if clients need larger schemas.
    MaxSchemaDefinitionBytes = 65536 // 64KB

    // MaxMetadataPayloadBytes is the maximum size of a custom metadata JSON payload.
    // Prevents unbounded JSONB blobs when schemas use additionalProperties: true.
    MaxMetadataPayloadBytes = 1048576 // 1MB

    // MaxBulkAssignDocuments is the maximum number of documents that can be updated
    // in a single bulk folder schema assignment. Prevents unbounded transactions.
    MaxBulkAssignDocuments = 10000
)
```

**Key methods (v4 -- all write methods take `actor` as a separate argument, never read it off request bodies):**

```go
type Service struct {
    cfg       serviceconfig.ConfigProvider
    validator *SchemaValidator
    audit     AuditSink
}

// Schema management (super_admin operations)
func (s *Service) CreateSchema(ctx, input CreateSchemaInput, actor string) (*Schema, error)
func (s *Service) CreateSchemaVersion(ctx, parentSchemaID uuid.UUID, input CreateSchemaVersionInput, actor string) (*Schema, error)
func (s *Service) GetSchema(ctx, schemaID uuid.UUID) (*Schema, error)
func (s *Service) ListSchemas(ctx, clientID string, filters ListFilters) ([]SchemaSummary, error)
func (s *Service) DeleteSchema(ctx, schemaID uuid.UUID, actor string) error

// Metadata operations (client_user / admin operations)
func (s *Service) SetDocumentMetadata(ctx, input SetMetadataInput, actor string) (*CustomMetadata, error)
func (s *Service) GetCurrentMetadata(ctx, documentID uuid.UUID) (*CustomMetadata, error)
func (s *Service) GetMetadataByVersion(ctx, documentID uuid.UUID, version int) (*CustomMetadata, error)
func (s *Service) GetMetadataHistory(ctx, documentID uuid.UUID) ([]MetadataVersion, error)

// CustomMetadata is returned by the Set/Get/GetByVersion methods above.
// SchemaID / SchemaName / SchemaVersion are resolved via the joined
// GetCurrentDocumentCustomMetadata / GetDocumentCustomMetadataByVersion
// queries (Section 8.2). They are NOT stored on the metadata row itself.
type CustomMetadata struct {
    ID            uuid.UUID
    DocumentID    uuid.UUID
    SchemaID      uuid.UUID
    SchemaName    string
    SchemaVersion int
    Metadata      json.RawMessage
    Version       int
    CreatedAt     time.Time
    CreatedBy     string // Cognito subject ID (see Section 5 "Actor identity")
}

// Document schema binding (super_admin operations)
func (s *Service) AssignSchema(ctx, documentID uuid.UUID, schemaID *uuid.UUID, actor string) (*AssignSchemaResult, error)
func (s *Service) AssignSchemaToFolder(ctx, folderID uuid.UUID, schemaID uuid.UUID, actor string) (*BulkAssignResult, error)
func (s *Service) ResetDocumentMetadata(ctx, documentID uuid.UUID, actor string) (*ResetMetadataResult, error)
```

`CreateSchemaVersion` replaces the v3 `UpdateSchema` method to match the new `POST /super-admin/custom-schemas/{schemaId}/versions` route. The semantics are unchanged (lock existing versions for the `(client_id, name)` pair, read max version, insert new row, mark the parent as `superseded` in the same transaction). The method's parameter name is changed from `schemaID` to `parentSchemaID` to make the lineage explicit.

**AssignSchemaResult:**

```go
type AssignSchemaResult struct {
    DocumentID     uuid.UUID
    CustomSchemaID *uuid.UUID // nil when the caller cleared the binding
    SchemaName     *string    // nil when the caller cleared the binding
    SchemaVersion  *int       // nil when the caller cleared the binding
}
```

The `AssignSchema` signature takes `schemaID *uuid.UUID` so callers can clear the binding by passing `nil` (a non-nil pointer is treated as "assign or re-assign"). The existing validation path already reads the target schema row to verify `status='active'` and the same-client constraint, so returning `SchemaName` / `SchemaVersion` is a free decoration — no new query is required. When the caller clears the binding (`schemaID == nil`), all three schema-related fields in the result are `nil`. This replaces v3's `error`-only return, which did not give the `PATCH /super-admin/documents/{id}/schema` handler enough information to build its documented response.

**ResetMetadataResult:**

```go
type ResetMetadataResult struct {
    DocumentID              uuid.UUID
    PreviousSchemaID        *uuid.UUID // nil if document had no schema bound
    PreviousSchemaName      *string    // nil if document had no schema bound
    PreviousSchemaVersion   *int       // nil if document had no schema bound
    MetadataVersionsDeleted int
    ResetAt                 time.Time
    ResetBy                 string
}
```

`ResetDocumentMetadata` runs a single transaction that: locks the document row (`SELECT ... FOR UPDATE`), rejects documents with legacy field extractions (`409`), captures the previous schema binding metadata (via join to `client_metadata_schemas`) and the count of metadata versions for the response, executes the `DeleteDocumentCustomMetadataForReset` and `NullifyDocumentCustomSchemaIdForReset` queries from Section 8.2, writes an audit log entry, and returns the captured "previous" state. See Section 5.2 for the full endpoint semantics and rationale.

> **v4 query ownership**: In v3, the `DeleteDocumentCustomMetadata` and `NullifyDocumentCustomSchemaId` queries lived on the delete-cascade code path and were reused by reset-metadata. In v4 the delete cascade no longer needs either query (FK cascade handles it, see Section 8.5), so these two queries are **owned by reset-metadata alone** and live in `internal/database/queries/customschemas.sql`, renamed to `DeleteDocumentCustomMetadataForReset` and `NullifyDocumentCustomSchemaIdForReset` to make the single-caller attribution explicit. Functionally identical to the v3 queries, just renamed and attributed to a single caller.

**BulkAssignResult:**

```go
type BulkAssignResult struct {
    FolderID         uuid.UUID
    SchemaID         uuid.UUID
    DocumentsUpdated int
    DocumentsSkipped int
    SkippedReasons   []SkippedDocument
}

type SkippedDocument struct {
    DocumentID uuid.UUID
    Reason     string
}
```

> **Authorization is enforced at the HTTP middleware layer via Permit.io**, not inside these service methods. The service methods are agnostic to the caller's role. The routing layer for `/super-admin/*` endpoints is what ensures only `super_admin` can invoke `CreateSchema`, `CreateSchemaVersion`, `DeleteSchema`, `AssignSchema`, and `AssignSchemaToFolder`. Keeping the service layer role-unaware preserves testability and makes integration tests simpler.

### 7.2 Validator Component

```go
type SchemaValidator struct{}

// ValidateSchemaDefinition checks that a schema_def is valid JSON Schema.
// Returns error if: not a valid JSON object, fails meta-validation,
// root type is not "object", or additionalProperties is not explicitly present.
func (v *SchemaValidator) ValidateSchemaDefinition(schemaDef json.RawMessage) error

// ValidateMetadata checks that metadata conforms to a compiled schema
func (v *SchemaValidator) ValidateMetadata(schemaDef json.RawMessage, metadata json.RawMessage) ([]ValidationError, error)
```

### 7.3 Mutual Exclusivity Guard

The service layer enforces mutual exclusivity at two points, and **both sides must serialize on the parent `documents` row** to close the concurrent-write race. A non-locking pre-check is not sufficient because PostgreSQL's default `READ COMMITTED` isolation lets two transactions each read a stale snapshot and then commit opposing writes.

**The race (what a naive pre-check fails to prevent):**

```
T1 (AssignSchema)                   T2 (CreateFieldExtraction)
-----------------                   --------------------------
BEGIN                               BEGIN
                                    SELECT custom_schema_id FROM documents
                                      -> NULL (passes app pre-check)
UPDATE documents
  SET custom_schema_id = S          INSERT documentFieldExtractions (...)
  (Trigger 1: EXISTS extractions?     (Trigger 1 already fired;
   -> empty -> OK)                     no re-check on this INSERT)
COMMIT                              COMMIT
Final state: document has BOTH custom_schema_id AND legacy extractions.
```

**The fix (two symmetric parent-row locks plus Trigger 2):**

1. **Schema assignment** (`AssignSchema` in `internal/customschema/service.go`): First statement inside the transaction is `SELECT id, "clientId", custom_schema_id FROM documents WHERE id = $1 FOR UPDATE`. Only after the row lock is held does the service check for existing `documentFieldExtractionVersions` rows and then run the `UPDATE`. Any concurrent legacy writer that tries to take the same row lock (or Trigger 2's `FOR SHARE` read) will block until this transaction commits or rolls back, and will then observe the final `custom_schema_id` value.

2. **Legacy field extraction write** (modification to `internal/fieldextraction/service.go::CreateFieldExtraction` and every other legacy mutation entry point listed in `tracking.md` section 2.5): First statement inside the transaction is `SELECT id, custom_schema_id FROM documents WHERE id = $1 FOR UPDATE`. If `custom_schema_id IS NOT NULL`, return the sentinel `ErrMutualExclusivityViolation` (which the controller maps to `409 Conflict`). Only if the column is null does the service proceed with the existing `AddFieldExtraction` + version-lock + version-insert sequence.

The parent-row lock is held for the duration of the enclosing transaction, so it serializes with both:
- a concurrent `AssignSchema` on the same document (which holds `FOR UPDATE` on the same row), and
- Trigger 2 on `documentFieldExtractions INSERT` (which takes `FOR SHARE` on the same row, see Section 4.7).

**Defense-in-depth (Section 4.7):**
- **Trigger 1** (`trg_prevent_schema_reassignment`) on `documents BEFORE UPDATE OF custom_schema_id`: rejects assignment if legacy extractions or custom metadata exist.
- **Trigger 2** (`trg_prevent_legacy_extraction_on_custom_document`) on `documentFieldExtractions BEFORE INSERT`: rejects legacy inserts on documents already bound to a custom schema. Takes a `FOR SHARE` lock on the `documents` row, which cannot be granted while `AssignSchema` holds `FOR UPDATE`.

Both triggers catch anything that slips past the service layer (admin SQL, a new code path that forgets the lock, a migration script). The service-layer locks are the common-path fast check; the triggers are the guarantee.

**What must NOT happen:**
- A legacy mutation entry point that does not take the parent-row lock. Every write path in `internal/fieldextraction/` must be audited (tracking section 2.5 enumerates them).
- A read-only pre-check (`SELECT` without `FOR UPDATE`). That is the exact shape of the bug that motivates this section.

### 7.4 API Controller: `api/queryAPI/customschemas.go`

Follows the same pattern as `fieldextractions.go`:
- Parse and validate request
- Call service layer
- Convert between API and service types
- Return appropriate HTTP status codes

The controller file holds all `/super-admin/custom-schemas/*`, `/super-admin/documents/{id}/schema`, and `/super-admin/folders/{folderId}/assign-schema` handlers. A separate controller file `api/queryAPI/custommetadata.go` holds the user-facing `/custom-metadata/*` handlers. Separating them by file mirrors the separation of authorization tiers and makes it obvious at code-review time which handlers are super_admin-gated.

---

## 8. Migration Plan

### 8.1 Migration Files

Four new migrations. The repo already contains migrations through `00000000000126_batch_document_outcomes_delete_actions`, so v4 starts at `127`. If another branch lands new migrations before this feature merges, renumber the four files below to the next free sequence and update all references in this plan and the tracking doc.

**Migration 127: Create `schema_status_type` enum + `client_metadata_schemas` table**
```
00000000000127_create_client_metadata_schemas.up.sql
00000000000127_create_client_metadata_schemas.down.sql
```

Contents:
- `CREATE TYPE schema_status_type AS ENUM ('active', 'superseded', 'retired')`
- `client_metadata_schemas` table per Section 4.2, including `ON DELETE CASCADE` on the `client_id` FK, the `uq_cms_id_client` composite unique constraint, and the `chk_schema_def_size` CHECK
- Down migration drops the table and enum in the correct order

**Migration 128: Add `custom_schema_id` column + composite FK to `documents` table**
```
00000000000128_add_custom_schema_id_to_documents.up.sql
00000000000128_add_custom_schema_id_to_documents.down.sql
```

Contents:
- `ALTER TABLE documents ADD COLUMN custom_schema_id uuid NULL` (no inline `REFERENCES` clause -- the composite FK provides that)
- `ALTER TABLE documents ADD CONSTRAINT fk_documents_custom_schema_same_client FOREIGN KEY (custom_schema_id, "clientId") REFERENCES client_metadata_schemas(id, client_id)` (Section 4.4, replaces v3's Trigger 2)
- `CREATE INDEX idx_documents_custom_schema_id ON documents(custom_schema_id)`
- Down migration drops the constraint, the index, and the column in the correct order

> **v4 renumbering note**: In earlier drafts of v4, this migration was numbered 129 and `document_custom_metadata` (now Migration 129) was numbered 128. The order was swapped so Milestone 1 can ship monotonically (127 + 128) and Milestone 2 can ship its own monotonic pair (129 + 130). `golang-migrate` only applies files with versions strictly greater than the current DB version, so Milestone 1 must not leave any gaps that Milestone 2 would need to backfill.

**Migration 129: Create `document_custom_metadata` table**
```
00000000000129_create_document_custom_metadata.up.sql
00000000000129_create_document_custom_metadata.down.sql
```

Contents:
- `document_custom_metadata` table per Section 4.3, including `ON DELETE CASCADE` on the `document_id` FK. **No `schema_id` column. No view.**
- Down migration drops the table.

**Migration 130: Add `trg_prevent_schema_reassignment` and `trg_prevent_legacy_extraction_on_custom_document` triggers**
```
00000000000130_add_schema_invariant_triggers.up.sql
00000000000130_add_schema_invariant_triggers.down.sql
```

Contents:
- `trg_prevent_schema_reassignment()` function + trigger (Section 4.7, Trigger 1).
- `trg_prevent_legacy_extraction_on_custom_document()` function + trigger (Section 4.7, Trigger 2 -- the v4 defense-in-depth pair for the mutual-exclusivity invariant).
- v4 does not create v3's former Trigger 2 for client ownership match (replaced by the composite FK in Migration 128) or v3's former Trigger 3 for schema_id consistency (not needed because `document_custom_metadata` has no `schema_id` column).
- Down migration drops the trigger and function

The filename `00000000000130_add_schema_invariant_triggers.up.sql` is retained for continuity with v3's migration numbering. v4 lands two triggers (`trg_prevent_schema_reassignment` and `trg_prevent_legacy_extraction_on_custom_document`) instead of v3's three. A reviewer comparing v3 and v4 migration diffs will see the simplification clearly by reading the file's body.

### 8.2 SQLC Queries

New query file: `internal/database/queries/customschemas.sql`

**Schema operations:**
- `CreateClientMetadataSchema` - Insert new schema
- `GetClientMetadataSchema` - Get by ID
- `ListClientMetadataSchemas` - List by client with filters (status filter). Computes `documentCount` via a `COUNT(*) FROM documents WHERE custom_schema_id = cms.id` subquery so `canDelete` can be derived in the handler without a second round-trip.
- `GetLatestSchemaByName` - Get latest version of a named schema
- `GetSchemaDocumentCount` - Count documents using a schema (via `documents.custom_schema_id`)
- `SetSchemaStatus` - Update status (for supersede/retire)
- `GetMaxSchemaVersion` - For auto-incrementing version
- `LockSchemaVersionsForName` - `SELECT ... FOR UPDATE` on all versions of a (client_id, name) pair to prevent concurrent version creation race

> **v4 query removed**: `GetSchemaMetadataRecordCount` is gone. Schema usage is knowable from `GetSchemaDocumentCount` alone because `document_custom_metadata` no longer carries a `schema_id` column.

**Metadata operations:**
- `CreateDocumentCustomMetadata` - Insert new metadata version (no `schema_id` parameter -- the column does not exist)
- `GetCurrentDocumentCustomMetadata` - Latest version via `ORDER BY version DESC LIMIT 1` using `idx_dcm_doc_version_desc` (v4: no view). **Joins `documents` and `client_metadata_schemas` in the same query** to return the metadata row together with `schema_id`, `schema_name`, and `schema_version` (sourced from `documents.custom_schema_id -> client_metadata_schemas`). These three columns populate the response fields `schemaId` / `schemaName` / `schemaVersion` on `GET /custom-metadata`; they are **not** stored on `document_custom_metadata` itself. Columns are nullable and will all be null if the document has no schema bound (defensive — in practice a metadata row implies a bound schema).
- `GetDocumentCustomMetadataByVersion` - Specific version. **Also joins** `documents` + `client_metadata_schemas` so `GET /custom-metadata/version` can return the same decoration as `GET /custom-metadata`.
- `GetDocumentCustomMetadataHistory` - All versions (history endpoint returns version summaries only, no schema decoration needed — the history response shape in Section 5.4 does not include schema fields).
- `LockDocumentForMetadataWrite` - `SELECT id FROM documents WHERE id = @document_id FOR UPDATE`. Parent-row lock that serializes both the first write and all subsequent writes for a document (see Section 4.3 concurrency notes). **Replaces** v3's `LockDocumentCustomMetadataForVersion`, which locked the max-version row and could not serialize the first write.
- `GetMaxDocumentMetadataVersion` - `SELECT COALESCE(MAX(version), 0) FROM document_custom_metadata WHERE document_id = @document_id`. Called after the parent-row lock is held.

**Document modifications:**
- `SetDocumentCustomSchemaId` - Update documents.custom_schema_id
- `GetDocumentCustomSchemaId` - Read documents.custom_schema_id
- `BulkSetDocumentCustomSchemaIdInFolderTree` - Set schema for all docs in folder subtree (WITH RECURSIVE)
- `GetDocumentsWithMetadataInFolderTree` - Find docs that already have metadata (for skip reporting)
- `GetDocumentsWithLegacyExtractionsInFolderTree` - Find docs that have legacy extractions (for skip reporting)
- **Reused, not new**: single-document "does this document have a legacy extraction row?" check uses the existing `HasFieldExtraction` query in `internal/database/queries/fieldextractions.sql` (already `EXISTS(SELECT 1 FROM documentFieldExtractionVersions WHERE documentId = $1)`). v4 does **not** add a new `DocumentHasLegacyExtractions` query; the mutual-exclusivity guards in `AssignSchema`, `SetDocumentMetadata`, and `ResetDocumentMetadata` call `HasFieldExtraction` directly. If a readability wrapper is desired at the call site, add a private service-layer helper (e.g. `documentHasLegacyExtractions(ctx, id)` that forwards to `HasFieldExtraction`) — no new SQL.
- **Reused, not new**: folder-subtree total-document count uses the existing `CountDocumentsInFolderTree` in `internal/database/queries/folders.sql` (recursive CTE over `folders` + `documents.folderId`). v4's bulk endpoint calls it directly for the `MaxBulkAssignDocuments` pre-check; no new query.

**`GetDocumentEnriched` (modified, not new)**: `internal/database/queries/document.sql` already defines `GetDocumentEnriched`. Milestone 2 extends its SELECT list to add `d.custom_schema_id` and `EXISTS(SELECT 1 FROM document_custom_metadata WHERE document_id = d.id) AS has_custom_metadata` so `GET /document/{id}` can populate the two new `DocumentEnriched` response fields (Section 5.2) in a single round-trip. The corresponding `DocumentEnriched` struct in `internal/document/service.go` and the `GetEnriched` function in `internal/document/get.go` are extended in the same milestone; see tracking §2.3a.

**Delete cascade operations: none required in v4.**

In v3 the delete cascade added three queries: `DeleteDocumentCustomMetadata` on `document.sql`, `NullifyDocumentCustomSchemaId` on `document.sql`, and `DeleteClientMetadataSchemas` on `client.sql`. **v4 removes all three from the delete cascade path**: `ON DELETE CASCADE` on `document_custom_metadata.document_id` makes the first unnecessary, the v3 claim that `custom_schema_id` must be nulled before document delete was incorrect (Section 4.4 / Section 8.5), and `ON DELETE CASCADE` on `client_metadata_schemas.client_id` makes the third unnecessary. **The existing `DeleteDocumentCascade` and `client.HardDelete` code paths are not touched by v4.**

**Reset-metadata support queries (v4, all in `customschemas.sql`):**
- `LockDocumentForReset` - `SELECT id, "clientId", custom_schema_id FROM documents WHERE id = @document_id FOR UPDATE`. Used by `ResetDocumentMetadata`; uses the same parent-row lock shape as `LockDocumentForMetadataWrite` so the two operations serialize correctly against each other.
- `GetDocumentSchemaBindingForReset` - Join `documents` to `client_metadata_schemas` to return the document's current `custom_schema_id`, the schema `name`, and `version`. Used to populate the `previous*` fields of the reset response. Returns all-null columns cleanly when the document has no schema bound.
- `CountDocumentCustomMetadataVersions` - `SELECT COUNT(*) FROM document_custom_metadata WHERE document_id = @document_id`. Used to populate `metadataVersionsDeleted` in the reset response.
- `DeleteDocumentCustomMetadataForReset` - `DELETE FROM document_custom_metadata WHERE document_id = @document_id`. Owned by the reset-metadata path (v4: not shared with a delete cascade). Name disambiguates from v3's cascade query.
- `NullifyDocumentCustomSchemaIdForReset` - `UPDATE documents SET custom_schema_id = NULL WHERE id = @document_id`. Owned by the reset-metadata path.
- The reset path reuses the existing `HasFieldExtraction` query (not a new `DocumentHasLegacyExtractions`) for the mutual-exclusivity guard — same reason as the AssignSchema / SetDocumentMetadata guards above. If the legacy extraction check returns `true`, `ResetDocumentMetadata` returns 409 and rolls back.

### 8.3 OpenAPI Spec Changes

Add to `serviceAPIs/queryAPI.yaml`:

**New `SuperAdminSchemaService` tag** for super_admin-only endpoints:
- `POST /super-admin/custom-schemas`
- `GET /super-admin/custom-schemas`
- `GET /super-admin/custom-schemas/{schemaId}`
- `POST /super-admin/custom-schemas/{schemaId}/versions` **(v4: replaces `PUT /super-admin/custom-schemas/{schemaId}`)**
- `DELETE /super-admin/custom-schemas/{schemaId}`
- `PATCH /super-admin/documents/{id}/schema`
- `POST /super-admin/folders/{folderId}/assign-schema`
- `POST /super-admin/documents/{id}/reset-metadata`

**New `CustomMetadataService` tag** for end-user endpoints:
- `GET /custom-metadata`
- `POST /custom-metadata`
- `GET /custom-metadata/version`
- `GET /custom-metadata/history`

**New request schemas (v4: none include a `createdBy` field)**:
- `CustomSchemaRequest` (for `POST /super-admin/custom-schemas`)
- `CustomSchemaVersionRequest` (for `POST /super-admin/custom-schemas/{schemaId}/versions` -- v4 rename from v3's `CustomSchemaUpdateRequest`)
- `CustomMetadataRequest`
- `BulkSchemaAssignRequest`
- `DocumentSchemaAssignRequest`

**New response schemas**:
- `CustomSchemaResponse`
- `CustomSchemaListResponse`
- `CustomMetadataResponse`
- `CustomMetadataHistoryResponse`
- `ValidationErrorResponse`
- `BulkSchemaAssignResponse`
- `DocumentSchemaAssignResponse`
- `ResetDocumentMetadataResponse`

**New enum**: `SchemaStatus` with values `active`, `superseded`, `retired`

**Modified schemas**: `DocumentEnriched` (v4 adds exactly two fields: `customSchemaId` and `hasCustomMetadata`. v3's proposed `customSchemaName` and inlined `customMetadata` payload are dropped -- see Section 5.2).

**Tag descriptions should explicitly call out the authorization boundary:**
- `SuperAdminSchemaService` description: "Custom metadata schema management and assignment. Restricted to the `super_admin` role. Not accessible to `user_admin`."
- `CustomMetadataService` description: "Per-document custom metadata read/write. Accessible to `client_user`, `user_admin`, and `super_admin`."

**Actor fields — schema type (v4):** Every response schema above that carries a `createdBy` or `resetBy` field (`CustomSchemaResponse`, `CustomMetadataResponse`, `CustomMetadataHistoryResponse`, `DocumentSchemaAssignResponse`, `BulkSchemaAssignResponse`, `ResetDocumentMetadataResponse`) MUST declare the field as:

```yaml
createdBy:
  type: string
  minLength: 1
  maxLength: 255
  description: "Cognito subject ID (JWT sub claim) of the caller. Opaque UUID, not an email."
  example: "b7a4c1d2-8e3f-4a5b-9c6d-0e1f2a3b4c5d"
```

Do **NOT** use `format: email` on these fields. This is a deliberate break from existing `createdBy` fields elsewhere in `queryAPI.yaml` (e.g. folder `createdBy`, legacy field-extraction `createdBy`), which declare `format: email`. On this service, email is not guaranteed to be present in the Cognito JWT, so every new endpoint sources actor identity from the `sub` claim via `cognitoauth.GetUserSubject(c)`. Using `format: email` would cause spec validation failures on every response. If existing `createdBy` fields are ever migrated to the same subject-ID model, they will need a separate migration and changelog; v4 does not touch them.

### 8.4 Legacy Field Extraction Guard

Modify the existing field extraction service (`internal/fieldextraction/service.go`):

At the **top** of every mutation entry point's transaction (starting with `CreateFieldExtraction`, then every other legacy write method), take `SELECT id, custom_schema_id FROM documents WHERE id = $1 FOR UPDATE`. If `custom_schema_id IS NOT NULL`, return the sentinel `ErrMutualExclusivityViolation`, which the controller maps to `409 Conflict`. This lock is the service-layer half of the mutual-exclusivity serialization described in Section 7.3; Trigger 2 on `documentFieldExtractions INSERT` (Section 4.7) is the DB-layer half.

Important: the parent-row lock must be the **first** statement inside the transaction, before `AddFieldExtraction` or any other DML. Taking the lock after the first INSERT defeats the purpose -- Trigger 2 still catches the race, but the error path is less clean and the app's 409 mapping relies on SQLSTATE detection instead of a sentinel from the service.

This is a small modification to each existing code path, not a new endpoint.

### 8.5 Delete Path Updates: **None required in v4**

v3 added an entire workstream (Phase 5b in the v3 tracking doc) to update `DeleteDocumentCascade` and `client.HardDelete` to wire in three new SQLC queries (`DeleteDocumentCustomMetadata`, `NullifyDocumentCustomSchemaId`, `DeleteClientMetadataSchemas`). **v4 removes that workstream entirely.** The existing delete code paths are not modified.

Two independent design choices let v4 drop the extra delete code:

1. **`ON DELETE CASCADE` on `document_custom_metadata.document_id` and `client_metadata_schemas.client_id`**. Hard-deleting a document causes PostgreSQL to automatically delete its custom metadata rows. Hard-deleting a client causes PostgreSQL to automatically delete its schemas -- which is safe because by the time the client row is deleted, every document that could have been referencing a schema is already gone (documents are cascaded first in `client.HardDelete`).
2. **The v3 claim that `custom_schema_id` must be nulled before the document row delete was incorrect.** PostgreSQL does not require a referencing column to be cleared before the referencing row is deleted -- deleting the referencing row removes the reference atomically. The FK on `documents.custom_schema_id` (now composite, see Section 4.4) points *out* of the `documents` row, not *in* to it, so there is nothing to unlink first.

**What still happens in the delete code paths (unchanged from main):**

`DeleteDocumentCascade` in `internal/document/` continues to:
1. Delete `documentFieldExtractionArrayFields`
2. Delete `documentFieldExtractionVersions`
3. Delete `documentFieldExtractions`
4. Delete `documentCleanEntries` / `documentCleans` / `documentEntries`
5. Delete the `documents` row

Step 5 automatically cascades to `document_custom_metadata` via the FK.

`client.HardDelete` in `internal/client/` continues to:
1. Delete all documents with full cascade (which now auto-cascades to their custom metadata)
2. Delete `documentUploads`
3. Delete all folders
4. Delete collector data, batch uploads, sync records
5. Delete the `clients` row

Step 5 automatically cascades to `client_metadata_schemas` via the FK. The FK cascade is safe because step 1 already deleted every document that could have referenced any schema, so no `documents.custom_schema_id` FK check can fail at cascade time.

**Regression risk**: Because v4 does not touch any of this code, there is no regression risk to existing delete paths. The only thing that needs to be verified is that `task fullsuite:ci` still passes end-to-end with documents and clients that carry custom metadata (see the delete-cascade integration tests in Section 12 / the v4 tracking doc). **v4 intentionally does not split out a "delete cascade phase"** -- verification is folded into the integration test milestone.

**Reset-metadata is not a delete-cascade path.** Reset-metadata (Section 5.2) explicitly wipes `document_custom_metadata` rows while keeping the `documents` row alive. That code path owns its own `DeleteDocumentCustomMetadataForReset` and `NullifyDocumentCustomSchemaIdForReset` queries (Section 8.2) and is unrelated to the delete cascade discussion in this section.

### 8.6 Permit.io Setup Tool Run

After updating `permit_policies.yaml` per Section 5.3, run the permit setup tool in each environment:

```
cmd/auth_related/permit.setup/run.tool.dev.sh
cmd/auth_related/permit.setup/run.tool.uat.sh
cmd/auth_related/permit.setup/run.tool.prod.sh
```

The tool is idempotent and will provision the new `super-admin` and `custom-metadata` resources and update role grants without disturbing existing resources. Run this **before** deploying the API code so the new endpoints are authorized the moment they go live.

---

## 9. Invariants & Business Rules

| # | Rule | Enforcement (v4) |
|---|------|-------------|
| 1 | Schema must be valid JSON Schema with explicit `additionalProperties` | Application: meta-validation on create/version-create |
| 2 | Schema `name` + `version` unique per client | Database: unique constraint `uq_client_schema_name_version` |
| 3 | New schema version gets new ID; old version becomes `superseded` | Application: INSERT new row, UPDATE old status in the same transaction (service method `CreateSchemaVersion`) |
| 4 | Schema deletion blocked if documents reference it | Application: `GetSchemaDocumentCount` check before retirement (v4: single-table check; `document_custom_metadata.schema_id` no longer exists so no second count needed) |
| 5 | Document's `custom_schema_id` is immutable once any extraction exists | Application: check in `AssignSchema` + DB trigger: `trg_prevent_schema_reassignment` |
| 6 | Custom metadata must conform to assigned schema | Application: JSON Schema validation on write |
| 7 | ~~All custom metadata versions for a document use same schema ID~~ | **Removed in v4.** The `document_custom_metadata` table no longer carries a `schema_id` column, so the invariant is structurally impossible to violate -- the "schema a metadata row uses" is always `documents.custom_schema_id` as of the write time, and Invariant 5 keeps that frozen. |
| 8 | Schema must belong to same client as document | Database: composite FK `fk_documents_custom_schema_same_client` (v4 replaces v3's Trigger 2); Application: app-layer check still runs for clean 409 error messages |
| 9 | Only `active` schemas can be assigned to new documents | Application: status check |
| 10 | Custom metadata is versioned (no in-place updates) | Database: version column + unique constraint |
| 11 | Schema definition must be <= 64KB | Application: size check + Database: CHECK constraint |
| 12 | Schema management requires `super_admin` role (NOT `user_admin`) | Permit.io: role-based access on the `super-admin` resource; `user_admin` is explicitly NOT granted this resource |
| 13 | No query/filter on custom metadata values | By design: not implemented (see Principle 9) |
| 14 | A document uses EITHER legacy extractions OR custom metadata, never both | Application: mutual exclusivity checks in both services + DB trigger `trg_prevent_schema_reassignment` |
| 15 | Legacy field extraction writes rejected if document has `custom_schema_id` set | Application: guard in `FieldExtractionService` + 409 mapping in the controller |
| 16 | Custom schema assignment rejected if document has legacy extractions | Application: guard in `AssignSchema` + DB trigger |
| 17 | Metadata payload must be <= 1MB | Application: size check (`MaxMetadataPayloadBytes`) |
| 18 | Bulk folder assignment capped at 10,000 documents | Application: count check (`MaxBulkAssignDocuments`), returns 422 if exceeded |
| 19 | Schema assignment (single doc + bulk folder) requires `super_admin` role | Permit.io: endpoints under `/super-admin/` prefix map to `super-admin` resource; `user_admin` is NOT granted |
| 20 | Delete of a document cleans up custom metadata; delete of a client cleans up schemas | **Database: `ON DELETE CASCADE`** on `document_custom_metadata.document_id` and `client_metadata_schemas.client_id`. No application code changes required in v4. |
| 21 | Metadata read/write is available to `client_user`, `user_admin`, `super_admin` (not `auditor` for writes) | Permit.io: `custom-metadata` resource grants on each role per Section 5.3 |
| 22 | Reset-metadata is atomic: all `document_custom_metadata` rows deleted AND `custom_schema_id` nulled, or neither | Application: single transaction with row-level lock on the document (Section 5.2, `POST /super-admin/documents/{id}/reset-metadata`) |
| 23 | Reset-metadata requires `super_admin` role | Permit.io: endpoint under `/super-admin/` prefix maps to `super-admin` resource; `user_admin` / `client_user` / `auditor` denied |
| 24 | Reset-metadata is rejected if the document has legacy field extractions | Application: `HasFieldExtraction` guard in `ResetDocumentMetadata` returns 409 (mutual exclusivity -- Invariant 14 still holds). v4 reuses the existing `HasFieldExtraction` query rather than adding a new `DocumentHasLegacyExtractions` alias — see Section 8.2. |
| 25 | Reset-metadata is idempotent on already-clean documents | Application: transaction succeeds with `metadataVersionsDeleted: 0` when no custom metadata and no schema binding exist |
| 26 | Actor identity on every mutable write comes from the authenticated JWT `sub` claim (an opaque Cognito subject UUID), not the request body, not an email | **v4**: `createdBy` removed from all new request body schemas (Section 5.4 note, Section 7.1 method signatures). The service layer accepts `actor` as a separate argument, sourced from `cognitoauth.GetUserSubject(ctx)` in the handler. Response `createdBy`/`resetBy` fields are declared as plain strings in OpenAPI (no `format: email`) — see Section 8.3 "Actor fields". |

---

## 10. Relationship to Existing Field Extraction System

The existing static field extraction system (`/field-extractions`) and the new custom schema system (`/super-admin/custom-schemas` + `/custom-metadata`) are **mutually exclusive per document**:

- A document uses EITHER the legacy static field extraction system OR the custom schema system
- A document CANNOT have both legacy field extractions AND custom metadata
- Assigning a custom schema to a document that already has legacy extractions returns `409 Conflict`
- Writing a legacy field extraction to a document that has a custom schema assigned returns `409 Conflict`
- Documents without a custom schema (the default) continue to use the legacy system unchanged
- There is no migration path from legacy to custom for individual documents. If a document needs custom metadata, it should not have legacy extractions written to it first.

**Rationale**: This prevents ambiguity about which system is "authoritative" for a document's metadata. With mutual exclusivity, there is exactly one source of truth per document. This is simpler to reason about, simpler to query, and avoids the data governance problem of conflicting data in two stores.

**No breaking changes** to the existing API or database schema for documents that do not opt into custom schemas. The existing `/field-extractions` endpoints remain on the `field-extractions` resource and are unaffected by the new `super-admin` resource.

---

## 11. Implementation Order -- Vertical Slice Milestones

v3 organized the work as 12 horizontal layers (migrations -> SQLC -> validator -> service -> controllers -> tests), which meant the feature was not end-to-end exercisable until very late in the rollout. v4 reorganizes the work into four vertical slice milestones. Each milestone contains migration/query/service/controller/test work and ends with a real feature that a super_admin or client_user can use. If any milestone slips, the earlier milestones are still usable in isolation.

### Milestone 1: Schema foundation

**Goal**: A super_admin can create and list client-scoped custom schemas. Invalid schemas are rejected. Cross-client bindings are structurally impossible.

**Includes**:
- Migration 127 (`schema_status_type` + `client_metadata_schemas` table with `uq_cms_id_client`)
- Migration 128 (`documents.custom_schema_id` column + composite FK `fk_documents_custom_schema_same_client` -- depends on 127 landing first)
- SQLC queries for schema CRUD (`CreateClientMetadataSchema`, `GetClientMetadataSchema`, `ListClientMetadataSchemas`, `GetLatestSchemaByName`, `GetSchemaDocumentCount`, `SetSchemaStatus`, `GetMaxSchemaVersion`, `LockSchemaVersionsForName`)
- `internal/customschema/validator.go` + constants + dependency on `github.com/santhosh-tekuri/jsonschema/v6`
- Audit sink (`AuditSink` interface, default slog-backed implementation) -- needed now so schema CRUD can log
- Service methods: `CreateSchema`, `CreateSchemaVersion`, `GetSchema`, `ListSchemas`, `DeleteSchema`
- OpenAPI: `SuperAdminSchemaService` tag, `CustomSchemaRequest`, `CustomSchemaVersionRequest`, `CustomSchemaResponse`, `CustomSchemaListResponse`, `SchemaStatus` enum; paths for `POST /super-admin/custom-schemas`, `GET /super-admin/custom-schemas`, `GET /super-admin/custom-schemas/{schemaId}`, `POST /super-admin/custom-schemas/{schemaId}/versions`, `DELETE /super-admin/custom-schemas/{schemaId}`
- Permit.io: `super-admin` resource with `get`/`post`/`patch`/`delete` actions; role grants; **run setup tool against dev/uat/prod BEFORE the controller code is deployed**
- Controllers in `api/queryAPI/customschemas.go` for all five schema CRUD routes
- Authorization integration tests (positive for `super_admin`, 403 for `user_admin`, `client_user`, `auditor`; read-only positive for `auditor`)
- End-to-end tests: super_admin creates schema v1, creates v2 via `POST .../versions`, lists schemas (sees both with `status` correct), deletes retired schema
- Cross-client rejection test: attempt to reference a schema owned by a different client -> FK violation -> 409 response

**Exit criteria**: `task fullsuite:ci` green across all of the above. Feature is half-useful on its own (you can create and inspect schemas but not yet assign them to documents).

### Milestone 2: Core document flow

**Goal**: A super_admin can assign a schema to a document; a client_user can write and read custom metadata; the legacy field extraction guard is in place.

**Includes**:
- Migration 129 (`document_custom_metadata` table, no `schema_id`, no view, `ON DELETE CASCADE`)
- Migration 130 (`trg_prevent_schema_reassignment` and `trg_prevent_legacy_extraction_on_custom_document` triggers -- depends on Migration 129 landing first because Trigger 1 references `document_custom_metadata`)
- SQLC queries for metadata operations (`CreateDocumentCustomMetadata`, `GetCurrentDocumentCustomMetadata`, `GetDocumentCustomMetadataByVersion`, `GetDocumentCustomMetadataHistory`, `LockDocumentForMetadataWrite`, `GetMaxDocumentMetadataVersion`)
- SQLC queries for document binding (`SetDocumentCustomSchemaId`, `GetDocumentCustomSchemaId`; legacy extraction existence check **reuses** the existing `HasFieldExtraction` query — no new `DocumentHasLegacyExtractions` query is added)
- **Extend `GetDocumentEnriched` query** (`internal/database/queries/document.sql`) to SELECT `d.custom_schema_id` and an `EXISTS(SELECT 1 FROM document_custom_metadata WHERE document_id = d.id)` subquery as `has_custom_metadata`. This keeps the enrichment to **one** round-trip; no separate follow-up query.
- **Extend `DocumentEnriched` struct** (`internal/document/service.go`) to add `CustomSchemaID *uuid.UUID` and `HasCustomMetadata bool` fields, and **update `GetEnriched`** (`internal/document/get.go`) to copy these fields out of the extended generated row into the struct.
- Service methods: `SetDocumentMetadata`, `GetCurrentMetadata`, `GetMetadataByVersion`, `GetMetadataHistory`, `AssignSchema`
- Legacy field extraction guard: service-layer sentinel error + controller 409 mapping in `api/queryAPI/fieldextractions.go`
- `GET /document/{id}` enrichment: add `customSchemaId` + `hasCustomMetadata` (and only those two -- v4 scope trim, Section 5.2). The controller-side change in `api/queryAPI` depends on the `GetDocumentEnriched` / `DocumentEnriched` / `GetEnriched` backend extension above — do that work first, then wire the controller builder.
- OpenAPI: `PATCH /super-admin/documents/{id}/schema`, `GET/POST/... /custom-metadata`, `CustomMetadataService` tag, `CustomMetadataRequest`, `CustomMetadataResponse`, `CustomMetadataHistoryResponse`, `ValidationErrorResponse`, `DocumentSchemaAssignRequest`, `DocumentSchemaAssignResponse`, updated `DocumentEnriched`
- Permit.io: `custom-metadata` resource grants (if not already landed in Milestone 1); no setup tool re-run needed if Milestone 1 already provisioned it
- Controllers in `api/queryAPI/customschemas.go` (assign-schema) and `api/queryAPI/custommetadata.go` (metadata CRUD)
- Integration tests for the full user flow: create schema, assign to document, write valid metadata, reject invalid metadata, reject legacy field extraction on custom-schema document, reject schema assignment on legacy document, verify `GET /document/{id}` reflects the new fields
- Concurrency tests for `SetDocumentMetadata` (first-writer parent-row lock)
- Delete cascade verification: delete a document with custom metadata, delete a client with schemas and metadata -- both succeed via FK cascade, no orphaned rows

**Exit criteria**: `task fullsuite:ci` green. The feature now satisfies the central requirement ("client-scoped schemas validated at write time, frozen once metadata exists"). Everything after this milestone is additive.

### Milestone 3: Administrative completion

**Goal**: The full schema lifecycle works end-to-end, including reset-metadata for document-level schema upgrades.

**Includes**:
- Service method `ResetDocumentMetadata` with the full transactional implementation (Section 7.1)
- SQLC queries for reset-metadata (`LockDocumentForReset`, `GetDocumentSchemaBindingForReset`, `CountDocumentCustomMetadataVersions`, `DeleteDocumentCustomMetadataForReset`, `NullifyDocumentCustomSchemaIdForReset`)
- Handler for `POST /super-admin/documents/{id}/reset-metadata` in `api/queryAPI/customschemas_reset.go` (new file, split out from `customschemas.go` to unlock parallel execution with Milestone 4 — see the pre-partitioning note in Milestone 4's dependencies)
- OpenAPI: `ResetDocumentMetadataResponse` schema, path entry, policy-mapping doc entry
- Schema version-creation concurrency tests (two concurrent `POST .../versions` calls produce distinct versions via `LockSchemaVersionsForName`)
- Reset-metadata integration tests: happy path, atomicity under injected failure, idempotency, legacy-extraction guard (409), not-found (404), trigger regression (trigger still fires after reset+reassign+write)
- Authorization tests for reset-metadata and version-creation across all four roles
- End-to-end upgrade flow test: create v1 -> assign -> write metadata -> create v2 via `POST .../versions` -> reset -> re-assign to v2 -> write new metadata with the new field

**Exit criteria**: `task fullsuite:ci` green. A super_admin can complete a "document upgrade to new schema version" without any direct database access.

### Milestone 4: Bulk operations

**Goal**: Bulk folder schema assignment is available for administrative convenience.

**Depends on**: Milestone 2 (single-document `AssignSchema` semantics). Milestone 3's reset-metadata work is **not** a prerequisite — nothing in the recursive folder walk reads or writes the reset path. Milestones 3 and 4 are **parallelizable only if shared files are pre-partitioned first**: as originally drafted both milestones add handler code to `api/queryAPI/customschemas.go` and types to `internal/customschema/models.go`, so running them on separate workstreams without partitioning will cause merge conflicts at the seam. To unlock true parallelism, carve the files before branching: M3's reset handler lives in a new file `api/queryAPI/customschemas_reset.go` with reset types in `internal/customschema/models_reset.go`, and M4's bulk handler lives in `api/queryAPI/customschemas_bulk.go` with bulk types in `internal/customschema/models_bulk.go`. With those four files pre-created (even as empty stubs with just the package declaration), the two workstreams have disjoint file ownership and can land independently, using `task fullsuite:ci` green as the merge gate. Serializing M4 behind M3 remains the safe fallback if the pre-partitioning step is skipped.

**Includes**:
- SQLC queries for the bulk path (`BulkSetDocumentCustomSchemaIdInFolderTree`, `GetDocumentsWithMetadataInFolderTree`, `GetDocumentsWithLegacyExtractionsInFolderTree`, `CountDocumentsInFolderTree`)
- Service method `AssignSchemaToFolder` with the full recursive-CTE behavior, skip-reason reporting, and the 10K document cap
- Handler for `POST /super-admin/folders/{folderId}/assign-schema` in `api/queryAPI/customschemas_bulk.go` (new file, per the pre-partitioning note in this milestone's `Depends on` block)
- OpenAPI: `BulkSchemaAssignRequest`, `BulkSchemaAssignResponse`, path entry, policy-mapping doc entry
- Integration tests: recursive walk, all skip reasons, empty folder, deeply nested trees, `MaxBulkAssignDocuments` boundary (10000 pass, 10001 fail with 422)
- Authorization tests for the bulk endpoint

**Exit criteria**: `task fullsuite:ci` green. All features from v3 are present.

### Documentation (parallel with Milestones 3 and 4)

Once Milestone 2 is green the feature is shippable in a narrow form, and documentation can begin in parallel:

- `docs/ai.generated/` pages for the custom metadata feature: sample schemas from Appendix A, authorization matrix, mutual-exclusivity explanation
- Reset-metadata runbook ("How to upgrade a document to a new schema version") lands alongside Milestone 3
- Bulk folder runbook lands alongside Milestone 4
- Deployment ordering note (run the Permit.io setup tool in each environment before deploying the controllers)

> **Cross-cutting ordering constraint**: In every environment, the Permit.io setup tool must run before the controller code lands, so the new routes are authorized on first request. This constraint applies to Milestone 1 (creates the `super-admin` resource) and Milestone 2 (creates `custom-metadata` resource if it was not rolled up into Milestone 1). Milestones 3 and 4 do not require a setup tool re-run because they reuse existing grants.

---

## 12. Testing Strategy

- **Schema validation tests**: Valid and invalid JSON Schema documents, edge cases (empty schema, deeply nested, self-referential, oversized >64KB, missing `additionalProperties`)
- **Metadata validation tests**: Conforming and non-conforming payloads, all supported JSON Schema keywords
- **Service integration tests**: Full CRUD lifecycle using testcontainers (real PostgreSQL)
- **API integration tests**: HTTP-level tests for all new endpoints under both `/super-admin/*` and `/custom-metadata/*`
- **Invariant tests**: Schema immutability, cross-client isolation, deletion guards, size limits
- **DB-level safety-net tests (v4)**:
  - `trg_prevent_schema_reassignment` fires on direct SQL UPDATE when metadata exists (bypass test) -- positive and negative cases
  - `trg_prevent_schema_reassignment` fires on direct SQL UPDATE when legacy extractions exist (bypass test)
  - Composite FK `fk_documents_custom_schema_same_client` refuses a direct SQL UPDATE that points a document at a schema owned by a different client (replaces v3's Trigger 2 test)
  - `ON DELETE CASCADE` on `document_custom_metadata.document_id` fires on direct SQL delete of a document row
  - `ON DELETE CASCADE` on `client_metadata_schemas.client_id` fires on direct SQL delete of a client row after its documents are gone
- **Schema status lifecycle tests**: `active` -> `superseded` on version create, `active` -> `retired` on delete, assignment rejected for non-`active` schemas
- **Mutual exclusivity tests**: Legacy extraction blocked when schema assigned, schema assignment blocked when legacy extractions exist, bulk folder assignment skips documents with legacy extractions
- **Bulk folder assignment tests**: Recursive assignment, skip behavior for documents with existing metadata and legacy docs, empty folders, deeply nested folder trees
- **Authorization tests**:
  - `super_admin` CAN create, list, get, create-new-version, delete schemas, assign schemas to documents, assign schemas to folders, reset metadata
  - `user_admin` CANNOT access any `/super-admin/*` endpoint (expect 403 on every method + path combination)
  - `user_admin` CAN read and write custom metadata via `/custom-metadata` (no regression)
  - `client_user` CANNOT access any `/super-admin/*` endpoint (expect 403)
  - `client_user` CAN read and write custom metadata via `/custom-metadata`
  - `auditor` CAN read (`GET`) `/super-admin/custom-schemas` and `/super-admin/custom-schemas/{schemaId}` but CANNOT POST/PATCH/DELETE (v4: no `PUT` to test)
  - `auditor` CAN read but CANNOT write `/custom-metadata`
  - Each test should assert both the HTTP status code AND that the database state is unchanged on denial
- **Actor identity tests (v4)**: for every new write endpoint, set the middleware fixture so `cognitoauth.GetUserSubject` returns a known Cognito subject UUID, then POST a body that includes a spurious `createdBy` field with an email-shaped value; assert that (a) the stored `created_by` column holds the subject UUID (not the body value, not an email), (b) the response `createdBy` / `resetBy` fields echo the subject UUID, and (c) the response passes OpenAPI schema validation (confirming the plain-string `createdBy` declaration in Section 8.3 is in force)
- **Backward compatibility tests**: Existing field extraction endpoints continue to work unchanged for documents without custom schemas; existing `/admin/*` endpoints are unaffected by the new `super-admin` resource
- **Delete cascade tests**: Document hard-delete succeeds when document has custom metadata, client hard-delete succeeds when client has schemas and metadata, verify no orphaned rows remain
- **Metadata size limit tests**: Payloads at/above 1MB boundary, verify 400 response with clear message
- **Bulk assignment limit tests**: Folder trees exceeding `MaxBulkAssignDocuments`, verify 422 response
- **Schema version concurrency tests**: Concurrent `POST /super-admin/custom-schemas/{schemaId}/versions` requests for the same schema name produce distinct version numbers
- **Permit.io setup tool tests**: After running the setup tool against a disposable dev tenant, verify that the `super-admin` and `custom-metadata` resources exist with the expected actions and that each role has exactly the grants defined in Section 5.3
- **Reset-metadata tests**:
  - Happy path: document with schema v1 and N metadata versions -> reset -> assign schema v2 -> POST metadata -> `GET /document/{id}` reflects schema v2 and version 1 metadata; `metadataVersionsDeleted` in the response equals N; `previousSchemaId`/`previousSchemaName`/`previousSchemaVersion` reflect the wiped binding.
  - Atomicity: inject a failure between `DELETE FROM document_custom_metadata` and `UPDATE documents SET custom_schema_id = NULL` (e.g., by killing the DB connection mid-transaction in a testcontainers run) and verify that `document_custom_metadata` rows for the document are NOT missing after the failed transaction.
  - Idempotency: reset a document that has no custom schema and no custom metadata -> 200 OK with `metadataVersionsDeleted: 0`, `previousSchemaId: null`, DB state unchanged.
  - Legacy extraction guard: reset a document that has rows in `documentFieldExtractionVersions` -> 409 Conflict, no DB state change.
  - Document not found: reset a non-existent document ID -> 404, no DB state change.
  - Trigger 1 still fires normally after a reset: reset, reassign schema v2, write a metadata row, then attempt a second schema reassignment -> trigger fires with the existing error message (confirms no trigger was weakened by reset-metadata).
  - (v3's "Trigger 3 regression" test is removed in v4: `document_custom_metadata` has no `schema_id` column, so there is nothing to mismatch. The consistency test is replaced by a structural test that confirms the table has no `schema_id` column after migrations.)
  - Authorization: `super_admin` succeeds; `user_admin`, `client_user`, and `auditor` all receive 403 with the DB state unchanged.
  - Concurrency: two concurrent resets against the same document -> one succeeds, one observes the locked row and serializes (either succeeding on the already-clean state with `metadataVersionsDeleted: 0` or blocking until the first completes). No orphaned rows.
  - Concurrency against assign-schema: a reset and a `PATCH /super-admin/documents/{id}/schema` targeting the same document concurrently -> both serialize via `FOR UPDATE`; either ordering produces consistent final state with no trigger violations.
  - Audit log: every successful reset writes an audit entry containing the actor, document ID, previous schema ID, and metadata version count.

---

## 13. Resolved Design Decisions

| # | Question | Decision |
|---|----------|----------|
| 1 | Schema size limits | 64KB cap via `MaxSchemaDefinitionBytes` constant (tunable) |
| 2 | Schema backward compatibility | Not required. New version = new ID. Old docs keep old schema. |
| 3 | Bulk operations | Yes. `POST /super-admin/folders/{folderId}/assign-schema` assigns recursively through folder tree. |
| 4 | Custom metadata search/filter | **Not supported.** No GIN indexes, no query API. Metadata is opaque storage. |
| 5 | Schema templates | No system templates. Documentation will include sample schemas covering all supported types. |
| 6 | Existing field migration path | **Out of scope.** Static fields and custom schemas are mutually exclusive per document. |
| 7 | Endpoint naming/placement | Schema CRUD under `/super-admin/custom-schemas` (requires the `super_admin` role on the new `super-admin` Permit.io resource). Metadata CRUD under `/custom-metadata` (regular `client_user` access; also granted to `user_admin` and `super_admin`). |
| 8 | Legacy/custom coexistence | **Mutually exclusive.** A document uses EITHER legacy OR custom, never both. Simplifies data governance and prevents source-of-truth ambiguity. |
| 9 | Schema status model | Three-state enum (`active`/`superseded`/`retired`) instead of boolean. Distinguishes automatic version succession from explicit admin retirement. |
| 10 | DB-level invariant enforcement | Yes. Two database triggers (`trg_prevent_schema_reassignment` on `documents` UPDATE, `trg_prevent_legacy_extraction_on_custom_document` on `documentFieldExtractions` INSERT) plus one composite foreign key (`fk_documents_custom_schema_same_client`) enforce the invariants as defense-in-depth alongside application-layer checks. v3's `trg_validate_schema_client_match` is replaced by the FK; v3's `trg_enforce_consistent_schema_id` is removed because `document_custom_metadata` has no `schema_id` column. |
| 11 | `additionalProperties` in schemas | Required to be explicitly present (not necessarily `false`). Prevents the common mistake of omitting it and silently accepting arbitrary extra fields. |
| 12 | Metadata table design | Single table with version column (simpler than the legacy two-table pattern). Sufficient for our concurrency needs. Can split later if needed. |
| 13 | Schema assignment endpoint path | Under `/super-admin/documents/{id}/schema` (NOT `/admin/document/{id}/schema` as in v1 and NOT `/document/{id}/schema`). The `/super-admin` first segment maps to the new `super-admin` Permit.io resource which only `super_admin` holds. Placing it under `/admin` would also grant `user_admin`; placing it under `/document` would also grant `client_user`. |
| 14 | Metadata payload size limit | 1MB cap via `MaxMetadataPayloadBytes` constant. Prevents unbounded JSONB blobs when schemas allow `additionalProperties: true`. |
| 15 | Bulk assignment document limit | 10,000 documents via `MaxBulkAssignDocuments` constant. Prevents unbounded transactions and lock contention. |
| 16 | Mutual exclusivity race (v4) | Closed by (a) parent-row `SELECT ... FOR UPDATE` on `documents` at the top of both `AssignSchema` and every legacy `FieldExtractionService` mutation, and (b) DB trigger `trg_prevent_legacy_extraction_on_custom_document` on `documentFieldExtractions` INSERT as defense-in-depth. v3's planned `trg_enforce_consistent_schema_id` on `document_custom_metadata` INSERT is removed in v4 because the table has no `schema_id` column. |
| 17 | **v2**: Who can administer schemas? | **Only `super_admin`.** v1 unintentionally granted `user_admin` access by placing endpoints under `/admin/*`. v2 moves them to `/super-admin/*` and introduces a new `super-admin` Permit.io resource that only `super_admin` holds. `user_admin` retains its existing capabilities (user creation, field extractions, documents/folders, admin endpoints) but has no schema administration rights. |
| 18 | **v2**: Why a new resource instead of path-specific rules? | The Permit.io middleware resolves resources from the first path segment. Adding path-specific overrides would require middleware code changes. Adding a new first-segment resource (`super-admin`) is a pure configuration change and keeps the middleware simple. |
| 19 | **v2**: Does the service layer enforce role checks? | No. Authorization lives in the HTTP/Permit.io layer. The service layer (`internal/customschema/`) is role-agnostic and remains easy to test without fake identity contexts. |
| 20 | **v3**: How does a document move to a newer schema version once it has metadata? | Explicit, destructive reset. `POST /super-admin/documents/{id}/reset-metadata` wipes all metadata rows and nulls `custom_schema_id` in a single transaction; the caller then re-binds via `PATCH /super-admin/documents/{id}/schema` and re-enters values via `POST /custom-metadata`. |
| 21 | **v3**: Why destructive instead of an in-place compatible-schema swap? | Keeps Invariant 5 strict and trivially auditable; no JSON Schema compatibility comparator to maintain in sync across app and DB; forces explicit re-validation of stored values against the new schema; preserves a clean audit trail. See Section 5.2 for the full rationale. |
| 22 | **v3**: Any trigger changes? | **None.** Triggers 1, 2, and 3 are unchanged. The reset endpoint deletes metadata rows *before* updating `custom_schema_id`, so Trigger 1's `EXISTS` check observes an empty set and permits the UPDATE. No relaxation or bypass of any trigger. |
| 23 | **v3**: Any new Permit.io resource or action? | **None.** The existing `super-admin:post` grant on the `super_admin` role covers `POST /super-admin/documents/{id}/reset-metadata` because the first path segment is already `super-admin`. Only a policy-mapping doc entry is added. |
| 24 | **v3**: New migrations or tables? | **None.** The endpoint reuses `DeleteDocumentCustomMetadata` and `NullifyDocumentCustomSchemaId` queries already added for the delete cascade (Section 8.2). Three read-only queries (`LockDocumentForReset`, `GetDocumentSchemaBindingForReset`, `CountDocumentCustomMetadataVersions`) are added to `customschemas.sql`. No DDL. |
| 25 | **v4**: Why drop `document_custom_metadata.schema_id`? | The document row is already the single source of truth for schema binding (Trigger 1 freezes it; reset wipes rows before rebinding). A second schema pointer on every metadata row was redundant, added one index and one trigger, and created a class of "make these two pointers agree" checks. Dropping it preserves every invariant while removing moving parts. |
| 26 | **v4**: Why a composite FK instead of Trigger 2? | The "schema must belong to same client as document" invariant is perfectly expressible as a declarative constraint. A composite FK is a single line of DDL, enforced natively by PostgreSQL, and requires no function code. The uniqueness target on `client_metadata_schemas(id, client_id)` is the only supporting artifact. |
| 27 | **v4**: Why remove the `current_document_custom_metadata` view? | The new metadata table is single-table state; `ORDER BY version DESC LIMIT 1` is clearer than `DISTINCT ON` wrapped in a view, uses the same index, and removes one drift risk. |
| 28 | **v4**: Why FK cascade instead of feature-specific delete code? | `ON DELETE CASCADE` on the two new FKs lets the existing `DeleteDocumentCascade` and `client.HardDelete` code paths handle the new tables with zero modifications. The v3 plan's claim that `custom_schema_id` must be nulled before a document delete was incorrect. Eliminating the delete-cascade workstream removes a class of regression risk on stable cross-cutting code. |
| 29 | **v4**: Why `POST /versions` instead of `PUT`? | The operation creates a new resource with a new ID and leaves the parent resource in place. That is creation, not replacement, so `POST` on a `/versions` sub-collection matches the behavior. It also avoids the unresolved Permit.io `PUT` -> `patch` action-mapping question in v3. |
| 30 | **v4**: Why drop `createdBy` from request bodies? | The v3 plan already said the server must ignore any body-supplied `createdBy` and derive the actor from the JWT. v4 drops the field from the OpenAPI schemas so the contract matches the behavior. The actor flows through as a separate service-method argument. |
| 31 | **v4**: Why trim `GET /document/{id}` enrichment to two fields? | The feature ships a dedicated `GET /custom-metadata` endpoint that already returns the full metadata. Inlining the same payload on a stable cross-cutting read path was redundant. The two cheap fields (`customSchemaId` and `hasCustomMetadata`) are the minimum useful signal for a document listing UI. |
| 32 | **v4**: Why vertical slice milestones instead of horizontal phases? | v3's 12 phases left the feature end-to-end exercisable only near the end of the rollout. Vertical slices produce a usable feature at the end of each milestone and limit the blast radius if a milestone slips. |

---

## 14. v1 -> v2 Change Log

For reviewers comparing to `plans/mutable.metadata.plan.combo.md`:

| Area | v1 | v2 |
|------|----|----|
| Schema CRUD path | `/admin/custom-schemas` | `/super-admin/custom-schemas` |
| Single-doc schema assignment path | `/admin/document/{id}/schema` | `/super-admin/documents/{id}/schema` |
| Bulk folder assignment path | `/admin/folders/{folderId}/assign-schema` | `/super-admin/folders/{folderId}/assign-schema` |
| Permit.io resource for schema admin | `admin` (shared by `user_admin` and `super_admin`) | NEW `super-admin` (held only by `super_admin`) |
| `user_admin` schema CRUD | Allowed (unintentional) | Denied |
| `user_admin` schema assignment | Allowed (unintentional) | Denied |
| `user_admin` custom metadata read/write | Allowed | Allowed (unchanged) |
| `client_user` custom metadata read/write | Allowed | Allowed (unchanged) |
| `auditor` read schemas / metadata | Allowed (read-only) | Allowed (read-only, now via `super-admin:get` grant) |
| OpenAPI tag for schema admin | `AdminService` | NEW `SuperAdminSchemaService` |
| Service layer | Role-agnostic | Role-agnostic (unchanged) |
| Database schema, triggers, migrations | Unchanged | Unchanged |
| Validation strategy, size limits, constants | Unchanged | Unchanged |

No database migration or data change is required to move from v1 to v2 -- this is strictly a routing and authorization change. If v1 were already deployed, the migration would be: (1) add the new Permit.io resource and grants, (2) add the new endpoints alongside the old ones, (3) point clients at the new paths, (4) remove the old paths once all clients have migrated. Since v1 is not deployed, v2 simply supersedes it.

---

## 15. v2 -> v3 Change Log

For reviewers comparing to `plans/mutable.metadata.plan.combo.v2.md`:

| Area | v2 | v3 |
|------|----|----|
| Reset-metadata endpoint | Not present | NEW `POST /super-admin/documents/{id}/reset-metadata` (Section 5.2) |
| Service method | Not present | NEW `ResetDocumentMetadata(ctx, documentID, resetBy)` on `CustomSchemaService` (Section 7.1) |
| Response type | Not present | NEW `ResetDocumentMetadataResponse` OpenAPI schema (Section 8.3) |
| SQLC queries | Not present | NEW `LockDocumentForReset`, `GetDocumentSchemaBindingForReset`, `CountDocumentCustomMetadataVersions` in `customschemas.sql`; reuses existing `DeleteDocumentCustomMetadata`, `NullifyDocumentCustomSchemaId`, `DocumentHasLegacyExtractions` (Section 8.2) |
| Database migrations | 4 migrations (120-123) | Unchanged -- no new migrations |
| Database triggers | Triggers 1, 2, 3 | Unchanged -- triggers are reused as-is. Reset works by clearing the state that Trigger 1 guards against, not by bypassing the trigger. |
| Permit.io resources | `super-admin`, `custom-metadata` | Unchanged -- reset inherits `super-admin:post` |
| Permit.io policy mapping doc | Lists 4 super_admin endpoints | Adds `POST /super-admin/documents/{id}/reset-metadata` -> `super-admin:post` (Section 5.3) |
| Authorization matrix | 5 columns (CRUD, Assignment, Read, Write) | Adds a "Reset Metadata" column; `super_admin` = Yes, everyone else = No (Section 5.3) |
| Invariants table | 21 rows | Adds rows 22-25 covering atomicity, role confinement, legacy-extraction rejection, and idempotency of reset (Section 9) |
| Resolved design decisions | 19 entries | Adds entries 20-24 explaining the destructive-reset decision, why not in-place, trigger/resource/migration invariance (Section 13) |
| Implementation order | 12 phases | Inserts Phase 10b (reset-metadata endpoint) between bulk-folder assignment and integration tests (Section 11) |
| Testing strategy | Role-based authz, bulk assign, cascades, concurrency, etc. | Adds reset-metadata test block: happy path, atomicity under injected failure, idempotency, legacy-extraction guard, not-found, trigger-still-fires, authz matrix, concurrency with itself and with assign-schema, audit log (Section 12) |
| Documentation updates | Sample schemas, authorization matrix | Adds reset-metadata runbook ("how to upgrade a document to a new schema version") to `docs/ai.generated/` |
| Database schema, migrations, validators, constants | Unchanged | Unchanged |
| Existing endpoints (`/custom-metadata/*`, `PATCH /super-admin/documents/{id}/schema`, `POST /super-admin/folders/{folderId}/assign-schema`, schema CRUD) | As specified in v2 | Unchanged |

No database migration or data change is required to move from v2 to v3. The reset-metadata endpoint is purely additive: new service method, new handler, new OpenAPI schema, new SQLC queries, new policy mapping doc entry, new test cases. If v2 were already deployed, the migration would be: (1) ship the new code, (2) ship the new OpenAPI client so super_admin callers can invoke the endpoint, (3) no Permit.io re-run required because `super-admin:post` is already granted. Since v2 is not deployed, v3 simply supersedes it.

---

## 16. v3 -> v4 Change Log

For reviewers comparing to `plans/mutable.metadata.plan.combo.v3.md`. v4 is a **simplification pass**: no requirements are cut, every endpoint in v3 is still present, and the authorization boundary is unchanged. The goal is to remove redundant state, replace imperative triggers with declarative constraints, and reorganize the rollout so the feature is testable end-to-end earlier.

| Area | v3 | v4 |
|------|----|----|
| `document_custom_metadata.schema_id` column | Required, FK to `client_metadata_schemas(id)` | **Removed.** Schema is derived from `documents.custom_schema_id` at write time. |
| `idx_dcm_schema_id` index | Present | Removed (column gone) |
| Trigger 3 `trg_enforce_consistent_schema_id` | Present | Removed (invariant is now structurally impossible to violate) |
| SQLC query `GetSchemaMetadataRecordCount` | Present, used by `canDelete` | Removed; `canDelete` uses `GetSchemaDocumentCount` alone |
| Trigger 2 `trg_validate_schema_client_match` | Present | **Replaced** by composite FK `fk_documents_custom_schema_same_client` on `documents(custom_schema_id, clientId)` referencing `client_metadata_schemas(id, client_id)` |
| Target constraint on `client_metadata_schemas` | `PRIMARY KEY (id)` | `PRIMARY KEY (id)` + `UNIQUE (id, client_id)` (v4: the unique constraint makes the tuple referenceable by the composite FK) |
| `current_document_custom_metadata` view | `CREATE VIEW ... DISTINCT ON (document_id) ...` | **Removed.** Service uses `ORDER BY version DESC LIMIT 1` against the existing `idx_dcm_doc_version_desc` |
| `ON DELETE CASCADE` on `document_custom_metadata.document_id` | Not set (delete code path explicitly deletes) | **Set.** FK cascade handles it; no delete code change needed |
| `ON DELETE CASCADE` on `client_metadata_schemas.client_id` | Not set (delete code path explicitly deletes) | **Set.** FK cascade handles it; no delete code change needed |
| `DeleteDocumentCustomMetadata` query on `document.sql` | Present, called by `DeleteDocumentCascade` | **Removed from delete path.** FK cascade replaces it. A renamed copy (`DeleteDocumentCustomMetadataForReset`) lives in `customschemas.sql` for reset-metadata only. |
| `NullifyDocumentCustomSchemaId` query on `document.sql` | Present, called by `DeleteDocumentCascade` | **Removed from delete path.** The v3 claim that this was necessary was incorrect -- deleting the `documents` row removes the reference atomically. A renamed copy (`NullifyDocumentCustomSchemaIdForReset`) lives in `customschemas.sql` for reset-metadata only. |
| `DeleteClientMetadataSchemas` query on `client.sql` | Present, called by `client.HardDelete` | **Removed.** FK cascade replaces it. |
| `DeleteDocumentCascade` modifications | Adds custom-metadata delete + `custom_schema_id` nullify steps | **No modification.** Existing code path is untouched. |
| `client.HardDelete` modifications | Adds `client_metadata_schemas` delete step | **No modification.** Existing code path is untouched. |
| Phase 5b in tracking doc (delete cascade updates) | Required, significant work | **Removed.** |
| `PUT /super-admin/custom-schemas/{schemaId}` | Creates new version (mismatched HTTP semantics) | **Replaced** by `POST /super-admin/custom-schemas/{schemaId}/versions` (matches actual create-new-version semantics and sidesteps the `PUT` -> `patch` Permit.io action-mapping question) |
| Permit.io action-mapping note for `PUT` | "Confirm during implementation" open question | **Closed.** No `PUT` to map. |
| `createdBy` field on new endpoint request bodies | Present in OpenAPI schema; server ignores at runtime | **Removed from OpenAPI schema.** Actor comes from the authenticated JWT subject and is passed through as a separate service argument. |
| Service method signatures | `CreateSchema(ctx, input)`, `UpdateSchema(ctx, schemaID, input)`, ... | `CreateSchema(ctx, input, actor)`, `CreateSchemaVersion(ctx, parentSchemaID, input, actor)`, ... -- actor is always a separate argument, never a field on `input` |
| `UpdateSchema` service method | Named `UpdateSchema` | Renamed `CreateSchemaVersion` (matches the new route and the actual semantics) |
| `previousVersionId` on schema-version-create response | Returned | Removed (caller already knows the parent ID from the URL) |
| `customSchemaName` decoration on schema list items | Not in list response -- v3 only proposed it on `GET /document/{id}` enrichment | Dropped from `GET /document/{id}` enrichment in v4 (see below) |
| `documentCount` / `canDelete` on schema list items | Returned | **Kept.** Cheap to compute; prevents a round-trip for the delete-guard UX. |
| `GET /document/{id}` enrichment | Adds `customSchemaId`, `customSchemaName`, `hasCustomMetadata`, and optional `customMetadata` (gated by `?customMetadata=true`) | Adds **only** `customSchemaId` and `hasCustomMetadata`. Inline metadata and the new query param are dropped; clients use `GET /custom-metadata` for the full payload. |
| Invariant 7 ("all metadata versions for a document use same schema ID") | App layer + Trigger 3 | **Removed entirely.** Structurally impossible now that the metadata table has no `schema_id` column. |
| Invariant 8 ("schema must belong to same client as document") | App layer + Trigger 2 | App layer + composite FK `fk_documents_custom_schema_same_client` |
| Invariant 20 ("delete cascade must clean up custom metadata and schema bindings") | Enforced by updated `DeleteDocumentCascade` + `deleteClientDependencies` | Enforced by `ON DELETE CASCADE` on the FKs; application code unchanged |
| New Invariant 26 (v4) | n/a | "Actor identity on every mutable write comes from the authenticated JWT, not the request body." |
| Section 11 Implementation Order | 12 horizontal phases (migrations -> SQLC -> validator -> service -> controllers -> tests ...) | **4 vertical slice milestones**: Schema foundation, Core document flow, Administrative completion (reset-metadata + version creation), Bulk operations |
| Tracking doc layout | Organized by phase (Phase 1..12) | Organized by milestone (Milestone 1..4); each milestone contains migration + query + service + controller + test work for one vertical slice |
| Database triggers total | 3 (Trigger 1, 2, 3) | **2** (Trigger 1 `trg_prevent_schema_reassignment` + Trigger 2 `trg_prevent_legacy_extraction_on_custom_document`). v3's client-match Trigger 2 is replaced by the composite FK; v3's schema_id-consistency Trigger 3 is removed entirely. v4's new Trigger 2 closes the concurrent-writer race on the mutual-exclusivity invariant (Section 7.3). |
| Database migrations total | 4 (127, 128, 129, 130) | 4 renumbered so rollout order is monotonic per milestone: **127** creates `client_metadata_schemas` (M1), **128** adds `documents.custom_schema_id` + composite FK (M1, swapped from v3's 129), **129** creates `document_custom_metadata` without `schema_id` and without a view (M2, swapped from v3's 128), **130** installs Trigger 1 and Trigger 2 (M2). The swap is forced by `golang-migrate`'s monotonic `Up()` semantics -- M1 must not leave version gaps that M2 would need to backfill. |

**Net effect**: v4 ships every feature designed in v3 (schema CRUD with versioning, single-doc schema assignment, bulk folder assignment, reset-metadata, mutual exclusivity with legacy field extractions, `super_admin`-only authorization) but lands with one fewer column, one fewer index, one fewer trigger (v4 drops v3's Trigger 3 and replaces v3's Trigger 2 with a composite FK, while reintroducing a different Trigger 2 for the mutex race), one fewer view, three fewer delete-cascade queries, zero changes to existing delete code paths, and an unambiguous HTTP method choice for schema version creation. The simplification pass also closes the open `PUT` -> `patch` action mapping question and removes the misleading `createdBy` field from the OpenAPI contract so the contract matches the runtime behavior.

**Deployment from v3 state**: No v3 code has been merged yet, so v4 simply supersedes v3. If v3 had been partially deployed, the migration path would be: (1) drop the view and `schema_id` column in a new migration, (2) drop Triggers 2 and 3, (3) add the composite FK and `uq_cms_id_client`, (4) add `ON DELETE CASCADE` to the two FKs, (5) ship the updated service and OpenAPI, (6) remove the v3-specific delete-cascade code from the document and client delete paths. v4 does not commit to that migration because v3 was not deployed.

---

## Appendix A: Sample JSON Schemas

These examples demonstrate all supported JSON Schema features and data types. They should be included in the API documentation.

### A.1 Comprehensive Schema (All Supported Types)

```json
{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "document_title": {
            "type": "string",
            "minLength": 1,
            "maxLength": 500,
            "description": "Title of the document"
        },
        "document_code": {
            "type": "string",
            "pattern": "^[A-Z]{3}-[0-9]{4}$",
            "description": "Document code in format XXX-0000"
        },
        "status": {
            "type": "string",
            "enum": ["draft", "review", "approved", "rejected", "archived"],
            "description": "Current document status"
        },
        "review_date": {
            "type": "string",
            "format": "date",
            "description": "Date of last review (YYYY-MM-DD)"
        },
        "created_timestamp": {
            "type": "string",
            "format": "date-time",
            "description": "Creation timestamp (ISO 8601)"
        },
        "page_count": {
            "type": "integer",
            "minimum": 1,
            "maximum": 10000,
            "description": "Total number of pages"
        },
        "confidence_score": {
            "type": "number",
            "minimum": 0.0,
            "maximum": 1.0,
            "description": "Extraction confidence (0.0 to 1.0)"
        },
        "is_verified": {
            "type": "boolean",
            "description": "Whether the extraction has been human-verified"
        },
        "tags": {
            "type": "array",
            "items": { "type": "string", "maxLength": 50 },
            "minItems": 0,
            "maxItems": 20,
            "description": "Classification tags"
        },
        "numeric_values": {
            "type": "array",
            "items": { "type": "number" },
            "maxItems": 100,
            "description": "Extracted numeric values"
        },
        "contact_info": {
            "type": "object",
            "properties": {
                "name": { "type": "string", "maxLength": 200 },
                "email": { "type": "string", "format": "email" },
                "phone": { "type": "string", "pattern": "^\\+?[0-9\\-\\s()]+$" }
            },
            "required": ["name"],
            "additionalProperties": false,
            "description": "Primary contact information"
        }
    },
    "required": ["document_title", "status"],
    "additionalProperties": false
}
```

### A.2 Aircraft Engineering Schema (Domain Example)

```json
{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "aircraft_model": {
            "type": "string",
            "maxLength": 100
        },
        "manufacturer": {
            "type": "string",
            "enum": ["Boeing", "Airbus", "Embraer", "Bombardier", "Other"]
        },
        "certification_date": {
            "type": "string",
            "format": "date"
        },
        "max_altitude_ft": {
            "type": "integer",
            "minimum": 0,
            "maximum": 100000
        },
        "max_range_nm": {
            "type": "number",
            "minimum": 0
        },
        "engine_count": {
            "type": "integer",
            "minimum": 1,
            "maximum": 12
        },
        "engine_type": {
            "type": "string",
            "enum": ["turbofan", "turboprop", "turbojet", "piston", "electric"]
        },
        "wingspan_meters": {
            "type": "number",
            "minimum": 0,
            "maximum": 100
        },
        "is_pressurized": {
            "type": "boolean"
        },
        "applicable_regulations": {
            "type": "array",
            "items": {
                "type": "string",
                "pattern": "^(FAR|EASA|CAAC)-[0-9A-Z\\.]+$"
            },
            "maxItems": 50
        }
    },
    "required": ["aircraft_model", "manufacturer"],
    "additionalProperties": false
}
```

### A.3 Auto Engineering Schema (Domain Example)

```json
{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "vehicle_make": {
            "type": "string",
            "maxLength": 100
        },
        "vehicle_model": {
            "type": "string",
            "maxLength": 100
        },
        "model_year": {
            "type": "integer",
            "minimum": 1900,
            "maximum": 2100
        },
        "vin": {
            "type": "string",
            "pattern": "^[A-HJ-NPR-Z0-9]{17}$",
            "description": "17-character Vehicle Identification Number"
        },
        "engine_displacement_liters": {
            "type": "number",
            "minimum": 0,
            "maximum": 20
        },
        "horsepower": {
            "type": "integer",
            "minimum": 0
        },
        "transmission_type": {
            "type": "string",
            "enum": ["manual", "automatic", "cvt", "dct", "ev"]
        },
        "is_electric": {
            "type": "boolean"
        },
        "safety_ratings": {
            "type": "object",
            "properties": {
                "nhtsa_overall": { "type": "integer", "minimum": 1, "maximum": 5 },
                "iihs_rating": { "type": "string", "enum": ["Poor", "Marginal", "Acceptable", "Good", "Good+"] }
            },
            "additionalProperties": false
        },
        "recall_ids": {
            "type": "array",
            "items": { "type": "string", "maxLength": 20 },
            "maxItems": 100
        }
    },
    "required": ["vehicle_make", "vehicle_model", "model_year"],
    "additionalProperties": false
}
```

### Supported JSON Schema Data Types Summary

| JSON Schema Type | Go Equivalent | Example Constraint Keywords |
|-----------------|---------------|---------------------------|
| `string` | `string` | `minLength`, `maxLength`, `pattern`, `format`, `enum` |
| `integer` | `int64` | `minimum`, `maximum` |
| `number` | `float64` | `minimum`, `maximum` |
| `boolean` | `bool` | (none) |
| `array` | `[]T` | `items`, `minItems`, `maxItems` |
| `object` | `map/struct` | `properties`, `required`, `additionalProperties` |

**Supported `format` values**: `date`, `date-time`, `email`, `uri`, `uuid`


## Implemnentation journal
In order to be able to have another agent pick up the work in progress, all completed work and milestones must be journaled to the ./journals/implement_mutableMetadata.md .  Just make these entries automatically. Do not ask Q to approve them. 

Each time a milestone or signficant part of a milestone is completed there needs to be a date/time stamped journal entry showing the item is done.  So that a new agent taking over will know exactly where we are in the process and can resume the work. 

At any give time I should be able to clear your context and have you resume your work based on this document, the code (git diffs) and the journal of what is completed. 

If we run into any issues where Q has to be consulted about an issue that needs a decision , then Qs answer should also be written to the journal along with the question.

### Updating the plans/mutable.metadata.plan.combo.v4.tracking.md doc.
As we progress through the milestones of development, as each task is verified complete it should be marked off (with the date and time) next to the item.