jacob-mathison/query-orchestration
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity
Files
a080ca59d8565bd5ccb84ed246dc11e69e876c31
query-orchestration/plans/mutable.metadata.plan.combo.v3.md
T

1730 lines
101 KiB
Markdown
Raw Normal View History

Merged in feature/mutable-metadata1 (pull request #221)
2026-04-16 23:11:26 +00:00
# Mutable Metadata & Per-Client Custom Schemas - Combined Design Plan (v3)
## Status: DRAFT - Combined Claude + Codex + Jay's recommendations (2026-03-24, rev 2026-03-31 post-review, rev 2026-04-10: schema administration restricted to `super_admin` role only, rev 2026-04-13: add reset-metadata escape hatch)
**Lineage**: This plan was informed by two original detailed specs done adversarily. One by claude Opus 4.6 and one by Codex 5.4 (xhigh) based on human written requirements. Each was human reviewed for common items and conflicting items. Where there were conflicts the human was the final decision maker. This resulting hybrid document represents an improved design over each of the source documents.
**v2 change summary (2026-04-10)**: In v1 all mutable-metadata administrative endpoints lived under `/admin/*`. Because the Permit.io middleware maps resources from the first path segment and both `user_admin` and `super_admin` hold full permissions on the `admin` resource, v1 exposed schema CRUD and schema assignment to the lesser (client-scoped) `user_admin` role. v2 moves every schema administration endpoint onto a new `/super-admin/*` path prefix backed by a new Permit.io resource named `super-admin` that only the `super_admin` role can access. User-facing custom metadata read/write endpoints (`/custom-metadata/*`) are unchanged. The underlying service layer, database schema, triggers, and validation strategy are unchanged -- this is strictly an authorization and routing change.
**v3 change summary (2026-04-13)**: Adds a single new super_admin endpoint, `POST /super-admin/documents/{id}/reset-metadata`, to support upgrading a document to a newer schema version when the document already has custom metadata written. Today, once custom metadata exists, Trigger 1 (`trg_prevent_schema_reassignment`) freezes `documents.custom_schema_id` permanently, and Trigger 3 (`trg_enforce_consistent_schema_id`) forbids new metadata versions from referencing a different `schema_id`. The practical effect is that documents cannot move to a revised schema even when the new schema is a strict additive superset (e.g., one new optional field). v3 adds an explicit opt-in path: reset-metadata wipes all `document_custom_metadata` rows for the document and nulls `documents.custom_schema_id` in a single transaction, after which the existing `PATCH /super-admin/documents/{id}/schema` and `POST /custom-metadata` endpoints can bind a new schema and re-enter values. **No new triggers, no changes to any existing trigger, no changes to existing schemas or endpoints.** The escape hatch is deliberately destructive (metadata version history for the document is lost) so the invariant "schema is frozen once metadata exists" remains true -- you are explicitly opting out of "metadata exists" first. Everything else in this plan is unchanged from v2.
---
## 1. Problem Statement
Today, all field extraction data is stored in a rigid 3-table schema with ~130 statically-defined columns (19 single-value fields + 112 array fields). Every client shares the same field definitions. This means:
- Clients cannot define domain-specific fields (e.g., "aircraft engineering" vs. "auto engineering" document classes)
- Adding new fields requires database migrations and code changes across the full stack
- The system cannot accommodate varying data shapes per document class within a single client
**Goal**: Allow each client to curate their own collections of document schemas, where each schema defines the custom fields (with types and constraints) that a document's metadata must conform to. Administration of these schemas (create, update, retire, list, assign to documents/folders) is a platform-level responsibility and must be restricted to the `super_admin` role, not delegated to the lower-privileged `user_admin` role.
---
## 2. Current Architecture Summary
### 2.1 Existing Tables
```
documentFieldExtractions -- 19 single-value fields (1:1 per document)
documentFieldExtractionVersions -- Version tracking with row-level locking
documentFieldExtractionArrayFields -- 112 array fields (1:N per extraction)
```
### 2.2 Existing API Endpoints
| Method | Path | Description |
|--------|------|-------------|
| POST | `/field-extractions` | Create versioned extraction (singleFields + arrayFields) |
| GET | `/field-extractions` | Get current (latest version) extraction for a document |
| GET | `/field-extractions/version` | Get extraction by specific version |
| GET | `/field-extractions/history` | List all versions for a document |
### 2.3 Key Relationships
- `documents.id` -> `documentFieldExtractions.documentId` (1:N via versions)
- `documentFieldExtractions.id` -> `documentFieldExtractionArrayFields.fieldExtractionId` (1:N)
- `documentFieldExtractions.id` -> `documentFieldExtractionVersions.fieldExtractionId` (1:N)
- Documents belong to clients via `documents.clientId`
- No per-client customization of field definitions exists today
### 2.4 Current Role Model (Relevant to This Plan)
Two admin-class roles exist today, both configured in `cmd/auth_related/permit.setup/permit_policies.yaml`:
- **`super_admin`**: Platform administrator. Holds full permissions across every resource (`admin`, `client`, `clients`, `document`, `documents`, `folders`, etc.). Intended for AArete-internal operators.
- **`user_admin`**: Client-scoped administrator. Primarily used for creating new client users via the admin user tool. Holds full permissions on the `admin` resource but intentionally does **not** hold permissions on `client`, `clients`, or `document` at the resource level.
Because the middleware extracts the resource from the first path segment (`/admin/foo` -> resource `admin`), any endpoint placed under `/admin/*` is accessible to both `super_admin` and `user_admin`. v1 of this plan placed schema administration under `/admin/*` and therefore exposed schema CRUD and assignment to `user_admin`. v2 corrects this.
---
## 3. Design Principles
1. **Additive, not destructive** - The existing static field extraction system continues to work unchanged for documents that do not opt into custom schemas. Custom schemas are an additive capability.
2. **Mutual exclusivity** - A document uses EITHER the legacy static field extraction system OR the custom schema system, never both. This prevents ambiguity about which system is authoritative for a document's metadata. Once a document is assigned a custom schema, the legacy field extraction endpoints reject writes for that document, and vice versa.
3. **Schema-as-data** - Schemas are stored as JSON Schema documents in the database, not as DDL. No migrations needed when clients add/modify schemas.
4. **Immutable schema versions** - When a schema is updated, a new version (with new ID) is created. Old versions remain for historical reference. No backward compatibility checking is required between versions since each version gets a distinct ID.
5. **Schema binding is permanent** - Once custom metadata is written to a document with a schema ID, that schema ID cannot be changed on the document. This guarantees data integrity.
6. **JSON blob storage** - Custom metadata is stored as JSONB in PostgreSQL, validated against the schema at write time. This avoids dynamic column creation.
7. **Client isolation** - Schemas are scoped to clients. A client cannot reference another client's schema.
8. **Size limits** - Schema definitions are capped at 64KB (`MaxSchemaDefinitionBytes = 65536`). Metadata payloads are capped at 1MB (`MaxMetadataPayloadBytes = 1048576`). Bulk folder assignments are capped at 10,000 documents (`MaxBulkAssignDocuments = 10000`). All are tunable constants.
9. **No JSONB search/filter** - The system does NOT support querying or filtering documents by custom metadata values. Custom metadata is opaque storage validated on write. If search is needed in the future, it would require GIN indexes on the JSONB columns and a dedicated query API (separate phase).
10. **`super_admin`-scoped schema management** - Schema CRUD operations, single-document schema assignment, and bulk folder-level schema assignment live under `/super-admin/*` and require the `super_admin` role. The lesser `user_admin` role does NOT have access to schema administration. Custom metadata read/write on documents (`/custom-metadata/*`) is available to regular `client_user` users and to both admin roles.
11. **Defense-in-depth** - Critical invariants (schema lock after first extraction, client/schema ownership match, schema_id consistency across metadata versions) are enforced at BOTH the application layer AND the database layer via triggers. The app-layer checks provide better error messages; the DB triggers are safety nets against bypass.
12. **No migration from static fields** - There is no migration path from the existing static field extraction data into the custom schema system. They are mutually exclusive per document. Out of scope.
---
## 4. Database Schema Design
### 4.1 New Type: `schema_status_type`
A three-state enum that distinguishes between different reasons a schema is no longer assignable:
```sql
CREATE TYPE schema_status_type AS ENUM ('active', 'superseded', 'retired');
```
- **`active`**: The schema version is current and can be assigned to new documents.
- **`superseded`**: A newer version of this schema (same `name`) was created. The schema is still valid for documents already using it, but should not be assigned to new documents. Set automatically when a new version is created via `PUT`.
- **`retired`**: An admin explicitly disabled this schema version. Functionally similar to `superseded` but indicates a deliberate administrative action rather than an automatic version succession.
### 4.2 New Table: `client_metadata_schemas`
Stores client-defined JSON Schema documents. Each row is an immutable version of a schema definition.
```sql
CREATE TABLE client_metadata_schemas (
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
client_id varchar(255) NOT NULL REFERENCES clients(clientId),
name varchar(255) NOT NULL,
description text,
schema_def jsonb NOT NULL, -- The JSON Schema document (max 64KB enforced at app layer)
version int NOT NULL DEFAULT 1, -- Auto-incremented per (client_id, name)
status schema_status_type NOT NULL DEFAULT 'active',
created_at timestamptz NOT NULL DEFAULT NOW(),
created_by varchar(255) NOT NULL,
CONSTRAINT uq_client_schema_name_version UNIQUE (client_id, name, version),
CONSTRAINT chk_schema_def_size CHECK (pg_column_size(schema_def) <= 70000) -- ~68KB to account for JSONB overhead; app layer enforces 64KB on raw JSON
);
CREATE INDEX idx_cms_client_id ON client_metadata_schemas(client_id);
CREATE INDEX idx_cms_client_name ON client_metadata_schemas(client_id, name);
CREATE INDEX idx_cms_client_status ON client_metadata_schemas(client_id, status);
```
> **Note**: The 64KB limit is enforced at the application layer (Go constant `MaxSchemaDefinitionBytes = 65536`) by checking `len(rawJSON)`. The database CHECK constraint uses `pg_column_size()` which measures on-disk JSONB storage (includes header overhead), so it is set slightly higher (~68KB) as a safety net. The app-layer check provides the user-facing 64KB limit and a better error message.
**Design decisions:**
- **`name` + `version`** scoped to client: Allows human-friendly naming ("aircraft-engineering-v3") with automatic versioning. When a schema is "updated," a new row is inserted with `version = max(version) + 1` for that `(client_id, name)`.
- **`schema_def` is JSONB**: Stores the full JSON Schema document. PostgreSQL can index into it if needed later.
- **`status` enum**: Three-state lifecycle. `superseded` is set automatically when a newer version is created. `retired` is set explicitly by admin action (soft-delete). Both states prevent assignment to new documents but allow existing documents to continue using the schema for validation. This is richer than a boolean `is_active` because it distinguishes "replaced by v2" from "admin decided this is bad."
- **`id` is the schema version ID**: This is what documents reference. Each update creates a new `id`. The `(client_id, name, version)` tuple provides the human-readable lineage.
### 4.3 New Table: `document_custom_metadata`
Stores the actual custom metadata for documents, validated against the referenced schema.
```sql
CREATE TABLE document_custom_metadata (
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
document_id uuid NOT NULL REFERENCES documents(id),
schema_id uuid NOT NULL REFERENCES client_metadata_schemas(id),
metadata jsonb NOT NULL, -- The actual custom field values
version int NOT NULL DEFAULT 1, -- Versioned like field extractions
created_at timestamptz NOT NULL DEFAULT NOW(),
created_by varchar(255) NOT NULL,
CONSTRAINT uq_doc_custom_metadata_version UNIQUE (document_id, version)
);
CREATE INDEX idx_dcm_document_id ON document_custom_metadata(document_id);
CREATE INDEX idx_dcm_schema_id ON document_custom_metadata(schema_id);
CREATE INDEX idx_dcm_doc_version_desc ON document_custom_metadata(document_id, version DESC); -- Enables efficient DISTINCT ON in current_document_custom_metadata view
```
**Design decisions:**
- **Single table with version column**: We use one table (rather than separate extraction + version tables like the legacy pattern) because the simpler design is sufficient. The legacy two-table pattern exists to support row-level locking on the version counter independently of the data row.
- **Concurrency strategy for version assignment**: `SELECT ... FOR UPDATE` on the max-version row correctly serializes the *second and subsequent* writes for a document, but it does **not** serialize the *first* write -- there is no row to lock yet, so two concurrent first-writers both see `max(version) = 0`, both try to INSERT `version = 1`, and one fails with a `UNIQUE (document_id, version)` violation. v3 handles this explicitly by locking the parent `documents` row with `SELECT id FROM documents WHERE id = $1 FOR UPDATE` *before* reading the max version. The document row is guaranteed to exist (enforced by the service layer's existence check and by the FK) so this lock serializes the first write as well as every subsequent write. As a belt-and-suspenders measure, the service layer also retries once on a `UNIQUE (document_id, version)` violation; if the retry still conflicts the request returns `500`. If this proves insufficient under load, splitting into two tables is a backward-compatible change.
- **`schema_id` is immutable per document**: Enforced at both the application layer and the database trigger (see Section 4.7). All versions of custom metadata for a document must reference the same schema ID.
- **`metadata` is JSONB**: Validated against `client_metadata_schemas.schema_def` at write time. Stored as a single blob for simplicity and query flexibility.
### 4.4 Alteration: `documents` Table
Add an optional schema binding column:
```sql
ALTER TABLE documents
ADD COLUMN custom_schema_id uuid NULL REFERENCES client_metadata_schemas(id);
CREATE INDEX idx_documents_custom_schema_id ON documents(custom_schema_id);
```
**Behavior:**
- `NULL` = document uses the legacy static field extraction system (default, backward-compatible)
- `NOT NULL` = document is opted into the custom schema system; legacy field extraction endpoints will reject writes for this document
- Once set AND custom metadata has been written, this field becomes immutable (enforced at app layer AND DB trigger)
- Can be set before metadata is written (pre-binding) and changed freely until first metadata write
- Must reference a schema owned by the same client as the document (cross-client reference check enforced at app layer AND DB trigger)
- Cannot be set on a document that already has legacy field extractions (returns 409 Conflict)
### 4.5 New View: `current_document_custom_metadata`
```sql
CREATE VIEW current_document_custom_metadata AS
SELECT DISTINCT ON (document_id)
id,
document_id,
schema_id,
metadata,
version,
created_at,
created_by
FROM document_custom_metadata
ORDER BY document_id, version DESC;
```
### 4.6 Entity Relationship Diagram
```
clients
|
|-- 1:N --> client_metadata_schemas (per-client schema definitions)
| |
| |-- referenced by --> documents.custom_schema_id
| |-- referenced by --> document_custom_metadata.schema_id
|
|-- 1:N --> documents
|
|-- 1:N --> document_custom_metadata (versioned JSONB blobs)
| [MUTUALLY EXCLUSIVE with legacy extractions]
|-- 1:N --> documentFieldExtractions (existing static fields)
[MUTUALLY EXCLUSIVE with custom metadata]
```
### 4.7 Database Triggers (Defense-in-Depth)
Three triggers enforce critical invariants at the database level, providing a safety net regardless of which code path modifies the data.
#### Trigger 1: Prevent schema reassignment after first extraction
Fires on `documents` BEFORE UPDATE of `custom_schema_id`. Prevents changes if the document has any extraction data (legacy or custom).
```sql
CREATE OR REPLACE FUNCTION trg_prevent_schema_reassignment()
RETURNS TRIGGER AS $$
BEGIN
-- Only fire if custom_schema_id is actually changing
IF OLD.custom_schema_id IS DISTINCT FROM NEW.custom_schema_id THEN
-- Check for existing custom metadata
IF EXISTS (
SELECT 1 FROM document_custom_metadata
WHERE document_id = NEW.id
LIMIT 1
) THEN
RAISE EXCEPTION 'Cannot change custom_schema_id on document % because custom metadata already exists', NEW.id;
END IF;
-- Check for existing legacy field extractions
IF EXISTS (
SELECT 1 FROM "documentFieldExtractionVersions"
WHERE "documentId" = NEW.id
LIMIT 1
) THEN
RAISE EXCEPTION 'Cannot change custom_schema_id on document % because legacy field extractions already exist', NEW.id;
END IF;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trg_documents_prevent_schema_reassignment
BEFORE UPDATE OF custom_schema_id ON documents
FOR EACH ROW
EXECUTE FUNCTION trg_prevent_schema_reassignment();
```
#### Trigger 2: Validate client/schema ownership match
Fires on `documents` BEFORE INSERT OR UPDATE of `custom_schema_id`. Ensures the referenced schema belongs to the same client as the document.
```sql
CREATE OR REPLACE FUNCTION trg_validate_schema_client_match()
RETURNS TRIGGER AS $$
DECLARE
schema_client_id varchar(255);
BEGIN
-- Only check when custom_schema_id is non-null
IF NEW.custom_schema_id IS NOT NULL THEN
SELECT client_id INTO schema_client_id
FROM client_metadata_schemas
WHERE id = NEW.custom_schema_id;
IF schema_client_id IS NULL THEN
RAISE EXCEPTION 'Schema % does not exist', NEW.custom_schema_id;
END IF;
IF schema_client_id != NEW."clientId" THEN
RAISE EXCEPTION 'Schema % belongs to client %, but document % belongs to client %',
NEW.custom_schema_id, schema_client_id, NEW.id, NEW."clientId";
END IF;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trg_documents_validate_schema_client
BEFORE INSERT OR UPDATE OF custom_schema_id ON documents
FOR EACH ROW
EXECUTE FUNCTION trg_validate_schema_client_match();
```
#### Trigger 3: Enforce consistent schema_id across metadata versions
Fires on `document_custom_metadata` BEFORE INSERT. Ensures all metadata versions for a document reference the same schema ID. This is the database-level safety net for Invariant 7.
```sql
CREATE OR REPLACE FUNCTION trg_enforce_consistent_schema_id()
RETURNS TRIGGER AS $$
DECLARE
existing_schema_id uuid;
BEGIN
SELECT schema_id INTO existing_schema_id
FROM document_custom_metadata
WHERE document_id = NEW.document_id
LIMIT 1;
IF existing_schema_id IS NOT NULL AND existing_schema_id != NEW.schema_id THEN
RAISE EXCEPTION 'All custom metadata for document % must use schema %, but got %',
NEW.document_id, existing_schema_id, NEW.schema_id;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trg_dcm_enforce_consistent_schema
BEFORE INSERT ON document_custom_metadata
FOR EACH ROW
EXECUTE FUNCTION trg_enforce_consistent_schema_id();
```
> **Note**: These triggers duplicate checks that also exist in the application service layer. This is intentional. The app-layer checks return structured HTTP error responses (400/409). The triggers are a last-resort safety net that fires even if a future code path, migration script, or admin query bypasses the service layer.
---
## 5. API Design
### 5.1 Schema Management Endpoints (`super_admin` only)
All schema CRUD endpoints live under the `/super-admin/custom-schemas` path prefix. The first path segment `super-admin` maps (via the existing first-segment resource extraction rule in the Permit.io middleware) to a new Permit.io resource named `super-admin`. Only the `super_admin` role holds permissions on this resource. The `user_admin` role does NOT have access.
These endpoints are grouped under a new OpenAPI tag `SuperAdminSchemaService` (distinct from the existing `AdminService`) to make the authorization boundary visible in the generated client and in Swagger.
#### `POST /super-admin/custom-schemas`
Create a new schema for a client.
**Request:**
```json
{
"clientId": "acme-corp",
"name": "aircraft-engineering",
"description": "Custom fields for aircraft engineering documents",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"aircraft_model": {
"type": "string",
"maxLength": 100
},
"certification_date": {
"type": "string",
"format": "date"
},
"max_altitude_ft": {
"type": "number",
"minimum": 0,
"maximum": 100000
},
"engine_count": {
"type": "integer",
"minimum": 1,
"maximum": 12
}
},
"required": ["aircraft_model"],
"additionalProperties": false
},
"createdBy": "admin@acme.com"
}
```
**Response (201):**
```json
{
"id": "019577a3-...",
"clientId": "acme-corp",
"name": "aircraft-engineering",
"description": "Custom fields for aircraft engineering documents",
"schema": { ... },
"version": 1,
"status": "active",
"createdAt": "2026-03-23T12:00:00Z",
"createdBy": "admin@acme.com"
}
```
**Validation:**
- The `schema` field MUST be a valid JSON Schema document (validated server-side using a JSON Schema meta-validator)
- The `schema` field must be <= 64KB in size (returns 400 with message referencing the limit)
- Root `type` must be `"object"`
- Root `additionalProperties` MUST be explicitly present (either `true` or `false`). If omitted, the server returns 400 with a message explaining that `additionalProperties` must be explicitly declared. This prevents the common mistake of omitting it and inadvertently accepting arbitrary extra fields.
- `name` must be unique within the client (for active schemas at the same version)
#### `PUT /super-admin/custom-schemas/{schemaId}`
Update an existing schema. Creates a new version with a new ID. The old version remains intact and its status is set to `superseded`. No backward compatibility check is performed -- the new version is treated as an independent schema that happens to share a name lineage.
**Request:**
```json
{
"description": "Updated aircraft engineering fields (added wingspan)",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"aircraft_model": { "type": "string", "maxLength": 100 },
"certification_date": { "type": "string", "format": "date" },
"max_altitude_ft": { "type": "number", "minimum": 0, "maximum": 100000 },
"engine_count": { "type": "integer", "minimum": 1, "maximum": 12 },
"wingspan_meters": { "type": "number", "minimum": 0 }
},
"required": ["aircraft_model"],
"additionalProperties": false
},
"createdBy": "admin@acme.com"
}
```
**Response (201):**
```json
{
"id": "019577b4-...",
"clientId": "acme-corp",
"name": "aircraft-engineering",
"version": 2,
"status": "active",
"previousVersionId": "019577a3-...",
...
}
```
**Behavior:**
- The `schemaId` in the path identifies the schema to "update" (really: create a new version of)
- The new version inherits the `name` from the previous version
- The old version's status is set to `superseded` (it remains valid for existing documents but should not be assigned to new ones)
- Returns the newly created schema version
- The `schema` field in the request body follows the same validation rules as `POST` (including the `additionalProperties` requirement)
- **Concurrency**: Uses `SELECT ... FOR UPDATE` on `client_metadata_schemas WHERE client_id = $1 AND name = $2` to lock existing versions before reading `max(version)`. This prevents two concurrent PUTs from computing the same next version number. The unique constraint `uq_client_schema_name_version` is the final safety net.
#### `GET /super-admin/custom-schemas`
List all schemas for a client.
**Query Parameters:**
- `clientId` (required): The client whose schemas to list
- `name` (optional): Filter by schema name
- `status` (optional): Filter by status (`active`, `superseded`, `retired`). Default: return only `active`.
- `includeAllVersions` (optional, default: false): Return all versions or just latest per name
- `limit` (optional, default: 50, max: 200): Maximum number of results to return
- `offset` (optional, default: 0): Number of results to skip for pagination
**Response (200):**
```json
{
"schemas": [
{
"id": "019577b4-...",
"clientId": "acme-corp",
"name": "aircraft-engineering",
"description": "...",
"version": 2,
"status": "active",
"documentCount": 45,
"canDelete": false,
"createdAt": "...",
"createdBy": "..."
},
{
"id": "019577c1-...",
"clientId": "acme-corp",
"name": "auto-engineering",
"description": "...",
"version": 1,
"status": "active",
"documentCount": 0,
"canDelete": true,
"createdAt": "...",
"createdBy": "..."
}
]
}
```
**Notes:**
- `documentCount` shows how many documents reference each schema version. Computed via a subquery or join.
- `canDelete` is `true` when `documentCount == 0` AND no `document_custom_metadata` rows reference the schema. This saves the client a round-trip to check before attempting delete.
#### `GET /super-admin/custom-schemas/{schemaId}`
Get a specific schema by ID (returns full schema definition).
**Response (200):**
```json
{
"id": "019577a3-...",
"clientId": "acme-corp",
"name": "aircraft-engineering",
"description": "...",
"schema": { ... },
"version": 1,
"status": "superseded",
"documentCount": 45,
"canDelete": false,
"createdAt": "...",
"createdBy": "..."
}
```
#### `DELETE /super-admin/custom-schemas/{schemaId}`
Retire a schema. Only succeeds if no documents reference it.
**Response (204):** No content on success.
**Response (409 Conflict):**
```json
{
"error": "Schema is in use by 45 documents and cannot be deleted"
}
```
**Behavior:**
- Sets `status = 'retired'`
- Fails with 409 if any documents reference this schema ID (via `documents.custom_schema_id` or `document_custom_metadata.schema_id`)
- Retired schemas still serve validation for existing documents but cannot be assigned to new documents
---
### 5.2 Document Schema Assignment Endpoints (`super_admin` only)
Schema assignment endpoints also live under `/super-admin/*` so that the same first-segment resource extraction rule confines them to the `super-admin` Permit.io resource. Placing them under `/admin/*` (v1) or under `/document/*` would expose them to `user_admin` or `client_user` respectively, which we explicitly do not want.
#### `PATCH /super-admin/documents/{id}/schema`
Assign or change a custom schema on a single document.
> **Path note**: The sub-segment uses plural `documents/{id}` (not `document/{id}` as in v1) to avoid any possible collision with the existing top-level `/document/{id}` resource path during code review or manual URL inspection. This has no authorization impact because the first segment (`super-admin`) is what the middleware uses to resolve the resource, but it makes the namespacing visually unambiguous.
**Request:**
```json
{
"customSchemaId": "019577a3-..."
}
```
**Response (200):**
```json
{
"documentId": "...",
"customSchemaId": "019577a3-...",
"schemaName": "aircraft-engineering",
"schemaVersion": 1
}
```
**Validation rules:**
- Schema must belong to the same client as the document (enforced at app layer + DB trigger)
- Schema must have status `active`
- If the document already has custom metadata written, the schema cannot be changed (returns 409, also enforced by DB trigger)
- If the document already has legacy field extractions, the schema cannot be assigned (returns 409 -- mutual exclusivity)
- Setting `customSchemaId` to `null` removes the binding (only if no custom metadata exists)
#### `POST /super-admin/folders/{folderId}/assign-schema` (Bulk Folder Assignment)
Assign a custom schema to ALL documents in a folder and all its subfolders (recursive).
**Request:**
```json
{
"customSchemaId": "019577a3-...",
"createdBy": "admin@acme.com"
}
```
**Response (200):**
```json
{
"folderId": "...",
"customSchemaId": "019577a3-...",
"schemaName": "aircraft-engineering",
"documentsUpdated": 87,
"documentsSkipped": 5,
"skippedReasons": [
{"documentId": "...", "reason": "Document already has custom metadata with a different schema"},
{"documentId": "...", "reason": "Document already has custom metadata with a different schema"},
{"documentId": "...", "reason": "Document already assigned to same schema (no-op)"},
{"documentId": "...", "reason": "Document has legacy field extractions"},
{"documentId": "...", "reason": "Document has legacy field extractions"}
]
}
```
**Behavior:**
- Uses `WITH RECURSIVE` CTE to walk the entire folder subtree (same pattern as folder deletion)
- For each document in the tree:
- If the document has no schema, no custom metadata, and no legacy extractions -> assign the schema
- If the document already has the same schema -> skip (no-op, counted in `documentsSkipped`)
- If the document has a different schema but no custom metadata yet -> reassign to new schema
- If the document has a different schema AND has custom metadata -> skip with reason (cannot change)
- If the document has legacy field extractions -> skip with reason (mutual exclusivity)
- Runs in a single transaction
- **Document count limit**: If the folder subtree contains more than `MaxBulkAssignDocuments` (10,000) documents, returns `422 Unprocessable Entity` with a message explaining the limit. This prevents unbounded transaction duration, excessive lock contention, and replication lag.
- Returns summary of what was updated and what was skipped (with reasons)
- Requires `super_admin` role (the `super-admin` resource grants neither `user_admin` nor `client_user` access)
#### `POST /super-admin/documents/{id}/reset-metadata`
Wipe all custom metadata and schema binding from a single document so the document can be re-bound to a different schema (typically a newer version of the same schema, e.g., after an additive field was added via `PUT /super-admin/custom-schemas/{schemaId}`).
This is the explicit, audited escape hatch for the "schema is frozen once metadata exists" invariant. Under normal circumstances, once `document_custom_metadata` has any rows for a document, Trigger 1 (`trg_prevent_schema_reassignment`) blocks `documents.custom_schema_id` from ever changing and Trigger 3 (`trg_enforce_consistent_schema_id`) prevents writing new metadata rows under a different schema. Reset removes the blocking state (the metadata rows themselves and the schema binding) in a single transaction so the existing assign-schema + write-metadata endpoints can operate from a clean slate. **No triggers are added, modified, or bypassed** -- after the metadata rows are deleted inside the transaction, the existing triggers simply no longer have anything to object to.
**Request:** empty body (path parameter `id` is the document ID).
**Response (200):**
```json
{
"documentId": "019577de-...",
"previousSchemaId": "019577a3-...",
"previousSchemaName": "aircraft-engineering",
"previousSchemaVersion": 1,
"metadataVersionsDeleted": 3,
"customSchemaId": null,
"hasCustomMetadata": false,
"resetAt": "2026-04-13T12:00:00Z",
"resetBy": "admin@acme.com"
}
```
The `previous*` fields describe the state *before* the reset so the caller can log or display what was wiped. `customSchemaId: null` and `hasCustomMetadata: false` confirm the post-reset state (same values the next `GET /document/{id}` would return).
**Transaction sequence** (single transaction, serializable-acceptable at read committed with the row locks below):
1. `SELECT id, "clientId", custom_schema_id FROM documents WHERE id = $1 FOR UPDATE` -- lock the document row so concurrent assigns/writes serialize behind the reset.
2. If the document does not exist, return `404 Not Found`.
3. Capture `previousSchemaId` (may be `NULL`) and join to `client_metadata_schemas` to capture `previousSchemaName` / `previousSchemaVersion` for the response (null if no schema was bound).
4. `SELECT COUNT(*) FROM document_custom_metadata WHERE document_id = $1` -- captured as `metadataVersionsDeleted` for the response.
5. `DELETE FROM document_custom_metadata WHERE document_id = $1`.
6. `UPDATE documents SET custom_schema_id = NULL WHERE id = $1`. Trigger 1 fires on this UPDATE; because step 5 already removed all metadata rows and the document is guaranteed (by Invariant 14 and the existing mutual-exclusivity guards) to have no legacy field extractions, the trigger's `EXISTS` checks both return false and the UPDATE succeeds.
7. Insert an audit log entry (same log sink used by `AssignSchema` / `DeleteSchema`) recording `actor`, `documentId`, `previousSchemaId`, `metadataVersionsDeleted`, and `resetAt`.
8. Commit.
**Behavior and guarantees:**
- **Atomic**: If any step fails, the transaction rolls back and the document is unchanged.
- **Idempotent on already-clean documents**: If the document has no custom metadata and `custom_schema_id IS NULL`, the endpoint succeeds and returns `metadataVersionsDeleted: 0`, `previousSchemaId: null`. This keeps retries safe.
- **Does NOT touch legacy field extractions**: The mutual-exclusivity invariant (Invariant 14) still holds. A document that has any legacy field extraction rows is **rejected with `409 Conflict` before any DELETE runs** -- see validation below. Reset-metadata is deliberately scoped to documents using the custom schema system; it is not a "force wipe everything" endpoint and will not silently no-op on legacy documents.
- **Does NOT touch the schema definitions**: `client_metadata_schemas` rows are untouched. Retiring the document's binding does not affect other documents using the same schema.
- **Does NOT re-assign the new schema**: This endpoint is deliberately scoped to "wipe" only. The caller must follow up with `PATCH /super-admin/documents/{id}/schema` to bind the new schema and `POST /custom-metadata` to re-enter values. Keeping the two operations separate preserves the existing endpoints' semantics and makes the audit trail explicit (one log entry for the reset, one for the re-assignment, one per metadata write).
- **History is lost**: All prior metadata versions for this document are deleted. If the caller needs to preserve values across the reset, they must `GET /custom-metadata` beforehand and re-POST after the new schema is bound. Consider this the v3 "release valve," not the normal path. The `metadataVersionsDeleted` count in the response is a safety check for callers who want to confirm how much history they gave up.
**Validation and error responses:**
- `404 Not Found` if the document does not exist or the caller's client scope does not include it.
- `409 Conflict` if the document has legacy field extraction rows (`documentFieldExtractionVersions`). Message: "Document uses the legacy field extraction system; reset-metadata only applies to documents using the custom schema system." This mirrors the mutual-exclusivity guard used elsewhere.
- `200 OK` on successful wipe (even if the document was already clean).
- `500` on unexpected DB errors. The transaction ensures partial state is impossible.
**Authorization:**
- Path lives under `/super-admin/*`, so the first-segment resource extraction rule maps the request to the `super-admin` Permit.io resource. Only the `super_admin` role holds `super-admin:post`, so `user_admin`, `client_user`, and `auditor` all receive `403`. No new Permit.io resource or action is required -- the existing `super-admin:post` grant already covers this route.
- The endpoint is also reflected in the documentation-only `policy_mappings` block in `permit_policies.yaml` (see Section 5.3).
**Why wipe-and-reassign instead of allowing a compatible schema swap in place?**
An earlier design considered relaxing Trigger 1 to permit reassignment when the new schema is a strict superset of the old one (additional optional fields, no type changes, no newly-required fields). That approach was rejected because:
1. **Invariant 5 stays clean.** "`custom_schema_id` is immutable once metadata exists" is a strong, trivially auditable rule. Introducing "immutable *except* when the new schema is compatible" replaces a one-line check with a structural JSON Schema comparator that must agree exactly across the app layer and the DB trigger. Keeping the trigger unchanged is worth the cost of a destructive reset for the edge case of in-place upgrades.
2. **No silent reinterpretation of stored data.** Deleting the metadata and requiring the caller to re-submit guarantees that the new schema is actually applied to the values being stored, with full validation. An in-place schema swap would leave the JSONB blobs untouched and implicitly promote them to the new schema -- which means a later schema tightening (e.g., a pattern constraint added to an existing field) would not catch violations in legacy rows.
3. **Audit clarity.** The reset leaves an explicit log entry indicating that a super_admin chose to discard prior metadata history for this document. An in-place swap would silently change the schema linkage with no indication that the on-disk data predates the new schema.
4. **Simpler to implement and test.** One new endpoint, zero migrations, zero trigger changes, reuses the SQLC queries already added for the delete cascade (`DeleteDocumentCustomMetadata`, `NullifyDocumentCustomSchemaId`, see Section 8.2). No JSON Schema compatibility comparator to build, maintain, or debug.
If a non-destructive upgrade path is needed in the future, it can be added as a separate v4 endpoint (`POST /super-admin/documents/{id}/upgrade-schema`) that performs a compatibility check and a targeted UPDATE of `document_custom_metadata.schema_id` (still requiring a narrowly scoped relaxation of Trigger 3). v3 does not commit to that design.
#### `GET /document/{id}` (Modified)
Add `customSchemaId` and `customMetadata` fields to `DocumentEnriched`:
```json
{
"id": "...",
"clientId": "acme-corp",
"hash": "...",
"customSchemaId": "019577a3-...",
"customSchemaName": "aircraft-engineering",
"hasCustomMetadata": true,
"customMetadata": { ... },
...existing fields...
}
```
**Query parameter:** `customMetadata` (boolean, default: false) - when true, includes the full custom metadata in the response (like the existing `textRecord` parameter).
> **Authorization note**: The `GET /document/{id}` endpoint stays on the existing `document` resource. Both admin roles, `client_user`, and `auditor` can read it today. Returning `customSchemaId` / `customSchemaName` / `customMetadata` on this endpoint is read-only and does not grant any administrative capability. The `super-admin` gating applies only to the write/assign/manage paths.
---
### 5.3 Authorization & Permit.io Changes
> **Security note -- accepted cross-tenant limitation.** The Permit.io check performed by `internal/cognitoauth/middleware.go` and `internal/cognitoauth/permitio.go` is `(user, action, first-path-segment resource)` -- it does not pass a document or client object. That means a caller with `custom-metadata:get`/`custom-metadata:post` (i.e. `client_user`, `user_admin`, `super_admin`, or read-only `auditor`) and knowledge of a document UUID could read or write that document's custom metadata regardless of which client owns the document. This is the same limitation already documented for the rest of the API in `cmd/auth_related/permit.setup/permit_policies.yaml` and is **NOT** addressed by v3.
>
> **Why this is acceptable for v3.** Production environments are deployed per-client: each client runs in its own isolated stack (separate database, separate services, separate Cognito pool), so there is no "other tenant" to cross into from inside a production environment. The limitation only materializes in shared non-production environments (dev/uat) where multiple clients' fixtures may co-exist in a single database. Treat those environments accordingly -- do not load customer-sensitive fixtures into a shared stack.
>
> **Out of scope for v3.** Adding application-level document -> client ownership enforcement to the custom-metadata service/controller is a separate initiative that also needs to retrofit the existing `documents`, `field-extractions`, and `folders` endpoints. If/when that work happens, the `custom-metadata` resource should be updated to receive the document and client objects in the Permit.io check, and the service should refuse cross-tenant reads/writes with `404 Not Found` (never `403`, to avoid leaking existence). Until then the table below describes only coarse role gating, not tenant scoping.
v2 splits admin-class authorization for mutable metadata into two tiers; v3 keeps the same split and simply adds the reset-metadata endpoint to the `super_admin`-only column:
| Role | Schema CRUD (`/super-admin/custom-schemas/*`) | Schema Assignment (`/super-admin/documents/{id}/schema`, `/super-admin/folders/{folderId}/assign-schema`) | Reset Metadata (`/super-admin/documents/{id}/reset-metadata`) | Metadata Read (`GET /custom-metadata/*`, `GET /document/{id}` enrichment) | Metadata Write (`POST /custom-metadata`) |
|------|-----------------------------------------------|-----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------|----------------------------------------------------------------------------|-------------------------------------------|
| `super_admin` | Yes | Yes | Yes | Yes | Yes |
| `user_admin` | **No** | **No** | **No** | Yes | Yes |
| `auditor` | Read-only (`GET` only) | No | No | Read-only | No |
| `client_user` | No | No | No | Yes | Yes |
Key change from v1: `user_admin` has NO schema CRUD and NO schema assignment rights. The role continues to function exactly as it does today for everything else (user creation, labels, folders, etc.). Reset-metadata (new in v3) follows the same confinement: only `super_admin` can call it, because it is destructive and structurally equivalent to a schema reassignment.
#### New Permit.io resource: `super-admin`
Add a new resource to `cmd/auth_related/permit.setup/permit_policies.yaml`:
```yaml
resources:
# ...existing resources unchanged...
- name: super-admin
description: Platform-level administrative functions restricted to super_admin role (custom metadata schema management, schema assignment)
actions:
- get
- post
- patch
- delete
```
#### Role permission updates
```yaml
roles:
- name: super_admin
description: Full system access for platform administrators
permissions:
# ...existing permissions unchanged...
super-admin:
- get
- post
- patch
- delete
custom-metadata:
- get
- post
- name: user_admin
description: User management - primarily for creating new client users
permissions:
# ...existing permissions unchanged (admin, documents, field-extractions, folders)...
# NOTE: user_admin is intentionally NOT granted super-admin:* permissions.
custom-metadata:
- get
- post
- name: auditor
description: Full read-only access across the entire system for auditing and compliance
permissions:
# ...existing permissions unchanged...
super-admin:
- get
custom-metadata:
- get
- name: client_user
description: Client-specific access for external users (requires tenant filtering)
permissions:
# ...existing permissions unchanged...
custom-metadata:
- get
- post
```
#### New custom-metadata resource
The user-facing custom metadata endpoints (`/custom-metadata/*`) use a separate `custom-metadata` resource (first path segment). This is unchanged from v1 except that it now also needs to be granted to `super_admin` explicitly (v1 relied on `admin:*` coverage which no longer covers the metadata path).
```yaml
resources:
- name: custom-metadata
description: Per-document custom metadata read/write (validated against client schemas)
actions:
- get
- post
```
#### Policy mapping documentation additions
Add to the documentation-only `policy_mappings` section at the bottom of `permit_policies.yaml`:
```yaml
super_admin_endpoints:
- path: /super-admin/custom-schemas
methods:
GET: super-admin:get
POST: super-admin:post
- path: /super-admin/custom-schemas/{schemaId}
methods:
GET: super-admin:get
PUT: super-admin:patch # PUT maps to patch action (create-new-version semantics)
DELETE: super-admin:delete
- path: /super-admin/documents/{id}/schema
methods:
PATCH: super-admin:patch
- path: /super-admin/folders/{folderId}/assign-schema
methods:
POST: super-admin:post
- path: /super-admin/documents/{id}/reset-metadata
methods:
POST: super-admin:post
custom_metadata_endpoints:
- path: /custom-metadata
methods:
GET: custom-metadata:get
POST: custom-metadata:post
- path: /custom-metadata/version
methods:
GET: custom-metadata:get
- path: /custom-metadata/history
methods:
GET: custom-metadata:get
```
> **PUT-to-patch action note**: The Permit.io middleware lowercases the HTTP method. Permit.io's default action vocabulary does not include `put`; the existing middleware either maps `PUT` to `patch` or requires the action to exist. Confirm during implementation whether the middleware needs a `put` action added to each resource or whether mapping `PUT` -> `patch` is acceptable. If a new action is needed, add `put` to both `super-admin.actions` and the `super_admin` role permissions. The same consideration applies to any other resources that currently have no `put` action.
#### Middleware impact
The Permit.io resource-extraction middleware uses the first path segment. Introducing `super-admin` as a resource requires no middleware code changes -- it is purely a configuration addition via the permit setup tool. The setup tool (`cmd/auth_related/permit.setup/`) must be re-run in each environment (dev, uat, prod) to provision the new resource and role permissions before these endpoints can pass authorization.
#### Deployment ordering
Because the authorization changes must land in Permit.io before the API routes are live, the deployment sequence is:
1. Update `permit_policies.yaml` with the new resources and role grants.
2. Run the permit setup tool against dev/uat/prod to provision the new resource and permissions.
3. Deploy the API code containing the new endpoints.
Step 2 must complete before step 3 in each environment, otherwise the new endpoints would either be rejected by Permit.io (if fail-closed) or accidentally allowed (if fail-open) during the window between code deploy and permit sync.
---
### 5.4 Custom Metadata CRUD Endpoints (User-Facing)
All under a new `CustomMetadataService` tag. These endpoints are NOT moved under `/super-admin/*` because they are intended for regular users (including `client_user`) to read and write the custom metadata values on documents. Schema administration and schema assignment are separated from value read/write precisely so the day-to-day data entry workload does not require super_admin.
> **Actor identity is authoritative, not client-supplied.** Several request bodies in this section and Section 5.2 show a `createdBy` field (and the reset-metadata response shows `resetBy`). These are shown in the examples for audit-log shape *only*. The handler MUST derive the actor identity from the authenticated JWT / Cognito subject (`cognitoauth.GetUserSubject`, or whichever helper the middleware exposes) and ignore any `createdBy` string supplied in the request body. Accepting a caller-supplied `createdBy` would let a compromised client write arbitrary audit trails, which defeats the audit goal.
>
> Applies to: `POST /super-admin/custom-schemas`, `PUT /super-admin/custom-schemas/{schemaId}`, `DELETE /super-admin/custom-schemas/{schemaId}`, `PATCH /super-admin/documents/{id}/schema`, `POST /super-admin/folders/{folderId}/assign-schema`, `POST /super-admin/documents/{id}/reset-metadata`, and `POST /custom-metadata`. In a future OpenAPI pass the `createdBy` request-body fields should be removed entirely; for v3 we keep the field in the schema for backward shape compatibility but drop its value on the server.
>
> **Audit sink**: Sections 5.2 and 8 reference "the same log sink used by `AssignSchema` / `DeleteSchema`." If that sink does not already exist as a first-class abstraction in `internal/customschema/` or an `audit` package, Phase 4 must define it (signature, where records are written, whether it is synchronous or queued) before Reset-metadata (Phase 4.5) and Schema CRUD (Phase 4.2) can log through it. The audit record must include at minimum: `actor` (derived from auth context), `action`, `resourceId`, `timestamp`, and an action-specific detail payload.
#### `POST /custom-metadata`
Create or update custom metadata for a document. Creates a new version.
**Request:**
```json
{
"documentId": "...",
"metadata": {
"aircraft_model": "Boeing 737-800",
"certification_date": "2024-06-15",
"max_altitude_ft": 41000,
"engine_count": 2
},
"createdBy": "analyst@acme.com"
}
```
**Response (201):**
```json
{
"id": "...",
"documentId": "...",
"schemaId": "019577a3-...",
"metadata": { ... },
"version": 1,
"createdAt": "...",
"createdBy": "analyst@acme.com"
}
```
**Validation:**
- Document must have a `custom_schema_id` assigned (400 if not -- a super_admin must have assigned the schema first)
- Document must NOT have any legacy field extractions (409 if it does -- mutual exclusivity, also enforced by DB trigger on schema assignment)
- The `metadata` payload must be <= 1MB in size (`MaxMetadataPayloadBytes = 1048576`). Returns 400 if exceeded.
- The `metadata` payload is validated against the JSON Schema stored in `client_metadata_schemas.schema_def`
- The request body does NOT include `schemaId` -- the server derives it from the document's `custom_schema_id`. This prevents client/server drift.
- If validation fails, returns 400 with detailed validation errors:
```json
{
"error": "Metadata does not conform to schema",
"validationErrors": [
{"field": "engine_count", "message": "must be >= 1, got 0"},
{"field": "aircraft_model", "message": "required field missing"}
]
}
```
- On first write, locks the document's `custom_schema_id` (makes it immutable via app-layer flag + DB trigger)
- Subsequent writes create new versions but must use the same schema
#### `GET /custom-metadata`
Get current (latest version) custom metadata for a document.
**Query Parameters:**
- `documentId` (required)
**Response (200):**
```json
{
"id": "...",
"documentId": "...",
"schemaId": "019577a3-...",
"schemaName": "aircraft-engineering",
"metadata": { ... },
"version": 3,
"createdAt": "...",
"createdBy": "..."
}
```
#### `GET /custom-metadata/version`
Get a specific version of custom metadata.
**Query Parameters:**
- `documentId` (required)
- `version` (required, >= 1)
#### `GET /custom-metadata/history`
Get version history for a document's custom metadata.
**Query Parameters:**
- `documentId` (required)
- `limit` (optional, default: 50, max: 200): Maximum number of versions to return
- `offset` (optional, default: 0): Number of versions to skip for pagination
**Response (200):**
```json
{
"versions": [
{"id": "...", "documentId": "...", "version": 3, "createdAt": "...", "createdBy": "..."},
{"id": "...", "documentId": "...", "version": 2, "createdAt": "...", "createdBy": "..."},
{"id": "...", "documentId": "...", "version": 1, "createdAt": "...", "createdBy": "..."}
]
}
```
---
## 6. Validation Strategy
### 6.1 JSON Schema Validation
Use a Go JSON Schema validation library for both:
1. **Schema validation** (meta-validation): When a super_admin submits a new schema, validate it against the JSON Schema meta-schema to ensure it is a well-formed JSON Schema document.
2. **Metadata validation**: When custom metadata is submitted, validate the payload against the document's assigned schema.
**Recommended library**: `github.com/santhosh-tekuri/jsonschema/v6`
Reasons:
- Supports JSON Schema draft 2020-12
- Active maintenance
- Good error reporting
- Supports custom formats
### 6.2 Schema Definition Requirements
When a schema definition is submitted (via `POST` or `PUT`), the server validates:
1. It is a valid JSON object
2. It compiles successfully as a JSON Schema document (meta-validation)
3. Root `type` is `"object"`
4. Root `additionalProperties` is explicitly present (either `true` or `false`). This is required because JSON Schema defaults `additionalProperties` to `true` when omitted, which silently accepts any extra fields. Requiring explicit declaration forces a conscious choice.
5. Size is <= 64KB
### 6.3 Supported JSON Schema Features
The system will support the following JSON Schema keywords for custom field definitions:
| Feature | JSON Schema Keyword | Example |
|---------|-------------------|---------|
| Data type | `type` | `"string"`, `"number"`, `"integer"`, `"boolean"`, `"array"`, `"object"` |
| Required fields | `required` | `["field1", "field2"]` |
| String length | `minLength`, `maxLength` | `"maxLength": 100` |
| Numeric range | `minimum`, `maximum` | `"minimum": 0` |
| Pattern matching | `pattern` | `"pattern": "^[A-Z]{2}$"` |
| Enum values | `enum` | `"enum": ["active", "inactive"]` |
| Date/time format | `format` | `"format": "date"` |
| Array items | `items`, `minItems`, `maxItems` | Nested array support |
| Nested objects | `properties` | Hierarchical field groups |
| No extra fields | `additionalProperties` | `false` to enforce strict schemas |
### 6.4 Validation Error Reporting
Validation errors will be returned as structured JSON with:
- The JSON pointer to the failing field (`/aircraft_model`)
- The constraint that was violated (`maxLength`)
- A human-readable message
---
## 7. Service Layer Design
### 7.1 New Service: `internal/customschema/`
```
internal/customschema/
service.go -- CustomSchemaService (CRUD + validation)
models.go -- Input/output types
validator.go -- JSON Schema meta-validation + payload validation
constants.go -- MaxSchemaDefinitionBytes and other limits
service_test.go -- Integration tests
```
**Constants (`constants.go`):**
```go
const (
// MaxSchemaDefinitionBytes is the maximum size of a JSON Schema definition.
// This limit can be increased if clients need larger schemas.
MaxSchemaDefinitionBytes = 65536 // 64KB
// MaxMetadataPayloadBytes is the maximum size of a custom metadata JSON payload.
// Prevents unbounded JSONB blobs when schemas use additionalProperties: true.
MaxMetadataPayloadBytes = 1048576 // 1MB
// MaxBulkAssignDocuments is the maximum number of documents that can be updated
// in a single bulk folder schema assignment. Prevents unbounded transactions.
MaxBulkAssignDocuments = 10000
)
```
**Key methods:**
```go
type Service struct {
cfg serviceconfig.ConfigProvider
validator *SchemaValidator
}
// Schema management (super_admin operations)
func (s *Service) CreateSchema(ctx, input CreateSchemaInput) (*Schema, error)
func (s *Service) UpdateSchema(ctx, schemaID uuid.UUID, input UpdateSchemaInput) (*Schema, error)
func (s *Service) GetSchema(ctx, schemaID uuid.UUID) (*Schema, error)
func (s *Service) ListSchemas(ctx, clientID string, filters ListFilters) ([]SchemaSummary, error)
func (s *Service) DeleteSchema(ctx, schemaID uuid.UUID) error
// Metadata operations (client_user / admin operations)
func (s *Service) SetDocumentMetadata(ctx, input SetMetadataInput) (*CustomMetadata, error)
func (s *Service) GetCurrentMetadata(ctx, documentID uuid.UUID) (*CustomMetadata, error)
func (s *Service) GetMetadataByVersion(ctx, documentID uuid.UUID, version int) (*CustomMetadata, error)
func (s *Service) GetMetadataHistory(ctx, documentID uuid.UUID) ([]MetadataVersion, error)
// Document schema binding (super_admin operations)
func (s *Service) AssignSchema(ctx, documentID uuid.UUID, schemaID uuid.UUID) error
func (s *Service) AssignSchemaToFolder(ctx, folderID uuid.UUID, schemaID uuid.UUID, createdBy string) (*BulkAssignResult, error)
func (s *Service) ResetDocumentMetadata(ctx, documentID uuid.UUID, resetBy string) (*ResetMetadataResult, error)
```
**ResetMetadataResult:**
```go
type ResetMetadataResult struct {
DocumentID uuid.UUID
PreviousSchemaID *uuid.UUID // nil if document had no schema bound
PreviousSchemaName *string // nil if document had no schema bound
PreviousSchemaVersion *int // nil if document had no schema bound
MetadataVersionsDeleted int
ResetAt time.Time
ResetBy string
}
```
`ResetDocumentMetadata` runs a single transaction that: locks the document row (`SELECT ... FOR UPDATE`), rejects documents with legacy field extractions (`409`), captures the previous schema binding metadata (via join to `client_metadata_schemas`) and the count of metadata versions for the response, executes the `DeleteDocumentCustomMetadata` and `NullifyDocumentCustomSchemaId` queries from Section 8.2 (already present for the delete cascade, reused here verbatim), writes an audit log entry, and returns the captured "previous" state. See Section 5.2 for the full endpoint semantics and rationale.
**BulkAssignResult:**
```go
type BulkAssignResult struct {
FolderID uuid.UUID
SchemaID uuid.UUID
DocumentsUpdated int
DocumentsSkipped int
SkippedReasons []SkippedDocument
}
type SkippedDocument struct {
DocumentID uuid.UUID
Reason string
}
```
> **Authorization is enforced at the HTTP middleware layer via Permit.io**, not inside these service methods. The service methods are agnostic to the caller's role. The routing layer for `/super-admin/*` endpoints is what ensures only `super_admin` can invoke `CreateSchema`, `UpdateSchema`, `DeleteSchema`, `AssignSchema`, and `AssignSchemaToFolder`. Keeping the service layer role-unaware preserves testability and makes integration tests simpler.
### 7.2 Validator Component
```go
type SchemaValidator struct{}
// ValidateSchemaDefinition checks that a schema_def is valid JSON Schema.
// Returns error if: not a valid JSON object, fails meta-validation,
// root type is not "object", or additionalProperties is not explicitly present.
func (v *SchemaValidator) ValidateSchemaDefinition(schemaDef json.RawMessage) error
// ValidateMetadata checks that metadata conforms to a compiled schema
func (v *SchemaValidator) ValidateMetadata(schemaDef json.RawMessage, metadata json.RawMessage) ([]ValidationError, error)
```
### 7.3 Mutual Exclusivity Guard
The service layer enforces mutual exclusivity at two points:
1. **Schema assignment** (`AssignSchema`): Before setting `custom_schema_id` on a document, check that no rows exist in `documentFieldExtractionVersions` for that document. If they do, return `409 Conflict`.
2. **Legacy field extraction write** (modification to existing `FieldExtractionService`): Before creating a new field extraction, check that `documents.custom_schema_id IS NULL` for the target document. If it is not null, return `409 Conflict` with a message explaining that the document uses the custom schema system.
> **Note**: The DB trigger on `documents.custom_schema_id` (Section 4.7, Trigger 1) also prevents schema assignment if legacy extractions exist. This is the defense-in-depth pattern: the service layer returns a clean 409; the trigger catches anything that slips through.
### 7.4 API Controller: `api/queryAPI/customschemas.go`
Follows the same pattern as `fieldextractions.go`:
- Parse and validate request
- Call service layer
- Convert between API and service types
- Return appropriate HTTP status codes
The controller file holds all `/super-admin/custom-schemas/*`, `/super-admin/documents/{id}/schema`, and `/super-admin/folders/{folderId}/assign-schema` handlers. A separate controller file `api/queryAPI/custommetadata.go` holds the user-facing `/custom-metadata/*` handlers. Separating them by file mirrors the separation of authorization tiers and makes it obvious at code-review time which handlers are super_admin-gated.
---
## 8. Migration Plan
### 8.1 Migration Files
Four new migrations. The repo already contains migrations through `00000000000126_batch_document_outcomes_delete_actions`, so v3 starts at `127`. If another branch lands new migrations before this feature merges, renumber the four files below to the next free sequence and update all references in this plan and the tracking doc.
**Migration 127: Create `schema_status_type` enum + `client_metadata_schemas` table**
```
00000000000127_create_client_metadata_schemas.up.sql
00000000000127_create_client_metadata_schemas.down.sql
```
**Migration 128: Create `document_custom_metadata` table + view**
```
00000000000128_create_document_custom_metadata.up.sql
00000000000128_create_document_custom_metadata.down.sql
```
**Migration 129: Add `custom_schema_id` to `documents` table**
```
00000000000129_add_custom_schema_id_to_documents.up.sql
00000000000129_add_custom_schema_id_to_documents.down.sql
```
**Migration 130: Add database triggers for invariant enforcement**
```
00000000000130_add_schema_invariant_triggers.up.sql
00000000000130_add_schema_invariant_triggers.down.sql
```
Contents:
- `trg_prevent_schema_reassignment` function + trigger (Section 4.7, Trigger 1)
- `trg_validate_schema_client_match` function + trigger (Section 4.7, Trigger 2)
- `trg_enforce_consistent_schema_id` function + trigger (Section 4.7, Trigger 3)
- Down migration drops triggers and functions
### 8.2 SQLC Queries
New query file: `internal/database/queries/customschemas.sql`
**Schema operations:**
- `CreateClientMetadataSchema` - Insert new schema
- `GetClientMetadataSchema` - Get by ID
- `ListClientMetadataSchemas` - List by client with filters (status filter)
- `GetLatestSchemaByName` - Get latest version of a named schema
- `GetSchemaDocumentCount` - Count documents using a schema
- `GetSchemaMetadataRecordCount` - Count metadata records referencing a schema (for `canDelete`)
- `SetSchemaStatus` - Update status (for supersede/retire)
- `GetMaxSchemaVersion` - For auto-incrementing version
- `LockSchemaVersionsForName` - `SELECT ... FOR UPDATE` on all versions of a (client_id, name) pair to prevent concurrent version creation race
**Metadata operations:**
- `CreateDocumentCustomMetadata` - Insert new metadata version
- `GetCurrentDocumentCustomMetadata` - Latest version (via view)
- `GetDocumentCustomMetadataByVersion` - Specific version
- `GetDocumentCustomMetadataHistory` - All versions
- `LockDocumentCustomMetadataForVersion` - Row-level locking (SELECT FOR UPDATE on max version)
**Document modifications:**
- `SetDocumentCustomSchemaId` - Update documents.custom_schema_id
- `GetDocumentCustomSchemaId` - Read documents.custom_schema_id
- `BulkSetDocumentCustomSchemaIdInFolderTree` - Set schema for all docs in folder subtree (WITH RECURSIVE)
- `GetDocumentsWithMetadataInFolderTree` - Find docs that already have metadata (for skip reporting)
- `GetDocumentsWithLegacyExtractionsInFolderTree` - Find docs that have legacy extractions (for skip reporting)
- `DocumentHasLegacyExtractions` - Check if a single document has any legacy field extraction versions
- `CountDocumentsInFolderTree` - Count total documents in a folder subtree (for `MaxBulkAssignDocuments` limit check)
**Delete cascade operations (additions to `document.sql` and `client.sql`):**
- `DeleteDocumentCustomMetadata` - Delete all custom metadata versions for a document
- `NullifyDocumentCustomSchemaId` - Set `custom_schema_id = NULL` on a document (before hard delete)
- `DeleteClientMetadataSchemas` - Delete all schemas for a client (after all documents deleted)
**Reset-metadata support queries (v3, additions to `customschemas.sql`):**
- `LockDocumentForReset` - `SELECT id, "clientId", custom_schema_id FROM documents WHERE id = @document_id FOR UPDATE` (used by `ResetDocumentMetadata`; reuses the same row-level lock shape that `AssignSchema` uses so the two operations serialize correctly against each other)
- `GetDocumentSchemaBindingForReset` - Join `documents` to `client_metadata_schemas` to return the document's current `custom_schema_id`, the schema `name`, and `version`. Used to populate the `previous*` fields of the reset response. Returns all-null columns cleanly when the document has no schema bound.
- `CountDocumentCustomMetadataVersions` - `SELECT COUNT(*) FROM document_custom_metadata WHERE document_id = @document_id`. Used to populate `metadataVersionsDeleted` in the reset response.
- `DocumentHasLegacyExtractions` is already listed above and is reused by the reset endpoint for the mutual-exclusivity guard (returns 409 if true).
- `DeleteDocumentCustomMetadata` and `NullifyDocumentCustomSchemaId` are already listed above (for the delete cascade) and are reused verbatim by the reset endpoint -- no new queries for the wipe itself.
### 8.3 OpenAPI Spec Changes
Add to `serviceAPIs/queryAPI.yaml`:
**New `SuperAdminSchemaService` tag** for super_admin-only endpoints:
- `POST /super-admin/custom-schemas`
- `GET /super-admin/custom-schemas`
- `GET /super-admin/custom-schemas/{schemaId}`
- `PUT /super-admin/custom-schemas/{schemaId}`
- `DELETE /super-admin/custom-schemas/{schemaId}`
- `PATCH /super-admin/documents/{id}/schema`
- `POST /super-admin/folders/{folderId}/assign-schema`
- `POST /super-admin/documents/{id}/reset-metadata` (v3)
**New `CustomMetadataService` tag** for end-user endpoints:
- `GET /custom-metadata`
- `POST /custom-metadata`
- `GET /custom-metadata/version`
- `GET /custom-metadata/history`
**New schemas**: `CustomSchemaRequest`, `CustomSchemaResponse`, `CustomSchemaListResponse`, `CustomMetadataRequest`, `CustomMetadataResponse`, `CustomMetadataHistoryResponse`, `ValidationErrorResponse`, `BulkSchemaAssignRequest`, `BulkSchemaAssignResponse`, `DocumentSchemaAssignRequest`, `DocumentSchemaAssignResponse`, `ResetDocumentMetadataResponse` (v3)
**New enum**: `SchemaStatus` with values `active`, `superseded`, `retired`
**Modified schemas**: `DocumentEnriched` (add `customSchemaId`, `customSchemaName`, `hasCustomMetadata`, `customMetadata` fields)
**Tag descriptions should explicitly call out the authorization boundary:**
- `SuperAdminSchemaService` description: "Custom metadata schema management and assignment. Restricted to the `super_admin` role. Not accessible to `user_admin`."
- `CustomMetadataService` description: "Per-document custom metadata read/write. Accessible to `client_user`, `user_admin`, and `super_admin`."
### 8.4 Legacy Field Extraction Guard
Add a check to the existing field extraction service (`internal/fieldextraction/service.go`):
Before creating a new field extraction, verify that `documents.custom_schema_id IS NULL` for the target document. If the document has a custom schema assigned, return a `409 Conflict` error explaining that the document uses the custom schema system and legacy field extractions are not allowed.
This is a small modification to the existing code path, not a new endpoint.
### 8.5 Delete Path Updates (Cascade Cleanup)
The new tables and FKs require updates to existing delete paths. Without these changes, hard-deleting a document or client will fail with FK violations.
#### Document Delete Cascade
The existing `DeleteDocumentCascade` function (in `internal/document/`) and the SQLC queries in `internal/database/queries/document.sql` must be updated to delete custom metadata before deleting the document:
**New SQLC queries to add to `document.sql`:**
- `DeleteDocumentCustomMetadata` - `DELETE FROM document_custom_metadata WHERE document_id = @document_id`
- `NullifyDocumentCustomSchemaId` - `UPDATE documents SET custom_schema_id = NULL WHERE id = @document_id`
**Updated delete order in `DeleteDocumentCascade`:**
1. Delete `documentFieldExtractionArrayFields` (existing)
2. Delete `documentFieldExtractionVersions` (existing)
3. Delete `documentFieldExtractions` (existing)
4. **Delete `document_custom_metadata`** (new -- must come before document row delete)
5. **Nullify `documents.custom_schema_id`** (new -- removes FK to `client_metadata_schemas` before document delete)
6. Delete `documentCleanEntries` / `documentCleans` / `documentEntries` (existing)
7. Delete `documents` row (existing)
> **Note**: Step 5 (nullify `custom_schema_id`) must happen before step 7 because the FK from `documents.custom_schema_id` to `client_metadata_schemas` would block deletion otherwise. The nullify is safe because step 4 already removed all custom metadata, and Trigger 1 only fires on changes when metadata exists.
#### Client Delete Cascade
The existing `client.HardDelete` and `deleteClientDependencies` functions must be updated to clean up schema data after all documents are deleted:
**New SQLC queries:**
- `DeleteClientMetadataSchemas` - `DELETE FROM client_metadata_schemas WHERE client_id = @client_id`
**Updated delete order in `client.HardDelete`:**
1. Delete all documents with full cascade (existing, now includes custom metadata cleanup per above)
2. Delete `documentUploads` (existing)
3. Delete all folders (existing)
4. **Delete `client_metadata_schemas`** (new -- safe after all documents are gone since no FKs reference these rows)
5. Delete collector data, batch uploads, sync records (existing)
6. Delete client row (existing)
### 8.6 Permit.io Setup Tool Run
After updating `permit_policies.yaml` per Section 5.3, run the permit setup tool in each environment:
```
cmd/auth_related/permit.setup/run.tool.dev.sh
cmd/auth_related/permit.setup/run.tool.uat.sh
cmd/auth_related/permit.setup/run.tool.prod.sh
```
The tool is idempotent and will provision the new `super-admin` and `custom-metadata` resources and update role grants without disturbing existing resources. Run this **before** deploying the API code so the new endpoints are authorized the moment they go live.
---
## 9. Invariants & Business Rules
| # | Rule | Enforcement |
|---|------|-------------|
| 1 | Schema must be valid JSON Schema with explicit `additionalProperties` | Application: meta-validation on create/update |
| 2 | Schema `name` + `version` unique per client | Database: unique constraint |
| 3 | Updated schema gets new ID and version; old version becomes `superseded` | Application: INSERT new row, UPDATE old status |
| 4 | Schema deletion blocked if documents reference it | Application: count check before retirement |
| 5 | Document's `custom_schema_id` is immutable once any extraction exists | Application: check in AssignSchema + DB trigger: `trg_prevent_schema_reassignment` |
| 6 | Custom metadata must conform to assigned schema | Application: JSON Schema validation on write |
| 7 | All custom metadata versions for a document use same schema ID | Application: consistency check on write + DB trigger: `trg_enforce_consistent_schema_id` |
| 8 | Schema must belong to same client as document | Application: cross-reference check + DB trigger: `trg_validate_schema_client_match` |
| 9 | Only `active` schemas can be assigned to new documents | Application: status check |
| 10 | Custom metadata is versioned (no in-place updates) | Database: version column + unique constraint |
| 11 | Schema definition must be <= 64KB | Application: size check + Database: CHECK constraint |
| 12 | Schema management requires `super_admin` role (NOT `user_admin`) | Permit.io: role-based access on new `super-admin` resource; `user_admin` is explicitly NOT granted this resource |
| 13 | No query/filter on custom metadata values | By design: not implemented (see Principle 9) |
| 14 | A document uses EITHER legacy extractions OR custom metadata, never both | Application: mutual exclusivity checks in both services + DB trigger: `trg_prevent_schema_reassignment` |
| 15 | Legacy field extraction writes rejected if document has `custom_schema_id` set | Application: guard in `FieldExtractionService` |
| 16 | Custom schema assignment rejected if document has legacy extractions | Application: guard in `AssignSchema` + DB trigger |
| 17 | Metadata payload must be <= 1MB | Application: size check (`MaxMetadataPayloadBytes`) |
| 18 | Bulk folder assignment capped at 10,000 documents | Application: count check (`MaxBulkAssignDocuments`), returns 422 if exceeded |
| 19 | Schema assignment (single doc + bulk folder) requires `super_admin` role | Permit.io: endpoints under `/super-admin/` prefix map to `super-admin` resource; `user_admin` is NOT granted |
| 20 | Delete cascade must clean up custom metadata and schema bindings | Application: updated `DeleteDocumentCascade` + `deleteClientDependencies` (Section 8.5) |
| 21 | Metadata read/write is available to `client_user`, `user_admin`, `super_admin` (not `auditor` for writes) | Permit.io: `custom-metadata` resource grants on each role per Section 5.3 |
| 22 | Reset-metadata is atomic: all `document_custom_metadata` rows deleted AND `custom_schema_id` nulled, or neither | Application: single transaction with row-level lock on the document (Section 5.2, `POST /super-admin/documents/{id}/reset-metadata`) |
| 23 | Reset-metadata requires `super_admin` role | Permit.io: endpoint under `/super-admin/` prefix maps to `super-admin` resource; `user_admin` / `client_user` / `auditor` denied |
| 24 | Reset-metadata is rejected if the document has legacy field extractions | Application: `DocumentHasLegacyExtractions` guard in `ResetDocumentMetadata` returns 409 (mutual exclusivity -- Invariant 14 still holds) |
| 25 | Reset-metadata is idempotent on already-clean documents | Application: transaction succeeds with `metadataVersionsDeleted: 0` when no custom metadata and no schema binding exist |
---
## 10. Relationship to Existing Field Extraction System
The existing static field extraction system (`/field-extractions`) and the new custom schema system (`/super-admin/custom-schemas` + `/custom-metadata`) are **mutually exclusive per document**:
- A document uses EITHER the legacy static field extraction system OR the custom schema system
- A document CANNOT have both legacy field extractions AND custom metadata
- Assigning a custom schema to a document that already has legacy extractions returns `409 Conflict`
- Writing a legacy field extraction to a document that has a custom schema assigned returns `409 Conflict`
- Documents without a custom schema (the default) continue to use the legacy system unchanged
- There is no migration path from legacy to custom for individual documents. If a document needs custom metadata, it should not have legacy extractions written to it first.
**Rationale**: This prevents ambiguity about which system is "authoritative" for a document's metadata. With mutual exclusivity, there is exactly one source of truth per document. This is simpler to reason about, simpler to query, and avoids the data governance problem of conflicting data in two stores.
**No breaking changes** to the existing API or database schema for documents that do not opt into custom schemas. The existing `/field-extractions` endpoints remain on the `field-extractions` resource and are unaffected by the new `super-admin` resource.
---
## 11. Implementation Order
| Phase | Work | Dependencies |
|-------|------|--------------|
| 1 | Database migrations (enum, tables, view, document column, triggers) | None |
| 2 | SQLC queries + `task generate` | Phase 1 |
| 3 | Schema validator component (`internal/customschema/validator.go`) | JSON Schema library added to go.mod |
| 4 | Service layer (`internal/customschema/service.go`) incl. bulk folder assignment + mutual exclusivity guards | Phase 2, 3 |
| 5a | Legacy field extraction guard (add check to existing `FieldExtractionService`) | Phase 2 |
| 5b | Delete cascade updates (`DeleteDocumentCascade`, `deleteClientDependencies`, new SQLC queries) | Phase 2 |
| 6 | OpenAPI spec updates (`SuperAdminSchemaService` + `CustomMetadataService` tags) | None (can parallel with 1-3) |
| 7a | Permit.io policy file updates (`permit_policies.yaml`: new `super-admin` + `custom-metadata` resources, role grants) | None (can parallel with 1-6) |
| 7b | Run permit setup tool against dev/uat/prod to provision new resources and role permissions | Phase 7a |
| 8 | API controllers + code generation (super_admin schema CRUD in `customschemas.go`, metadata CRUD in `custommetadata.go`) | Phase 4, 6 |
| 9 | Document endpoint modifications (`PATCH /super-admin/documents/{id}/schema`, `GET /document/{id}` enrichment) | Phase 4, 8 |
| 10 | Bulk folder schema assignment endpoint (`POST /super-admin/folders/{folderId}/assign-schema`) | Phase 4, 8 |
| 10b | Reset-metadata endpoint (`POST /super-admin/documents/{id}/reset-metadata`) -- service method `ResetDocumentMetadata`, handler in `customschemas.go`, OpenAPI `ResetDocumentMetadataResponse` schema, policy mapping entry | Phase 4, 5b (reuses `DeleteDocumentCustomMetadata` + `NullifyDocumentCustomSchemaId` already landed for delete cascade), 8 |
| 11 | Integration tests (including explicit role-based authorization tests per Section 12) | Phase 10b, 7b |
| 12 | Documentation updates (`docs/ai.generated/`) incl. sample schemas, authorization matrix, reset-metadata runbook ("how to upgrade a document to a new schema version") | Phase 11 |
> **Ordering constraint**: Phase 7b (permit setup run) must complete in each environment before Phase 8 code is deployed to that environment. Otherwise the new endpoints will be rejected by Permit.io (fail-closed) until the setup runs.
---
## 12. Testing Strategy
- **Schema validation tests**: Valid and invalid JSON Schema documents, edge cases (empty schema, deeply nested, self-referential, oversized >64KB, missing `additionalProperties`)
- **Metadata validation tests**: Conforming and non-conforming payloads, all supported JSON Schema keywords
- **Service integration tests**: Full CRUD lifecycle using testcontainers (real PostgreSQL)
- **API integration tests**: HTTP-level tests for all new endpoints under both `/super-admin/*` and `/custom-metadata/*`
- **Invariant tests**: Schema immutability, cross-client isolation, deletion guards, size limits
- **DB trigger tests**: Verify triggers fire correctly when bypassing the service layer (direct SQL inserts/updates via testcontainers)
- **Schema status lifecycle tests**: `active` -> `superseded` on version create, `active` -> `retired` on delete, assignment rejected for non-`active` schemas
- **Mutual exclusivity tests**: Legacy extraction blocked when schema assigned, schema assignment blocked when legacy extractions exist, bulk folder assignment skips documents with legacy extractions
- **Bulk folder assignment tests**: Recursive assignment, skip behavior for locked docs and legacy docs, empty folders, deeply nested folder trees
- **Authorization tests (expanded for v2)**:
- `super_admin` CAN create, list, get, update, delete schemas, assign schemas to documents, assign schemas to folders
- `user_admin` CANNOT access any `/super-admin/*` endpoint (expect 403 on every method + path combination)
- `user_admin` CAN read and write custom metadata via `/custom-metadata` (no regression)
- `client_user` CANNOT access any `/super-admin/*` endpoint (expect 403)
- `client_user` CAN read and write custom metadata via `/custom-metadata`
- `auditor` CAN read (`GET`) `/super-admin/custom-schemas` and `/super-admin/custom-schemas/{schemaId}` but CANNOT POST/PUT/PATCH/DELETE
- `auditor` CAN read but CANNOT write `/custom-metadata`
- Each test should assert both the HTTP status code AND that the database state is unchanged on denial
- **Backward compatibility tests**: Existing field extraction endpoints continue to work unchanged for documents without custom schemas; existing `/admin/*` endpoints are unaffected by the new `super-admin` resource
- **Delete cascade tests**: Document hard-delete succeeds when document has custom metadata, client hard-delete succeeds when client has schemas and metadata, verify no orphaned rows remain
- **Metadata size limit tests**: Payloads at/above 1MB boundary, verify 400 response with clear message
- **Bulk assignment limit tests**: Folder trees exceeding `MaxBulkAssignDocuments`, verify 422 response
- **Schema version concurrency tests**: Concurrent PUT requests for same schema name produce distinct version numbers
- **Permit.io setup tool tests**: After running the setup tool against a disposable dev tenant, verify that the `super-admin` and `custom-metadata` resources exist with the expected actions and that each role has exactly the grants defined in Section 5.3
- **Reset-metadata tests (v3)**:
- Happy path: document with schema v1 and N metadata versions -> reset -> assign schema v2 -> POST metadata -> `GET /document/{id}` reflects schema v2 and version 1 metadata; `metadataVersionsDeleted` in the response equals N; `previousSchemaId`/`previousSchemaName`/`previousSchemaVersion` reflect the wiped binding.
- Atomicity: inject a failure between `DELETE FROM document_custom_metadata` and `UPDATE documents SET custom_schema_id = NULL` (e.g., by killing the DB connection mid-transaction in a testcontainers run) and verify that `document_custom_metadata` rows for the document are NOT missing after the failed transaction.
- Idempotency: reset a document that has no custom schema and no custom metadata -> 200 OK with `metadataVersionsDeleted: 0`, `previousSchemaId: null`, DB state unchanged.
- Legacy extraction guard: reset a document that has rows in `documentFieldExtractionVersions` -> 409 Conflict, no DB state change.
- Document not found: reset a non-existent document ID -> 404, no DB state change.
- Trigger 1 still fires normally after a reset: reset, reassign schema v2, write a metadata row, then attempt a second schema reassignment -> trigger fires with the existing error message (confirms no trigger was weakened).
- Trigger 3 still fires normally after a reset: reset, reassign schema v2, write a metadata row under schema v2, then attempt a direct INSERT into `document_custom_metadata` referencing schema v1 -> trigger fires with the existing error message.
- Authorization: `super_admin` succeeds; `user_admin`, `client_user`, and `auditor` all receive 403 with the DB state unchanged.
- Concurrency: two concurrent resets against the same document -> one succeeds, one observes the locked row and serializes (either succeeding on the already-clean state with `metadataVersionsDeleted: 0` or blocking until the first completes). No orphaned rows.
- Concurrency against assign-schema: a reset and a `PATCH /super-admin/documents/{id}/schema` targeting the same document concurrently -> both serialize via `FOR UPDATE`; either ordering produces consistent final state with no trigger violations.
- Audit log: every successful reset writes an audit entry containing the actor, document ID, previous schema ID, and metadata version count.
---
## 13. Resolved Design Decisions
| # | Question | Decision |
|---|----------|----------|
| 1 | Schema size limits | 64KB cap via `MaxSchemaDefinitionBytes` constant (tunable) |
| 2 | Schema backward compatibility | Not required. New version = new ID. Old docs keep old schema. |
| 3 | Bulk operations | Yes. `POST /super-admin/folders/{folderId}/assign-schema` assigns recursively through folder tree. |
| 4 | Custom metadata search/filter | **Not supported.** No GIN indexes, no query API. Metadata is opaque storage. |
| 5 | Schema templates | No system templates. Documentation will include sample schemas covering all supported types. |
| 6 | Existing field migration path | **Out of scope.** Static fields and custom schemas are mutually exclusive per document. |
| 7 | Endpoint naming/placement | Schema CRUD under `/super-admin/custom-schemas` (requires the `super_admin` role on the new `super-admin` Permit.io resource). Metadata CRUD under `/custom-metadata` (regular `client_user` access; also granted to `user_admin` and `super_admin`). |
| 8 | Legacy/custom coexistence | **Mutually exclusive.** A document uses EITHER legacy OR custom, never both. Simplifies data governance and prevents source-of-truth ambiguity. |
| 9 | Schema status model | Three-state enum (`active`/`superseded`/`retired`) instead of boolean. Distinguishes automatic version succession from explicit admin retirement. |
| 10 | DB-level invariant enforcement | Yes. Three database triggers enforce schema lock, client ownership match, and consistent schema_id as defense-in-depth alongside application-layer checks. |
| 11 | `additionalProperties` in schemas | Required to be explicitly present (not necessarily `false`). Prevents the common mistake of omitting it and silently accepting arbitrary extra fields. |
| 12 | Metadata table design | Single table with version column (simpler than the legacy two-table pattern). Sufficient for our concurrency needs. Can split later if needed. |
| 13 | Schema assignment endpoint path | Under `/super-admin/documents/{id}/schema` (NOT `/admin/document/{id}/schema` as in v1 and NOT `/document/{id}/schema`). The `/super-admin` first segment maps to the new `super-admin` Permit.io resource which only `super_admin` holds. Placing it under `/admin` would also grant `user_admin`; placing it under `/document` would also grant `client_user`. |
| 14 | Metadata payload size limit | 1MB cap via `MaxMetadataPayloadBytes` constant. Prevents unbounded JSONB blobs when schemas allow `additionalProperties: true`. |
| 15 | Bulk assignment document limit | 10,000 documents via `MaxBulkAssignDocuments` constant. Prevents unbounded transactions and lock contention. |
| 16 | Schema_id consistency trigger | DB trigger `trg_enforce_consistent_schema_id` on `document_custom_metadata` INSERT enforces same schema across all metadata versions for a document (defense-in-depth for Invariant 7). |
| 17 | **v2**: Who can administer schemas? | **Only `super_admin`.** v1 unintentionally granted `user_admin` access by placing endpoints under `/admin/*`. v2 moves them to `/super-admin/*` and introduces a new `super-admin` Permit.io resource that only `super_admin` holds. `user_admin` retains its existing capabilities (user creation, field extractions, documents/folders, admin endpoints) but has no schema administration rights. |
| 18 | **v2**: Why a new resource instead of path-specific rules? | The Permit.io middleware resolves resources from the first path segment. Adding path-specific overrides would require middleware code changes. Adding a new first-segment resource (`super-admin`) is a pure configuration change and keeps the middleware simple. |
| 19 | **v2**: Does the service layer enforce role checks? | No. Authorization lives in the HTTP/Permit.io layer. The service layer (`internal/customschema/`) is role-agnostic and remains easy to test without fake identity contexts. |
| 20 | **v3**: How does a document move to a newer schema version once it has metadata? | Explicit, destructive reset. `POST /super-admin/documents/{id}/reset-metadata` wipes all metadata rows and nulls `custom_schema_id` in a single transaction; the caller then re-binds via `PATCH /super-admin/documents/{id}/schema` and re-enters values via `POST /custom-metadata`. |
| 21 | **v3**: Why destructive instead of an in-place compatible-schema swap? | Keeps Invariant 5 strict and trivially auditable; no JSON Schema compatibility comparator to maintain in sync across app and DB; forces explicit re-validation of stored values against the new schema; preserves a clean audit trail. See Section 5.2 for the full rationale. |
| 22 | **v3**: Any trigger changes? | **None.** Triggers 1, 2, and 3 are unchanged. The reset endpoint deletes metadata rows *before* updating `custom_schema_id`, so Trigger 1's `EXISTS` check observes an empty set and permits the UPDATE. No relaxation or bypass of any trigger. |
| 23 | **v3**: Any new Permit.io resource or action? | **None.** The existing `super-admin:post` grant on the `super_admin` role covers `POST /super-admin/documents/{id}/reset-metadata` because the first path segment is already `super-admin`. Only a policy-mapping doc entry is added. |
| 24 | **v3**: New migrations or tables? | **None.** The endpoint reuses `DeleteDocumentCustomMetadata` and `NullifyDocumentCustomSchemaId` queries already added for the delete cascade (Section 8.2). Three read-only queries (`LockDocumentForReset`, `GetDocumentSchemaBindingForReset`, `CountDocumentCustomMetadataVersions`) are added to `customschemas.sql`. No DDL. |
---
## 14. v1 -> v2 Change Log
For reviewers comparing to `plans/mutable.metadata.plan.combo.md`:
| Area | v1 | v2 |
|------|----|----|
| Schema CRUD path | `/admin/custom-schemas` | `/super-admin/custom-schemas` |
| Single-doc schema assignment path | `/admin/document/{id}/schema` | `/super-admin/documents/{id}/schema` |
| Bulk folder assignment path | `/admin/folders/{folderId}/assign-schema` | `/super-admin/folders/{folderId}/assign-schema` |
| Permit.io resource for schema admin | `admin` (shared by `user_admin` and `super_admin`) | NEW `super-admin` (held only by `super_admin`) |
| `user_admin` schema CRUD | Allowed (unintentional) | Denied |
| `user_admin` schema assignment | Allowed (unintentional) | Denied |
| `user_admin` custom metadata read/write | Allowed | Allowed (unchanged) |
| `client_user` custom metadata read/write | Allowed | Allowed (unchanged) |
| `auditor` read schemas / metadata | Allowed (read-only) | Allowed (read-only, now via `super-admin:get` grant) |
| OpenAPI tag for schema admin | `AdminService` | NEW `SuperAdminSchemaService` |
| Service layer | Role-agnostic | Role-agnostic (unchanged) |
| Database schema, triggers, migrations | Unchanged | Unchanged |
| Validation strategy, size limits, constants | Unchanged | Unchanged |
No database migration or data change is required to move from v1 to v2 -- this is strictly a routing and authorization change. If v1 were already deployed, the migration would be: (1) add the new Permit.io resource and grants, (2) add the new endpoints alongside the old ones, (3) point clients at the new paths, (4) remove the old paths once all clients have migrated. Since v1 is not deployed, v2 simply supersedes it.
---
## 15. v2 -> v3 Change Log
For reviewers comparing to `plans/mutable.metadata.plan.combo.v2.md`:
| Area | v2 | v3 |
|------|----|----|
| Reset-metadata endpoint | Not present | NEW `POST /super-admin/documents/{id}/reset-metadata` (Section 5.2) |
| Service method | Not present | NEW `ResetDocumentMetadata(ctx, documentID, resetBy)` on `CustomSchemaService` (Section 7.1) |
| Response type | Not present | NEW `ResetDocumentMetadataResponse` OpenAPI schema (Section 8.3) |
| SQLC queries | Not present | NEW `LockDocumentForReset`, `GetDocumentSchemaBindingForReset`, `CountDocumentCustomMetadataVersions` in `customschemas.sql`; reuses existing `DeleteDocumentCustomMetadata`, `NullifyDocumentCustomSchemaId`, `DocumentHasLegacyExtractions` (Section 8.2) |
| Database migrations | 4 migrations (120-123) | Unchanged -- no new migrations |
| Database triggers | Triggers 1, 2, 3 | Unchanged -- triggers are reused as-is. Reset works by clearing the state that Trigger 1 guards against, not by bypassing the trigger. |
| Permit.io resources | `super-admin`, `custom-metadata` | Unchanged -- reset inherits `super-admin:post` |
| Permit.io policy mapping doc | Lists 4 super_admin endpoints | Adds `POST /super-admin/documents/{id}/reset-metadata` -> `super-admin:post` (Section 5.3) |
| Authorization matrix | 5 columns (CRUD, Assignment, Read, Write) | Adds a "Reset Metadata" column; `super_admin` = Yes, everyone else = No (Section 5.3) |
| Invariants table | 21 rows | Adds rows 22-25 covering atomicity, role confinement, legacy-extraction rejection, and idempotency of reset (Section 9) |
| Resolved design decisions | 19 entries | Adds entries 20-24 explaining the destructive-reset decision, why not in-place, trigger/resource/migration invariance (Section 13) |
| Implementation order | 12 phases | Inserts Phase 10b (reset-metadata endpoint) between bulk-folder assignment and integration tests (Section 11) |
| Testing strategy | Role-based authz, bulk assign, cascades, concurrency, etc. | Adds reset-metadata test block: happy path, atomicity under injected failure, idempotency, legacy-extraction guard, not-found, trigger-still-fires, authz matrix, concurrency with itself and with assign-schema, audit log (Section 12) |
| Documentation updates | Sample schemas, authorization matrix | Adds reset-metadata runbook ("how to upgrade a document to a new schema version") to `docs/ai.generated/` |
| Database schema, migrations, validators, constants | Unchanged | Unchanged |
| Existing endpoints (`/custom-metadata/*`, `PATCH /super-admin/documents/{id}/schema`, `POST /super-admin/folders/{folderId}/assign-schema`, schema CRUD) | As specified in v2 | Unchanged |
No database migration or data change is required to move from v2 to v3. The reset-metadata endpoint is purely additive: new service method, new handler, new OpenAPI schema, new SQLC queries, new policy mapping doc entry, new test cases. If v2 were already deployed, the migration would be: (1) ship the new code, (2) ship the new OpenAPI client so super_admin callers can invoke the endpoint, (3) no Permit.io re-run required because `super-admin:post` is already granted. Since v2 is not deployed, v3 simply supersedes it.
---
## Appendix A: Sample JSON Schemas
These examples demonstrate all supported JSON Schema features and data types. They should be included in the API documentation.
### A.1 Comprehensive Schema (All Supported Types)
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"document_title": {
"type": "string",
"minLength": 1,
"maxLength": 500,
"description": "Title of the document"
},
"document_code": {
"type": "string",
"pattern": "^[A-Z]{3}-[0-9]{4}$",
"description": "Document code in format XXX-0000"
},
"status": {
"type": "string",
"enum": ["draft", "review", "approved", "rejected", "archived"],
"description": "Current document status"
},
"review_date": {
"type": "string",
"format": "date",
"description": "Date of last review (YYYY-MM-DD)"
},
"created_timestamp": {
"type": "string",
"format": "date-time",
"description": "Creation timestamp (ISO 8601)"
},
"page_count": {
"type": "integer",
"minimum": 1,
"maximum": 10000,
"description": "Total number of pages"
},
"confidence_score": {
"type": "number",
"minimum": 0.0,
"maximum": 1.0,
"description": "Extraction confidence (0.0 to 1.0)"
},
"is_verified": {
"type": "boolean",
"description": "Whether the extraction has been human-verified"
},
"tags": {
"type": "array",
"items": { "type": "string", "maxLength": 50 },
"minItems": 0,
"maxItems": 20,
"description": "Classification tags"
},
"numeric_values": {
"type": "array",
"items": { "type": "number" },
"maxItems": 100,
"description": "Extracted numeric values"
},
"contact_info": {
"type": "object",
"properties": {
"name": { "type": "string", "maxLength": 200 },
"email": { "type": "string", "format": "email" },
"phone": { "type": "string", "pattern": "^\\+?[0-9\\-\\s()]+$" }
},
"required": ["name"],
"additionalProperties": false,
"description": "Primary contact information"
}
},
"required": ["document_title", "status"],
"additionalProperties": false
}
```
### A.2 Aircraft Engineering Schema (Domain Example)
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"aircraft_model": {
"type": "string",
"maxLength": 100
},
"manufacturer": {
"type": "string",
"enum": ["Boeing", "Airbus", "Embraer", "Bombardier", "Other"]
},
"certification_date": {
"type": "string",
"format": "date"
},
"max_altitude_ft": {
"type": "integer",
"minimum": 0,
"maximum": 100000
},
"max_range_nm": {
"type": "number",
"minimum": 0
},
"engine_count": {
"type": "integer",
"minimum": 1,
"maximum": 12
},
"engine_type": {
"type": "string",
"enum": ["turbofan", "turboprop", "turbojet", "piston", "electric"]
},
"wingspan_meters": {
"type": "number",
"minimum": 0,
"maximum": 100
},
"is_pressurized": {
"type": "boolean"
},
"applicable_regulations": {
"type": "array",
"items": {
"type": "string",
"pattern": "^(FAR|EASA|CAAC)-[0-9A-Z\\.]+$"
},
"maxItems": 50
}
},
"required": ["aircraft_model", "manufacturer"],
"additionalProperties": false
}
```
### A.3 Auto Engineering Schema (Domain Example)
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"vehicle_make": {
"type": "string",
"maxLength": 100
},
"vehicle_model": {
"type": "string",
"maxLength": 100
},
"model_year": {
"type": "integer",
"minimum": 1900,
"maximum": 2100
},
"vin": {
"type": "string",
"pattern": "^[A-HJ-NPR-Z0-9]{17}$",
"description": "17-character Vehicle Identification Number"
},
"engine_displacement_liters": {
"type": "number",
"minimum": 0,
"maximum": 20
},
"horsepower": {
"type": "integer",
"minimum": 0
},
"transmission_type": {
"type": "string",
"enum": ["manual", "automatic", "cvt", "dct", "ev"]
},
"is_electric": {
"type": "boolean"
},
"safety_ratings": {
"type": "object",
"properties": {
"nhtsa_overall": { "type": "integer", "minimum": 1, "maximum": 5 },
"iihs_rating": { "type": "string", "enum": ["Poor", "Marginal", "Acceptable", "Good", "Good+"] }
},
"additionalProperties": false
},
"recall_ids": {
"type": "array",
"items": { "type": "string", "maxLength": 20 },
"maxItems": 100
}
},
"required": ["vehicle_make", "vehicle_model", "model_year"],
"additionalProperties": false
}
```
### Supported JSON Schema Data Types Summary
| JSON Schema Type | Go Equivalent | Example Constraint Keywords |
|-----------------|---------------|---------------------------|
| `string` | `string` | `minLength`, `maxLength`, `pattern`, `format`, `enum` |
| `integer` | `int64` | `minimum`, `maximum` |
| `number` | `float64` | `minimum`, `maximum` |
| `boolean` | `bool` | (none) |
| `array` | `[]T` | `items`, `minItems`, `maxItems` |
| `object` | `map/struct` | `properties`, `required`, `additionalProperties` |
**Supported `format` values**: `date`, `date-time`, `email`, `uri`, `uuid`
## Implemnentation journal
In order to be able to have another agent pick up the work in progress, all completed work and milestones must be journaled to the ./journals/implement_mutableMetadata.md
Each time a milestone or signficant part of a milestone is completed there needs to be a date/time stamped journal entry showing the item is done. So that a new agent taking over will know exactly where we are in the process and can resume the work.
At any give time I should be able to clear your context and have you resume your work based on this document, the code (git diffs) and the journal of what is completed.
If we run into any issues where Q has to be consulted about an issue that needs a decision , then Qs answer should also be written to the journal along with the question.
### Updating the plans/mutable.metadata.plan.combo.v3.tracking.md doc.
As we progress through the phases of development , as each task is verified complete it should be marked off (with the date and time) next to the item.
Reference in New Issue Copy Permalink