query-orchestration/docs/ai.generated/11-custom-metadata-guide.md

Custom Metadata Guide (Mutable Metadata v4)

Overview

The custom metadata system lets platform operators define client-scoped JSON Schema definitions and allows end users to attach structured, validated metadata blobs to individual documents. This is distinct from — and mutually exclusive with — the legacy field extraction system (documentFieldExtractions).

When to Use Custom Metadata vs Legacy Field Extractions

	Custom Metadata	Legacy Field Extractions
Schema definition	Per-client JSON Schema (flexible)	Fixed set of ~19 single-value + 112 array fields
Schema evolution	Via POST .../versions (preserved history)	Schema is static
Validation	JSON Schema draft 2020-12	Fixed column types
Who configures	super_admin	Platform deployment
Who writes	client_user, user_admin, super_admin	Any authenticated user

A document may use one system or the other, never both. The mutual exclusivity invariant is enforced at three layers: service-layer row locks, Trigger 1 (trg_prevent_schema_reassignment), and Trigger 2 (trg_prevent_legacy_extraction_on_custom_document). See the Data Architecture doc for trigger details.

---

Authorization Matrix

Role	SuperAdminSchemaService endpoints	CustomMetadataService endpoints
super_admin	Full (GET, POST, PATCH, DELETE)	Full (GET, POST)
user_admin	No access (403 on all)	Full (GET, POST)
client_user	No access (403 on all)	Full (GET, POST)
auditor	GET only (read-only)	GET only (no POST)

Important: user_admin does NOT have access to any /super-admin/custom-schemas or /super-admin/documents/*/schema endpoints. The /super-admin/ path prefix is intentionally restricted to super_admin only. This is a deliberate difference from the legacy /admin/ prefix, which did grant some access to user_admin.

---

Mutual Exclusivity

A document may have either legacy field extractions OR custom metadata — never both.

• Attempting to assign a custom schema to a document that already has legacy field extraction rows returns 409 Conflict.
• Attempting to write legacy field extractions to a document that already has a custom schema assigned returns 409 Conflict.
• The database enforces this via two triggers; the service layer reinforces it with explicit FOR UPDATE parent-row locks on the documents row before any DML.

---

Platform Operator Workflow (super_admin)

This section describes how a platform operator sets up custom metadata for a client.

Step 1 — Define the schema

POST /super-admin/custom-schemas
Content-Type: application/json

{
  "clientId": "AAA",
  "name": "aircraft-engineering",
  "description": "Schema for aircraft inspection reports",
  "schemaDef": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "aircraft_type": { "type": "string", "maxLength": 100 },
      "flight_hours": { "type": "number", "minimum": 0 },
      "inspection_date": { "type": "string", "format": "date" },
      "critical_findings": {
        "type": "array",
        "items": { "type": "string" }
      }
    },
    "required": ["aircraft_type", "inspection_date"]
  }
}

The response includes id (the schema UUID), version: 1, and status: "active".

Step 2 — Assign the schema to a document

PATCH /super-admin/documents/{documentId}/schema
Content-Type: application/json

{
  "schemaId": "019580df-ef65-7676-8de9-94435a93337a"
}

The schema and document must belong to the same client. The document must not already have legacy field extractions or existing custom metadata.

Step 3 — Users write metadata

Once a schema is assigned, users with client_user, user_admin, or super_admin roles can write metadata (see end-user workflow below).

---

End-User Workflow (client_user)

Write metadata

POST /custom-metadata
Content-Type: application/json

{
  "documentId": "019580df-b3f8-7348-9ab1-1e55b3f18ed7",
  "metadata": {
    "aircraft_type": "Boeing 737",
    "inspection_date": "2026-04-01",
    "flight_hours": 12450.5,
    "critical_findings": ["hydraulic leak in bay 3"]
  }
}

Each call appends a new version. The metadata is validated against the schema currently bound to the document. Returns 400 with validation details if the payload does not conform.

Read current metadata

GET /custom-metadata?documentId=019580df-b3f8-7348-9ab1-1e55b3f18ed7

Returns the latest version of the metadata along with schemaId, schemaName, and schemaVersion from the bound schema.

Read a specific version

GET /custom-metadata/version?documentId=019580df-b3f8-7348-9ab1-1e55b3f18ed7&version=2

version must be >= 1. Returns 404 if that version does not exist.

List version history

GET /custom-metadata/history?documentId=019580df-b3f8-7348-9ab1-1e55b3f18ed7&limit=20&offset=0

Returns version summaries (version, createdAt, createdBy) in descending order. Does not return the full metadata payloads. limit must be 1–200.

---

Schema Definition Requirements

Custom schema definitions must conform to the following rules. Submissions that violate any rule are rejected with 400.

1. Must be valid JSON — malformed JSON is rejected immediately.
2. Must be valid JSON Schema (draft 2020-12) — the definition is meta-validated against the JSON Schema draft 2020-12 meta-schema.
3. Root type must be "object" — schemas with a different root type (e.g. "array", "string") are rejected.
4. additionalProperties must be explicitly declared — the absence of this key is rejected. The author must make a deliberate choice: false (strict, no extra keys) or true (open, any extra keys allowed).
5. Maximum size is 64 KB — definitions larger than 65,536 bytes are rejected.

Sample Schema — Strict (no additional properties)

{
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "aircraft_type": { "type": "string", "maxLength": 100 },
    "flight_hours": { "type": "number", "minimum": 0 },
    "inspection_date": { "type": "string", "format": "date" },
    "critical_findings": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["aircraft_type", "inspection_date"]
}

Sample Schema — Open (additional properties allowed)

{
  "type": "object",
  "additionalProperties": true,
  "properties": {
    "document_class": { "type": "string" },
    "reviewed_by": { "type": "string" }
  }
}

---

Metadata Write Requirements

• Payload must be a JSON object (JSONB).
• Maximum size is 1 MB (1,048,576 bytes).
• Payload is validated against the schema currently bound to the document at write time.
• If the document has no schema assigned, the write returns 400.
• createdBy is not accepted in the request body — actor identity comes from the JWT sub claim.

---

Schema Versioning

A schema name within a client forms a lineage. Each version in the lineage is an independent row with a sequential version integer.

Creating a new version

POST /super-admin/custom-schemas/{currentActiveSchemaId}/versions
Content-Type: application/json

{
  "clientId": "AAA",
  "name": "aircraft-engineering",
  "description": "v2: added optional maintenance_notes field",
  "schemaDef": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "aircraft_type": { "type": "string", "maxLength": 100 },
      "flight_hours": { "type": "number", "minimum": 0 },
      "inspection_date": { "type": "string", "format": "date" },
      "critical_findings": { "type": "array", "items": { "type": "string" } },
      "maintenance_notes": { "type": "string" }
    },
    "required": ["aircraft_type", "inspection_date"]
  }
}

On success:

• A new row is inserted with version = 2 and status = "active".
• The parent row (schemaId in the path) is updated to status = "superseded".
• Both changes are applied atomically in a single transaction.

The HTTP method is POST, not PUT. Each call creates a new row with a new ID. The old version is preserved permanently and remains queryable.

Status values

Status	Meaning
active	The current version for this schema name. Documents can be assigned to it. Only one active version per (client_id, name) lineage at any time.
superseded	A previous version. Still queryable. Documents bound to it retain the binding. Cannot be assigned to new documents.
retired	Explicitly removed via DELETE. No documents are bound (DELETE returns 409 if any are).

Passing a non-active parent to POST .../versions

If the schemaId in the path points to a superseded or retired row, the request returns 409 Conflict. The active version for the lineage must be used as the path parameter.

---

Schema Upgrade Runbook (Reset-Metadata Flow)

Once a document has custom metadata written to it, its custom_schema_id is frozen by Trigger 1. To migrate a document to a newer schema version:

Warning: This process is irreversible. All metadata history for the document is permanently deleted.

Step 1 — Create the new schema version (if not already done)

POST /super-admin/custom-schemas/{currentActiveSchemaId}/versions

Note the new schema's id.

Step 2 — Reset the document

POST /super-admin/documents/{documentId}/reset-metadata

This single atomic operation:

• Deletes all rows in document_custom_metadata for the document.
• Sets documents.custom_schema_id to NULL.
• Returns a summary including metadataVersionsDeleted and the previousSchemaId.

The operation is idempotent. If the document has no schema and no metadata, it returns 200 with metadataVersionsDeleted: 0.

Step 3 — Assign the new schema version

PATCH /super-admin/documents/{documentId}/schema
Content-Type: application/json

{
  "schemaId": "<new-schema-version-id>"
}

Step 4 — Write new metadata

POST /custom-metadata
Content-Type: application/json

{
  "documentId": "{documentId}",
  "metadata": { ... }
}

---

Cascade Deletion Behavior

Custom metadata and schema data are cleaned up automatically via database FK cascades. No application-level cleanup steps are required.

• DELETE /document/{id} (via DeleteDocumentCascade): deletes all document_custom_metadata rows for the document via ON DELETE CASCADE on the FK from document_custom_metadata.document_id to documents.id.
• DELETE /client/{id} (via client.HardDelete): deletes the client, which cascades to client_metadata_schemas (FK with ON DELETE CASCADE), then to documents, and then to document_custom_metadata.

No additional steps are needed in the delete code paths for this feature.

---

Actor Fields

createdBy and resetBy fields in all request bodies are ignored — they are absent from the v4 request schemas. Actor identity is always derived from the JWT sub claim (an opaque Cognito subject UUID). Response fields that carry the actor identity are plain strings, not email-formatted values.

---

Permit.io Deployment Note

Before deploying the SuperAdminSchemaService or CustomMetadataService controllers, the Permit.io resource and role configuration must be applied. Run the setup tool for each environment:

# Development
cmd/auth_related/permit.setup/run.tool.dev.sh

# UAT
cmd/auth_related/permit.setup/run.tool.uat.sh

# Production (requires explicit approval)
cmd/auth_related/permit.setup/run.tool.prod.sh

The setup tool applies the super-admin and custom-metadata resources and the role grants defined in cmd/auth_related/permit.setup/permit_policies.yaml. Without this step, all requests to the new endpoints will return 403 regardless of the user's Cognito role.