Merged in feature/mutable-metadata1 (pull request #221)

M1, M2 and M3 complete * M1, M2 and M3 complete * review changes * docs * docs
2026-04-16 23:11:26 +00:00
parent ad7b21f3a2
commit 17fc813823
164 changed files with 44700 additions and 371 deletions
@@ -29,6 +29,8 @@ The API is organized into the following service groups:
 | AdminService           | Operations related to user management and administration    |
 | EulaService            | Operations related to End User License Agreement management and tracking |
 | UISettingsService      | Operations related to UI-scoped key-value settings (per user, client, or global) |
+| SuperAdminSchemaService | Operations related to custom schema management (super_admin only)              |
+| CustomMetadataService  | Operations related to per-document custom metadata (client_user and above)       |

 ## Authentication and Authorization

@@ -1783,6 +1785,369 @@ Soft-deletes a UI setting (sets `isDeleted` to true). Record is retained. Return

 ---

+## Schema Management (SuperAdminSchemaService)
+
+All endpoints in this group require the `super_admin` role. Any other role receives `403 Forbidden`. The Permit.io resource is `super-admin`; auditor is granted `get` only.
+
+### `POST /super-admin/custom-schemas`
+
+Creates a new custom schema for a client. The schema definition must be a valid JSON Schema draft 2020-12 object with `additionalProperties` explicitly declared and a root `type` of `object`. Maximum definition size is 64 KB.
+
+**Security**: Requires JWT authentication. Requires `super_admin` role.
+
+**Request Body** (`CustomSchemaRequest`):
+
+```json
+{
+  "clientId": "AAA",
+  "name": "aircraft-engineering",
+  "description": "Schema for aircraft inspection documents",
+  "schemaDef": {
+    "type": "object",
+    "additionalProperties": false,
+    "properties": {
+      "aircraft_type": { "type": "string", "maxLength": 100 },
+      "inspection_date": { "type": "string", "format": "date" }
+    },
+    "required": ["aircraft_type", "inspection_date"]
+  }
+}
+```
+
+**Response** (`201 Created`, `CustomSchemaResponse`):
+
+```json
+{
+  "id": "019580df-ef65-7676-8de9-94435a93337a",
+  "clientId": "AAA",
+  "name": "aircraft-engineering",
+  "description": "Schema for aircraft inspection documents",
+  "version": 1,
+  "status": "active",
+  "schemaDef": { ... },
+  "createdAt": "2026-04-14T10:00:00Z",
+  "createdBy": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
+}
+```
+
+**Notes**:
+- `createdBy` is populated from the JWT `sub` claim (a Cognito subject UUID), not accepted from the request body.
+- `schemaDef` must have `type: "object"` at the root.
+- `additionalProperties` must be explicitly `true` or `false`.
+
+**Error Responses**:
+- `400`: Invalid or oversized schema definition
+- `401`: Authentication required
+- `403`: Insufficient permissions
+- `409`: A schema with this name already exists for the client (at version 1)
+
+### `GET /super-admin/custom-schemas`
+
+Lists schemas for a client with optional filtering.
+
+**Security**: Requires JWT authentication. Requires `super_admin` role (`auditor` may use GET).
+
+**Query Parameters**:
+
+| Parameter | Type | Required | Default | Description |
+|---|---|---|---|---|
+| clientId | string | Yes | — | Client ID to filter schemas |
+| name | string | No | — | Filter by schema name |
+| status | string | No | `active` | Filter by status: `active`, `superseded`, `retired`, `any` |
+| includeAllVersions | boolean | No | `false` | When `true`, returns all versions per name; otherwise returns only the latest active version |
+| limit | integer | No | 20 | Pagination page size (1–100) |
+| offset | integer | No | 0 | Pagination offset |
+
+**Response** (`200 OK`, `CustomSchemaListResponse`):
+
+```json
+{
+  "schemas": [
+    {
+      "id": "019580df-ef65-7676-8de9-94435a93337a",
+      "clientId": "AAA",
+      "name": "aircraft-engineering",
+      "version": 2,
+      "status": "active",
+      "documentCount": 14,
+      "canDelete": false,
+      "createdAt": "2026-04-14T10:00:00Z",
+      "createdBy": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
+    }
+  ],
+  "totalCount": 1
+}
+```
+
+**Notes**:
+- Default list (no `status`, no `includeAllVersions`) returns only the latest active version per name.
+- `canDelete` is `false` when any document has `custom_schema_id` pointing to this schema.
+- `documentCount` is the count of documents currently bound to this specific schema version.
+
+### `GET /super-admin/custom-schemas/{schemaId}`
+
+Retrieves a single schema by ID, including the full `schemaDef`.
+
+**Security**: Requires JWT authentication. Requires `super_admin` role (`auditor` may use GET).
+
+**Path Parameters**:
+
+- `schemaId`: Schema UUID
+
+**Response** (`200 OK`, `CustomSchemaResponse`): Full schema object including `schemaDef`.
+
+**Error Responses**:
+- `404`: Schema not found
+
+### `POST /super-admin/custom-schemas/{schemaId}/versions`
+
+Creates a new version of an existing schema. The parent schema must be `active`. On success: a new row is inserted with an incremented `version` number and `status=active`; the parent row's `status` is set to `superseded`. Both changes are applied in a single transaction.
+
+**Security**: Requires JWT authentication. Requires `super_admin` role.
+
+**Path Parameters**:
+
+- `schemaId`: UUID of the current active schema (the parent version)
+
+**Request Body** (`CustomSchemaVersionRequest`): Same shape as `CustomSchemaRequest` body (name, description, schemaDef — no `createdBy`).
+
+**Response** (`201 Created`, `CustomSchemaResponse`): The new schema version row.
+
+**Notes**:
+- This is a `POST`, not a `PUT` — it creates a new row with a new ID. The old version is preserved and queryable.
+- Passing a `schemaId` that is not `active` (e.g. `superseded` or `retired`) returns `409`.
+- `createdBy` is derived from JWT, not the request body.
+
+**Error Responses**:
+- `400`: Invalid schema definition
+- `404`: Parent schema not found
+- `409`: Parent schema is not active (already superseded or retired)
+
+### `DELETE /super-admin/custom-schemas/{schemaId}`
+
+Retires a schema. Sets `status` to `retired`. This operation is reversible only via a new version.
+
+**Security**: Requires JWT authentication. Requires `super_admin` role.
+
+**Path Parameters**:
+
+- `schemaId`: Schema UUID
+
+**Response**: `204 No Content`
+
+**Error Responses**:
+- `404`: Schema not found
+- `409`: One or more documents are currently bound to this schema (`documentCount > 0`)
+
+### `PATCH /super-admin/documents/{id}/schema`
+
+Assigns or clears a custom schema on a document. Once a document has custom metadata written to it, the schema binding is frozen — this endpoint will return `409` if metadata exists. To upgrade a schema, first use `POST .../reset-metadata`.
+
+**Security**: Requires JWT authentication. Requires `super_admin` role.
+
+**Path Parameters**:
+
+- `id`: Document UUID
+
+**Request Body** (`DocumentSchemaAssignRequest`):
+
+```json
+{
+  "schemaId": "019580df-ef65-7676-8de9-94435a93337a"
+}
+```
+
+Pass `"schemaId": null` to clear an existing schema binding (allowed only when no metadata exists).
+
+**Response** (`200 OK`, `DocumentSchemaAssignResponse`):
+
+```json
+{
+  "documentId": "019580df-b3f8-7348-9ab1-1e55b3f18ed7",
+  "customSchemaId": "019580df-ef65-7676-8de9-94435a93337a",
+  "schemaName": "aircraft-engineering",
+  "schemaVersion": 2
+}
+```
+
+**Error Responses**:
+- `400`: Schema is not active or document already has legacy field extractions
+- `404`: Document or schema not found
+- `409`: Document already has custom metadata (schema binding is frozen)
+
+### `POST /super-admin/documents/{id}/reset-metadata`
+
+Atomically deletes all custom metadata rows for a document and nullifies `custom_schema_id`. This is the required first step when upgrading a document to a newer schema version. The operation is irreversible — metadata history is lost.
+
+**Security**: Requires JWT authentication. Requires `super_admin` role.
+
+**Path Parameters**:
+
+- `id`: Document UUID
+
+**Request Body**: None.
+
+**Response** (`200 OK`, `DocumentMetadataResetResponse`):
+
+```json
+{
+  "documentId": "019580df-b3f8-7348-9ab1-1e55b3f18ed7",
+  "previousSchemaId": "019580df-ef65-7676-8de9-94435a93337a",
+  "previousSchemaName": "aircraft-engineering",
+  "previousSchemaVersion": 1,
+  "metadataVersionsDeleted": 3,
+  "resetAt": "2026-04-15T09:30:00Z",
+  "resetBy": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
+}
+```
+
+This endpoint is idempotent. If the document has no schema binding and no metadata, `metadataVersionsDeleted` is `0` and all `previous*` fields are `null`.
+
+**Error Responses**:
+- `404`: Document not found
+- `409`: Document has legacy field extractions (mutual exclusivity — cannot reset a legacy document)
+
+---
+
+## Custom Metadata Operations (CustomMetadataService)
+
+These endpoints are available to `client_user`, `user_admin`, and `super_admin` for both read and write. `auditor` has read-only access (GET endpoints only). The Permit.io resource is `custom-metadata`.
+
+### `POST /custom-metadata`
+
+Writes a new metadata version for a document. The document must already have a custom schema assigned. The metadata payload is validated against the bound schema. Each call appends a new version; the latest version is what `GET /custom-metadata` returns.
+
+**Security**: Requires JWT authentication. Requires `client_user`, `user_admin`, or `super_admin` role.
+
+**Request Body** (`CustomMetadataRequest`):
+
+```json
+{
+  "documentId": "019580df-b3f8-7348-9ab1-1e55b3f18ed7",
+  "metadata": {
+    "aircraft_type": "Boeing 737",
+    "inspection_date": "2026-04-01",
+    "flight_hours": 12450.5
+  }
+}
+```
+
+Maximum payload size is 1 MB.
+
+**Response** (`201 Created`, `CustomMetadataResponse`):
+
+```json
+{
+  "id": "019580df-ef65-7676-8de9-94435a93337c",
+  "documentId": "019580df-b3f8-7348-9ab1-1e55b3f18ed7",
+  "schemaId": "019580df-ef65-7676-8de9-94435a93337a",
+  "schemaName": "aircraft-engineering",
+  "schemaVersion": 2,
+  "version": 4,
+  "metadata": { ... },
+  "createdAt": "2026-04-15T10:00:00Z",
+  "createdBy": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
+}
+```
+
+**Notes**:
+- `createdBy` is derived from JWT, not the request body.
+- Returns `400` with validation errors if the payload does not conform to the schema.
+
+**Error Responses**:
+- `400`: Document has no schema assigned, payload exceeds 1 MB, or payload fails schema validation
+- `409`: Document has legacy field extractions (mutually exclusive with custom metadata)
+
+### `GET /custom-metadata`
+
+Returns the current (latest) metadata version for a document.
+
+**Security**: Requires JWT authentication. Requires `client_user`, `user_admin`, `super_admin`, or `auditor` role.
+
+**Query Parameters**:
+
+- `documentId` (required): Document UUID
+
+**Response** (`200 OK`, `CustomMetadataResponse`): The latest version row.
+
+**Error Responses**:
+- `404`: Document not found or no metadata exists for the document
+
+### `GET /custom-metadata/version`
+
+Returns a specific historical metadata version for a document.
+
+**Security**: Requires JWT authentication. Requires `client_user`, `user_admin`, `super_admin`, or `auditor` role.
+
+**Query Parameters**:
+
+| Parameter | Type | Required | Description |
+|---|---|---|---|
+| documentId | uuid | Yes | Document UUID |
+| version | integer | Yes | Version number (>= 1) |
+
+**Response** (`200 OK`, `CustomMetadataResponse`): The requested version row.
+
+**Error Responses**:
+- `400`: `version` is less than 1
+- `404`: Document not found or the specified version does not exist
+
+### `GET /custom-metadata/history`
+
+Returns a paginated list of metadata version summaries for a document, ordered by version descending (newest first). Summaries include `version`, `createdAt`, and `createdBy` but not the full metadata payload.
+
+**Security**: Requires JWT authentication. Requires `client_user`, `user_admin`, `super_admin`, or `auditor` role.
+
+**Query Parameters**:
+
+| Parameter | Type | Required | Default | Description |
+|---|---|---|---|---|
+| documentId | uuid | Yes | — | Document UUID |
+| limit | integer | No | 20 | Number of versions to return (1–200) |
+| offset | integer | No | 0 | Pagination offset (>= 0) |
+
+**Response** (`200 OK`, `CustomMetadataHistoryResponse`):
+
+```json
+{
+  "versions": [
+    {
+      "version": 3,
+      "createdAt": "2026-04-15T12:00:00Z",
+      "createdBy": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
+    },
+    {
+      "version": 2,
+      "createdAt": "2026-04-14T15:00:00Z",
+      "createdBy": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
+    }
+  ]
+}
+```
+
+**Error Responses**:
+- `400`: `limit` is out of range (must be 1–200) or `offset` is negative
+- `404`: Document not found
+
+**Notes**:
+- If the document exists but has no metadata, returns `200` with an empty `versions` array.
+- `limit=201` is clamped to 200.
+
+---
+
+## GET /document/{id} — Custom Metadata Enrichment
+
+`GET /document/{id}` (documented under [Document Operations](#document-operations-documentsservice)) now returns two additional fields on the `DocumentEnriched` response:
+
+| Field | Type | Description |
+|---|---|---|
+| `customSchemaId` | uuid or null | UUID of the custom schema currently bound to the document. Null if no schema has been assigned. |
+| `hasCustomMetadata` | boolean | `true` if at least one custom metadata version exists for the document. |
+
+These fields are present on every `GET /document/{id}` response regardless of whether the caller has any custom metadata role grants. A document that has never been assigned a schema returns `customSchemaId: null, hasCustomMetadata: false`.
+
+---
+
 ## Request/Response Schemas

 ### Standard Response Headers
@@ -1970,6 +2335,20 @@ The following table shows what data is passed to Permit.io for each protected en
 | `GET`                           | `/ui-settings/{id}`                      | -                   | -        | Auth-only (no Permit.io)|
 | `PATCH`                         | `/ui-settings/{id}`                      | -                   | -        | Auth-only (no Permit.io)|
 | `DELETE`                        | `/ui-settings/{id}`                      | -                   | -        | Auth-only (no Permit.io)|
+| **Schema Management (SuperAdminSchemaService)** |
+| `POST`                          | `/super-admin/custom-schemas`                     | `super-admin` | `post`   | Create new schema (super_admin only) |
+| `GET`                           | `/super-admin/custom-schemas`                     | `super-admin` | `get`    | List schemas with filters            |
+| `GET`                           | `/super-admin/custom-schemas/{schemaId}`           | `super-admin` | `get`    | Get a schema by ID                   |
+| `POST`                          | `/super-admin/custom-schemas/{schemaId}/versions`  | `super-admin` | `post`   | Create new version (supersedes parent) |
+| `DELETE`                        | `/super-admin/custom-schemas/{schemaId}`           | `super-admin` | `delete` | Retire schema (409 if documents bound) |
+| **Document Schema Assignment (SuperAdminSchemaService)** |
+| `PATCH`                         | `/super-admin/documents/{id}/schema`               | `super-admin` | `patch`  | Assign (or clear) custom schema for a document |
+| `POST`                          | `/super-admin/documents/{id}/reset-metadata`       | `super-admin` | `post`   | Wipe custom metadata history and clear schema binding |
+| **Custom Metadata Operations (CustomMetadataService)** |
+| `GET`                           | `/custom-metadata`                                 | `custom-metadata` | `get`  | Get current (latest) metadata version for a document |
+| `POST`                          | `/custom-metadata`                                 | `custom-metadata` | `post` | Write new validated metadata version for a document |
+| `GET`                           | `/custom-metadata/version`                         | `custom-metadata` | `get`  | Get a specific historical metadata version |
+| `GET`                           | `/custom-metadata/history`                         | `custom-metadata` | `get`  | List metadata version summaries (paged, descending) |

 ### Permit.io Configuration Requirements

@@ -1988,6 +2367,8 @@ To configure Permit.io to work with this API, administrators must create:
 | `field-extractions` | Field extraction operations                    |
 | `export`            | Export operations                              |
 | `admin`             | User administration and EULA admin operations  |
+| `super-admin`       | Schema management — gates all `/super-admin/custom-schemas` routes (super_admin CRUD; auditor read-only) |
+| `custom-metadata`   | Custom document metadata operations (all roles except auditor can read and write; auditor read-only) |

 #### Actions

@@ -2006,12 +2387,23 @@ The following roles are defined in the Permit.io configuration (`cmd/auth_relate
 | Role           | Resources                                                                              | Actions                          | Use Case                                |
 | -------------- | -------------------------------------------------------------------------------------- | -------------------------------- | --------------------------------------- |
 | `super_admin`  | All resources                                                                          | All actions including `delete`   | Full administrative access              |
-| `user_admin`   | `admin`, `documents`, `field-extractions`, `folders`                                   | `get`, `post`, `patch`, `delete` (admin only) | User management                |
-| `auditor`      | `admin`, `client`, `clients`, `document`, `export`, `documents`, `field-extractions`, `folders`, `labels` | `get` (and `head` for admin)     | Read-only access for auditing           |
-| `client_user`  | `client`, `clients`, `document`, `export`, `documents`, `field-extractions`, `folders`, `labels` | `get`, `post`, `patch` (varies by resource) | Client-specific access         |
+| `user_admin`   | `admin`, `documents`, `field-extractions`, `folders`, `custom-metadata`                | `get`, `post`, `patch`, `delete` (admin only); `get`, `post` (custom-metadata) | User management |
+| `auditor`      | `admin`, `client`, `clients`, `document`, `export`, `documents`, `field-extractions`, `folders`, `labels`, `super-admin`, `custom-metadata` | `get` (and `head` for admin) | Read-only access for auditing |
+| `client_user`  | `client`, `clients`, `document`, `export`, `documents`, `field-extractions`, `folders`, `labels`, `custom-metadata` | `get`, `post`, `patch` (varies by resource); `get`, `post` (custom-metadata) | Client-specific access |

 **Delete permissions**: Only `super_admin` has `delete` access to `client`, `document`, and `folders` resources. The `client_user` role does NOT have `client:delete` or any other delete permissions on these resources.

+**Schema management permissions** (`super-admin` and `custom-metadata` resources):
+
+| Role          | `super-admin` resource              | `custom-metadata` resource |
+| ------------- | ----------------------------------- | -------------------------- |
+| `super_admin` | `get`, `post`, `patch`, `delete`    | `get`, `post`              |
+| `user_admin`  | none                                | `get`, `post`              |
+| `auditor`     | `get`                               | `get`                      |
+| `client_user` | none                                | `get`, `post`              |
+
+The `super-admin` resource gates all `/super-admin/custom-schemas` CRUD endpoints. A `user_admin` has no `super-admin` grant and receives 403 on every schema management route.
+
 ### Authorization Error Responses

 When authorization fails, the API returns:
@@ -2040,6 +2432,22 @@ When authorization fails, the API returns:

 ---

+## Mutable Metadata v4 — Design Decisions
+
+The following decisions shaped the Milestone 1 schema CRUD implementation and apply to all subsequent milestones.
+
+- **`POST .../versions` instead of `PUT`**: A `PUT` route implies in-place replacement, which would overwrite the existing row and lose the ability to audit what the previous definition was. `POST /super-admin/custom-schemas/{schemaId}/versions` creates a new row with a new `id`, increments the `version` counter, and marks the parent row as `superseded` — all in a single transaction. The old row is preserved permanently and remains queryable.
+
+- **`createdBy` is not in request bodies**: The actor identity comes from the JWT `sub` claim (the Cognito subject UUID), extracted server-side by the handler. Accepting `createdBy` in the request body would allow any caller to impersonate a different actor. The `sub` claim is cryptographically bound to the token and cannot be spoofed by the client.
+
+- **Composite FK replaces the v3 trigger**: v3 used a trigger (`trg_validate_schema_client_match`) to prevent a document from being bound to a schema owned by a different client. v4 replaces this with a composite foreign key `fk_documents_custom_schema_same_client` on `(custom_schema_id, clientId)` referencing `client_metadata_schemas(id, client_id)`. Structural enforcement fires before any application code runs, catches the violation even on direct SQL updates, and requires no ongoing maintenance.
+
+- **`additionalProperties` must be explicitly declared**: JSON Schema's default behavior when `additionalProperties` is absent is to allow all additional properties. Requiring an explicit declaration forces the schema author to make a deliberate choice and prevents accidental data loss due to schema drift.
+
+- **`documentFieldExtractions` and custom metadata are mutually exclusive**: A document may either have legacy field extractions or custom schema metadata — never both. This is enforced at three layers: service-layer `FOR UPDATE` parent-row locks, a Postgres trigger (`trg_prevent_schema_reassignment`), and a second trigger (`trg_prevent_legacy_extraction_on_custom_document`). Defense-in-depth ensures the invariant holds even if one layer is bypassed.
+
+---
+
 ## Code Generation

 ### OpenAPI Integration