# Mutable Metadata v4 - Implementation Tracking **Source plan**: `plans/mutable.metadata.plan.combo.v4.md` **Journal (details / decisions / blockers)**: `./journals/implement_mutableMetadata.md` **Purpose**: One-glance status of every task needed to ship v4. Each checkbox is a discrete, verifiable unit of work. TDD rule: write the test first, watch it fail, make it pass, check the box. If a task requires Q's input or produces a decision, log it to the journal and keep the checkbox state accurate here. ## How v4 differs from v3 tracking v3 organized work as 12 horizontal phases (migrations first, then queries, then service, then controllers, then tests). v4 reorganizes into **four vertical slice milestones**. Each milestone lands migrations, queries, service methods, controllers, OpenAPI, Permit.io updates, and integration tests for exactly one slice of feature behavior. At the end of every milestone the feature is end-to-end usable in its current scope -- not a half-built stack. The simplification pass (see Section 16 of the plan) also removed: - v3's `trg_validate_schema_client_match` (v3 labelled "Trigger 2") -- replaced by the composite FK `fk_documents_custom_schema_same_client`. Note: v4 introduces a **different** Trigger 2 (`trg_prevent_legacy_extraction_on_custom_document`) — see Section 2.2; do not confuse the two. - v3's `trg_enforce_consistent_schema_id` (v3 labelled "Trigger 3") -- made structurally impossible by removing `document_custom_metadata.schema_id`. v4 has no Trigger 3. - `current_document_custom_metadata` view -- replaced by a `LIMIT 1` query - The entire delete-cascade workstream (v3 Phase 5b) -- replaced by `ON DELETE CASCADE` on the new FKs - `PUT /super-admin/custom-schemas/{schemaId}` -- replaced by `POST .../versions` - `createdBy` request-body fields -- actor comes from the JWT - `customSchemaName` and inlined `customMetadata` on `GET /document/{id}` -- only `customSchemaId` and `hasCustomMetadata` remain Every task below reflects those decisions. ## How to use this document - `[ ]` = not started - `[~]` = in progress - `[x]` = done AND tests passing (`task fullsuite:ci` clean for the affected code paths) - `[!]` = blocked -- add a short `(blocked: )` suffix and open a journal entry - `[-]` = consciously skipped -- add a short `(skipped: )` suffix - Every task line ends with a `test:` marker indicating where the passing test(s) live or what test type proves it. - When a whole milestone is green, flip the milestone header to `COMPLETE` and date-stamp it. - Never mark a milestone complete until `task fullsuite:ci` ends with `All coverage checks passed!` with no `FAIL` lines. --- ## Milestone roll-up (high-level) | Milestone | Title | Status | Started | Completed | |-----------|-------|--------|---------|-----------| | 0-INT | **BLOCKER**: Fix `task fullsuite:ci` regression (container race in `api/queryAPI`) | COMPLETE | 2026-04-15 | 2026-04-15 | | 1 | Schema foundation (create/list/version/delete schemas; super_admin gating) | COMPLETE | 2026-04-14 | 2026-04-14 | | 2 | Core document flow (assign schema; write/read metadata; legacy guard; `GET /document/{id}` enrichment) | COMPLETE | 2026-04-14 | 2026-04-15 | | 3 | Administrative completion (reset-metadata; schema version concurrency; full authz matrix) | COMPLETE | 2026-04-15 | 2026-04-15 | | 4 | Bulk operations (`POST /super-admin/folders/{folderId}/assign-schema`; 10K cap; skip reasons) | NOT STARTED | | | | Docs | Documentation updates (runs in parallel with Milestones 3 + 4) | NOT STARTED | | | --- ## Milestone 0-INT — Fix fullsuite:ci regression — COMPLETE (2026-04-15) **Goal**: `task fullsuite:ci` passes clean with `All coverage checks passed!` and no `FAIL` lines. This is a prerequisite for completing any other milestone. **Background**: The mutable metadata branch caused widespread duplicate-key failures in test runs because the reused postgres container retained data between runs. A truncation fix was applied to `internal/test/database.go` (see journal entry for 2026-04-15), resolving the duplicate-key issues but introducing a new regression: `api/queryAPI` tests intermittently fail with `"unable to ping database"` under parallel execution due to a container concurrency race in testcontainers `Reuse: true`. **What is already done**: - `[x]` Truncation fix in `internal/test/database.go` — resolves duplicate-key violations across all packages **What remains**: - `[x]` Diagnose actual failure: Q's run showed the container race was resolved by Docker state cleanup; the remaining failure was function coverage below 60% on `ResetDocumentMetadata` and `mapResetError` in `api/queryAPI/customschemas_reset.go` — both at 0% because no tests existed. - `[-]` Mutex fix — not needed; container race was accumulated Docker state, not a code bug. - `[x]` Write tests: added `TestMapResetError` (white-box, `package queryapi`) to `custommetadata_errors_test.go`; created `api/queryAPI/customschemas_reset_test.go` (`package queryapi_test`) with 5 integration tests covering 401/200/200-idempotent/404/409. All pass. - `[x]` Run `task fullsuite:ci` — passes with `All coverage checks passed!`, Total Coverage 85.4%, 2026-04-15. - `[x]` Flip this milestone to COMPLETE and date-stamp. --- ## Milestone 1 - Schema foundation — COMPLETE (2026-04-14) **Goal**: A super_admin can create, list, get, create-new-version, and delete client-scoped custom schemas. Invalid schemas are rejected. Cross-client schema bindings are structurally impossible at the database level. The `super-admin` Permit.io resource exists and gates every new schema endpoint. **Dependencies**: None. **Exit criteria**: - `task fullsuite:ci` green - End-to-end integration test proves super_admin can POST a schema, POST a new version, `GET /super-admin/custom-schemas?clientId=...&name=aircraft-engineering&includeAllVersions=true&status=any` returns both rows with the correct statuses (v1 `superseded`, v2 `active`), `GET` with the default query (no `status`, no `includeAllVersions`) returns only v2 `active`, DELETE removes a retired schema, and `user_admin` / `client_user` / `auditor` are correctly gated on every route. - Direct-SQL test proves the composite FK refuses a cross-client schema binding. ### 1.1 Migration 127 - `schema_status_type` enum + `client_metadata_schemas` > **Numbering note**: The repo contains migrations through `00000000000126`. If another branch lands new migrations before this feature merges, renumber this block and all references in the plan and tracking docs. - [x] Create `internal/database/migrations/00000000000127_create_client_metadata_schemas.up.sql` with `CREATE TYPE schema_status_type AS ENUM ('active', 'superseded', 'retired')` -- test: migration applies cleanly on empty DB - [x] Create `client_metadata_schemas` table per Section 4.2 of the plan -- columns, PK, FK to `clients(clientId) ON DELETE CASCADE`, `schema_def jsonb`, `version`, `status`, timestamps, `created_by` -- test: `\d client_metadata_schemas` shows all columns and the cascade FK - [x] Add `CONSTRAINT uq_client_schema_name_version UNIQUE (client_id, name, version)` -- test: direct SQL INSERT of two rows with same `(client_id, name, version)` fails - [x] **v4**: Add `CONSTRAINT uq_cms_id_client UNIQUE (id, client_id)` (required as the target of the composite FK in Milestone 1.3) -- test: constraint visible in `\d client_metadata_schemas` - [x] Add `CONSTRAINT chk_schema_def_size CHECK (pg_column_size(schema_def) <= 70000)` -- test: insert of a schema_def larger than 70000 bytes fails with CHECK violation - [x] Add indexes `idx_cms_client_id`, `idx_cms_client_name`, `idx_cms_client_status` -- test: `\di client_metadata_schemas` shows the three indexes - [x] Create `00000000000127_create_client_metadata_schemas.down.sql` dropping the table and enum in correct order -- test: up then down leaves DB state byte-identical to pre-migration - [x] Run `task db:generate` after migration lands -- test: no errors ### 1.2 Schema validator component + dependency - [x] Add `github.com/santhosh-tekuri/jsonschema/v6` to `go.mod` -- test: `go mod tidy` clean, build clean - [x] Create `internal/customschema/constants.go` with `MaxSchemaDefinitionBytes = 65536`, `MaxMetadataPayloadBytes = 1048576`, `MaxBulkAssignDocuments = 10000` -- test: constants referenced by tests - [x] Create `internal/customschema/validator.go` with `SchemaValidator` struct -- test: compiles - [x] Implement `ValidateSchemaDefinition(schemaDef json.RawMessage) error` -- test: valid JSON Schema draft 2020-12 accepted - [x] Reject schema definitions that are not valid JSON -- test: TDD test with malformed JSON - [x] Reject schema definitions that fail JSON Schema meta-validation -- test: TDD test with `{"type": "nonsense"}` - [x] Reject schema definitions where root `type` is not `"object"` -- test: TDD test with `{"type": "array"}` - [x] Reject schema definitions where root `additionalProperties` is not explicitly present -- test: TDD test with schema missing the key - [x] Accept schema definitions where root `additionalProperties` is explicitly `true` or `false` -- test: TDD tests for both - [x] Reject schema definitions larger than `MaxSchemaDefinitionBytes` -- test: TDD test with 64KB+1 payload - [x] `go test ./internal/customschema/...` green (validator only) -- test: package level ### 1.3 Migration 128 - `documents.custom_schema_id` column + composite FK > **v4 monotonic ordering**: Milestone 1 ships migrations 127 and 128. Milestone 2 ships 129 and 130. The filename numbering must match rollout order because `golang-migrate`'s `Up()` only applies versions strictly greater than the current DB version -- if Milestone 1 lands a 129 file, a later Milestone 2 file numbered 128 will be silently skipped or flagged as out of order. Earlier drafts of v4 had 128 and 129 swapped; do not revert that swap. - [x] Create `00000000000128_add_custom_schema_id_to_documents.up.sql` -- `ALTER TABLE documents ADD COLUMN custom_schema_id uuid NULL` (**no inline `REFERENCES` clause**; the composite FK below handles it) -- test: column present and nullable - [x] Add composite FK: `ALTER TABLE documents ADD CONSTRAINT fk_documents_custom_schema_same_client FOREIGN KEY (custom_schema_id, clientId) REFERENCES client_metadata_schemas(id, client_id)` -- test: constraint visible; direct SQL INSERT of a document with `custom_schema_id` pointing at a schema owned by a different client fails with FK violation. **v4 implementation note**: the plan Section 4.4 SQL writes `"clientId"` quoted, but the `documents.clientId` column was declared unquoted in migration 5, so Postgres stored it as lowercase `clientid`. A quoted `"clientId"` in the FK declaration is a case-sensitive literal lookup that fails with `column "clientId" referenced in foreign key constraint does not exist`. The migration uses **unquoted** `clientId` which folds to `clientid` at parse time and matches. This same gotcha applies to any future DDL or trigger that references `documents."clientId"`. - [x] Create `idx_documents_custom_schema_id` -- test: index present - [x] Write `down.sql` that drops the index, the constraint, and the column in correct order -- test: round trip clean - [x] Verify existing documents rows have `custom_schema_id = NULL` after migration -- test: `SELECT COUNT(*) FROM documents WHERE custom_schema_id IS NOT NULL` returns 0 ### 1.4 SQLC queries for schema CRUD New query file: `internal/database/queries/customschemas.sql` - [x] `CreateClientMetadataSchema` -- INSERT returning full row -- test: generated code compiles, query name appears in `*.sql.go` - [x] `GetClientMetadataSchema` (by id) -- test: generated - [x] `ListClientMetadataSchemas` (filters: client_id required, name optional, status optional with default `active`, pagination limit/offset; includes `documentCount` subquery against `documents.custom_schema_id`) -- test: generated; unit test confirms `documentCount` populated correctly - [x] `GetLatestSchemaByName` -- test: generated - [x] `GetSchemaDocumentCount` (count of `documents.custom_schema_id = $1`) -- test: generated - [x] `SetSchemaStatus` (update status, used for supersede and retire) -- test: generated - [x] `GetMaxSchemaVersion` (for auto-increment on version creation) -- test: generated - [x] `LockSchemaVersionsForName` (`SELECT ... FOR UPDATE` on all versions of a `(client_id, name)` pair) -- test: generated - [x] `task generate` clean; no diff after re-run -- test: idempotent > **Note**: `GetSchemaMetadataRecordCount` is deliberately **not** in this list. v3 used it for `canDelete`, but v4's `canDelete` is derived from `GetSchemaDocumentCount` alone (the metadata table has no `schema_id` column). ### 1.5 Audit sink > **Why in Milestone 1**: Schema CRUD needs to log schema.create, schema.update, schema.delete. Defining the sink now lets Milestone 1's service methods land with audit wired in, and Milestones 2/3/4 reuse the same sink without back-fill. - [x] Decide sink location: either `internal/customschema/audit.go` (feature-local) or `internal/audit/` (shared) -- test: decision recorded in journal (feature-local chosen 2026-04-14) - [x] Define `AuditRecord` struct: `Actor string`, `Action string` (enum: `schema.create|schema.update|schema.delete|schema.assign|metadata.write|metadata.reset|folder.assign_schema`), `ResourceID uuid.UUID`, `ClientID uuid.UUID`, `Timestamp time.Time`, `Details map[string]any` -- test: compiles - [x] Define `AuditSink` interface with `Record(ctx, AuditRecord) error` -- test: compiles - [x] Provide a default implementation (structured `slog` with `audit=true` attribute, or a DB-backed appender if Q decides) -- test: unit test verifies a recorded event lands in the sink (`TestSlogAuditSink_Record` — 4 subtests green) - [x] Wire the sink into `customschema.Service` via constructor arg -- test: compiles (Service.New(cfg, validator, audit) landed in §1.6) ### 1.6 Service layer - schema CRUD - [x] Create `internal/customschema/service.go` with `Service` struct holding `cfg`, `validator`, `audit` -- test: compiles - [x] Create `internal/customschema/models.go` with `Schema`, `SchemaSummary`, `CreateSchemaInput`, `CreateSchemaVersionInput`, `ListFilters` -- test: compiles - [x] **v4**: `CreateSchemaInput` and `CreateSchemaVersionInput` do **not** carry a `CreatedBy` field. Every write method accepts `actor string` as a separate argument. -- test: unit test proves the service method signature has `actor string` and not a body field - [x] `CreateSchema(ctx, input, actor)` -- meta-validates, enforces 64KB, enforces explicit `additionalProperties`, enforces `name` unique within client for active schemas at version 1 -- test: happy path - [x] `CreateSchema` rejects invalid JSON Schema -- test: TDD - [x] `CreateSchema` rejects oversize definition -- test: TDD - [x] `CreateSchema` rejects duplicate name for same client at version 1 -- test: TDD (service returns `ErrSchemaNameConflict`, wraps pg `23505` / `uq_client_schema_name_version`) - [x] `CreateSchema` writes `schema.create` audit entry -- test: TDD - [x] `CreateSchemaVersion(ctx, parentSchemaID, input, actor)` -- uses `LockSchemaVersionsForName` -> verifies the path `parentSchemaID` matches the single row with `status='active'` for `(client_id, name)` -> `GetMaxSchemaVersion` -> INSERT new version -> `SetSchemaStatus(parent, 'superseded')` inside a single tx -- test: happy path produces v2 - [x] `CreateSchemaVersion` concurrency: two concurrent calls against same `(client_id, name)` produce distinct version numbers -- test: TDD with goroutines against testcontainer DB (v4 resolution: winner writes v2, loser returns `ErrSchemaParentNotActive` — no retry loop; the `FOR UPDATE` serializes the two tx and the loser observes the parent as superseded) - [x] `CreateSchemaVersion` rejects when parent does not exist -- test: TDD (returns `ErrSchemaNotFound`) - [x] **v4 non-active parent rejection**: `CreateSchemaVersion` rejects with 409 when the path `parentSchemaID` points at a `superseded` row while a different row is the active version for the lineage (e.g. v2 is active, caller passes v1's ID) -- test: TDD asserts 409, asserts no new row inserted, asserts v2 still has `status='active'` and v1 still has `status='superseded'` (service returns `ErrSchemaParentNotActive`) - [x] **v4 retired parent rejection**: `CreateSchemaVersion` rejects with 409 when the path `parentSchemaID` points at a `retired` row -- test: TDD asserts 409, asserts no new row inserted, asserts the retired row's status is unchanged (service returns `ErrSchemaParentRetired`) - [x] **v4 active-parent invariant**: After `CreateSchemaVersion` succeeds there is exactly one row with `status='active'` for the `(client_id, name)` lineage -- test: TDD asserts the invariant after happy path, after two sequential calls (v1 -> v2 -> v3), and after the two rejection cases above - [x] `CreateSchemaVersion` rejects invalid schema definition (same validator rules as create) -- test: TDD - [x] `CreateSchemaVersion` writes `schema.update` audit entry -- test: TDD (`ResourceID` is the new version's ID; `Details` carries `name`, `version`, `parent_schema_id`) - [x] `GetSchema(ctx, schemaID)` -- test: returns full row including `schema_def` - [x] `GetSchema` 404 when not found -- test: TDD (returns `ErrSchemaNotFound` via `errors.Is(err, pgx.ErrNoRows)`) - [x] `ListSchemas(ctx, clientID, filters)` -- default returns only `active` and only the latest version per `(client_id, name)` -- test: TDD - [x] `ListSchemas` filter by `name`, by `status` (including the `any` sentinel for "all statuses"), by `includeAllVersions=true`, with pagination -- test: TDD per case, including one case that sets both `status=any` and `includeAllVersions=true` against a lineage that has one active + one superseded row and asserts both rows are returned - [x] `ListSchemas` populates `documentCount` and `canDelete` -- test: TDD - [x] `ListFilters` accepts an `Status` field with zero-value meaning "active" (default) and a dedicated `StatusAny` sentinel (or `*schemaStatusType` with `nil` meaning "any") so handlers can distinguish "filter omitted" from "caller asked for everything" — test: unit test on the mapping from query-string `status=any` to the service-layer value - [x] `DeleteSchema(ctx, schemaID, actor)` -- sets status to `retired` -- test: happy path - [x] `DeleteSchema` returns 409 if any document references the schema via `custom_schema_id` -- test: TDD (returns `ErrSchemaInUse`) - [x] `DeleteSchema` writes `schema.delete` audit entry -- test: TDD - [x] `go test ./internal/customschema/...` green -- test: package level - [x] `task test:race` green for package -- test: no races (`go test -race ./internal/customschema/... → ok 8.303s`) ### 1.7 OpenAPI spec - schema CRUD - [x] Add `SuperAdminSchemaService` tag with description explicitly stating the `super_admin`-only boundary -- test: spec lints - [x] Add `SchemaStatus` enum (`active`/`superseded`/`retired`) -- test: codegen produces Go type - [x] `CustomSchemaRequest` schema (no `createdBy` field) -- test: codegen - [x] `CustomSchemaVersionRequest` schema (no `createdBy`; v4 rename from v3's `CustomSchemaUpdateRequest`) -- test: codegen - [x] `CustomSchemaResponse` schema with `createdBy` declared as plain string (**not** `format: email`) per plan Section 8.3 "Actor fields" -- test: codegen; spec lint - [x] Spec check: assert that none of the new response schemas declare `format: email` on a `createdBy` / `resetBy` field -- test: grep or programmatic spec check in `task openapi:lint` (verified via `grep -nE 'format: ?email' serviceAPIs/queryAPI.yaml | grep -iE 'createdBy|resetBy'` → 0 matches) - [x] `CustomSchemaListResponse` schema with `documentCount` and `canDelete` fields -- test: codegen - [x] `POST /super-admin/custom-schemas` path -- test: codegen produces handler interface - [x] `GET /super-admin/custom-schemas` (with query params) -- test: codegen - [x] `GET /super-admin/custom-schemas/{schemaId}` -- test: codegen - [x] **v4**: `POST /super-admin/custom-schemas/{schemaId}/versions` path (replaces v3's `PUT /super-admin/custom-schemas/{schemaId}`) -- test: codegen - [x] `DELETE /super-admin/custom-schemas/{schemaId}` -- test: codegen - [x] `task generate` clean -- test: no errors - [x] Generated controllers compile -- test: `go build ./...` (stub handlers land in `api/queryAPI/customschemas_stub.go` returning `http.StatusNotImplemented`; real handlers arrive in §1.9) ### 1.8 Permit.io policy file + setup tool run > **Isolation model (v4)**: Production is deployed **one client per stack** (separate DB, services, Cognito pool). The whole stack is the client scope, so the Permit.io role/action/resource-type check is the complete authorization story on production — there is no other client to cross into. Shared dev/uat stacks are test data only (see plan Section 5.3). Do **not** add application-level document->client ownership checks to the custom-metadata or super-admin routes in v4; there is no cross-client data to protect from inside a production stack, and retrofitting such a check would be an API-wide initiative out of scope for this feature. - [x] Add `super-admin` resource to `cmd/auth_related/permit.setup/permit_policies.yaml` with `get`, `post`, `patch`, `delete` actions -- test: YAML valid (verified via `python3 -c 'yaml.safe_load(...)'` 2026-04-14) - [x] Add `custom-metadata` resource with `get`, `post` actions (provisioning it now lets Milestone 2 ship without another setup-tool run) -- test: YAML valid - [x] Grant `super_admin` role: `super-admin: [get, post, patch, delete]`, `custom-metadata: [get, post]` -- test: YAML valid - [x] Confirm `user_admin` role NOT granted `super-admin` -- test: visual diff (YAML parser confirmed `user_admin super-admin: None`) - [x] Grant `user_admin` role: `custom-metadata: [get, post]` -- test: YAML valid - [x] Grant `auditor` role: `super-admin: [get]`, `custom-metadata: [get]` -- test: YAML valid - [x] Grant `client_user` role: `custom-metadata: [get, post]` -- test: YAML valid - [x] Add documentation-only `policy_mappings` entries for every `/super-admin/*` path including `/versions` and `/reset-metadata` -- test: YAML valid - [x] Add documentation-only `policy_mappings` entries for every `/custom-metadata/*` path -- test: YAML valid - [x] **v4**: No `put` action needed on `super-admin` -- v4 replaced the v3 `PUT` route with `POST .../versions`. Record this decision in the journal so no one re-adds it. -- test: journal entry (recorded) - [x] Run `cmd/auth_related/permit.setup/run.tool.dev.sh` -- test: exit code 0 (Q ran 2026-04-14; confirmed dev environment updated) - [x] Verify `super-admin` and `custom-metadata` resources exist in dev with expected actions -- test: confirmed by Q running the setup tool successfully 2026-04-14 - [x] Verify `super_admin` role has expected grants and `user_admin` has NO `super-admin` grants in dev -- test: confirmed by Q 2026-04-14 - [!] Run `run.tool.uat.sh` after dev verified -- test: exit 0 (deferred until feature complete per Q directive 2026-04-14: "the others do not need to be run until this feature is completed") - [!] Same verification pass on uat -- test: list + role inspect (deferred until feature complete) - [!] Run `run.tool.prod.sh` after uat verified AND only with Q's explicit go-ahead -- test: exit 0, Q confirmation logged in journal (deferred until feature complete) - [!] Same verification pass on prod -- test: list + role inspect (deferred until feature complete) - [x] Journal entry with timestamps of each env run (dev: 2026-04-14; uat/prod: deferred) ### 1.9 Schema CRUD controllers - [x] Create `api/queryAPI/customschemas.go` skeleton with handler receivers wired to `customschema.Service` -- test: compiles (replaced the §1.7 stub file) - [x] **v4 actor extraction helper**: every handler in this file derives actor identity via `cognitoauth.GetUserSubject(c)` (returns the JWT `sub` claim — an opaque Cognito subject UUID, **not** an email) and passes it to the service as a separate argument; any `createdBy` on the request body is ignored (actually absent from v4 schemas, but the test codifies the intent) -- test: handler unit test sets a JWT subject via middleware fixture to a fixed UUID, posts a body that includes an extra `createdBy` key with a different email-shaped value, asserts the service receives the JWT subject UUID and the body key has no effect, and asserts the response `createdBy` is the UUID (not the body value, not an email) (handler tests set claims via `ctx.Set("user_claims", ...)` mirroring `eulaHandlers_test.go`) - [x] `POST /super-admin/custom-schemas` handler -- parse, validate, derive actor, call `CreateSchema`, map errors to 400/409/500 -- test: TDD (`TestCreateCustomSchema_Handler` 5 subtests) - [x] `GET /super-admin/custom-schemas` handler with query params -- test: TDD (`TestListCustomSchemas_Handler` 4 subtests) - [x] `GET /super-admin/custom-schemas/{schemaId}` handler -- test: TDD (`TestGetCustomSchema_Handler` 2 subtests) - [x] **v4**: `POST /super-admin/custom-schemas/{schemaId}/versions` handler -- derives actor, calls `CreateSchemaVersion` -- test: TDD (`TestCreateCustomSchemaVersion_Handler` 3 subtests) - [x] `DELETE /super-admin/custom-schemas/{schemaId}` handler -- test: TDD including 409 path (`TestDeleteCustomSchema_Handler` 3 subtests) - [x] `go test ./api/queryAPI/customschemas_test.go` green -- test: package (17 subtests PASS under scoped -run invocations) ### 1.10 Milestone 1 integration tests - [x] End-to-end test: super_admin POSTs schema v1 -> GET returns it with version 1, active -- test: integration against testcontainer DB (`TestMilestone1_CreateV1AndGet`) - [x] End-to-end test: super_admin POSTs schema v2 via `POST .../versions` -> `GET /super-admin/custom-schemas?clientId=...&name=...&includeAllVersions=true&status=any` returns v1 (status `superseded`) and v2 (status `active`). Also assert the default `GET` (no `includeAllVersions`, no `status`) returns **only** v2 (the default filter is `includeAllVersions=false` + `status=active`, which intentionally hides superseded predecessors from the common admin UX). -- test: integration (`TestMilestone1_CreateV2ListSemantics` with 2 subtests `default_list_returns_active_latest_only` and `include_all_versions_status_any_returns_both`) - [x] End-to-end test: DELETE on a schema with documents bound returns 409 -- test: integration (`TestMilestone1_DeleteBoundReturns409`; asserts row stays `active` in DB on denial) - [x] End-to-end test: DELETE on an unreferenced schema succeeds and sets status `retired` -- test: integration (`TestMilestone1_DeleteUnreferencedRetires`) - [x] Direct-SQL cross-client rejection test: attempt an `UPDATE documents SET custom_schema_id = 'schema-owned-by-client-B' WHERE id = 'doc-owned-by-client-A'` and assert the composite FK fires -- test: integration (`TestMilestone1_CrossClientFKRejection`; asserts the error text mentions `fk_documents_custom_schema_same_client` and docA.custom_schema_id is still NULL) - [x] Authorization matrix for Milestone 1 endpoints: `super_admin` positive on all methods; `user_admin` 403 on every route; `client_user` 403; `auditor` GET-only on the two read routes, 403 on write routes -- test: one integration test per (role, route, method) cell, each asserting status + DB unchanged on denial (`TestMilestone1_AuthorizationMatrix` — 4 roles × 5 routes = 20 subtests; `runPermitGate` reproduces the `performPermitIOAuthorization` choke-point logic in-test using only the exported `cognitoauth` symbols `GetUserSubject`, `GetResourceFromRoute`, `GetActionFromMethod`, `PermitChecker`) - [x] `task fullsuite:ci` clean with `All coverage checks passed!` -- test: full run (55 packages OK, 0 FAIL, Total Coverage 86.0%, 2026-04-14 evening run captured in `/tmp/fullsuite-ci.log`) - [x] Journal entry: Milestone 1 exit snapshot (test counts, coverage numbers, timestamp, commit SHA) (see 2026-04-14 Milestone 1 COMPLETE entry in `journals/implement_mutableMetadata.md`) --- ## Milestone 2 - Core document flow **Goal**: A super_admin can bind a schema to a document. A client_user (or higher) can write and read validated custom metadata on that document. The legacy field extraction path rejects writes to documents that have been opted into the custom schema system. `GET /document/{id}` surfaces two new fields for any reader. The mutual-exclusivity invariant is proven end-to-end. **Dependencies**: Milestone 1 complete. **Exit criteria**: - Full round-trip test: create schema -> assign to document -> POST valid metadata -> GET returns it -> attempt invalid metadata (400) -> attempt legacy field extraction on same document (409). - FK-cascade delete test: delete a document that has custom metadata, verify rows are cleaned up automatically. - `task fullsuite:ci` green. ### 2.1 Migration 129 - `document_custom_metadata` table - [x] Create `00000000000129_create_document_custom_metadata.up.sql` with `document_custom_metadata` table per Section 4.3 of the plan -- columns: `id`, `document_id`, `metadata`, `version`, `created_at`, `created_by`. **No `schema_id` column.** -- test: `\d document_custom_metadata` confirms column set - [x] FK on `document_id` uses `ON DELETE CASCADE` -- test: direct SQL delete of a document row removes its metadata rows - [x] Add `CONSTRAINT uq_doc_custom_metadata_version UNIQUE (document_id, version)` -- test: direct SQL duplicate `(document_id, version)` INSERT fails - [x] Add `idx_dcm_doc_version_desc ON document_custom_metadata(document_id, version DESC)` (the only index v4 needs on this table) -- test: index present - [x] **v4**: No `idx_dcm_schema_id` (column does not exist). **No view.** Record this explicitly in the migration file comment so a future reader comparing to v3 sees why. -- test: visual - [x] Create corresponding `down.sql` dropping the table -- test: up/down round trip clean ### 2.2 Migration 130 - schema invariant triggers (v4: Trigger 1 + Trigger 2) - [x] Create `00000000000130_add_schema_invariant_triggers.up.sql` -- test: migration applies cleanly - [x] Define `trg_prevent_schema_reassignment()` function per Section 4.7 Trigger 1 -- test: function exists in `pg_proc` - [x] Attach trigger to `documents` BEFORE UPDATE OF `custom_schema_id` -- test: trigger listed in `\d documents` - [x] Trigger 1 unit test: insert document, assign schema, write custom metadata, attempt UPDATE `custom_schema_id` -- expect `RAISE EXCEPTION` with "custom metadata already exists" message - [x] Trigger 1 unit test: insert document, add legacy extraction row, attempt UPDATE `custom_schema_id` -- expect `RAISE EXCEPTION` with "legacy field extractions already exist" message - [x] Trigger 1 unit test: insert document with no metadata and no extractions, UPDATE `custom_schema_id` from schema_a to schema_b -- expect success (pre-binding change allowed) - [x] Define `trg_prevent_legacy_extraction_on_custom_document()` function per Section 4.7 Trigger 2 -- reads `documents.custom_schema_id` with `FOR SHARE` and raises `check_violation` if non-null -- test: function exists in `pg_proc` - [x] Attach trigger to `documentFieldExtractions` BEFORE INSERT -- test: trigger listed in `\d "documentFieldExtractions"` - [x] Trigger 2 unit test: insert document with `custom_schema_id` set, attempt direct-SQL INSERT into `documentFieldExtractions` -- expect `RAISE EXCEPTION` with "bound to custom schema" message and SQLSTATE `23514` (`check_violation`) - [x] Trigger 2 unit test: insert document with `custom_schema_id = NULL`, INSERT into `documentFieldExtractions` -- expect success (no-op guard path) - [x] Trigger 2 concurrency test: start tx A running `AssignSchema`-style `SELECT documents FOR UPDATE` + `UPDATE documents SET custom_schema_id = S`, then in tx B run `INSERT documentFieldExtractions` targeting the same document -- tx B's trigger's `FOR SHARE` must block until tx A commits; after tx A commits tx B's INSERT must fail with the check_violation -- test: TDD with goroutines against testcontainer DB - [x] **v4**: Migration intentionally does NOT define v3's `trg_validate_schema_client_match` (replaced by the composite FK in Milestone 1.3) -- record this in a migration-file comment for future readers - [x] **v4**: Migration intentionally does NOT define v3's `trg_enforce_consistent_schema_id` (not needed -- `document_custom_metadata` has no `schema_id` column) -- record this in a migration-file comment - [x] Write `down.sql` dropping both triggers and both functions in correct order -- test: up/down round trip clean - [x] Run `task generate` after migrations land -- test: clean - [x] Run `task test:unit:short` -- test: all existing tests still pass (no regression) ### 2.3 SQLC queries for metadata + document binding - [x] `CreateDocumentCustomMetadata` (no `schema_id` parameter) -- test: generated code compiles - [x] `GetCurrentDocumentCustomMetadata` -- `ORDER BY version DESC LIMIT 1` using `idx_dcm_doc_version_desc` (**v4: no view**). **Joins `documents` and `client_metadata_schemas`** to also return `schema_id`, `schema_name`, `schema_version` so the `GET /custom-metadata` handler can populate its response in a single round-trip -- test: generated; unit test asserts the returned row carries the schema name/version for a metadata row whose parent document has a bound schema - [x] `GetDocumentCustomMetadataByVersion` -- same join as above so `GET /custom-metadata/version` returns matching schema decoration -- test: generated - [x] `GetDocumentCustomMetadataHistory` (with limit/offset) -- no schema join; the history response returns version summaries only -- test: generated - [x] **v4**: `LockDocumentForMetadataWrite` (`SELECT id FROM documents WHERE id = @document_id FOR UPDATE`; parent-row lock that serializes first and subsequent writes) -- test: generated. **Replaces** v3's `LockDocumentCustomMetadataForVersion`. - [x] `GetMaxDocumentMetadataVersion` -- `SELECT COALESCE(MAX(version), 0) FROM document_custom_metadata WHERE document_id = @document_id` -- test: generated - [x] `SetDocumentCustomSchemaId` -- test: generated - [x] `GetDocumentCustomSchemaId` -- test: generated - [x] **Reuse existing query**: `HasFieldExtraction` already exists in `internal/database/queries/fieldextractions.sql` (`EXISTS(SELECT 1 FROM documentFieldExtractionVersions WHERE documentId = $1)`) and is semantically "does this document have any legacy field-extraction version row?" — exactly the check the mutual-exclusivity guard and `AssignSchema` legacy-extraction rejection need. Do **not** add a new `DocumentHasLegacyExtractions` query. Call `HasFieldExtraction` from the service layer. If readability at the call site is a concern, wrap the call in a private service-layer helper named `documentHasLegacyExtractions(ctx, id)` that just forwards to `HasFieldExtraction` — no new SQL, no new generated code -- test: service unit test confirms `AssignSchema` and the metadata write path both reject documents that have an existing `documentFieldExtractionVersions` row - [x] `task generate` clean -- test: idempotent > **v4**: No delete-cascade queries are added in this milestone. `DeleteDocumentCustomMetadata`, `NullifyDocumentCustomSchemaId`, and `DeleteClientMetadataSchemas` from v3 are **not** created -- the FK cascades replace them. ### 2.3a Extend `internal/document` backend for `GET /document/{id}` enrichment > **Why this subsection exists**: Plan Section 5.2 declares that `GET /document/{id}` gains two new response fields (`customSchemaId`, `hasCustomMetadata`). The controller-side builder in §2.7 reads those values out of the `document.DocumentEnriched` struct, but as of the start of Milestone 2 the struct has neither field and the `GetDocumentEnriched` SQL query (`internal/database/queries/document.sql:63`) does not return either column. Without the three changes below, the §2.7 builder has nothing to read and the §2.8 integration tests will fail on the enrichment assertions. Do this subsection **before** §2.7. - [x] Extend the `GetDocumentEnriched` query in `internal/database/queries/document.sql` to add `d.custom_schema_id` to the SELECT list and a correlated `EXISTS(SELECT 1 FROM document_custom_metadata WHERE document_id = d.id) AS has_custom_metadata` subquery — no join to `document_custom_metadata` (the `EXISTS` keeps the plan flat and index-friendly). -- test: unit test against testcontainer DB asserts the query returns the two new columns with the right values for (a) a document with no schema bound, (b) a document with a schema bound but zero metadata rows, (c) a document with a schema bound and >=1 metadata rows - [x] `task db:generate` clean — confirm the generated `GetDocumentEnriched` row type in `internal/database/repository/` now has `CustomSchemaID` and `HasCustomMetadata` fields -- test: `go build ./internal/database/repository/...` clean; `grep -n CustomSchemaID internal/database/repository/document.sql.go` finds the new field - [x] Extend the `DocumentEnriched` struct in `internal/document/service.go` to add `CustomSchemaID *uuid.UUID` (nullable because most documents are not schema-bound) and `HasCustomMetadata bool`. Place them adjacent to `HasTextRecord` and `FileSizeBytes` so the field layout of the struct stays topically grouped -- test: `go build ./internal/document/...` clean; unit test instantiates the struct and asserts the two new fields zero-value to `nil` and `false` - [x] Update `GetEnriched` in `internal/document/get.go` to copy `docEnriched.CustomSchemaID` and `docEnriched.HasCustomMetadata` from the extended generated row into the returned `DocumentEnriched` struct — **no** separate follow-up query, no extra round-trip to the DB -- test: service-level unit test against testcontainer DB asserts the returned struct carries the right values for all three fixture cases (no schema, schema no metadata, schema with metadata). Assert in a second test that the function issues **exactly** the same number of DB queries before and after the change (i.e., no regression from "one enriched read" to "enriched read + follow-up exists check") - [x] Run existing tests for `internal/document/` and `api/queryAPI/document*` — every pre-existing test that reads `DocumentEnriched` must still pass without modification; the two new fields are additive. -- test: `go test ./internal/document/... ./api/queryAPI/... -run Document` green ### 2.4 Service layer - metadata + document binding - [x] Add `CustomMetadata`, `MetadataVersion`, `SetMetadataInput` types to `models.go` -- test: compiles. **`SetMetadataInput` has no `CreatedBy` field.** **`CustomMetadata` carries `SchemaID`, `SchemaName`, `SchemaVersion`** populated from the joined query in 2.3 (Section 7.1 of the plan has the struct shape). - [x] `SetDocumentMetadata(ctx, input, actor)` -- derives schema from `documents.custom_schema_id`, never trusts client -- test: happy path - [x] `SetDocumentMetadata` rejects when document has no `custom_schema_id` -- test: TDD (400) - [x] `SetDocumentMetadata` rejects when document has legacy extractions -- test: TDD (409) - [x] `SetDocumentMetadata` validates metadata against schema via `validator.ValidateMetadata` -- test: TDD with conforming and non-conforming payloads - [x] `SetDocumentMetadata` enforces `MaxMetadataPayloadBytes` -- test: TDD - [x] `SetDocumentMetadata` serializes version assignment by taking `LockDocumentForMetadataWrite` on the parent row, then `GetMaxDocumentMetadataVersion`, then INSERT `version = max+1` -- test: TDD confirms the parent-row lock is acquired before the version read - [x] `SetDocumentMetadata` concurrency: two concurrent writers against the **same** document that already has >=1 metadata row -- both succeed, produce distinct versions, no unique-violation -- test: TDD with goroutines against testcontainer DB - [x] `SetDocumentMetadata` concurrency: two concurrent **first** writers against a document with zero metadata rows -- both succeed, produce `version=1` and `version=2` in some order -- test: TDD with goroutines, proves the parent-row lock is what serializes the first write - [x] `SetDocumentMetadata` retries once on `UNIQUE (document_id, version)` violation as a belt-and-suspenders fallback -- test: TDD with an injected unique-violation on the first INSERT attempt - [x] `SetDocumentMetadata` writes `metadata.write` audit entry -- test: TDD - [x] `GetCurrentMetadata(ctx, documentID)` -- test: uses the LIMIT 1 query, returns latest - [x] `GetMetadataByVersion(ctx, documentID, version)` -- test: TDD - [x] `GetMetadataHistory(ctx, documentID)` with pagination -- test: TDD - [x] Add `AssignSchemaResult` type to `models.go` per plan Section 7.1 (`DocumentID uuid.UUID`, `CustomSchemaID *uuid.UUID`, `SchemaName *string`, `SchemaVersion *int`) -- test: compiles - [x] `AssignSchema(ctx, documentID uuid.UUID, schemaID *uuid.UUID, actor string) (*AssignSchemaResult, error)` — signature takes `schemaID` as a pointer so `nil` clears the binding and returns an `AssignSchemaResult` with all three schema fields `nil` -- test: happy path single-document assignment returns a populated result - [x] `AssignSchema` populates `SchemaName` and `SchemaVersion` on the result from the same schema row read it already does for the `active` / same-client validation — no new query is added for this decoration -- test: TDD asserts result fields match the target schema row - [x] `AssignSchema` takes `SELECT id, "clientId", custom_schema_id FROM documents WHERE id = $1 FOR UPDATE` as the **first** statement inside the transaction, before any legacy-extraction existence check or the `UPDATE documents` -- this is the symmetric parent-row lock for the mutual-exclusivity invariant described in plan Section 7.3 -- test: TDD; code review confirms the lock precedes all other DML - [x] `AssignSchema` rejects if schema and document belong to different clients -- the composite FK does this at the DB layer; the app-layer check converts the FK violation to a clean 409 -- test: TDD - [x] `AssignSchema` rejects if schema status is not `active` -- test: TDD - [x] `AssignSchema` rejects if document already has custom metadata (matches Trigger 1 behavior; expect 409) -- test: TDD - [x] `AssignSchema` rejects if document has legacy extractions (service-layer check under the parent-row lock plus Trigger 1 as backstop; expect 409) -- test: TDD - [x] `AssignSchema` allows setting `schemaID == nil` to clear binding when no metadata exists; result has `CustomSchemaID`, `SchemaName`, `SchemaVersion` all `nil` -- test: TDD - [x] `AssignSchema` writes `schema.assign` audit entry -- test: TDD - [x] `go test ./internal/customschema/...` green -- test: full package green for Milestones 1+2 service methods ### 2.5 Legacy field extraction guard + controller 409 mapping > **v4 concurrency requirement**: A non-locking `SELECT documents.custom_schema_id` is NOT sufficient. Plan Section 7.3 walks through the race: two transactions each read a stale snapshot under `READ COMMITTED`, both pass the pre-check, and both commit opposing writes. The fix is a `SELECT ... FOR UPDATE` on the parent `documents` row as the **first** statement inside the tx, mirrored by the same lock in `AssignSchema`. Trigger 2 (Milestone 2.2) is the DB-layer backstop. - [x] Locate the existing `FieldExtractionService.CreateFieldExtraction` entry point (`internal/fieldextraction/service.go:27`) -- test: grep and read code - [x] Add a new SQLC query `LockDocumentForLegacyExtractionWrite` (`SELECT id, custom_schema_id FROM documents WHERE id = @document_id FOR UPDATE`) in `internal/database/queries/document.sql` -- test: generated code compiles - [x] Modify `CreateFieldExtraction`: after `Begin(ctx)` and **before** any other DML, call `LockDocumentForLegacyExtractionWrite`. The existing `AddFieldExtraction` call must move to **after** the lock acquisition. -- test: code review confirms no DML runs before the lock - [x] If the lock query returns `pgx.ErrNoRows`, return a typed sentinel error `ErrDocumentNotFound` (controller maps to 404) -- test: TDD service test - [x] If the locked row has `custom_schema_id IS NOT NULL`, return a typed sentinel error `ErrMutualExclusivityViolation` -- test: TDD service test asserts `errors.Is(err, ErrMutualExclusivityViolation)` - [x] Service-layer concurrency test: two goroutines, one calls `AssignSchema` (which will itself take `FOR UPDATE` -- see 2.4) and the other calls `CreateFieldExtraction`. Run both orderings. Whichever commits second must fail -- `AssignSchema`-then-`CreateFieldExtraction` fails with `ErrMutualExclusivityViolation`; `CreateFieldExtraction`-then-`AssignSchema` fails with the Trigger 1 EXCEPTION converted to 409. Neither ordering may leave the document in the "both systems active" state. -- test: TDD with goroutines against testcontainer DB; assertion checks final DB state in addition to returned errors - [x] Service-layer concurrency test (DB trigger backstop): deliberately skip the `FOR UPDATE` call path in a test-only harness, run the same race, and confirm Trigger 2 still catches it by surfacing a PostgreSQL `check_violation` error with SQLSTATE `23514`. This proves the defense-in-depth works even if the service-layer lock is ever removed by a regression. -- test: TDD - [x] **Regression guard (v4 — not a present-day audit)**: at the time of this plan, `FieldExtractionService` has exactly one mutator — `CreateFieldExtraction` at `internal/fieldextraction/service.go:27`. The other four methods (`GetCurrentFieldExtraction`, `GetFieldExtractionHistory`, `GetFieldExtractionArrayFields`, `GetFieldExtractionByVersion`) are reads. v3 carried a generic "audit every other mutation entry point" task that pointed at methods (`UpdateFieldExtraction`, `AppendValueTo*`) that do not exist and have never existed. Instead of re-running a phantom audit, add a **regression test** that enforces the rule going forward: list the exported methods of `FieldExtractionService` via reflection (or, simpler, a hand-maintained allow-list), and fail the test if any new exported method with a mutating name prefix (`Create`, `Update`, `Append`, `Delete`, `Insert`, `Set`, `Reset`) is added without being added to a second allow-list that says "this method has been reviewed and it takes `LockDocumentForLegacyExtractionWrite` before any DML." -- test: unit test in `internal/fieldextraction/` that fails loudly with a message pointing at this tracking line if a new mutator is added without the review flag - [x] Existing field extraction happy-path tests still green -- test: `go test ./internal/fieldextraction/...` - [x] Update `CreateFieldExtraction` handler in `api/queryAPI/fieldextractions.go` to branch on `ErrMutualExclusivityViolation` -> `echo.NewHTTPError(http.StatusConflict, ...)` and `ErrDocumentNotFound` -> 404 -- test: handler unit tests assert both status codes and body messages - [x] Handler also converts a PostgreSQL `check_violation` (SQLSTATE `23514`) surfaced from Trigger 2 into the same 409 response so a future bypass of the service-layer lock still returns the documented error shape -- test: TDD handler test with a fabricated `*pgconn.PgError` - [x] Preserve the existing mapping of non-sentinel service errors to `500` -- test: TDD handler test with a generic error path - [x] End-to-end HTTP test: create document, assign custom schema, call `POST /field-extractions`, assert response status is `409` and JSON body carries the documented message -- test: integration against testcontainer DB - [x] End-to-end HTTP race test: fire `POST /super-admin/documents/{id}/schema` and `POST /field-extractions` concurrently against the same document, assert the final DB state has exactly one of `custom_schema_id IS NOT NULL` XOR a `documentFieldExtractionVersions` row, never both -- test: integration ### 2.6 OpenAPI spec - metadata + document binding - [x] Add `CustomMetadataService` tag with description stating `client_user` + `user_admin` + `super_admin` access -- test: spec lints - [x] `CustomMetadataRequest` (no `createdBy`) -- test: codegen - [x] `CustomMetadataResponse` -- test: codegen - [x] `CustomMetadataHistoryResponse` -- test: codegen - [x] `ValidationErrorResponse` -- test: codegen - [x] `DocumentSchemaAssignRequest` (no `createdBy`) -- test: codegen - [x] `DocumentSchemaAssignResponse` -- test: codegen - [x] **v4**: Modify `DocumentEnriched` to add **only** `customSchemaId` and `hasCustomMetadata`. **Do NOT** add `customSchemaName` or `customMetadata`. **Do NOT** add a new `?customMetadata=true` query parameter. -- test: codegen produces exactly two new fields - [x] `PATCH /super-admin/documents/{id}/schema` path -- test: codegen - [x] `GET /custom-metadata` path -- test: codegen - [x] `POST /custom-metadata` path -- test: codegen - [x] `GET /custom-metadata/version` path -- test: codegen - [x] `GET /custom-metadata/history` path -- test: codegen - [x] `task generate` clean -- test: no errors ### 2.7 Controllers - metadata + assign schema + document enrichment - [x] Create `api/queryAPI/custommetadata.go` skeleton -- test: compiles - [x] **v4 actor extraction**: `POST /custom-metadata` derives actor identity from the JWT and passes it to the service as an argument -- test: TDD handler test asserts body `createdBy` (if provided) does not override - [x] `POST /custom-metadata` handler -- test: TDD - [x] `GET /custom-metadata` handler -- test: TDD - [x] `GET /custom-metadata/version` handler -- test: TDD - [x] `GET /custom-metadata/history` handler -- test: TDD - [x] Add `PATCH /super-admin/documents/{id}/schema` handler to `api/queryAPI/customschemas.go` -- derives actor -- test: TDD - [x] Extend `DocumentEnriched` builder to include `customSchemaId` and `hasCustomMetadata` unconditionally — reads the values directly from the `document.DocumentEnriched` struct (populated in §2.3a); **no** new DB query or service call at this layer. -- test: TDD - [x] Document without any custom schema returns `customSchemaId: null`, `hasCustomMetadata: false` -- test: TDD - [x] Document with schema bound but no metadata returns `customSchemaId: `, `hasCustomMetadata: false` -- test: TDD - [x] Document with schema + metadata returns populated fields -- test: TDD - [x] Authorization: confirm `GET /document/{id}` still uses the existing `document` resource, not `super-admin` -- test: role matrix test in 2.8 - [x] `go build ./...` clean -- test: compile ### 2.8 Milestone 2 integration tests - [x] End-to-end: create schema -> assign to document (PATCH) -> POST valid metadata -> `GET /custom-metadata?documentId=...` returns the stored row with populated `schemaId`, `schemaName`, `schemaVersion` (from the §2.3 joined query) and the `createdBy` field equal to the JWT subject UUID -- test: integration - [x] End-to-end: POST invalid metadata -> 400 with validation errors -- test: integration - [x] End-to-end: POST metadata to document without assigned schema -> 400 -- test: integration - [x] End-to-end: assign schema to document that has legacy extractions -> 409 -- test: integration - [x] End-to-end: write legacy field extraction to document that has custom schema -> 409 -- test: integration - [x] End-to-end `GET /custom-metadata/version`: POST metadata three times on the same document (version 1, 2, 3), then `GET /custom-metadata/version?documentId=...&version=2` returns exactly the v2 payload (not v1, not v3) with populated `schemaId`/`schemaName`/`schemaVersion` from the §2.3 joined query -- test: integration - [x] End-to-end `GET /custom-metadata/version` not-found: `version=99` on a document with only 3 versions returns 404; `documentId` for a non-existent document returns 404; `version < 1` (e.g. `0`) returns 400 (plan Section 5.4 declares the parameter `>= 1`) -- test: integration per case - [x] End-to-end `GET /custom-metadata/history` happy path: after three POSTs, `GET /custom-metadata/history?documentId=...` returns all three version summaries in descending `version` order and each summary carries `version`, `createdAt`, `createdBy` (subject UUID) -- test: integration - [x] End-to-end `GET /custom-metadata/history` pagination: POST 5 metadata versions, call with `limit=2&offset=0` (returns v5, v4), `limit=2&offset=2` (returns v3, v2), `limit=2&offset=4` (returns v1) — assert no duplicates and correct ordering across the page boundary -- test: integration - [x] `GET /custom-metadata/history` edge cases (v4 — behavior committed in plan Section 5.4 edge-case table, not up to the implementer): one integration test per case, each asserting the exact status code and response body declared in the plan — - `limit=0` → 400 with message `"limit must be between 1 and 200"` - `limit=-1` → 400 with the same message - `limit=201` → 200, clamped to 200 rows (seed 205 versions, assert exactly 200 returned, newest first) - `offset=-1` → 400 with message `"offset must be >= 0"` - `documentId` = random non-existent UUID → 404 with message `"document not found"` - `documentId` exists but has zero metadata rows → 200 with `{"versions": []}` (and `Content-Type: application/json`) -- test: integration per case - [x] Authorization matrix for Milestone 2 endpoints: one integration test per `(role, route, method)` cell across **all four** `/custom-metadata` routes (`GET`, `POST`, `GET /version`, `GET /history`) plus `PATCH /super-admin/documents/{id}/schema`. Expected matrix: `super_admin` positive on every cell; `user_admin` 403 on `PATCH /super-admin/...` and positive on all four `/custom-metadata` routes; `client_user` 403 on `PATCH /super-admin/...` and positive on all four `/custom-metadata` routes; `auditor` 403 on `PATCH /super-admin/...` and on `POST /custom-metadata`, positive on the three `/custom-metadata` GET routes (`GET`, `GET /version`, `GET /history`) -- test: one integration test per (role, route, method) cell, each asserting status + DB unchanged on denial - [x] **v4 FK-cascade delete test**: create document with custom metadata -> call existing `DeleteDocumentCascade` -> verify `document_custom_metadata` rows for that document are gone (via the ON DELETE CASCADE, without touching the delete code path) -- test: integration - [x] **v4 FK-cascade client delete test**: create client with schemas and a document with custom metadata -> call `client.HardDelete` -> verify `document_custom_metadata` and `client_metadata_schemas` rows are gone -- test: integration - [x] Metadata payload at 1MB boundary: accept 1048576, reject 1048577 -- test: integration - [x] Concurrency: two concurrent first-writers against same document succeed -- test: integration (service test in 2.4 covers it; re-verify at HTTP level) - [x] `task fullsuite:ci` clean -- test: full run - [x] Journal entry: Milestone 2 exit snapshot ### 2.9 Pre-partitioning carve-out for M3/M4 parallel execution > **Purpose**: Milestones 3 and 4 both add handler code and service-layer types that were originally drafted into `api/queryAPI/customschemas.go` and `internal/customschema/models.go`. If both milestones run in parallel on separate branches against those two shared files, the merge at the end produces conflicts at the seam. This section creates four empty stub files so each milestone owns disjoint files from the start. **Do this once, at the end of Milestone 2, before branching M3 and M4 onto separate workstreams.** If M3 and M4 will be run serially, this section can be skipped (the handlers can land in `customschemas.go` directly); only commit the stubs if parallel execution is actually planned. - [x] Create `api/queryAPI/customschemas_reset.go` as an empty stub containing only `package queryAPI` -- test: `go build ./...` clean - [x] Create `api/queryAPI/customschemas_bulk.go` as an empty stub containing only `package queryAPI` -- test: `go build ./...` clean - [x] Create `internal/customschema/models_reset.go` as an empty stub containing only `package customschema` -- test: `go build ./...` clean - [x] Create `internal/customschema/models_bulk.go` as an empty stub containing only `package customschema` -- test: `go build ./...` clean - [x] Commit the four stubs on the shared base branch (the branch M3 and M4 will both fork from), so neither workstream has to create them -- test: `git log` shows the four files as committed before either milestone branch exists - [ ] Update Milestone 3 owner and Milestone 4 owner: M3 only edits `customschemas_reset.go` and `models_reset.go`; M4 only edits `customschemas_bulk.go` and `models_bulk.go`; neither touches `customschemas.go` or `models.go` for new types, only for imports and registration wiring if required -- test: code review of each milestone's final diff --- ## Milestone 3 - Administrative completion **Goal**: Reset-metadata is exposed so super_admin can upgrade a document to a newer schema version. Schema version-creation concurrency is tested end-to-end. The full authorization matrix (including reset-metadata and version-creation) is locked in. **Dependencies**: Milestone 2 complete. **Exit criteria**: - Full upgrade flow works: assign v1 -> write metadata -> create v2 -> reset -> assign v2 -> write new metadata. - Reset is atomic, idempotent, rejects legacy documents, returns 404 on missing document. - `task fullsuite:ci` green. ### 3.1 SQLC queries for reset-metadata - [x] `LockDocumentForReset` (`SELECT id, "clientId", custom_schema_id FROM documents WHERE id = @document_id FOR UPDATE`) -- test: generated - [x] `GetDocumentSchemaBindingForReset` (join `documents` -> `client_metadata_schemas`, returns `(custom_schema_id, schema_name, schema_version)` or all-null) -- test: generated - [x] `CountDocumentCustomMetadataVersions` -- test: generated - [x] **v4**: `DeleteDocumentCustomMetadataForReset` (`DELETE FROM document_custom_metadata WHERE document_id = @document_id`) -- owned by the reset path, name disambiguates from any future cascade query -- test: generated - [x] **v4**: `NullifyDocumentCustomSchemaIdForReset` (`UPDATE documents SET custom_schema_id = NULL WHERE id = @document_id`) -- owned by the reset path -- test: generated - [x] `task generate` clean -- test: idempotent ### 3.2 Service method - ResetDocumentMetadata - [x] Add `ResetMetadataResult` type to `internal/customschema/models_reset.go` (the M3-owned stub from §2.9; fall back to `internal/customschema/models.go` only if M3/M4 will run serially and §2.9 was skipped) with `DocumentID`, `PreviousSchemaID *uuid.UUID`, `PreviousSchemaName *string`, `PreviousSchemaVersion *int`, `MetadataVersionsDeleted int`, `ResetAt time.Time`, `ResetBy string` -- test: compiles - [x] `ResetDocumentMetadata(ctx, documentID, actor)` -- single transaction: `LockDocumentForReset`, `GetDocumentSchemaBindingForReset`, `CountDocumentCustomMetadataVersions`, `DeleteDocumentCustomMetadataForReset`, `NullifyDocumentCustomSchemaIdForReset`, audit log -- test: happy path wipe with schema bound + N metadata versions - [x] Reset returns populated `previous*` fields when document had a schema -- test: TDD - [x] Reset returns `metadataVersionsDeleted` matching actual deleted count -- test: TDD - [x] Reset is idempotent: already-clean document returns `metadataVersionsDeleted: 0`, `previousSchemaId: nil` -- test: TDD - [x] Reset rejects documents with legacy field extractions (409) -- test: TDD - [x] Reset returns 404 on non-existent document -- test: TDD - [x] Reset atomicity: inject failure between DELETE and UPDATE -- test: transaction rolls back, metadata rows still present - [x] Reset + reassign + write: after reset, Trigger 1 still fires on second schema change under new metadata -- test: TDD regression - [x] (v3's "Trigger 3 regression" test is removed in v4: the table has no `schema_id` column. Replace with a structural test that confirms `document_custom_metadata` has no `schema_id` column after migrations.) -- test: TDD schema-structure test - [x] Reset writes `metadata.reset` audit log entry containing actor, documentID, previousSchemaId, metadataVersionsDeleted -- test: TDD reads the audit sink - [x] Reset concurrency: two concurrent resets on same document serialize via `FOR UPDATE` -- test: TDD - [x] Reset concurrency: reset vs `AssignSchema` on same document serialize -- test: TDD - [x] **v4 Reset concurrency: reset vs `SetDocumentMetadata` on same document serialize.** This is the pair that matters most in practice: plan Section 7.1 / 8.2 say `LockDocumentForReset` and `LockDocumentForMetadataWrite` use the same parent-row `SELECT ... FOR UPDATE` shape specifically so these two operations cannot interleave. Test both orderings with goroutines against testcontainer DB: 1. **reset-then-write**: start reset (holds `FOR UPDATE`), race `SetDocumentMetadata` for the same document. The writer must block until reset commits; once it unblocks it must observe `custom_schema_id = NULL` and return the "no schema assigned" error (400/409), not a successful write into a stale schema. Assert final DB state has zero metadata rows for the document. 2. **write-then-reset**: start `SetDocumentMetadata` (holds `FOR UPDATE`, inserts a new metadata row), race reset for the same document. The reset must block until the write commits; once it unblocks it must delete the just-written row and return `metadataVersionsDeleted >= 1`. Assert final DB state has zero metadata rows and `custom_schema_id = NULL`. Neither ordering may leave an orphaned metadata row or a partially-reset document. -- test: TDD with goroutines against testcontainer DB; assertions cover both ordering outcomes ### 3.3 OpenAPI + handler - reset-metadata - [x] `DocumentMetadataResetResponse` schema (no request body schema needed; empty body) -- test: codegen (NOTE: named `DocumentMetadataResetResponse` not `ResetDocumentMetadataResponse` to avoid codegen name conflict; see journal 2026-04-15) - [x] `POST /super-admin/documents/{id}/reset-metadata` path -- test: codegen - [x] `task generate` clean -- test: no errors - [x] Add handler to `api/queryAPI/customschemas_reset.go` (the M3-owned stub from §2.9; fall back to `api/queryAPI/customschemas.go` only if M3/M4 will run serially and §2.9 was skipped) -- parses document ID, derives actor from JWT, calls service, maps errors -- test: TDD - [x] Handler maps success to 200 with `ResetDocumentMetadataResponse` -- test: `TestM3_ResetDocumentMetadata_HappyPath` + `TestM3_ResetDocumentMetadata_IdempotentOnCleanDoc` - [x] Handler maps "document not found" to 404 -- test: `TestM3_ResetDocumentMetadata_DocumentNotFound` + `TestMapResetError/document_not_found` - [x] Handler maps "legacy extractions exist" to 409 -- test: `TestM3_ResetDocumentMetadata_LegacyExtractionsConflict` + `TestMapResetError/mutual_exclusivity_violation` - [x] Handler maps unexpected DB errors to 500 -- test: `TestMapResetError/unknown_error_returns_500` ### 3.4 Milestone 3 integration tests - [x] End-to-end upgrade flow: bind schema v1 -> POST metadata -> `POST .../versions` to create v2 -> reset-metadata -> PATCH schema to v2 -> POST new metadata with added optional field -- test: `TestM3_FullUpgradeFlow` in `api/queryAPI/customschemas_reset_test.go` - [x] End-to-end: reset on already-clean document returns 200 with `metadataVersionsDeleted: 0` -- test: `TestM3_ResetDocumentMetadata_IdempotentOnCleanDoc` - [x] Schema version concurrency: two concurrent `POST .../versions` calls for same `(client_id, name)` produce distinct versions -- test: `TestM3_SchemaVersionConcurrency` in `api/queryAPI/customschemas_integration_test.go` - [x] Authorization matrix for reset-metadata and version creation: `super_admin` positive; `user_admin` 403; `client_user` 403; `auditor` 403 -- test: `TestM3_AuthMatrix_Reset` in `api/queryAPI/customschemas_reset_test.go` (4 subtests: super_admin/user_admin/client_user/auditor) - [x] `task fullsuite:ci` clean -- test: full run (85.4%, All coverage checks passed!, no FAIL lines, 2026-04-15) - [x] Journal entry: Milestone 3 exit snapshot (2026-04-15) --- ## Milestone 4 - Bulk operations **Goal**: Bulk folder schema assignment is available for administrative convenience. **Dependencies**: Milestone 2 complete. Milestone 4 only needs the single-document `AssignSchema` semantics (committed in Milestone 2), not the `ResetDocumentMetadata` work in Milestone 3 — nothing in the recursive folder walk reads or writes the reset path. Milestones 3 and 4 **can run in parallel on separate workstreams only after the shared files are pre-partitioned**: without that step, both milestones would edit `api/queryAPI/customschemas.go` and `internal/customschema/models.go` concurrently and produce merge conflicts at the seam. The pre-partition carve-out (see section 2.9 below) creates four new stub files so each milestone owns disjoint files: M3 owns `api/queryAPI/customschemas_reset.go` and `internal/customschema/models_reset.go`; M4 owns `api/queryAPI/customschemas_bulk.go` and `internal/customschema/models_bulk.go`. With the carve-out in place the merge gate is `task fullsuite:ci` green after both land, and serializing M4 behind M3 adds roughly one milestone of wall-clock time for no technical reason. Without the carve-out, serialize M4 behind M3 as the safe fallback. **Exit criteria**: - Recursive walk covers a 3-deep folder tree correctly, handles all skip reasons, enforces the 10K document cap, runs in a single transaction. - `task fullsuite:ci` green. ### 4.1 SQLC queries for bulk folder assignment - [ ] `BulkSetDocumentCustomSchemaIdInFolderTree` (`WITH RECURSIVE`) -- test: generated; recursive CTE compiles - [ ] `GetDocumentsWithMetadataInFolderTree` -- test: generated - [ ] `GetDocumentsWithLegacyExtractionsInFolderTree` -- test: generated - [ ] **Reuse existing query**: `CountDocumentsInFolderTree` already exists in `internal/database/queries/folders.sql` (a recursive CTE over `folders` + `documents.folderId`) and is exactly the shape the `MaxBulkAssignDocuments` pre-check needs. Do **not** add a new query; call the existing one from the Milestone 4 service method. Task: unit test that confirms the bulk service's cap check is wired to the existing `CountDocumentsInFolderTree` generated method -- test: unit test asserts the right method is called and returns a count that matches a seeded fixture - [ ] `task generate` clean -- test: idempotent ### 4.2 Service method - AssignSchemaToFolder - [ ] Add `BulkAssignResult`, `SkippedDocument` types to `internal/customschema/models_bulk.go` (the M4-owned stub from §2.9; fall back to `internal/customschema/models.go` only if M3/M4 will run serially and §2.9 was skipped) -- test: compiles. **No `createdBy` on the input.** - [ ] `AssignSchemaToFolder(ctx, folderID, schemaID, actor)` -- walks subtree with `WITH RECURSIVE` and returns `BulkAssignResult` -- test: happy path on a 3-deep tree - [ ] Bulk assignment skips documents with different schema + existing metadata with `SkippedDocument` reason -- test: TDD - [ ] Bulk assignment skips documents with legacy extractions with reason -- test: TDD - [ ] Bulk assignment no-ops documents already on target schema with reason -- test: TDD - [ ] Bulk assignment reassigns documents with different schema but no metadata -- test: TDD - [ ] Bulk assignment returns 422 when subtree exceeds `MaxBulkAssignDocuments` -- test: TDD (constant override for test) - [ ] Bulk assignment runs in single transaction; simulated mid-walk failure rolls back everything -- test: TDD - [ ] Bulk assignment writes `folder.assign_schema` audit entry with counts -- test: TDD ### 4.3 OpenAPI + handler - bulk folder assignment - [ ] `BulkSchemaAssignRequest` schema (no `createdBy`) -- test: codegen - [ ] `BulkSchemaAssignResponse` schema -- test: codegen - [ ] `POST /super-admin/folders/{folderId}/assign-schema` path -- test: codegen - [ ] `task generate` clean -- test: no errors - [ ] Handler in `api/queryAPI/customschemas_bulk.go` (the M4-owned stub from §2.9; fall back to `api/queryAPI/customschemas.go` only if M3/M4 will run serially and §2.9 was skipped) wired to `AssignSchemaToFolder` -- derives actor from JWT -- test: compiles ### 4.4 Milestone 4 integration tests - [ ] Recursive walk on a 3-deep folder tree produces correct `documentsUpdated` count -- test: integration - [ ] Skip reason: "different schema + existing metadata" -- test: integration - [ ] Skip reason: "already assigned to same schema (no-op)" -- test: integration - [ ] Skip reason: "has legacy field extractions" -- test: integration - [ ] Reassignment on document with different schema but no metadata -- test: integration - [ ] Subtree > 10K documents returns 422 with documented message -- test: integration (constant override for test) - [ ] Bulk folder exactly 10000 docs: succeeds -- test: integration - [ ] Empty folder returns 200 with 0 counts -- test: integration - [ ] Deeply nested (10+ levels) tree walks correctly -- test: integration - [ ] Authorization matrix for bulk folder endpoint: `super_admin` positive; `user_admin` 403; `client_user` 403; `auditor` 403 -- test: integration per cell - [ ] `task fullsuite:ci` clean -- test: full run - [ ] Journal entry: Milestone 4 exit snapshot --- ## Documentation (parallel with Milestones 3 + 4) **Goal**: Every doc that references endpoints, env vars, authorization, or the feature is updated. **Dependencies**: Milestone 2 complete (feature is shippable in narrow form); documentation can track Milestones 3 and 4 in real time. - [x] Update `docs/ai.generated/README.md` index if new doc files added -- test: review (entry 11 added pointing to `11-custom-metadata-guide.md`) - [x] Authorization matrix table published in a dedicated doc page (include the v4 change that `user_admin` has no schema CRUD) -- test: review (`11-custom-metadata-guide.md` Authorization Matrix section) - [x] Sample schemas from Appendix A of the plan published as copyable examples -- test: review (`11-custom-metadata-guide.md` Schema Definition Requirements section, two examples) - [x] `CustomMetadataService` usage guide (end-user perspective) -- test: review (`11-custom-metadata-guide.md` End-User Workflow section + `03-api-documentation.md` CustomMetadataService section) - [x] `SuperAdminSchemaService` usage guide (platform-operator perspective), including the `POST /versions` route -- test: review (`11-custom-metadata-guide.md` Platform Operator Workflow + `03-api-documentation.md` SuperAdminSchemaService section) - [x] Reset-metadata runbook: "How to upgrade a document to a newer schema version" with the full wipe -> reassign -> re-POST flow and a warning about metadata history loss -- test: review (`11-custom-metadata-guide.md` Schema Upgrade Runbook section) - [x] Mutual exclusivity explanation (legacy vs custom) in the document processing guide -- test: review (`11-custom-metadata-guide.md` Mutual Exclusivity section) - [x] **v4**: Note that `DeleteDocumentCascade` and `client.HardDelete` do NOT need custom-metadata-specific steps because of the FK cascades -- test: review (`11-custom-metadata-guide.md` Cascade Deletion Behavior section) - [x] Permit.io deployment ordering note (run setup tool before deploying controllers) added to the deployment runbook -- test: review (`11-custom-metadata-guide.md` Permit.io Deployment Note section) - [x] **v4**: Document the HTTP method choice: `POST /super-admin/custom-schemas/{schemaId}/versions`, not `PUT` -- test: review (`03-api-documentation.md` and `11-custom-metadata-guide.md` Schema Versioning section) - [x] **v4**: Document that `createdBy` is not accepted from request bodies; actor comes from the JWT -- test: review (`03-api-documentation.md` Notes on POST /super-admin/custom-schemas; `11-custom-metadata-guide.md` Actor Fields section) - [x] Any new env vars? (none expected for v4; confirm.) -- test: review (confirmed: no new env vars in v4) - [ ] Generate new API status report / architecture diagrams if the pipeline regenerates them -- test: review - [ ] Journal entry marking the feature shipped --- ## Cross-milestone reminders - Before starting any milestone, re-read the relevant section of `plans/mutable.metadata.plan.combo.v4.md`. - TDD is mandatory per project rules: write the test, watch it fail, make it pass. Never write multiple tests before running any. - No mocks without Q's explicit permission. Use testcontainers for real Postgres. - Keep the journal current. When a checkbox flips to `[x]`, add a timestamped line to `./journals/implement_mutableMetadata.md` with the commit SHA and test command that proved it. - If any migration, trigger, FK, or service invariant seems to conflict with the plan, STOP and ask Q before modifying it. The defense-in-depth design depends on the composite FK and Trigger 1 being exactly as written. - **v4 reminder — things NOT to add (they are v3 leftovers and mean you are regressing):** - Do NOT add a `schema_id` column to `document_custom_metadata`. - Do NOT add v3's `trg_validate_schema_client_match` trigger — it is replaced by the composite FK `fk_documents_custom_schema_same_client` (plan Section 4.7, Milestone 1.3). - Do NOT add v3's `trg_enforce_consistent_schema_id` trigger — it is structurally unnecessary because `document_custom_metadata` has no `schema_id` column in v4. - **Reminder**: v4 DOES add two triggers — `trg_prevent_schema_reassignment` (v3's Trigger 1, renamed for clarity) and `trg_prevent_legacy_extraction_on_custom_document` (new in v4, see Section 2.2). Those are **correct** and must be present. - Do NOT add a `current_document_custom_metadata` view. - Do NOT add `DeleteDocumentCustomMetadata` / `NullifyDocumentCustomSchemaId` / `DeleteClientMetadataSchemas` to the existing delete code paths. - Do NOT add a `PUT /super-admin/custom-schemas/{schemaId}` route. - Do NOT accept `createdBy` in new request bodies. - Do NOT add `customSchemaName` or `customMetadata` to `GET /document/{id}`. - Do NOT declare `format: email` on `createdBy` / `resetBy` in the new OpenAPI response schemas — those are plain-string Cognito subject IDs (plan Section 8.3 "Actor fields"). - If any of those appear in your diff, something has regressed to v3.