query-orchestration/plans/mutable.metadata.plan.combo.v3.tracking.md

# Mutable Metadata v3 - Implementation Tracking

**Source plan**: `plans/mutable.metadata.plan.combo.v3.md`
**Journal (details / decisions / blockers)**: `./journals/implement_mutableMetadata.md`
**Purpose**: One-glance status of every task needed to ship v3. Each checkbox is a discrete, verifiable unit of work. TDD rule: write the test first, watch it fail, make it pass, check the box. If a task requires Q's input or produces a decision, log it to the journal and keep the checkbox state accurate here.

## How to use this document

- `[ ]` = not started
- `[~]` = in progress
- `[x]` = done AND tests passing (`task fullsuite:ci` clean for the affected code paths)
- `[!]` = blocked -- add a short `(blocked: <why>)` suffix and open a journal entry
- `[-]` = consciously skipped -- add a short `(skipped: <why>)` suffix
- Every task line ends with a `test:` marker indicating where the passing test(s) live or what test type proves it.
- When a whole phase is green, flip the phase header to `COMPLETE` and date-stamp it.
- Never mark a phase complete until `task fullsuite:ci` ends with `All coverage checks passed!` with no `FAIL` lines.

---

## Phase roll-up (high-level)

| Phase | Title | Status | Started | Completed |
|-------|-------|--------|---------|-----------|
| 1     | Database migrations | NOT STARTED |  |  |
| 2     | SQLC queries + generate | NOT STARTED |  |  |
| 3     | Schema validator component | NOT STARTED |  |  |
| 4     | Service layer (schema CRUD, metadata, assign, bulk, reset) | NOT STARTED |  |  |
| 5a    | Legacy field extraction guard | NOT STARTED |  |  |
| 5b    | Delete cascade updates | NOT STARTED |  |  |
| 6     | OpenAPI spec updates | NOT STARTED |  |  |
| 7a    | Permit.io policy file updates | NOT STARTED |  |  |
| 7b    | Run permit setup tool (dev/uat/prod) | NOT STARTED |  |  |
| 8     | API controllers + codegen | NOT STARTED |  |  |
| 9     | Document endpoint modifications | NOT STARTED |  |  |
| 10    | Bulk folder schema assignment endpoint | NOT STARTED |  |  |
| 10b   | Reset-metadata endpoint (v3) | NOT STARTED |  |  |
| 11    | Integration & authorization tests | NOT STARTED |  |  |
| 12    | Documentation updates | NOT STARTED |  |  |

---

## Phase 1 - Database migrations

**Goal**: Land all DDL needed for the feature. No Go code touched in this phase.
**Dependencies**: None.
**Exit criteria**: Migrations up/down cleanly on a fresh database AND on a database with existing data; `task db:generate` runs clean; triggers verifiable via direct SQL.

> **Migration numbering note**: The repo already contains migrations through `00000000000126_batch_document_outcomes_delete_actions`. v3 therefore starts at `127` to avoid filename collisions. If another branch lands new migrations before this feature merges, renumber this block to the next free sequence and update the references below.

### 1.1 Migration 127 - `schema_status_type` enum + `client_metadata_schemas`

- [ ] Create `internal/database/migrations/00000000000127_create_client_metadata_schemas.up.sql` with `CREATE TYPE schema_status_type AS ENUM ('active', 'superseded', 'retired')` -- test: migration applies cleanly on empty DB
- [ ] Same migration creates `client_metadata_schemas` table per Section 4.2 (columns, PK, FK to `clients`, `schema_def jsonb`, `version`, `status`, timestamps, `created_by`) -- test: `\d client_metadata_schemas` shows all columns
- [ ] Add `CONSTRAINT uq_client_schema_name_version UNIQUE (client_id, name, version)` -- test: direct SQL INSERT of two rows with same `(client_id, name, version)` fails
- [ ] Add `CONSTRAINT chk_schema_def_size CHECK (pg_column_size(schema_def) <= 70000)` -- test: insert of a schema_def larger than 70000 bytes fails with CHECK violation
- [ ] Add indexes `idx_cms_client_id`, `idx_cms_client_name`, `idx_cms_client_status` -- test: `\di client_metadata_schemas` shows the three indexes
- [ ] Create `00000000000127_create_client_metadata_schemas.down.sql` dropping the table and enum in correct order -- test: up then down leaves DB state byte-identical to pre-migration
- [ ] Run `task db:generate` after migration lands -- test: no errors

### 1.2 Migration 128 - `document_custom_metadata` table + view

- [ ] Create `00000000000128_create_document_custom_metadata.up.sql` with `document_custom_metadata` table per Section 4.3 (columns, FKs, `metadata jsonb`, version column) -- test: migration applies cleanly
- [ ] Add `CONSTRAINT uq_doc_custom_metadata_version UNIQUE (document_id, version)` -- test: direct SQL duplicate `(document_id, version)` INSERT fails
- [ ] Add indexes `idx_dcm_document_id`, `idx_dcm_schema_id`, `idx_dcm_doc_version_desc` -- test: indexes present
- [ ] Create `current_document_custom_metadata` view per Section 4.5 (`DISTINCT ON (document_id) ... ORDER BY document_id, version DESC`) -- test: insert 3 versions for same doc, `SELECT * FROM current_document_custom_metadata WHERE document_id = X` returns only version 3
- [ ] Create corresponding `down.sql` (drop view then table) -- test: up/down round trip clean

### 1.3 Migration 129 - Add `custom_schema_id` to `documents`

- [ ] Create `00000000000129_add_custom_schema_id_to_documents.up.sql` -- `ALTER TABLE documents ADD COLUMN custom_schema_id uuid NULL REFERENCES client_metadata_schemas(id)` -- test: column present, nullable, FK enforced
- [ ] Create `idx_documents_custom_schema_id` -- test: index present
- [ ] Write `down.sql` that drops index then column -- test: round trip clean
- [ ] Verify existing documents rows have `custom_schema_id = NULL` after migration -- test: `SELECT COUNT(*) FROM documents WHERE custom_schema_id IS NOT NULL` returns 0

### 1.4 Migration 130 - Invariant triggers

- [ ] Create `00000000000130_add_schema_invariant_triggers.up.sql` -- test: migration applies cleanly
- [ ] Define `trg_prevent_schema_reassignment()` function per Section 4.7 Trigger 1 -- test: function exists in `pg_proc`
- [ ] Attach trigger to `documents` BEFORE UPDATE OF `custom_schema_id` -- test: trigger listed in `\d documents`
- [ ] Trigger 1 unit test: insert document, assign schema, write custom metadata, attempt UPDATE `custom_schema_id` -- expect `RAISE EXCEPTION` with "custom metadata already exists" message
- [ ] Trigger 1 unit test: insert document, add legacy extraction row, attempt UPDATE `custom_schema_id` -- expect `RAISE EXCEPTION` with "legacy field extractions already exist" message
- [ ] Trigger 1 unit test: insert document with no metadata and no extractions, UPDATE `custom_schema_id` from schema_a to schema_b -- expect success (pre-binding change allowed)
- [ ] Define `trg_validate_schema_client_match()` function per Section 4.7 Trigger 2 -- test: function exists
- [ ] Attach trigger to `documents` BEFORE INSERT OR UPDATE OF `custom_schema_id` -- test: trigger listed
- [ ] Trigger 2 unit test: attempt to assign a schema owned by client A to a document owned by client B -- expect exception
- [ ] Trigger 2 unit test: attempt to assign a non-existent schema ID -- expect "Schema % does not exist"
- [ ] Trigger 2 unit test: same-client assignment -- expect success
- [ ] Define `trg_enforce_consistent_schema_id()` function per Section 4.7 Trigger 3 -- test: function exists
- [ ] Attach trigger to `document_custom_metadata` BEFORE INSERT -- test: trigger listed
- [ ] Trigger 3 unit test: insert metadata v1 under schema_a, attempt direct SQL INSERT of metadata v2 under schema_b for same document -- expect exception
- [ ] Trigger 3 unit test: insert metadata v1 and v2 under same schema -- expect success
- [ ] Write `down.sql` dropping triggers and functions in correct order -- test: up/down round trip clean

### 1.5 Phase 1 integration

- [ ] Run `task generate` -- test: clean
- [ ] Run `task test:unit:short` -- test: all existing tests still pass (no regression)
- [ ] Commit Phase 1 on a dedicated branch or commit cluster; journal entry recording migration numbers used

---

## Phase 2 - SQLC queries + code generation

**Goal**: All DB access the feature needs is expressed as SQLC-generated Go.
**Dependencies**: Phase 1.
**Exit criteria**: `task generate` clean; generated `*.sql.go` files compile; every query listed below exists and has a typed wrapper.

### 2.1 New file `internal/database/queries/customschemas.sql`

**Schema CRUD queries:**

- [ ] `CreateClientMetadataSchema` -- INSERT returning full row -- test: generated code compiles, query name appears in `*.sql.go`
- [ ] `GetClientMetadataSchema` (by id) -- test: generated
- [ ] `ListClientMetadataSchemas` (filters: client_id required, name optional, status optional with default `active`, pagination limit/offset) -- test: generated
- [ ] `GetLatestSchemaByName` -- test: generated
- [ ] `GetSchemaDocumentCount` (count of `documents.custom_schema_id = $1`) -- test: generated
- [ ] `GetSchemaMetadataRecordCount` (count of `document_custom_metadata.schema_id = $1`) -- test: generated
- [ ] `SetSchemaStatus` (update status, used for supersede and retire) -- test: generated
- [ ] `GetMaxSchemaVersion` (for auto-increment on PUT) -- test: generated
- [ ] `LockSchemaVersionsForName` (`SELECT ... FOR UPDATE` on all versions of a `(client_id, name)` pair) -- test: generated

**Metadata CRUD queries:**

- [ ] `CreateDocumentCustomMetadata` -- test: generated
- [ ] `GetCurrentDocumentCustomMetadata` (via view) -- test: generated
- [ ] `GetDocumentCustomMetadataByVersion` -- test: generated
- [ ] `GetDocumentCustomMetadataHistory` (with limit/offset) -- test: generated
- [ ] `LockDocumentCustomMetadataForVersion` (`SELECT ... FOR UPDATE` on max version) -- test: generated

**Document-binding queries:**

- [ ] `SetDocumentCustomSchemaId` -- test: generated
- [ ] `GetDocumentCustomSchemaId` -- test: generated
- [ ] `BulkSetDocumentCustomSchemaIdInFolderTree` (`WITH RECURSIVE`) -- test: generated; recursive CTE compiles
- [ ] `GetDocumentsWithMetadataInFolderTree` -- test: generated
- [ ] `GetDocumentsWithLegacyExtractionsInFolderTree` -- test: generated
- [ ] `DocumentHasLegacyExtractions` (boolean, single doc) -- test: generated
- [ ] `CountDocumentsInFolderTree` (for `MaxBulkAssignDocuments` check) -- test: generated

**Reset-metadata support queries (v3):**

- [ ] `LockDocumentForReset` (`SELECT id, "clientId", custom_schema_id FROM documents WHERE id = $1 FOR UPDATE`) -- test: generated
- [ ] `GetDocumentSchemaBindingForReset` (join `documents` -> `client_metadata_schemas`, returns `(custom_schema_id, schema_name, schema_version)` or all-null) -- test: generated
- [ ] `CountDocumentCustomMetadataVersions` -- test: generated

### 2.2 Query additions for delete cascade (Phase 5b prep)

- [ ] Add `DeleteDocumentCustomMetadata` to `document.sql` -- test: generated
- [ ] Add `NullifyDocumentCustomSchemaId` to `document.sql` -- test: generated
- [ ] Add `DeleteClientMetadataSchemas` to `client.sql` -- test: generated

### 2.3 Phase 2 integration

- [ ] `task generate` clean -- test: no diff after re-run (idempotent)
- [ ] Generated files compile: `go build ./...` -- test: clean
- [ ] Commit Phase 2 with journal entry

---

## Phase 3 - Schema validator component

**Goal**: A standalone `internal/customschema/validator.go` that can meta-validate JSON Schema definitions and validate metadata payloads against a compiled schema.
**Dependencies**: JSON Schema library added to `go.mod`.
**Exit criteria**: Validator unit tests cover all supported types in Section 6.3 and the explicit-`additionalProperties` rule.

### 3.1 Dependency

- [ ] Add `github.com/santhosh-tekuri/jsonschema/v6` to `go.mod` -- test: `go mod tidy` clean, build clean

### 3.2 Constants file

- [ ] Create `internal/customschema/constants.go` with `MaxSchemaDefinitionBytes = 65536`, `MaxMetadataPayloadBytes = 1048576`, `MaxBulkAssignDocuments = 10000` -- test: constants referenced by tests

### 3.3 Validator implementation

- [ ] Create `internal/customschema/validator.go` with `SchemaValidator` struct -- test: compiles
- [ ] Implement `ValidateSchemaDefinition(schemaDef json.RawMessage) error` -- test: valid JSON Schema draft 2020-12 accepted
- [ ] Reject schema definitions that are not valid JSON -- test: TDD test with malformed JSON
- [ ] Reject schema definitions that fail JSON Schema meta-validation -- test: TDD test with `{"type": "nonsense"}`
- [ ] Reject schema definitions where root `type` is not `"object"` -- test: TDD test with `{"type": "array"}`
- [ ] Reject schema definitions where root `additionalProperties` is not explicitly present -- test: TDD test with schema missing the key
- [ ] Accept schema definitions where root `additionalProperties` is explicitly `true` -- test: TDD test
- [ ] Accept schema definitions where root `additionalProperties` is explicitly `false` -- test: TDD test
- [ ] Reject schema definitions larger than `MaxSchemaDefinitionBytes` -- test: TDD test with 64KB+1 payload
- [ ] Implement `ValidateMetadata(schemaDef, metadata json.RawMessage) ([]ValidationError, error)` -- test: compiles
- [ ] Conforming metadata returns empty error slice -- test: Appendix A.1 sample schema + conforming payload
- [ ] Missing required field produces `ValidationError` with field pointer and message -- test: TDD
- [ ] `maxLength` violation produces structured error -- test: TDD
- [ ] `minimum`/`maximum` violation produces structured error -- test: TDD
- [ ] `pattern` violation produces structured error -- test: TDD
- [ ] `enum` violation produces structured error -- test: TDD
- [ ] `format: date` / `date-time` / `email` / `uri` / `uuid` enforced -- test: one TDD test per format
- [ ] `additionalProperties: false` rejects extra fields -- test: TDD
- [ ] Nested object validation works (e.g., `contact_info` from A.1) -- test: TDD
- [ ] Array `items` / `minItems` / `maxItems` enforced -- test: TDD
- [ ] Metadata larger than `MaxMetadataPayloadBytes` rejected -- test: TDD with 1MB+1 payload

### 3.4 Phase 3 integration

- [ ] `go test ./internal/customschema/...` green -- test: full package green
- [ ] `task test:race` green for the package -- test: no races
- [ ] Journal entry

---

## Phase 4 - Service layer

**Goal**: `internal/customschema/service.go` exposes every operation the HTTP layer will call. Role-agnostic. Real DB in tests (testcontainers, no mocks per project rules).
**Dependencies**: Phase 2 (SQLC), Phase 3 (validator).
**Exit criteria**: Service integration tests cover every method and every documented error path.

### 4.1 Skeleton and models

- [ ] Create `internal/customschema/service.go` with `Service` struct holding `cfg serviceconfig.ConfigProvider` and `validator *SchemaValidator` -- test: compiles
- [ ] Create `internal/customschema/models.go` with `Schema`, `SchemaSummary`, `CreateSchemaInput`, `UpdateSchemaInput`, `ListFilters`, `CustomMetadata`, `MetadataVersion`, `SetMetadataInput`, `BulkAssignResult`, `SkippedDocument`, `ResetMetadataResult` -- test: compiles
- [ ] Every `*Input` struct that historically carried `CreatedBy string` is renamed or documented so the service expects the actor to arrive as a **separate function argument** (e.g. `Service.CreateSchema(ctx, input, actor string)`). The service must never read `CreatedBy` off a request body model. -- test: compiles + unit test that asserts passing an input with a non-empty `CreatedBy` is ignored in favor of the `actor` argument (or, cleaner, the struct field is removed entirely)

### 4.1a Audit sink definition (new, required before 4.2 and 4.5 can log)

- [ ] Decide sink location: either `internal/customschema/audit.go` (feature-local) or `internal/audit/` (shared) -- test: decision recorded in journal
- [ ] Define `AuditRecord` struct: `Actor string`, `Action string` (enum: `schema.create|schema.update|schema.delete|schema.assign|metadata.write|metadata.reset`), `ResourceID uuid.UUID`, `ClientID uuid.UUID`, `Timestamp time.Time`, `Details map[string]any` -- test: compiles
- [ ] Define `AuditSink` interface with `Record(ctx, AuditRecord) error` -- test: compiles
- [ ] Provide a default implementation (structured `slog` with `audit=true` attribute, or a DB-backed appender if Q decides) -- test: unit test verifies a recorded event lands in the sink
- [ ] Wire the sink into `customschema.Service` via `serviceconfig.ConfigProvider` or a constructor arg -- test: compiles

### 4.2 Schema CRUD methods

- [ ] `CreateSchema(ctx, input)` -- meta-validates, enforces 64KB, enforces explicit `additionalProperties`, enforces `name` unique within client for active schemas at version 1 -- test: happy path
- [ ] `CreateSchema` rejects invalid JSON Schema -- test: TDD
- [ ] `CreateSchema` rejects oversize definition -- test: TDD
- [ ] `CreateSchema` rejects duplicate name for same client -- test: TDD
- [ ] `UpdateSchema(ctx, schemaID, input)` -- uses `LockSchemaVersionsForName` -> `GetMaxSchemaVersion` -> INSERT new version -> SetSchemaStatus(old, 'superseded') inside a single tx -- test: happy path produces v2
- [ ] `UpdateSchema` concurrency: two concurrent calls against same `(client_id, name)` produce distinct version numbers -- test: TDD with goroutines against testcontainer DB
- [ ] `UpdateSchema` rejects when old version does not exist -- test: TDD
- [ ] `UpdateSchema` rejects invalid schema definition (same validator rules as create) -- test: TDD
- [ ] `GetSchema(ctx, schemaID)` -- test: returns full row including `schema_def`
- [ ] `GetSchema` 404 when not found -- test: TDD
- [ ] `ListSchemas(ctx, clientID, filters)` -- default returns only `active` -- test: TDD
- [ ] `ListSchemas` filter by `name` -- test: TDD
- [ ] `ListSchemas` filter by `status` -- test: TDD
- [ ] `ListSchemas` `includeAllVersions=true` returns all versions -- test: TDD
- [ ] `ListSchemas` pagination (`limit`, `offset`) -- test: TDD
- [ ] `ListSchemas` populates `documentCount` and `canDelete` fields -- test: TDD
- [ ] `DeleteSchema(ctx, schemaID)` -- sets status to `retired` -- test: happy path
- [ ] `DeleteSchema` returns 409 if any document references the schema via `custom_schema_id` -- test: TDD
- [ ] `DeleteSchema` returns 409 if any `document_custom_metadata.schema_id` row references it -- test: TDD

### 4.3 Metadata methods

- [ ] `SetDocumentMetadata(ctx, input)` -- derives `schemaId` from `documents.custom_schema_id`, never trusts client -- test: happy path
- [ ] `SetDocumentMetadata` rejects when document has no `custom_schema_id` -- test: TDD (400)
- [ ] `SetDocumentMetadata` rejects when document has legacy extractions -- test: TDD (409)
- [ ] `SetDocumentMetadata` validates metadata against schema via `validator.ValidateMetadata` -- test: TDD with conforming and non-conforming payloads
- [ ] `SetDocumentMetadata` enforces `MaxMetadataPayloadBytes` -- test: TDD
- [ ] `SetDocumentMetadata` serializes version assignment by first taking `SELECT id FROM documents WHERE id = $1 FOR UPDATE` on the parent row, then reading `GetMaxDocumentMetadataVersion`, then INSERTing `version = max+1` -- test: TDD confirms the parent-row lock is acquired before the version read
- [ ] `SetDocumentMetadata` concurrency: two concurrent writers against the **same** document that already has >=1 metadata row -- both succeed, produce distinct versions, no unique-violation -- test: TDD with goroutines against testcontainer DB
- [ ] `SetDocumentMetadata` concurrency: two concurrent **first** writers against a document with zero metadata rows -- both succeed, produce `version=1` and `version=2` in some order, no unique-violation surfaces to the caller -- test: TDD with goroutines, proves the parent-row lock (not just `FOR UPDATE` on the non-existent max-version row) is what serializes the first write
- [ ] `SetDocumentMetadata` retries once on `UNIQUE (document_id, version)` violation as a belt-and-suspenders fallback (should be unreachable once the parent lock is in place) -- test: TDD with an injected unique-violation on the first INSERT attempt
- [ ] `SetDocumentMetadata` subsequent writes must use same schema (consistency with Trigger 3) -- test: TDD
- [ ] `GetCurrentMetadata(ctx, documentID)` -- test: uses view, returns latest
- [ ] `GetMetadataByVersion(ctx, documentID, version)` -- test: TDD
- [ ] `GetMetadataHistory(ctx, documentID)` with pagination -- test: TDD

### 4.4 Document schema binding methods

- [ ] `AssignSchema(ctx, documentID, schemaID)` -- test: happy path single-document assignment
- [ ] `AssignSchema` rejects if schema and document belong to different clients (app-layer + trigger 2) -- test: TDD
- [ ] `AssignSchema` rejects if schema status is not `active` -- test: TDD
- [ ] `AssignSchema` rejects if document already has custom metadata (matches Trigger 1 behavior; expect 409) -- test: TDD
- [ ] `AssignSchema` rejects if document has legacy extractions -- test: TDD
- [ ] `AssignSchema` allows setting `customSchemaId=nil` to clear binding when no metadata exists -- test: TDD
- [ ] `AssignSchemaToFolder(ctx, folderID, schemaID, createdBy)` -- test: walks subtree with `WITH RECURSIVE` and returns `BulkAssignResult`
- [ ] Bulk assignment skips documents with different schema + existing metadata with `SkippedDocument` reason -- test: TDD
- [ ] Bulk assignment skips documents with legacy extractions with reason -- test: TDD
- [ ] Bulk assignment no-ops documents already on target schema with reason -- test: TDD
- [ ] Bulk assignment reassigns documents with different schema but no metadata -- test: TDD
- [ ] Bulk assignment returns 422 when subtree exceeds `MaxBulkAssignDocuments` -- test: TDD with 10001 docs (or a smaller constant override for tests)
- [ ] Bulk assignment runs in single transaction -- test: verify partial-failure rolls back everything

### 4.5 Reset-metadata method (v3)

- [ ] `ResetDocumentMetadata(ctx, documentID, resetBy)` -- single transaction, `LockDocumentForReset`, `GetDocumentSchemaBindingForReset`, `CountDocumentCustomMetadataVersions`, `DeleteDocumentCustomMetadata`, `NullifyDocumentCustomSchemaId`, audit log -- test: happy path wipe with schema bound + N metadata versions
- [ ] Reset returns populated `previous*` fields when document had a schema -- test: TDD
- [ ] Reset returns `metadataVersionsDeleted` matching actual deleted count -- test: TDD
- [ ] Reset is idempotent: already-clean document returns `metadataVersionsDeleted: 0`, `previousSchemaId: nil` -- test: TDD
- [ ] Reset rejects documents with legacy field extractions (409) -- test: TDD
- [ ] Reset returns 404 on non-existent document -- test: TDD
- [ ] Reset atomicity: inject failure between DELETE and UPDATE -- test: transaction rolls back, metadata rows still present
- [ ] Reset + reassign + write: after reset, Trigger 1 still fires on second schema change under new metadata -- test: TDD regression
- [ ] Reset + reassign + write: after reset, Trigger 3 still fires on mismatched-schema direct INSERT -- test: TDD regression
- [ ] Reset writes audit log entry containing actor, documentID, previousSchemaId, metadataVersionsDeleted -- test: TDD reads the audit sink
- [ ] Reset concurrency: two concurrent resets on same document serialize via `FOR UPDATE` -- test: TDD
- [ ] Reset concurrency: reset vs `AssignSchema` on same document serialize -- test: TDD

### 4.6 Phase 4 integration

- [ ] `go test ./internal/customschema/...` green -- test: full package green
- [ ] `task test:race` green for package -- test: no races
- [ ] `task test:unit:short` still green globally -- test: no cross-package regressions
- [ ] Journal entry

---

## Phase 5a - Legacy field extraction guard

**Goal**: The existing `FieldExtractionService` refuses to write to any document that has `custom_schema_id IS NOT NULL`, and the HTTP controller surfaces that refusal as `409 Conflict`.
**Dependencies**: Phase 2.
**Exit criteria**: Existing field extraction tests still pass; new service-level test proves the guard rejects; new controller-level test proves the HTTP response is `409` (not `500`).

### 5a.1 Service-layer guard

- [ ] Locate the existing `FieldExtractionService.CreateFieldExtraction` entry point (currently `internal/fieldextraction/service.go`) -- test: `grep` and read code
- [ ] Add a pre-check: read `documents.custom_schema_id` for target document -- test: happy path unchanged (null binding)
- [ ] If `custom_schema_id IS NOT NULL`, return a typed sentinel error (e.g. `ErrMutualExclusivityViolation`) with message referencing mutual exclusivity -- test: TDD service test asserts `errors.Is(err, ErrMutualExclusivityViolation)`
- [ ] Existing field extraction happy-path tests still green -- test: `go test ./internal/fieldextraction/...`

### 5a.2 Controller-layer 409 mapping (REQUIRED to deliver the documented response)

**Why this sub-phase exists**: today `api/queryAPI/fieldextractions.go:CreateFieldExtraction` maps every service error to `http.StatusInternalServerError`. Adding only the service pre-check is not enough -- without controller error mapping the new guard still surfaces as a `500`. Phase 5a is not complete until the HTTP layer returns `409`.

- [ ] Update `CreateFieldExtraction` handler in `api/queryAPI/fieldextractions.go` to branch on the service sentinel error and return `echo.NewHTTPError(http.StatusConflict, ...)` with the mutual-exclusivity message -- test: handler unit test asserts status 409 and body message
- [ ] Audit the other `FieldExtractionService` handler entry points in `api/queryAPI/fieldextractions.go` (Update, AppendValueTo*, etc.) and apply the same 409 mapping where those methods also run the guard -- test: TDD handler test per entry point
- [ ] Preserve the existing mapping of non-sentinel service errors to `500` -- test: TDD handler test with a generic error path
- [ ] End-to-end HTTP test: create document, assign custom schema, call `POST /field-extractions`, assert response status is `409` and JSON body carries the documented message -- test: integration against testcontainer DB
- [ ] Journal entry

---

## Phase 5b - Delete cascade updates

**Goal**: Hard-deleting a document or client works even when custom metadata and schemas are present.
**Dependencies**: Phase 2.
**Exit criteria**: Document and client delete cascades succeed with custom metadata present; no orphaned rows.

### 5b.1 Document delete cascade

- [ ] Locate `DeleteDocumentCascade` implementation -- test: found
- [ ] Insert step: `DeleteDocumentCustomMetadata` between legacy extractions delete and document row delete -- test: TDD
- [ ] Insert step: `NullifyDocumentCustomSchemaId` before document row delete -- test: TDD
- [ ] Document delete succeeds for a document with N metadata versions and a schema bound -- test: TDD integration
- [ ] Document delete succeeds for a document with no custom metadata (regression) -- test: TDD
- [ ] Post-delete: `SELECT COUNT(*) FROM document_custom_metadata WHERE document_id = deleted_id` returns 0 -- test: TDD
- [ ] Post-delete: `client_metadata_schemas` rows referenced by the document are NOT deleted (only the binding is) -- test: TDD

### 5b.2 Client delete cascade

- [ ] Locate `client.HardDelete` / `deleteClientDependencies` -- test: found
- [ ] Insert step: `DeleteClientMetadataSchemas` after all documents are deleted, before client row delete -- test: TDD
- [ ] Client hard-delete succeeds when client has schemas and documents with custom metadata -- test: TDD integration
- [ ] Post-delete: `SELECT COUNT(*) FROM client_metadata_schemas WHERE client_id = deleted_id` returns 0 -- test: TDD
- [ ] Post-delete: no orphaned `document_custom_metadata` rows -- test: TDD
- [ ] Regression: client delete still works for clients with no custom metadata -- test: TDD
- [ ] Journal entry

---

## Phase 6 - OpenAPI spec updates

**Goal**: `serviceAPIs/queryAPI.yaml` reflects all new endpoints and schemas; code generation produces clean client/server stubs.
**Dependencies**: None for spec authoring; must land before Phase 8 codegen.
**Exit criteria**: `task generate` produces updated `api/queryAPI/*` and `pkg/queryAPI/*` without errors.

### 6.1 New tags

- [ ] Add `SuperAdminSchemaService` tag with description explicitly stating the `super_admin`-only boundary -- test: spec lints
- [ ] Add `CustomMetadataService` tag with description stating `client_user` + `user_admin` + `super_admin` access -- test: spec lints

### 6.2 Schemas

- [ ] `CustomSchemaRequest` -- test: codegen produces Go type
- [ ] `CustomSchemaResponse` -- test: codegen
- [ ] `CustomSchemaListResponse` -- test: codegen
- [ ] `CustomMetadataRequest` -- test: codegen
- [ ] `CustomMetadataResponse` -- test: codegen
- [ ] `CustomMetadataHistoryResponse` -- test: codegen
- [ ] `ValidationErrorResponse` -- test: codegen
- [ ] `BulkSchemaAssignRequest` -- test: codegen
- [ ] `BulkSchemaAssignResponse` -- test: codegen
- [ ] `DocumentSchemaAssignRequest` -- test: codegen
- [ ] `DocumentSchemaAssignResponse` -- test: codegen
- [ ] `ResetDocumentMetadataResponse` (v3) -- test: codegen
- [ ] `SchemaStatus` enum (`active`/`superseded`/`retired`) -- test: codegen
- [ ] Modify `DocumentEnriched` to add `customSchemaId`, `customSchemaName`, `hasCustomMetadata`, `customMetadata` -- test: codegen

### 6.3 Paths

- [ ] `POST /super-admin/custom-schemas` -- test: codegen produces handler interface
- [ ] `GET /super-admin/custom-schemas` (with query params) -- test: codegen
- [ ] `GET /super-admin/custom-schemas/{schemaId}` -- test: codegen
- [ ] `PUT /super-admin/custom-schemas/{schemaId}` -- test: codegen
- [ ] `DELETE /super-admin/custom-schemas/{schemaId}` -- test: codegen
- [ ] `PATCH /super-admin/documents/{id}/schema` -- test: codegen
- [ ] `POST /super-admin/folders/{folderId}/assign-schema` -- test: codegen
- [ ] `POST /super-admin/documents/{id}/reset-metadata` (v3) -- test: codegen
- [ ] `GET /custom-metadata` -- test: codegen
- [ ] `POST /custom-metadata` -- test: codegen
- [ ] `GET /custom-metadata/version` -- test: codegen
- [ ] `GET /custom-metadata/history` -- test: codegen
- [ ] Modify `GET /document/{id}` to document new fields and the `customMetadata` query parameter -- test: codegen

### 6.4 Phase 6 integration

- [ ] `task generate` clean -- test: no errors
- [ ] Generated controllers compile -- test: `go build ./...`
- [ ] Journal entry

---

## Phase 7a - Permit.io policy file updates

**Goal**: `permit_policies.yaml` declares the new `super-admin` and `custom-metadata` resources and grants the correct role permissions.
**Dependencies**: None.
**Exit criteria**: YAML validates; setup tool dry-run shows the expected resource/role diff.

> **Accepted cross-tenant limitation**: the authorization check performed by the middleware is role/action/resource-type only -- it does not pass a document or client object to Permit.io. Any caller with `custom-metadata:*` grants can therefore read or write *any* document's custom metadata if they know the UUID. This is unchanged from the rest of the API and is **explicitly accepted for v3** because production is deployed per-client (one isolated stack per client, separate DB). The limitation only matters in shared non-prod environments. See Section 5.3 of the plan for the full rationale. **Do not add application-level tenant checks to the custom-metadata service in v3** -- that is a separate, API-wide initiative.

- [ ] Add `super-admin` resource with `get`, `post`, `patch`, `delete` actions -- test: YAML valid
- [ ] Add `custom-metadata` resource with `get`, `post` actions -- test: YAML valid
- [ ] Grant `super_admin` role: `super-admin: [get, post, patch, delete]`, `custom-metadata: [get, post]` -- test: YAML valid
- [ ] Confirm `user_admin` role NOT granted `super-admin` -- test: visual diff against v1 intent; test-assertion in Phase 11
- [ ] Grant `user_admin` role: `custom-metadata: [get, post]` -- test: YAML valid
- [ ] Grant `auditor` role: `super-admin: [get]`, `custom-metadata: [get]` -- test: YAML valid
- [ ] Grant `client_user` role: `custom-metadata: [get, post]` -- test: YAML valid
- [ ] Add documentation-only `policy_mappings` entries for every `/super-admin/*` path including `POST /super-admin/documents/{id}/reset-metadata` -- test: YAML valid
- [ ] Add documentation-only `policy_mappings` entries for every `/custom-metadata/*` path -- test: YAML valid
- [ ] Resolve PUT-to-patch question per Section 5.3 footnote: either confirm middleware maps PUT -> patch OR add `put` action to `super-admin` resource and grant it. Record decision in the journal. -- test: decision logged
- [ ] Journal entry

---

## Phase 7b - Run Permit.io setup tool (dev / uat / prod)

**Goal**: New resources and grants provisioned in every environment BEFORE Phase 8 code is deployed to that environment.
**Dependencies**: Phase 7a.
**Exit criteria**: Setup tool run succeeds in each env; spot-check via permit.io admin UI or `list_roles.sh` matches YAML.

- [ ] Run `cmd/auth_related/permit.setup/run.tool.dev.sh` -- test: exit code 0
- [ ] Verify `super-admin` resource exists in dev Permit.io tenant -- test: list
- [ ] Verify `custom-metadata` resource exists in dev -- test: list
- [ ] Verify `super_admin` role has expected grants in dev -- test: role inspect
- [ ] Verify `user_admin` role does NOT have `super-admin` grants in dev -- test: role inspect
- [ ] Run `run.tool.uat.sh` after dev verified -- test: exit 0
- [ ] Same verification pass on uat -- test: list + role inspect
- [ ] Run `run.tool.prod.sh` after uat verified AND only with Q's explicit go-ahead -- test: exit 0, Q confirmation logged in journal
- [ ] Same verification pass on prod -- test: list + role inspect
- [ ] Journal entry with timestamps of each env run

---

## Phase 8 - API controllers + codegen

**Goal**: HTTP handlers thin-wrap the service layer; every generated interface method is implemented.
**Dependencies**: Phase 4, Phase 6.
**Exit criteria**: Every new endpoint compiles and routes resolve; handler unit tests green.

### 8.1 Schema CRUD controller (`api/queryAPI/customschemas.go`)

- [ ] File skeleton created with handler receivers wired to `customschema.Service` -- test: compiles
- [ ] **Actor extraction helper**: every handler in this file derives actor identity via `cognitoauth.GetUserSubject(c)` (or the existing middleware helper) and passes it to the service as a separate argument; any `createdBy` on the request body is ignored -- test: handler unit test sets a JWT subject via middleware fixture, posts a body with a different `createdBy`, asserts the service receives the JWT subject
- [ ] `POST /super-admin/custom-schemas` handler -- parse, validate, derive actor from JWT, call `CreateSchema`, map errors to 400/409/500 -- test: TDD handler test
- [ ] `GET /super-admin/custom-schemas` handler with query params -- test: TDD
- [ ] `GET /super-admin/custom-schemas/{schemaId}` handler -- test: TDD
- [ ] `PUT /super-admin/custom-schemas/{schemaId}` handler -- derives actor from JWT -- test: TDD
- [ ] `DELETE /super-admin/custom-schemas/{schemaId}` handler -- derives actor from JWT -- test: TDD including 409 path
- [ ] `PATCH /super-admin/documents/{id}/schema` handler -- derives actor from JWT -- test: TDD
- [ ] `POST /super-admin/folders/{folderId}/assign-schema` handler -- derives actor from JWT -- test: TDD (covered more deeply in Phase 10)
- [ ] `POST /super-admin/documents/{id}/reset-metadata` handler (v3) -- derives actor from JWT -- test: TDD (covered more deeply in Phase 10b)

### 8.2 User-facing custom metadata controller (`api/queryAPI/custommetadata.go`)

- [ ] File skeleton -- test: compiles
- [ ] **Actor extraction**: `POST /custom-metadata` derives actor identity from the JWT and ignores any body-supplied `createdBy` -- test: TDD handler test asserts body `createdBy` does not override
- [ ] `POST /custom-metadata` handler -- test: TDD
- [ ] `GET /custom-metadata` handler -- test: TDD
- [ ] `GET /custom-metadata/version` handler -- test: TDD
- [ ] `GET /custom-metadata/history` handler -- test: TDD

### 8.3 Phase 8 integration

- [ ] All new handler files compile -- test: `go build ./...`
- [ ] Handler unit tests green -- test: `go test ./api/queryAPI/...`
- [ ] Journal entry

---

## Phase 9 - Document endpoint modifications

**Goal**: `GET /document/{id}` surfaces custom schema and metadata when requested.
**Dependencies**: Phase 4, Phase 8.
**Exit criteria**: Existing tests unchanged; new fields populated correctly for documents with and without custom schemas.

- [ ] Extend `DocumentEnriched` builder to include `customSchemaId`, `customSchemaName`, `hasCustomMetadata` unconditionally -- test: TDD
- [ ] Extend builder to include `customMetadata` when `?customMetadata=true` -- test: TDD
- [ ] Authorization: confirm `GET /document/{id}` still uses the existing `document` resource, not `super-admin` -- test: role matrix test in Phase 11
- [ ] Document without any custom schema returns `customSchemaId: null`, `hasCustomMetadata: false` -- test: TDD
- [ ] Document with schema bound but no metadata returns `customSchemaId: <id>`, `hasCustomMetadata: false` -- test: TDD
- [ ] Document with schema + metadata returns populated fields -- test: TDD
- [ ] `PATCH /super-admin/documents/{id}/schema` handler delegates to `AssignSchema` -- test: TDD
- [ ] Assign-schema 409 paths surface correct error messages -- test: TDD
- [ ] Journal entry

---

## Phase 10 - Bulk folder schema assignment endpoint

**Goal**: `POST /super-admin/folders/{folderId}/assign-schema` behaves per Section 5.2 including all skip reasons and the 10K limit.
**Dependencies**: Phase 4 (`AssignSchemaToFolder`), Phase 8.
**Exit criteria**: Recursive walk correct, skip reasons complete, document count limit enforced.

- [ ] Handler wired to `AssignSchemaToFolder` -- test: compiles
- [ ] Recursive walk on a 3-deep folder tree produces correct `documentsUpdated` count -- test: TDD
- [ ] Skip reason: "different schema + existing metadata" -- test: TDD
- [ ] Skip reason: "already assigned to same schema (no-op)" -- test: TDD
- [ ] Skip reason: "has legacy field extractions" -- test: TDD
- [ ] Reassignment on document with different schema but no metadata -- test: TDD
- [ ] Subtree > 10K documents returns 422 with the documented message -- test: TDD (constant override for test)
- [ ] Empty folder returns 200 with 0 counts -- test: TDD
- [ ] Deeply nested (10+ levels) tree walks correctly -- test: TDD
- [ ] Single transaction: simulated mid-walk failure rolls back -- test: TDD
- [ ] Journal entry

---

## Phase 10b - Reset-metadata endpoint (v3)

**Goal**: `POST /super-admin/documents/{id}/reset-metadata` exposes `Service.ResetDocumentMetadata` via HTTP with correct auth, response shape, and error mapping. (Deep service-level tests already live in Phase 4.5; this phase validates the HTTP wiring.)
**Dependencies**: Phase 4.5 (service method), Phase 5b (queries reused), Phase 8 (controller skeleton).
**Exit criteria**: Handler + integration tests green; authorization matrix for the endpoint verified in Phase 11.

- [ ] Handler parses document ID path parameter -- test: TDD
- [ ] Handler extracts caller identity for `resetBy` field -- test: TDD
- [ ] Handler calls `Service.ResetDocumentMetadata` -- test: TDD
- [ ] Handler maps success to 200 with `ResetDocumentMetadataResponse` -- test: TDD
- [ ] Handler maps "document not found" to 404 -- test: TDD
- [ ] Handler maps "legacy extractions exist" to 409 with mutual-exclusivity message -- test: TDD
- [ ] Handler maps unexpected DB errors to 500 -- test: TDD
- [ ] End-to-end HTTP test: bind schema v1, POST metadata, PUT schema -> v2 created, reset-metadata, PATCH schema to v2, POST new metadata with added optional field -- test: integration test exercises the full upgrade flow
- [ ] End-to-end HTTP test: reset on already-clean document returns 200 with `metadataVersionsDeleted: 0` -- test: integration
- [ ] Journal entry

---

## Phase 11 - Integration and authorization tests

**Goal**: Every role's access to every new endpoint is verified. Every cross-cutting invariant has a test. `task fullsuite:ci` ends with `All coverage checks passed!`.
**Dependencies**: Phase 10b and Phase 7b in each environment used for testing.
**Exit criteria**: 80% minimum coverage across new packages; authz matrix 100% covered; no `FAIL` in `fullsuite:ci` output.

### 11.1 Role authorization matrix (every row = one test)

- [ ] `super_admin` CAN `POST /super-admin/custom-schemas` -- test: integration
- [ ] `super_admin` CAN `GET /super-admin/custom-schemas` -- test: integration
- [ ] `super_admin` CAN `GET /super-admin/custom-schemas/{id}` -- test: integration
- [ ] `super_admin` CAN `PUT /super-admin/custom-schemas/{id}` -- test: integration
- [ ] `super_admin` CAN `DELETE /super-admin/custom-schemas/{id}` -- test: integration
- [ ] `super_admin` CAN `PATCH /super-admin/documents/{id}/schema` -- test: integration
- [ ] `super_admin` CAN `POST /super-admin/folders/{folderId}/assign-schema` -- test: integration
- [ ] `super_admin` CAN `POST /super-admin/documents/{id}/reset-metadata` -- test: integration
- [ ] `user_admin` gets 403 on every `/super-admin/*` path+method combination (8 tests) -- test: integration, one per route
- [ ] `user_admin` CAN read + write `/custom-metadata` (no regression) -- test: integration
- [ ] `client_user` gets 403 on every `/super-admin/*` path+method combination (8 tests) -- test: integration
- [ ] `client_user` CAN read + write `/custom-metadata` -- test: integration
- [ ] `auditor` CAN `GET` on `/super-admin/custom-schemas` and `/super-admin/custom-schemas/{id}` -- test: integration
- [ ] `auditor` gets 403 on every `/super-admin/*` write path (POST/PUT/PATCH/DELETE) -- test: integration
- [ ] `auditor` CAN `GET /custom-metadata`, CANNOT `POST /custom-metadata` -- test: integration
- [ ] Every 403 test asserts DB state is unchanged on denial -- test: embedded in each

### 11.2 Mutual exclusivity

- [ ] Legacy write blocked when document has custom schema -- test: integration
- [ ] Schema assignment blocked when document has legacy extractions -- test: integration
- [ ] Bulk folder assign skips legacy-extracting documents -- test: integration
- [ ] Reset-metadata rejected on legacy-extracting document -- test: integration

### 11.3 Schema lifecycle

- [ ] `active` -> `superseded` on PUT create new version -- test: integration
- [ ] `active` -> `retired` on DELETE -- test: integration
- [ ] Assignment rejected for non-`active` status -- test: integration
- [ ] Schema version concurrency (concurrent PUTs) produces distinct versions -- test: integration

### 11.4 Delete cascade

- [ ] Document hard-delete with custom metadata -- test: integration
- [ ] Client hard-delete with schemas and metadata -- test: integration
- [ ] No orphaned rows post-delete -- test: integration

### 11.5 Size and count limits

- [ ] Metadata payload at 1MB boundary: accept 1048576, reject 1048577 -- test: integration
- [ ] Schema definition at 64KB boundary: accept 65536, reject 65537 -- test: integration
- [ ] Bulk folder > 10000 docs: 422 -- test: integration
- [ ] Bulk folder exactly 10000 docs: succeeds -- test: integration

### 11.6 Trigger regression (defense-in-depth)

- [ ] Direct SQL bypass of service layer: attempt UPDATE `custom_schema_id` on a document with metadata -- trigger fires
- [ ] Direct SQL bypass: INSERT `document_custom_metadata` with mismatched `schema_id` -- trigger fires
- [ ] Direct SQL bypass: assign cross-client schema -- trigger fires
- [ ] After reset + reassign + new metadata, all three triggers still fire on repeat attempts -- test: integration

### 11.7 Backward compatibility

- [ ] Existing `/field-extractions` endpoints unchanged for documents with no custom schema -- test: integration
- [ ] Existing `/admin/*` endpoints unaffected by `super-admin` resource -- test: integration
- [ ] `DocumentEnriched` for documents without custom schemas has null new fields, no regression on existing fields -- test: integration

### 11.8 Permit.io setup tool verification

- [ ] Run setup tool against a disposable dev tenant and assert both resources exist with expected actions -- test: tool output + API inspection
- [ ] Assert each role's grants match Section 5.3 exactly (including the negative case for `user_admin`) -- test: tool output

### 11.9 Coverage & full suite

- [ ] `internal/customschema/` >= 80% coverage -- test: `go test -cover`
- [ ] `api/queryAPI/customschemas.go` + `custommetadata.go` >= 80% coverage -- test: `go test -cover`
- [ ] `task test:race` green -- test: no race reports
- [ ] `task test:perf` checked for regressions in existing hot paths -- test: review output
- [ ] `task fullsuite:ci` ends with `All coverage checks passed!` and NO `FAIL` lines -- test: full run
- [ ] Journal entry with test counts, coverage numbers, and any flaky test notes

---

## Phase 12 - Documentation updates

**Goal**: Every doc that references endpoints, env vars, authorization, or the delete cascade is updated.
**Dependencies**: Phase 11.
**Exit criteria**: `docs/ai.generated/` regenerated; manual review of key runbooks complete.

- [ ] Update `docs/ai.generated/README.md` index if new doc files added -- test: review
- [ ] Authorization matrix table published in a dedicated doc page -- test: review
- [ ] Sample schemas from Appendix A published as copyable examples -- test: review
- [ ] `CustomMetadataService` usage guide (end-user perspective) -- test: review
- [ ] `SuperAdminSchemaService` usage guide (platform-operator perspective) -- test: review
- [ ] Reset-metadata runbook: "How to upgrade a document to a newer schema version" with the full wipe -> reassign -> re-POST flow and a warning about metadata history loss -- test: review
- [ ] Mutual exclusivity explanation (legacy vs custom) in the document processing guide -- test: review
- [ ] Delete cascade documentation updated -- test: review
- [ ] Permit.io deployment ordering note (Phase 7b before Phase 8) added to the deployment runbook -- test: review
- [ ] Any new env vars? (none expected for v3; confirm.) -- test: review
- [ ] Generate new API status report / architecture diagrams if the pipeline regenerates them -- test: review
- [ ] Journal entry marking the feature shipped

---

## Cross-phase reminders

- Before starting any phase, re-read the relevant section of `plans/mutable.metadata.plan.combo.v3.md`.
- TDD is mandatory per project rules: write the test, watch it fail, make it pass. Never write multiple tests before running any.
- No mocks without Q's explicit permission. Use testcontainers for real Postgres.
- Keep the journal current. When a checkbox flips to `[x]`, add a timestamped line to `./journals/implement_mutableMetadata.md` with the commit SHA and test command that proved it.
- If any trigger, migration, or service invariant seems to conflict with the plan, STOP and ask Q before modifying it. The defense-in-depth design depends on the triggers being exactly as written.
- `task fullsuite:ci` must be green at the end of every phase, not just the end of the project. Do not stack broken phases.