Files

T

Jay Brown 17fc813823 Merged in feature/mutable-metadata1 (pull request #221 )

M1, M2 and M3 complete

* M1, M2 and M3 complete

* review changes

* docs

* docs

2026-04-16 23:11:26 +00:00

72 KiB

Raw Blame History

Mutable Metadata v4 - Implementation Tracking

Source plan: plans/mutable.metadata.plan.combo.v4.md Journal (details / decisions / blockers): ./journals/implement_mutableMetadata.md Purpose: One-glance status of every task needed to ship v4. Each checkbox is a discrete, verifiable unit of work. TDD rule: write the test first, watch it fail, make it pass, check the box. If a task requires Q's input or produces a decision, log it to the journal and keep the checkbox state accurate here.

How v4 differs from v3 tracking

v3 organized work as 12 horizontal phases (migrations first, then queries, then service, then controllers, then tests). v4 reorganizes into four vertical slice milestones. Each milestone lands migrations, queries, service methods, controllers, OpenAPI, Permit.io updates, and integration tests for exactly one slice of feature behavior. At the end of every milestone the feature is end-to-end usable in its current scope -- not a half-built stack.

The simplification pass (see Section 16 of the plan) also removed:

v3's trg_validate_schema_client_match (v3 labelled "Trigger 2") -- replaced by the composite FK fk_documents_custom_schema_same_client. Note: v4 introduces a different Trigger 2 (trg_prevent_legacy_extraction_on_custom_document) — see Section 2.2; do not confuse the two.
v3's trg_enforce_consistent_schema_id (v3 labelled "Trigger 3") -- made structurally impossible by removing document_custom_metadata.schema_id. v4 has no Trigger 3.
current_document_custom_metadata view -- replaced by a LIMIT 1 query
The entire delete-cascade workstream (v3 Phase 5b) -- replaced by ON DELETE CASCADE on the new FKs
PUT /super-admin/custom-schemas/{schemaId} -- replaced by POST .../versions
createdBy request-body fields -- actor comes from the JWT
customSchemaName and inlined customMetadata on GET /document/{id} -- only customSchemaId and hasCustomMetadata remain

Every task below reflects those decisions.

How to use this document

[ ] = not started
[~] = in progress
[x] = done AND tests passing (task fullsuite:ci clean for the affected code paths)
[!] = blocked -- add a short (blocked: <why>) suffix and open a journal entry
[-] = consciously skipped -- add a short (skipped: <why>) suffix
Every task line ends with a test: marker indicating where the passing test(s) live or what test type proves it.
When a whole milestone is green, flip the milestone header to COMPLETE and date-stamp it.
Never mark a milestone complete until task fullsuite:ci ends with All coverage checks passed! with no FAIL lines.

Milestone roll-up (high-level)

Milestone	Title	Status	Started	Completed
0-INT	BLOCKER: Fix `task fullsuite:ci` regression (container race in `api/queryAPI`)	COMPLETE	2026-04-15	2026-04-15
1	Schema foundation (create/list/version/delete schemas; super_admin gating)	COMPLETE	2026-04-14	2026-04-14
2	Core document flow (assign schema; write/read metadata; legacy guard; `GET /document/{id}` enrichment)	COMPLETE	2026-04-14	2026-04-15
3	Administrative completion (reset-metadata; schema version concurrency; full authz matrix)	COMPLETE	2026-04-15	2026-04-15
4	Bulk operations (`POST /super-admin/folders/{folderId}/assign-schema`; 10K cap; skip reasons)	NOT STARTED
Docs	Documentation updates (runs in parallel with Milestones 3 + 4)	NOT STARTED

Milestone 0-INT — Fix fullsuite:ci regression — COMPLETE (2026-04-15)

Goal: task fullsuite:ci passes clean with All coverage checks passed! and no FAIL lines. This is a prerequisite for completing any other milestone.

Background: The mutable metadata branch caused widespread duplicate-key failures in test runs because the reused postgres container retained data between runs. A truncation fix was applied to internal/test/database.go (see journal entry for 2026-04-15), resolving the duplicate-key issues but introducing a new regression: api/queryAPI tests intermittently fail with "unable to ping database" under parallel execution due to a container concurrency race in testcontainers Reuse: true.

What is already done:

[x] Truncation fix in internal/test/database.go — resolves duplicate-key violations across all packages

What remains:

[x] Diagnose actual failure: Q's run showed the container race was resolved by Docker state cleanup; the remaining failure was function coverage below 60% on ResetDocumentMetadata and mapResetError in api/queryAPI/customschemas_reset.go — both at 0% because no tests existed.
[-] Mutex fix — not needed; container race was accumulated Docker state, not a code bug.
[x] Write tests: added TestMapResetError (white-box, package queryapi) to custommetadata_errors_test.go; created api/queryAPI/customschemas_reset_test.go (package queryapi_test) with 5 integration tests covering 401/200/200-idempotent/404/409. All pass.
[x] Run task fullsuite:ci — passes with All coverage checks passed!, Total Coverage 85.4%, 2026-04-15.
[x] Flip this milestone to COMPLETE and date-stamp.

Milestone 1 - Schema foundation — COMPLETE (2026-04-14)

Goal: A super_admin can create, list, get, create-new-version, and delete client-scoped custom schemas. Invalid schemas are rejected. Cross-client schema bindings are structurally impossible at the database level. The super-admin Permit.io resource exists and gates every new schema endpoint.

Dependencies: None.

Exit criteria:

task fullsuite:ci green
End-to-end integration test proves super_admin can POST a schema, POST a new version, GET /super-admin/custom-schemas?clientId=...&name=aircraft-engineering&includeAllVersions=true&status=any returns both rows with the correct statuses (v1 superseded, v2 active), GET with the default query (no status, no includeAllVersions) returns only v2 active, DELETE removes a retired schema, and user_admin / client_user / auditor are correctly gated on every route.
Direct-SQL test proves the composite FK refuses a cross-client schema binding.

1.1 Migration 127 - `schema_status_type` enum + `client_metadata_schemas`

Numbering note: The repo contains migrations through 00000000000126. If another branch lands new migrations before this feature merges, renumber this block and all references in the plan and tracking docs.

Create internal/database/migrations/00000000000127_create_client_metadata_schemas.up.sql with CREATE TYPE schema_status_type AS ENUM ('active', 'superseded', 'retired') -- test: migration applies cleanly on empty DB
Create client_metadata_schemas table per Section 4.2 of the plan -- columns, PK, FK to clients(clientId) ON DELETE CASCADE, schema_def jsonb, version, status, timestamps, created_by -- test: \d client_metadata_schemas shows all columns and the cascade FK
Add CONSTRAINT uq_client_schema_name_version UNIQUE (client_id, name, version) -- test: direct SQL INSERT of two rows with same (client_id, name, version) fails
v4: Add CONSTRAINT uq_cms_id_client UNIQUE (id, client_id) (required as the target of the composite FK in Milestone 1.3) -- test: constraint visible in \d client_metadata_schemas
Add CONSTRAINT chk_schema_def_size CHECK (pg_column_size(schema_def) <= 70000) -- test: insert of a schema_def larger than 70000 bytes fails with CHECK violation
Add indexes idx_cms_client_id, idx_cms_client_name, idx_cms_client_status -- test: \di client_metadata_schemas shows the three indexes
Create 00000000000127_create_client_metadata_schemas.down.sql dropping the table and enum in correct order -- test: up then down leaves DB state byte-identical to pre-migration
Run task db:generate after migration lands -- test: no errors

1.2 Schema validator component + dependency

Add github.com/santhosh-tekuri/jsonschema/v6 to go.mod -- test: go mod tidy clean, build clean
Create internal/customschema/constants.go with MaxSchemaDefinitionBytes = 65536, MaxMetadataPayloadBytes = 1048576, MaxBulkAssignDocuments = 10000 -- test: constants referenced by tests
Create internal/customschema/validator.go with SchemaValidator struct -- test: compiles
Implement ValidateSchemaDefinition(schemaDef json.RawMessage) error -- test: valid JSON Schema draft 2020-12 accepted
Reject schema definitions that are not valid JSON -- test: TDD test with malformed JSON
Reject schema definitions that fail JSON Schema meta-validation -- test: TDD test with {"type": "nonsense"}
Reject schema definitions where root type is not "object" -- test: TDD test with {"type": "array"}
Reject schema definitions where root additionalProperties is not explicitly present -- test: TDD test with schema missing the key
Accept schema definitions where root additionalProperties is explicitly true or false -- test: TDD tests for both
Reject schema definitions larger than MaxSchemaDefinitionBytes -- test: TDD test with 64KB+1 payload
go test ./internal/customschema/... green (validator only) -- test: package level

1.3 Migration 128 - `documents.custom_schema_id` column + composite FK

v4 monotonic ordering: Milestone 1 ships migrations 127 and 128. Milestone 2 ships 129 and 130. The filename numbering must match rollout order because golang-migrate's Up() only applies versions strictly greater than the current DB version -- if Milestone 1 lands a 129 file, a later Milestone 2 file numbered 128 will be silently skipped or flagged as out of order. Earlier drafts of v4 had 128 and 129 swapped; do not revert that swap.

Create 00000000000128_add_custom_schema_id_to_documents.up.sql -- ALTER TABLE documents ADD COLUMN custom_schema_id uuid NULL (no inline REFERENCES clause; the composite FK below handles it) -- test: column present and nullable
Add composite FK: ALTER TABLE documents ADD CONSTRAINT fk_documents_custom_schema_same_client FOREIGN KEY (custom_schema_id, clientId) REFERENCES client_metadata_schemas(id, client_id) -- test: constraint visible; direct SQL INSERT of a document with custom_schema_id pointing at a schema owned by a different client fails with FK violation. v4 implementation note: the plan Section 4.4 SQL writes "clientId" quoted, but the documents.clientId column was declared unquoted in migration 5, so Postgres stored it as lowercase clientid. A quoted "clientId" in the FK declaration is a case-sensitive literal lookup that fails with column "clientId" referenced in foreign key constraint does not exist. The migration uses unquoted clientId which folds to clientid at parse time and matches. This same gotcha applies to any future DDL or trigger that references documents."clientId".
Create idx_documents_custom_schema_id -- test: index present
Write down.sql that drops the index, the constraint, and the column in correct order -- test: round trip clean
Verify existing documents rows have custom_schema_id = NULL after migration -- test: SELECT COUNT(*) FROM documents WHERE custom_schema_id IS NOT NULL returns 0

1.4 SQLC queries for schema CRUD

New query file: internal/database/queries/customschemas.sql

CreateClientMetadataSchema -- INSERT returning full row -- test: generated code compiles, query name appears in *.sql.go
GetClientMetadataSchema (by id) -- test: generated
ListClientMetadataSchemas (filters: client_id required, name optional, status optional with default active, pagination limit/offset; includes documentCount subquery against documents.custom_schema_id) -- test: generated; unit test confirms documentCount populated correctly
GetLatestSchemaByName -- test: generated
GetSchemaDocumentCount (count of documents.custom_schema_id = $1) -- test: generated
SetSchemaStatus (update status, used for supersede and retire) -- test: generated
GetMaxSchemaVersion (for auto-increment on version creation) -- test: generated
LockSchemaVersionsForName (SELECT ... FOR UPDATE on all versions of a (client_id, name) pair) -- test: generated
task generate clean; no diff after re-run -- test: idempotent

Note

: GetSchemaMetadataRecordCount is deliberately not in this list. v3 used it for canDelete, but v4's canDelete is derived from GetSchemaDocumentCount alone (the metadata table has no schema_id column).

1.5 Audit sink

Why in Milestone 1: Schema CRUD needs to log schema.create, schema.update, schema.delete. Defining the sink now lets Milestone 1's service methods land with audit wired in, and Milestones 2/3/4 reuse the same sink without back-fill.

Decide sink location: either internal/customschema/audit.go (feature-local) or internal/audit/ (shared) -- test: decision recorded in journal (feature-local chosen 2026-04-14)
Define AuditRecord struct: Actor string, Action string (enum: schema.create|schema.update|schema.delete|schema.assign|metadata.write|metadata.reset|folder.assign_schema), ResourceID uuid.UUID, ClientID uuid.UUID, Timestamp time.Time, Details map[string]any -- test: compiles
Define AuditSink interface with Record(ctx, AuditRecord) error -- test: compiles
Provide a default implementation (structured slog with audit=true attribute, or a DB-backed appender if Q decides) -- test: unit test verifies a recorded event lands in the sink (TestSlogAuditSink_Record — 4 subtests green)
Wire the sink into customschema.Service via constructor arg -- test: compiles (Service.New(cfg, validator, audit) landed in §1.6)

1.6 Service layer - schema CRUD

Create internal/customschema/service.go with Service struct holding cfg, validator, audit -- test: compiles
Create internal/customschema/models.go with Schema, SchemaSummary, CreateSchemaInput, CreateSchemaVersionInput, ListFilters -- test: compiles
v4: CreateSchemaInput and CreateSchemaVersionInput do not carry a CreatedBy field. Every write method accepts actor string as a separate argument. -- test: unit test proves the service method signature has actor string and not a body field
CreateSchema(ctx, input, actor) -- meta-validates, enforces 64KB, enforces explicit additionalProperties, enforces name unique within client for active schemas at version 1 -- test: happy path
CreateSchema rejects invalid JSON Schema -- test: TDD
CreateSchema rejects oversize definition -- test: TDD
CreateSchema rejects duplicate name for same client at version 1 -- test: TDD (service returns ErrSchemaNameConflict, wraps pg 23505 / uq_client_schema_name_version)
CreateSchema writes schema.create audit entry -- test: TDD
CreateSchemaVersion(ctx, parentSchemaID, input, actor) -- uses LockSchemaVersionsForName -> verifies the path parentSchemaID matches the single row with status='active' for (client_id, name) -> GetMaxSchemaVersion -> INSERT new version -> SetSchemaStatus(parent, 'superseded') inside a single tx -- test: happy path produces v2
CreateSchemaVersion concurrency: two concurrent calls against same (client_id, name) produce distinct version numbers -- test: TDD with goroutines against testcontainer DB (v4 resolution: winner writes v2, loser returns ErrSchemaParentNotActive — no retry loop; the FOR UPDATE serializes the two tx and the loser observes the parent as superseded)
CreateSchemaVersion rejects when parent does not exist -- test: TDD (returns ErrSchemaNotFound)
v4 non-active parent rejection: CreateSchemaVersion rejects with 409 when the path parentSchemaID points at a superseded row while a different row is the active version for the lineage (e.g. v2 is active, caller passes v1's ID) -- test: TDD asserts 409, asserts no new row inserted, asserts v2 still has status='active' and v1 still has status='superseded' (service returns ErrSchemaParentNotActive)
v4 retired parent rejection: CreateSchemaVersion rejects with 409 when the path parentSchemaID points at a retired row -- test: TDD asserts 409, asserts no new row inserted, asserts the retired row's status is unchanged (service returns ErrSchemaParentRetired)
v4 active-parent invariant: After CreateSchemaVersion succeeds there is exactly one row with status='active' for the (client_id, name) lineage -- test: TDD asserts the invariant after happy path, after two sequential calls (v1 -> v2 -> v3), and after the two rejection cases above
CreateSchemaVersion rejects invalid schema definition (same validator rules as create) -- test: TDD
CreateSchemaVersion writes schema.update audit entry -- test: TDD (ResourceID is the new version's ID; Details carries name, version, parent_schema_id)
GetSchema(ctx, schemaID) -- test: returns full row including schema_def
GetSchema 404 when not found -- test: TDD (returns ErrSchemaNotFound via errors.Is(err, pgx.ErrNoRows))
ListSchemas(ctx, clientID, filters) -- default returns only active and only the latest version per (client_id, name) -- test: TDD
ListSchemas filter by name, by status (including the any sentinel for "all statuses"), by includeAllVersions=true, with pagination -- test: TDD per case, including one case that sets both status=any and includeAllVersions=true against a lineage that has one active + one superseded row and asserts both rows are returned
ListSchemas populates documentCount and canDelete -- test: TDD
ListFilters accepts an Status field with zero-value meaning "active" (default) and a dedicated StatusAny sentinel (or *schemaStatusType with nil meaning "any") so handlers can distinguish "filter omitted" from "caller asked for everything" — test: unit test on the mapping from query-string status=any to the service-layer value
DeleteSchema(ctx, schemaID, actor) -- sets status to retired -- test: happy path
DeleteSchema returns 409 if any document references the schema via custom_schema_id -- test: TDD (returns ErrSchemaInUse)
DeleteSchema writes schema.delete audit entry -- test: TDD
go test ./internal/customschema/... green -- test: package level
task test:race green for package -- test: no races (go test -race ./internal/customschema/... → ok 8.303s)

1.7 OpenAPI spec - schema CRUD

Add SuperAdminSchemaService tag with description explicitly stating the super_admin-only boundary -- test: spec lints
Add SchemaStatus enum (active/superseded/retired) -- test: codegen produces Go type
CustomSchemaRequest schema (no createdBy field) -- test: codegen
CustomSchemaVersionRequest schema (no createdBy; v4 rename from v3's CustomSchemaUpdateRequest) -- test: codegen
CustomSchemaResponse schema with createdBy declared as plain string (not format: email) per plan Section 8.3 "Actor fields" -- test: codegen; spec lint
Spec check: assert that none of the new response schemas declare format: email on a createdBy / resetBy field -- test: grep or programmatic spec check in task openapi:lint (verified via grep -nE 'format: ?email' serviceAPIs/queryAPI.yaml | grep -iE 'createdBy|resetBy' → 0 matches)
CustomSchemaListResponse schema with documentCount and canDelete fields -- test: codegen
POST /super-admin/custom-schemas path -- test: codegen produces handler interface
GET /super-admin/custom-schemas (with query params) -- test: codegen
GET /super-admin/custom-schemas/{schemaId} -- test: codegen
v4: POST /super-admin/custom-schemas/{schemaId}/versions path (replaces v3's PUT /super-admin/custom-schemas/{schemaId}) -- test: codegen
DELETE /super-admin/custom-schemas/{schemaId} -- test: codegen
task generate clean -- test: no errors
Generated controllers compile -- test: go build ./... (stub handlers land in api/queryAPI/customschemas_stub.go returning http.StatusNotImplemented; real handlers arrive in §1.9)

1.8 Permit.io policy file + setup tool run

Isolation model (v4): Production is deployed one client per stack (separate DB, services, Cognito pool). The whole stack is the client scope, so the Permit.io role/action/resource-type check is the complete authorization story on production — there is no other client to cross into. Shared dev/uat stacks are test data only (see plan Section 5.3). Do not add application-level document->client ownership checks to the custom-metadata or super-admin routes in v4; there is no cross-client data to protect from inside a production stack, and retrofitting such a check would be an API-wide initiative out of scope for this feature.

Add super-admin resource to cmd/auth_related/permit.setup/permit_policies.yaml with get, post, patch, delete actions -- test: YAML valid (verified via python3 -c 'yaml.safe_load(...)' 2026-04-14)
Add custom-metadata resource with get, post actions (provisioning it now lets Milestone 2 ship without another setup-tool run) -- test: YAML valid
Grant super_admin role: super-admin: [get, post, patch, delete], custom-metadata: [get, post] -- test: YAML valid
Confirm user_admin role NOT granted super-admin -- test: visual diff (YAML parser confirmed user_admin super-admin: None)
Grant user_admin role: custom-metadata: [get, post] -- test: YAML valid
Grant auditor role: super-admin: [get], custom-metadata: [get] -- test: YAML valid
Grant client_user role: custom-metadata: [get, post] -- test: YAML valid
Add documentation-only policy_mappings entries for every /super-admin/* path including /versions and /reset-metadata -- test: YAML valid
Add documentation-only policy_mappings entries for every /custom-metadata/* path -- test: YAML valid
v4: No put action needed on super-admin -- v4 replaced the v3 PUT route with POST .../versions. Record this decision in the journal so no one re-adds it. -- test: journal entry (recorded)
Run cmd/auth_related/permit.setup/run.tool.dev.sh -- test: exit code 0 (Q ran 2026-04-14; confirmed dev environment updated)
Verify super-admin and custom-metadata resources exist in dev with expected actions -- test: confirmed by Q running the setup tool successfully 2026-04-14
Verify super_admin role has expected grants and user_admin has NO super-admin grants in dev -- test: confirmed by Q 2026-04-14
[!] Run run.tool.uat.sh after dev verified -- test: exit 0 (deferred until feature complete per Q directive 2026-04-14: "the others do not need to be run until this feature is completed")
[!] Same verification pass on uat -- test: list + role inspect (deferred until feature complete)
[!] Run run.tool.prod.sh after uat verified AND only with Q's explicit go-ahead -- test: exit 0, Q confirmation logged in journal (deferred until feature complete)
[!] Same verification pass on prod -- test: list + role inspect (deferred until feature complete)
Journal entry with timestamps of each env run (dev: 2026-04-14; uat/prod: deferred)

1.9 Schema CRUD controllers

Create api/queryAPI/customschemas.go skeleton with handler receivers wired to customschema.Service -- test: compiles (replaced the §1.7 stub file)
v4 actor extraction helper: every handler in this file derives actor identity via cognitoauth.GetUserSubject(c) (returns the JWT sub claim — an opaque Cognito subject UUID, not an email) and passes it to the service as a separate argument; any createdBy on the request body is ignored (actually absent from v4 schemas, but the test codifies the intent) -- test: handler unit test sets a JWT subject via middleware fixture to a fixed UUID, posts a body that includes an extra createdBy key with a different email-shaped value, asserts the service receives the JWT subject UUID and the body key has no effect, and asserts the response createdBy is the UUID (not the body value, not an email) (handler tests set claims via ctx.Set("user_claims", ...) mirroring eulaHandlers_test.go)
POST /super-admin/custom-schemas handler -- parse, validate, derive actor, call CreateSchema, map errors to 400/409/500 -- test: TDD (TestCreateCustomSchema_Handler 5 subtests)
GET /super-admin/custom-schemas handler with query params -- test: TDD (TestListCustomSchemas_Handler 4 subtests)
GET /super-admin/custom-schemas/{schemaId} handler -- test: TDD (TestGetCustomSchema_Handler 2 subtests)
v4: POST /super-admin/custom-schemas/{schemaId}/versions handler -- derives actor, calls CreateSchemaVersion -- test: TDD (TestCreateCustomSchemaVersion_Handler 3 subtests)
DELETE /super-admin/custom-schemas/{schemaId} handler -- test: TDD including 409 path (TestDeleteCustomSchema_Handler 3 subtests)
go test ./api/queryAPI/customschemas_test.go green -- test: package (17 subtests PASS under scoped -run invocations)

1.10 Milestone 1 integration tests

End-to-end test: super_admin POSTs schema v1 -> GET returns it with version 1, active -- test: integration against testcontainer DB (TestMilestone1_CreateV1AndGet)
End-to-end test: super_admin POSTs schema v2 via POST .../versions -> GET /super-admin/custom-schemas?clientId=...&name=...&includeAllVersions=true&status=any returns v1 (status superseded) and v2 (status active). Also assert the default GET (no includeAllVersions, no status) returns only v2 (the default filter is includeAllVersions=false + status=active, which intentionally hides superseded predecessors from the common admin UX). -- test: integration (TestMilestone1_CreateV2ListSemantics with 2 subtests default_list_returns_active_latest_only and include_all_versions_status_any_returns_both)
End-to-end test: DELETE on a schema with documents bound returns 409 -- test: integration (TestMilestone1_DeleteBoundReturns409; asserts row stays active in DB on denial)
End-to-end test: DELETE on an unreferenced schema succeeds and sets status retired -- test: integration (TestMilestone1_DeleteUnreferencedRetires)
Direct-SQL cross-client rejection test: attempt an UPDATE documents SET custom_schema_id = 'schema-owned-by-client-B' WHERE id = 'doc-owned-by-client-A' and assert the composite FK fires -- test: integration (TestMilestone1_CrossClientFKRejection; asserts the error text mentions fk_documents_custom_schema_same_client and docA.custom_schema_id is still NULL)
Authorization matrix for Milestone 1 endpoints: super_admin positive on all methods; user_admin 403 on every route; client_user 403; auditor GET-only on the two read routes, 403 on write routes -- test: one integration test per (role, route, method) cell, each asserting status + DB unchanged on denial (TestMilestone1_AuthorizationMatrix — 4 roles × 5 routes = 20 subtests; runPermitGate reproduces the performPermitIOAuthorization choke-point logic in-test using only the exported cognitoauth symbols GetUserSubject, GetResourceFromRoute, GetActionFromMethod, PermitChecker)
task fullsuite:ci clean with All coverage checks passed! -- test: full run (55 packages OK, 0 FAIL, Total Coverage 86.0%, 2026-04-14 evening run captured in /tmp/fullsuite-ci.log)
Journal entry: Milestone 1 exit snapshot (test counts, coverage numbers, timestamp, commit SHA) (see 2026-04-14 Milestone 1 COMPLETE entry in journals/implement_mutableMetadata.md)

Milestone 2 - Core document flow

Goal: A super_admin can bind a schema to a document. A client_user (or higher) can write and read validated custom metadata on that document. The legacy field extraction path rejects writes to documents that have been opted into the custom schema system. GET /document/{id} surfaces two new fields for any reader. The mutual-exclusivity invariant is proven end-to-end.

Dependencies: Milestone 1 complete.

Exit criteria:

Full round-trip test: create schema -> assign to document -> POST valid metadata -> GET returns it -> attempt invalid metadata (400) -> attempt legacy field extraction on same document (409).
FK-cascade delete test: delete a document that has custom metadata, verify rows are cleaned up automatically.
task fullsuite:ci green.

2.1 Migration 129 - `document_custom_metadata` table

Create 00000000000129_create_document_custom_metadata.up.sql with document_custom_metadata table per Section 4.3 of the plan -- columns: id, document_id, metadata, version, created_at, created_by. No schema_id column. -- test: \d document_custom_metadata confirms column set
FK on document_id uses ON DELETE CASCADE -- test: direct SQL delete of a document row removes its metadata rows
Add CONSTRAINT uq_doc_custom_metadata_version UNIQUE (document_id, version) -- test: direct SQL duplicate (document_id, version) INSERT fails
Add idx_dcm_doc_version_desc ON document_custom_metadata(document_id, version DESC) (the only index v4 needs on this table) -- test: index present
v4: No idx_dcm_schema_id (column does not exist). No view. Record this explicitly in the migration file comment so a future reader comparing to v3 sees why. -- test: visual
Create corresponding down.sql dropping the table -- test: up/down round trip clean

2.2 Migration 130 - schema invariant triggers (v4: Trigger 1 + Trigger 2)

Create 00000000000130_add_schema_invariant_triggers.up.sql -- test: migration applies cleanly
Define trg_prevent_schema_reassignment() function per Section 4.7 Trigger 1 -- test: function exists in pg_proc
Attach trigger to documents BEFORE UPDATE OF custom_schema_id -- test: trigger listed in \d documents
Trigger 1 unit test: insert document, assign schema, write custom metadata, attempt UPDATE custom_schema_id -- expect RAISE EXCEPTION with "custom metadata already exists" message
Trigger 1 unit test: insert document, add legacy extraction row, attempt UPDATE custom_schema_id -- expect RAISE EXCEPTION with "legacy field extractions already exist" message
Trigger 1 unit test: insert document with no metadata and no extractions, UPDATE custom_schema_id from schema_a to schema_b -- expect success (pre-binding change allowed)
Define trg_prevent_legacy_extraction_on_custom_document() function per Section 4.7 Trigger 2 -- reads documents.custom_schema_id with FOR SHARE and raises check_violation if non-null -- test: function exists in pg_proc
Attach trigger to documentFieldExtractions BEFORE INSERT -- test: trigger listed in \d "documentFieldExtractions"
Trigger 2 unit test: insert document with custom_schema_id set, attempt direct-SQL INSERT into documentFieldExtractions -- expect RAISE EXCEPTION with "bound to custom schema" message and SQLSTATE 23514 (check_violation)
Trigger 2 unit test: insert document with custom_schema_id = NULL, INSERT into documentFieldExtractions -- expect success (no-op guard path)
Trigger 2 concurrency test: start tx A running AssignSchema-style SELECT documents FOR UPDATE + UPDATE documents SET custom_schema_id = S, then in tx B run INSERT documentFieldExtractions targeting the same document -- tx B's trigger's FOR SHARE must block until tx A commits; after tx A commits tx B's INSERT must fail with the check_violation -- test: TDD with goroutines against testcontainer DB
v4: Migration intentionally does NOT define v3's trg_validate_schema_client_match (replaced by the composite FK in Milestone 1.3) -- record this in a migration-file comment for future readers
v4: Migration intentionally does NOT define v3's trg_enforce_consistent_schema_id (not needed -- document_custom_metadata has no schema_id column) -- record this in a migration-file comment
Write down.sql dropping both triggers and both functions in correct order -- test: up/down round trip clean
Run task generate after migrations land -- test: clean
Run task test:unit:short -- test: all existing tests still pass (no regression)

2.3 SQLC queries for metadata + document binding

CreateDocumentCustomMetadata (no schema_id parameter) -- test: generated code compiles
GetCurrentDocumentCustomMetadata -- ORDER BY version DESC LIMIT 1 using idx_dcm_doc_version_desc (v4: no view). Joins documents and client_metadata_schemas to also return schema_id, schema_name, schema_version so the GET /custom-metadata handler can populate its response in a single round-trip -- test: generated; unit test asserts the returned row carries the schema name/version for a metadata row whose parent document has a bound schema
GetDocumentCustomMetadataByVersion -- same join as above so GET /custom-metadata/version returns matching schema decoration -- test: generated
GetDocumentCustomMetadataHistory (with limit/offset) -- no schema join; the history response returns version summaries only -- test: generated
v4: LockDocumentForMetadataWrite (SELECT id FROM documents WHERE id = @document_id FOR UPDATE; parent-row lock that serializes first and subsequent writes) -- test: generated. Replaces v3's LockDocumentCustomMetadataForVersion.
GetMaxDocumentMetadataVersion -- SELECT COALESCE(MAX(version), 0) FROM document_custom_metadata WHERE document_id = @document_id -- test: generated
SetDocumentCustomSchemaId -- test: generated
GetDocumentCustomSchemaId -- test: generated
Reuse existing query: HasFieldExtraction already exists in internal/database/queries/fieldextractions.sql (EXISTS(SELECT 1 FROM documentFieldExtractionVersions WHERE documentId = $1)) and is semantically "does this document have any legacy field-extraction version row?" — exactly the check the mutual-exclusivity guard and AssignSchema legacy-extraction rejection need. Do not add a new DocumentHasLegacyExtractions query. Call HasFieldExtraction from the service layer. If readability at the call site is a concern, wrap the call in a private service-layer helper named documentHasLegacyExtractions(ctx, id) that just forwards to HasFieldExtraction — no new SQL, no new generated code -- test: service unit test confirms AssignSchema and the metadata write path both reject documents that have an existing documentFieldExtractionVersions row
task generate clean -- test: idempotent

v4: No delete-cascade queries are added in this milestone. DeleteDocumentCustomMetadata, NullifyDocumentCustomSchemaId, and DeleteClientMetadataSchemas from v3 are not created -- the FK cascades replace them.

2.3a Extend `internal/document` backend for `GET /document/{id}` enrichment

Why this subsection exists: Plan Section 5.2 declares that GET /document/{id} gains two new response fields (customSchemaId, hasCustomMetadata). The controller-side builder in §2.7 reads those values out of the document.DocumentEnriched struct, but as of the start of Milestone 2 the struct has neither field and the GetDocumentEnriched SQL query (internal/database/queries/document.sql:63) does not return either column. Without the three changes below, the §2.7 builder has nothing to read and the §2.8 integration tests will fail on the enrichment assertions. Do this subsection before §2.7.

Extend the GetDocumentEnriched query in internal/database/queries/document.sql to add d.custom_schema_id to the SELECT list and a correlated EXISTS(SELECT 1 FROM document_custom_metadata WHERE document_id = d.id) AS has_custom_metadata subquery — no join to document_custom_metadata (the EXISTS keeps the plan flat and index-friendly). -- test: unit test against testcontainer DB asserts the query returns the two new columns with the right values for (a) a document with no schema bound, (b) a document with a schema bound but zero metadata rows, (c) a document with a schema bound and >=1 metadata rows
task db:generate clean — confirm the generated GetDocumentEnriched row type in internal/database/repository/ now has CustomSchemaID and HasCustomMetadata fields -- test: go build ./internal/database/repository/... clean; grep -n CustomSchemaID internal/database/repository/document.sql.go finds the new field
Extend the DocumentEnriched struct in internal/document/service.go to add CustomSchemaID *uuid.UUID (nullable because most documents are not schema-bound) and HasCustomMetadata bool. Place them adjacent to HasTextRecord and FileSizeBytes so the field layout of the struct stays topically grouped -- test: go build ./internal/document/... clean; unit test instantiates the struct and asserts the two new fields zero-value to nil and false
Update GetEnriched in internal/document/get.go to copy docEnriched.CustomSchemaID and docEnriched.HasCustomMetadata from the extended generated row into the returned DocumentEnriched struct — no separate follow-up query, no extra round-trip to the DB -- test: service-level unit test against testcontainer DB asserts the returned struct carries the right values for all three fixture cases (no schema, schema no metadata, schema with metadata). Assert in a second test that the function issues exactly the same number of DB queries before and after the change (i.e., no regression from "one enriched read" to "enriched read + follow-up exists check")
Run existing tests for internal/document/ and api/queryAPI/document* — every pre-existing test that reads DocumentEnriched must still pass without modification; the two new fields are additive. -- test: go test ./internal/document/... ./api/queryAPI/... -run Document green

2.4 Service layer - metadata + document binding

Add CustomMetadata, MetadataVersion, SetMetadataInput types to models.go -- test: compiles. SetMetadataInput has no CreatedBy field. CustomMetadata carries SchemaID, SchemaName, SchemaVersion populated from the joined query in 2.3 (Section 7.1 of the plan has the struct shape).
SetDocumentMetadata(ctx, input, actor) -- derives schema from documents.custom_schema_id, never trusts client -- test: happy path
SetDocumentMetadata rejects when document has no custom_schema_id -- test: TDD (400)
SetDocumentMetadata rejects when document has legacy extractions -- test: TDD (409)
SetDocumentMetadata validates metadata against schema via validator.ValidateMetadata -- test: TDD with conforming and non-conforming payloads
SetDocumentMetadata enforces MaxMetadataPayloadBytes -- test: TDD
SetDocumentMetadata serializes version assignment by taking LockDocumentForMetadataWrite on the parent row, then GetMaxDocumentMetadataVersion, then INSERT version = max+1 -- test: TDD confirms the parent-row lock is acquired before the version read
SetDocumentMetadata concurrency: two concurrent writers against the same document that already has >=1 metadata row -- both succeed, produce distinct versions, no unique-violation -- test: TDD with goroutines against testcontainer DB
SetDocumentMetadata concurrency: two concurrent first writers against a document with zero metadata rows -- both succeed, produce version=1 and version=2 in some order -- test: TDD with goroutines, proves the parent-row lock is what serializes the first write
SetDocumentMetadata retries once on UNIQUE (document_id, version) violation as a belt-and-suspenders fallback -- test: TDD with an injected unique-violation on the first INSERT attempt
SetDocumentMetadata writes metadata.write audit entry -- test: TDD
GetCurrentMetadata(ctx, documentID) -- test: uses the LIMIT 1 query, returns latest
GetMetadataByVersion(ctx, documentID, version) -- test: TDD
GetMetadataHistory(ctx, documentID) with pagination -- test: TDD
Add AssignSchemaResult type to models.go per plan Section 7.1 (DocumentID uuid.UUID, CustomSchemaID *uuid.UUID, SchemaName *string, SchemaVersion *int) -- test: compiles
AssignSchema(ctx, documentID uuid.UUID, schemaID *uuid.UUID, actor string) (*AssignSchemaResult, error) — signature takes schemaID as a pointer so nil clears the binding and returns an AssignSchemaResult with all three schema fields nil -- test: happy path single-document assignment returns a populated result
AssignSchema populates SchemaName and SchemaVersion on the result from the same schema row read it already does for the active / same-client validation — no new query is added for this decoration -- test: TDD asserts result fields match the target schema row
AssignSchema takes SELECT id, "clientId", custom_schema_id FROM documents WHERE id = $1 FOR UPDATE as the first statement inside the transaction, before any legacy-extraction existence check or the UPDATE documents -- this is the symmetric parent-row lock for the mutual-exclusivity invariant described in plan Section 7.3 -- test: TDD; code review confirms the lock precedes all other DML
AssignSchema rejects if schema and document belong to different clients -- the composite FK does this at the DB layer; the app-layer check converts the FK violation to a clean 409 -- test: TDD
AssignSchema rejects if schema status is not active -- test: TDD
AssignSchema rejects if document already has custom metadata (matches Trigger 1 behavior; expect 409) -- test: TDD
AssignSchema rejects if document has legacy extractions (service-layer check under the parent-row lock plus Trigger 1 as backstop; expect 409) -- test: TDD
AssignSchema allows setting schemaID == nil to clear binding when no metadata exists; result has CustomSchemaID, SchemaName, SchemaVersion all nil -- test: TDD
AssignSchema writes schema.assign audit entry -- test: TDD
go test ./internal/customschema/... green -- test: full package green for Milestones 1+2 service methods

2.5 Legacy field extraction guard + controller 409 mapping

v4 concurrency requirement: A non-locking SELECT documents.custom_schema_id is NOT sufficient. Plan Section 7.3 walks through the race: two transactions each read a stale snapshot under READ COMMITTED, both pass the pre-check, and both commit opposing writes. The fix is a SELECT ... FOR UPDATE on the parent documents row as the first statement inside the tx, mirrored by the same lock in AssignSchema. Trigger 2 (Milestone 2.2) is the DB-layer backstop.

Locate the existing FieldExtractionService.CreateFieldExtraction entry point (internal/fieldextraction/service.go:27) -- test: grep and read code
Add a new SQLC query LockDocumentForLegacyExtractionWrite (SELECT id, custom_schema_id FROM documents WHERE id = @document_id FOR UPDATE) in internal/database/queries/document.sql -- test: generated code compiles
Modify CreateFieldExtraction: after Begin(ctx) and before any other DML, call LockDocumentForLegacyExtractionWrite. The existing AddFieldExtraction call must move to after the lock acquisition. -- test: code review confirms no DML runs before the lock
If the lock query returns pgx.ErrNoRows, return a typed sentinel error ErrDocumentNotFound (controller maps to 404) -- test: TDD service test
If the locked row has custom_schema_id IS NOT NULL, return a typed sentinel error ErrMutualExclusivityViolation -- test: TDD service test asserts errors.Is(err, ErrMutualExclusivityViolation)
Service-layer concurrency test: two goroutines, one calls AssignSchema (which will itself take FOR UPDATE -- see 2.4) and the other calls CreateFieldExtraction. Run both orderings. Whichever commits second must fail -- AssignSchema-then-CreateFieldExtraction fails with ErrMutualExclusivityViolation; CreateFieldExtraction-then-AssignSchema fails with the Trigger 1 EXCEPTION converted to 409. Neither ordering may leave the document in the "both systems active" state. -- test: TDD with goroutines against testcontainer DB; assertion checks final DB state in addition to returned errors
Service-layer concurrency test (DB trigger backstop): deliberately skip the FOR UPDATE call path in a test-only harness, run the same race, and confirm Trigger 2 still catches it by surfacing a PostgreSQL check_violation error with SQLSTATE 23514. This proves the defense-in-depth works even if the service-layer lock is ever removed by a regression. -- test: TDD
Regression guard (v4 — not a present-day audit): at the time of this plan, FieldExtractionService has exactly one mutator — CreateFieldExtraction at internal/fieldextraction/service.go:27. The other four methods (GetCurrentFieldExtraction, GetFieldExtractionHistory, GetFieldExtractionArrayFields, GetFieldExtractionByVersion) are reads. v3 carried a generic "audit every other mutation entry point" task that pointed at methods (UpdateFieldExtraction, AppendValueTo*) that do not exist and have never existed. Instead of re-running a phantom audit, add a regression test that enforces the rule going forward: list the exported methods of FieldExtractionService via reflection (or, simpler, a hand-maintained allow-list), and fail the test if any new exported method with a mutating name prefix (Create, Update, Append, Delete, Insert, Set, Reset) is added without being added to a second allow-list that says "this method has been reviewed and it takes LockDocumentForLegacyExtractionWrite before any DML." -- test: unit test in internal/fieldextraction/ that fails loudly with a message pointing at this tracking line if a new mutator is added without the review flag
Existing field extraction happy-path tests still green -- test: go test ./internal/fieldextraction/...
Update CreateFieldExtraction handler in api/queryAPI/fieldextractions.go to branch on ErrMutualExclusivityViolation -> echo.NewHTTPError(http.StatusConflict, ...) and ErrDocumentNotFound -> 404 -- test: handler unit tests assert both status codes and body messages
Handler also converts a PostgreSQL check_violation (SQLSTATE 23514) surfaced from Trigger 2 into the same 409 response so a future bypass of the service-layer lock still returns the documented error shape -- test: TDD handler test with a fabricated *pgconn.PgError
Preserve the existing mapping of non-sentinel service errors to 500 -- test: TDD handler test with a generic error path
End-to-end HTTP test: create document, assign custom schema, call POST /field-extractions, assert response status is 409 and JSON body carries the documented message -- test: integration against testcontainer DB
End-to-end HTTP race test: fire POST /super-admin/documents/{id}/schema and POST /field-extractions concurrently against the same document, assert the final DB state has exactly one of custom_schema_id IS NOT NULL XOR a documentFieldExtractionVersions row, never both -- test: integration

2.6 OpenAPI spec - metadata + document binding

Add CustomMetadataService tag with description stating client_user + user_admin + super_admin access -- test: spec lints
CustomMetadataRequest (no createdBy) -- test: codegen
CustomMetadataResponse -- test: codegen
CustomMetadataHistoryResponse -- test: codegen
ValidationErrorResponse -- test: codegen
DocumentSchemaAssignRequest (no createdBy) -- test: codegen
DocumentSchemaAssignResponse -- test: codegen
v4: Modify DocumentEnriched to add only customSchemaId and hasCustomMetadata. Do NOT add customSchemaName or customMetadata. Do NOT add a new ?customMetadata=true query parameter. -- test: codegen produces exactly two new fields
PATCH /super-admin/documents/{id}/schema path -- test: codegen
GET /custom-metadata path -- test: codegen
POST /custom-metadata path -- test: codegen
GET /custom-metadata/version path -- test: codegen
GET /custom-metadata/history path -- test: codegen
task generate clean -- test: no errors

2.7 Controllers - metadata + assign schema + document enrichment

Create api/queryAPI/custommetadata.go skeleton -- test: compiles
v4 actor extraction: POST /custom-metadata derives actor identity from the JWT and passes it to the service as an argument -- test: TDD handler test asserts body createdBy (if provided) does not override
POST /custom-metadata handler -- test: TDD
GET /custom-metadata handler -- test: TDD
GET /custom-metadata/version handler -- test: TDD
GET /custom-metadata/history handler -- test: TDD
Add PATCH /super-admin/documents/{id}/schema handler to api/queryAPI/customschemas.go -- derives actor -- test: TDD
Extend DocumentEnriched builder to include customSchemaId and hasCustomMetadata unconditionally — reads the values directly from the document.DocumentEnriched struct (populated in §2.3a); no new DB query or service call at this layer. -- test: TDD
Document without any custom schema returns customSchemaId: null, hasCustomMetadata: false -- test: TDD
Document with schema bound but no metadata returns customSchemaId: <id>, hasCustomMetadata: false -- test: TDD
Document with schema + metadata returns populated fields -- test: TDD
Authorization: confirm GET /document/{id} still uses the existing document resource, not super-admin -- test: role matrix test in 2.8
go build ./... clean -- test: compile

2.8 Milestone 2 integration tests

End-to-end: create schema -> assign to document (PATCH) -> POST valid metadata -> GET /custom-metadata?documentId=... returns the stored row with populated schemaId, schemaName, schemaVersion (from the §2.3 joined query) and the createdBy field equal to the JWT subject UUID -- test: integration
End-to-end: POST invalid metadata -> 400 with validation errors -- test: integration
End-to-end: POST metadata to document without assigned schema -> 400 -- test: integration
End-to-end: assign schema to document that has legacy extractions -> 409 -- test: integration
End-to-end: write legacy field extraction to document that has custom schema -> 409 -- test: integration
End-to-end GET /custom-metadata/version: POST metadata three times on the same document (version 1, 2, 3), then GET /custom-metadata/version?documentId=...&version=2 returns exactly the v2 payload (not v1, not v3) with populated schemaId/schemaName/schemaVersion from the §2.3 joined query -- test: integration
End-to-end GET /custom-metadata/version not-found: version=99 on a document with only 3 versions returns 404; documentId for a non-existent document returns 404; version < 1 (e.g. 0) returns 400 (plan Section 5.4 declares the parameter >= 1) -- test: integration per case
End-to-end GET /custom-metadata/history happy path: after three POSTs, GET /custom-metadata/history?documentId=... returns all three version summaries in descending version order and each summary carries version, createdAt, createdBy (subject UUID) -- test: integration
End-to-end GET /custom-metadata/history pagination: POST 5 metadata versions, call with limit=2&offset=0 (returns v5, v4), limit=2&offset=2 (returns v3, v2), limit=2&offset=4 (returns v1) — assert no duplicates and correct ordering across the page boundary -- test: integration
GET /custom-metadata/history edge cases (v4 — behavior committed in plan Section 5.4 edge-case table, not up to the implementer): one integration test per case, each asserting the exact status code and response body declared in the plan — - limit=0 → 400 with message "limit must be between 1 and 200" - limit=-1 → 400 with the same message - limit=201 → 200, clamped to 200 rows (seed 205 versions, assert exactly 200 returned, newest first) - offset=-1 → 400 with message "offset must be >= 0" - documentId = random non-existent UUID → 404 with message "document not found" - documentId exists but has zero metadata rows → 200 with {"versions": []} (and Content-Type: application/json) -- test: integration per case
Authorization matrix for Milestone 2 endpoints: one integration test per (role, route, method) cell across all four /custom-metadata routes (GET, POST, GET /version, GET /history) plus PATCH /super-admin/documents/{id}/schema. Expected matrix: super_admin positive on every cell; user_admin 403 on PATCH /super-admin/... and positive on all four /custom-metadata routes; client_user 403 on PATCH /super-admin/... and positive on all four /custom-metadata routes; auditor 403 on PATCH /super-admin/... and on POST /custom-metadata, positive on the three /custom-metadata GET routes (GET, GET /version, GET /history) -- test: one integration test per (role, route, method) cell, each asserting status + DB unchanged on denial
v4 FK-cascade delete test: create document with custom metadata -> call existing DeleteDocumentCascade -> verify document_custom_metadata rows for that document are gone (via the ON DELETE CASCADE, without touching the delete code path) -- test: integration
v4 FK-cascade client delete test: create client with schemas and a document with custom metadata -> call client.HardDelete -> verify document_custom_metadata and client_metadata_schemas rows are gone -- test: integration
Metadata payload at 1MB boundary: accept 1048576, reject 1048577 -- test: integration
Concurrency: two concurrent first-writers against same document succeed -- test: integration (service test in 2.4 covers it; re-verify at HTTP level)
task fullsuite:ci clean -- test: full run
Journal entry: Milestone 2 exit snapshot

2.9 Pre-partitioning carve-out for M3/M4 parallel execution

Purpose: Milestones 3 and 4 both add handler code and service-layer types that were originally drafted into api/queryAPI/customschemas.go and internal/customschema/models.go. If both milestones run in parallel on separate branches against those two shared files, the merge at the end produces conflicts at the seam. This section creates four empty stub files so each milestone owns disjoint files from the start. Do this once, at the end of Milestone 2, before branching M3 and M4 onto separate workstreams. If M3 and M4 will be run serially, this section can be skipped (the handlers can land in customschemas.go directly); only commit the stubs if parallel execution is actually planned.

Create api/queryAPI/customschemas_reset.go as an empty stub containing only package queryAPI -- test: go build ./... clean
Create api/queryAPI/customschemas_bulk.go as an empty stub containing only package queryAPI -- test: go build ./... clean
Create internal/customschema/models_reset.go as an empty stub containing only package customschema -- test: go build ./... clean
Create internal/customschema/models_bulk.go as an empty stub containing only package customschema -- test: go build ./... clean
Commit the four stubs on the shared base branch (the branch M3 and M4 will both fork from), so neither workstream has to create them -- test: git log shows the four files as committed before either milestone branch exists
Update Milestone 3 owner and Milestone 4 owner: M3 only edits customschemas_reset.go and models_reset.go; M4 only edits customschemas_bulk.go and models_bulk.go; neither touches customschemas.go or models.go for new types, only for imports and registration wiring if required -- test: code review of each milestone's final diff

Milestone 3 - Administrative completion

Goal: Reset-metadata is exposed so super_admin can upgrade a document to a newer schema version. Schema version-creation concurrency is tested end-to-end. The full authorization matrix (including reset-metadata and version-creation) is locked in.

Dependencies: Milestone 2 complete.

Exit criteria:

Full upgrade flow works: assign v1 -> write metadata -> create v2 -> reset -> assign v2 -> write new metadata.
Reset is atomic, idempotent, rejects legacy documents, returns 404 on missing document.
task fullsuite:ci green.

3.1 SQLC queries for reset-metadata

LockDocumentForReset (SELECT id, "clientId", custom_schema_id FROM documents WHERE id = @document_id FOR UPDATE) -- test: generated
GetDocumentSchemaBindingForReset (join documents -> client_metadata_schemas, returns (custom_schema_id, schema_name, schema_version) or all-null) -- test: generated
CountDocumentCustomMetadataVersions -- test: generated
v4: DeleteDocumentCustomMetadataForReset (DELETE FROM document_custom_metadata WHERE document_id = @document_id) -- owned by the reset path, name disambiguates from any future cascade query -- test: generated
v4: NullifyDocumentCustomSchemaIdForReset (UPDATE documents SET custom_schema_id = NULL WHERE id = @document_id) -- owned by the reset path -- test: generated
task generate clean -- test: idempotent

3.2 Service method - ResetDocumentMetadata

Add ResetMetadataResult type to internal/customschema/models_reset.go (the M3-owned stub from §2.9; fall back to internal/customschema/models.go only if M3/M4 will run serially and §2.9 was skipped) with DocumentID, PreviousSchemaID *uuid.UUID, PreviousSchemaName *string, PreviousSchemaVersion *int, MetadataVersionsDeleted int, ResetAt time.Time, ResetBy string -- test: compiles
ResetDocumentMetadata(ctx, documentID, actor) -- single transaction: LockDocumentForReset, GetDocumentSchemaBindingForReset, CountDocumentCustomMetadataVersions, DeleteDocumentCustomMetadataForReset, NullifyDocumentCustomSchemaIdForReset, audit log -- test: happy path wipe with schema bound + N metadata versions
Reset returns populated previous* fields when document had a schema -- test: TDD
Reset returns metadataVersionsDeleted matching actual deleted count -- test: TDD
Reset is idempotent: already-clean document returns metadataVersionsDeleted: 0, previousSchemaId: nil -- test: TDD
Reset rejects documents with legacy field extractions (409) -- test: TDD
Reset returns 404 on non-existent document -- test: TDD
Reset atomicity: inject failure between DELETE and UPDATE -- test: transaction rolls back, metadata rows still present
Reset + reassign + write: after reset, Trigger 1 still fires on second schema change under new metadata -- test: TDD regression
(v3's "Trigger 3 regression" test is removed in v4: the table has no schema_id column. Replace with a structural test that confirms document_custom_metadata has no schema_id column after migrations.) -- test: TDD schema-structure test
Reset writes metadata.reset audit log entry containing actor, documentID, previousSchemaId, metadataVersionsDeleted -- test: TDD reads the audit sink
Reset concurrency: two concurrent resets on same document serialize via FOR UPDATE -- test: TDD
Reset concurrency: reset vs AssignSchema on same document serialize -- test: TDD
v4 Reset concurrency: reset vs SetDocumentMetadata on same document serialize. This is the pair that matters most in practice: plan Section 7.1 / 8.2 say LockDocumentForReset and LockDocumentForMetadataWrite use the same parent-row SELECT ... FOR UPDATE shape specifically so these two operations cannot interleave. Test both orderings with goroutines against testcontainer DB: 1. reset-then-write: start reset (holds FOR UPDATE), race SetDocumentMetadata for the same document. The writer must block until reset commits; once it unblocks it must observe custom_schema_id = NULL and return the "no schema assigned" error (400/409), not a successful write into a stale schema. Assert final DB state has zero metadata rows for the document. 2. write-then-reset: start SetDocumentMetadata (holds FOR UPDATE, inserts a new metadata row), race reset for the same document. The reset must block until the write commits; once it unblocks it must delete the just-written row and return metadataVersionsDeleted >= 1. Assert final DB state has zero metadata rows and custom_schema_id = NULL. Neither ordering may leave an orphaned metadata row or a partially-reset document. -- test: TDD with goroutines against testcontainer DB; assertions cover both ordering outcomes

3.3 OpenAPI + handler - reset-metadata

DocumentMetadataResetResponse schema (no request body schema needed; empty body) -- test: codegen (NOTE: named DocumentMetadataResetResponse not ResetDocumentMetadataResponse to avoid codegen name conflict; see journal 2026-04-15)
POST /super-admin/documents/{id}/reset-metadata path -- test: codegen
task generate clean -- test: no errors
Add handler to api/queryAPI/customschemas_reset.go (the M3-owned stub from §2.9; fall back to api/queryAPI/customschemas.go only if M3/M4 will run serially and §2.9 was skipped) -- parses document ID, derives actor from JWT, calls service, maps errors -- test: TDD
Handler maps success to 200 with ResetDocumentMetadataResponse -- test: TestM3_ResetDocumentMetadata_HappyPath + TestM3_ResetDocumentMetadata_IdempotentOnCleanDoc
Handler maps "document not found" to 404 -- test: TestM3_ResetDocumentMetadata_DocumentNotFound + TestMapResetError/document_not_found
Handler maps "legacy extractions exist" to 409 -- test: TestM3_ResetDocumentMetadata_LegacyExtractionsConflict + TestMapResetError/mutual_exclusivity_violation
Handler maps unexpected DB errors to 500 -- test: TestMapResetError/unknown_error_returns_500

3.4 Milestone 3 integration tests

End-to-end upgrade flow: bind schema v1 -> POST metadata -> POST .../versions to create v2 -> reset-metadata -> PATCH schema to v2 -> POST new metadata with added optional field -- test: TestM3_FullUpgradeFlow in api/queryAPI/customschemas_reset_test.go
End-to-end: reset on already-clean document returns 200 with metadataVersionsDeleted: 0 -- test: TestM3_ResetDocumentMetadata_IdempotentOnCleanDoc
Schema version concurrency: two concurrent POST .../versions calls for same (client_id, name) produce distinct versions -- test: TestM3_SchemaVersionConcurrency in api/queryAPI/customschemas_integration_test.go
Authorization matrix for reset-metadata and version creation: super_admin positive; user_admin 403; client_user 403; auditor 403 -- test: TestM3_AuthMatrix_Reset in api/queryAPI/customschemas_reset_test.go (4 subtests: super_admin/user_admin/client_user/auditor)
task fullsuite:ci clean -- test: full run (85.4%, All coverage checks passed!, no FAIL lines, 2026-04-15)
Journal entry: Milestone 3 exit snapshot (2026-04-15)

Milestone 4 - Bulk operations

Goal: Bulk folder schema assignment is available for administrative convenience.

Dependencies: Milestone 2 complete. Milestone 4 only needs the single-document AssignSchema semantics (committed in Milestone 2), not the ResetDocumentMetadata work in Milestone 3 — nothing in the recursive folder walk reads or writes the reset path. Milestones 3 and 4 can run in parallel on separate workstreams only after the shared files are pre-partitioned: without that step, both milestones would edit api/queryAPI/customschemas.go and internal/customschema/models.go concurrently and produce merge conflicts at the seam. The pre-partition carve-out (see section 2.9 below) creates four new stub files so each milestone owns disjoint files: M3 owns api/queryAPI/customschemas_reset.go and internal/customschema/models_reset.go; M4 owns api/queryAPI/customschemas_bulk.go and internal/customschema/models_bulk.go. With the carve-out in place the merge gate is task fullsuite:ci green after both land, and serializing M4 behind M3 adds roughly one milestone of wall-clock time for no technical reason. Without the carve-out, serialize M4 behind M3 as the safe fallback.

Exit criteria:

Recursive walk covers a 3-deep folder tree correctly, handles all skip reasons, enforces the 10K document cap, runs in a single transaction.
task fullsuite:ci green.

4.1 SQLC queries for bulk folder assignment

BulkSetDocumentCustomSchemaIdInFolderTree (WITH RECURSIVE) -- test: generated; recursive CTE compiles
GetDocumentsWithMetadataInFolderTree -- test: generated
GetDocumentsWithLegacyExtractionsInFolderTree -- test: generated
Reuse existing query: CountDocumentsInFolderTree already exists in internal/database/queries/folders.sql (a recursive CTE over folders + documents.folderId) and is exactly the shape the MaxBulkAssignDocuments pre-check needs. Do not add a new query; call the existing one from the Milestone 4 service method. Task: unit test that confirms the bulk service's cap check is wired to the existing CountDocumentsInFolderTree generated method -- test: unit test asserts the right method is called and returns a count that matches a seeded fixture
task generate clean -- test: idempotent

4.2 Service method - AssignSchemaToFolder

Add BulkAssignResult, SkippedDocument types to internal/customschema/models_bulk.go (the M4-owned stub from §2.9; fall back to internal/customschema/models.go only if M3/M4 will run serially and §2.9 was skipped) -- test: compiles. No createdBy on the input.
AssignSchemaToFolder(ctx, folderID, schemaID, actor) -- walks subtree with WITH RECURSIVE and returns BulkAssignResult -- test: happy path on a 3-deep tree
Bulk assignment skips documents with different schema + existing metadata with SkippedDocument reason -- test: TDD
Bulk assignment skips documents with legacy extractions with reason -- test: TDD
Bulk assignment no-ops documents already on target schema with reason -- test: TDD
Bulk assignment reassigns documents with different schema but no metadata -- test: TDD
Bulk assignment returns 422 when subtree exceeds MaxBulkAssignDocuments -- test: TDD (constant override for test)
Bulk assignment runs in single transaction; simulated mid-walk failure rolls back everything -- test: TDD
Bulk assignment writes folder.assign_schema audit entry with counts -- test: TDD

4.3 OpenAPI + handler - bulk folder assignment

BulkSchemaAssignRequest schema (no createdBy) -- test: codegen
BulkSchemaAssignResponse schema -- test: codegen
POST /super-admin/folders/{folderId}/assign-schema path -- test: codegen
task generate clean -- test: no errors
Handler in api/queryAPI/customschemas_bulk.go (the M4-owned stub from §2.9; fall back to api/queryAPI/customschemas.go only if M3/M4 will run serially and §2.9 was skipped) wired to AssignSchemaToFolder -- derives actor from JWT -- test: compiles

4.4 Milestone 4 integration tests

Recursive walk on a 3-deep folder tree produces correct documentsUpdated count -- test: integration
Skip reason: "different schema + existing metadata" -- test: integration
Skip reason: "already assigned to same schema (no-op)" -- test: integration
Skip reason: "has legacy field extractions" -- test: integration
Reassignment on document with different schema but no metadata -- test: integration
Subtree > 10K documents returns 422 with documented message -- test: integration (constant override for test)
Bulk folder exactly 10000 docs: succeeds -- test: integration
Empty folder returns 200 with 0 counts -- test: integration
Deeply nested (10+ levels) tree walks correctly -- test: integration
Authorization matrix for bulk folder endpoint: super_admin positive; user_admin 403; client_user 403; auditor 403 -- test: integration per cell
task fullsuite:ci clean -- test: full run
Journal entry: Milestone 4 exit snapshot

Documentation (parallel with Milestones 3 + 4)

Goal: Every doc that references endpoints, env vars, authorization, or the feature is updated. Dependencies: Milestone 2 complete (feature is shippable in narrow form); documentation can track Milestones 3 and 4 in real time.

Update docs/ai.generated/README.md index if new doc files added -- test: review (entry 11 added pointing to 11-custom-metadata-guide.md)
Authorization matrix table published in a dedicated doc page (include the v4 change that user_admin has no schema CRUD) -- test: review (11-custom-metadata-guide.md Authorization Matrix section)
Sample schemas from Appendix A of the plan published as copyable examples -- test: review (11-custom-metadata-guide.md Schema Definition Requirements section, two examples)
CustomMetadataService usage guide (end-user perspective) -- test: review (11-custom-metadata-guide.md End-User Workflow section + 03-api-documentation.md CustomMetadataService section)
SuperAdminSchemaService usage guide (platform-operator perspective), including the POST /versions route -- test: review (11-custom-metadata-guide.md Platform Operator Workflow + 03-api-documentation.md SuperAdminSchemaService section)
Reset-metadata runbook: "How to upgrade a document to a newer schema version" with the full wipe -> reassign -> re-POST flow and a warning about metadata history loss -- test: review (11-custom-metadata-guide.md Schema Upgrade Runbook section)
Mutual exclusivity explanation (legacy vs custom) in the document processing guide -- test: review (11-custom-metadata-guide.md Mutual Exclusivity section)
v4: Note that DeleteDocumentCascade and client.HardDelete do NOT need custom-metadata-specific steps because of the FK cascades -- test: review (11-custom-metadata-guide.md Cascade Deletion Behavior section)
Permit.io deployment ordering note (run setup tool before deploying controllers) added to the deployment runbook -- test: review (11-custom-metadata-guide.md Permit.io Deployment Note section)
v4: Document the HTTP method choice: POST /super-admin/custom-schemas/{schemaId}/versions, not PUT -- test: review (03-api-documentation.md and 11-custom-metadata-guide.md Schema Versioning section)
v4: Document that createdBy is not accepted from request bodies; actor comes from the JWT -- test: review (03-api-documentation.md Notes on POST /super-admin/custom-schemas; 11-custom-metadata-guide.md Actor Fields section)
Any new env vars? (none expected for v4; confirm.) -- test: review (confirmed: no new env vars in v4)
Generate new API status report / architecture diagrams if the pipeline regenerates them -- test: review
Journal entry marking the feature shipped

Cross-milestone reminders

Before starting any milestone, re-read the relevant section of plans/mutable.metadata.plan.combo.v4.md.
TDD is mandatory per project rules: write the test, watch it fail, make it pass. Never write multiple tests before running any.
No mocks without Q's explicit permission. Use testcontainers for real Postgres.
Keep the journal current. When a checkbox flips to [x], add a timestamped line to ./journals/implement_mutableMetadata.md with the commit SHA and test command that proved it.
If any migration, trigger, FK, or service invariant seems to conflict with the plan, STOP and ask Q before modifying it. The defense-in-depth design depends on the composite FK and Trigger 1 being exactly as written.
v4 reminder — things NOT to add (they are v3 leftovers and mean you are regressing):
- Do NOT add a schema_id column to document_custom_metadata.
- Do NOT add v3's trg_validate_schema_client_match trigger — it is replaced by the composite FK fk_documents_custom_schema_same_client (plan Section 4.7, Milestone 1.3).
- Do NOT add v3's trg_enforce_consistent_schema_id trigger — it is structurally unnecessary because document_custom_metadata has no schema_id column in v4.
- Reminder: v4 DOES add two triggers — trg_prevent_schema_reassignment (v3's Trigger 1, renamed for clarity) and trg_prevent_legacy_extraction_on_custom_document (new in v4, see Section 2.2). Those are correct and must be present.
- Do NOT add a current_document_custom_metadata view.
- Do NOT add DeleteDocumentCustomMetadata / NullifyDocumentCustomSchemaId / DeleteClientMetadataSchemas to the existing delete code paths.
- Do NOT add a PUT /super-admin/custom-schemas/{schemaId} route.
- Do NOT accept createdBy in new request bodies.
- Do NOT add customSchemaName or customMetadata to GET /document/{id}.
- Do NOT declare format: email on createdBy / resetBy in the new OpenAPI response schemas — those are plain-string Cognito subject IDs (plan Section 8.3 "Actor fields").
- If any of those appear in your diff, something has regressed to v3.

72 KiB Raw Blame History Unescape Escape

Mutable Metadata v4 - Implementation Tracking

How v4 differs from v3 tracking

How to use this document

Milestone roll-up (high-level)

Milestone 0-INT — Fix fullsuite:ci regression — COMPLETE (2026-04-15)

Milestone 1 - Schema foundation — COMPLETE (2026-04-14)

1.1 Migration 127 - schema_status_type enum + client_metadata_schemas

1.2 Schema validator component + dependency

1.3 Migration 128 - documents.custom_schema_id column + composite FK

1.4 SQLC queries for schema CRUD

1.5 Audit sink

1.6 Service layer - schema CRUD

1.7 OpenAPI spec - schema CRUD

1.8 Permit.io policy file + setup tool run

1.9 Schema CRUD controllers

1.10 Milestone 1 integration tests

Milestone 2 - Core document flow

2.1 Migration 129 - document_custom_metadata table

2.2 Migration 130 - schema invariant triggers (v4: Trigger 1 + Trigger 2)

2.3 SQLC queries for metadata + document binding

2.3a Extend internal/document backend for GET /document/{id} enrichment

2.4 Service layer - metadata + document binding

2.5 Legacy field extraction guard + controller 409 mapping

2.6 OpenAPI spec - metadata + document binding

2.7 Controllers - metadata + assign schema + document enrichment

2.8 Milestone 2 integration tests

2.9 Pre-partitioning carve-out for M3/M4 parallel execution

Milestone 3 - Administrative completion

3.1 SQLC queries for reset-metadata

3.2 Service method - ResetDocumentMetadata

3.3 OpenAPI + handler - reset-metadata

3.4 Milestone 3 integration tests

Milestone 4 - Bulk operations

4.1 SQLC queries for bulk folder assignment

4.2 Service method - AssignSchemaToFolder

4.3 OpenAPI + handler - bulk folder assignment

4.4 Milestone 4 integration tests

Documentation (parallel with Milestones 3 + 4)

Cross-milestone reminders

72 KiB

Raw Blame History

1.1 Migration 127 - `schema_status_type` enum + `client_metadata_schemas`

1.3 Migration 128 - `documents.custom_schema_id` column + composite FK

2.1 Migration 129 - `document_custom_metadata` table

2.3a Extend `internal/document` backend for `GET /document/{id}` enrichment