jacob-mathison/query-orchestration
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity
Files
7b820b18c2b4ec316f1d9ee2f509abc6623985b9
query-orchestration/plans/mutable.metadata.plan.combo.v3.tracking.md
T
Jay Brown 17fc813823 Merged in feature/mutable-metadata1 (pull request #221) 
M1, M2 and M3 complete

* M1, M2 and M3 complete

* review changes

* docs

* docs
2026-04-16 23:11:26 +00:00

46 KiB
Raw Blame History

Mutable Metadata v3 - Implementation Tracking

Source plan: plans/mutable.metadata.plan.combo.v3.md Journal (details / decisions / blockers): ./journals/implement_mutableMetadata.md Purpose: One-glance status of every task needed to ship v3. Each checkbox is a discrete, verifiable unit of work. TDD rule: write the test first, watch it fail, make it pass, check the box. If a task requires Q's input or produces a decision, log it to the journal and keep the checkbox state accurate here.

How to use this document

  • [ ] = not started
  • [~] = in progress
  • [x] = done AND tests passing (task fullsuite:ci clean for the affected code paths)
  • [!] = blocked -- add a short (blocked: <why>) suffix and open a journal entry
  • [-] = consciously skipped -- add a short (skipped: <why>) suffix
  • Every task line ends with a test: marker indicating where the passing test(s) live or what test type proves it.
  • When a whole phase is green, flip the phase header to COMPLETE and date-stamp it.
  • Never mark a phase complete until task fullsuite:ci ends with All coverage checks passed! with no FAIL lines.

Phase roll-up (high-level)

Phase Title Status Started Completed
1 Database migrations NOT STARTED
2 SQLC queries + generate NOT STARTED
3 Schema validator component NOT STARTED
4 Service layer (schema CRUD, metadata, assign, bulk, reset) NOT STARTED
5a Legacy field extraction guard NOT STARTED
5b Delete cascade updates NOT STARTED
6 OpenAPI spec updates NOT STARTED
7a Permit.io policy file updates NOT STARTED
7b Run permit setup tool (dev/uat/prod) NOT STARTED
8 API controllers + codegen NOT STARTED
9 Document endpoint modifications NOT STARTED
10 Bulk folder schema assignment endpoint NOT STARTED
10b Reset-metadata endpoint (v3) NOT STARTED
11 Integration & authorization tests NOT STARTED
12 Documentation updates NOT STARTED

Phase 1 - Database migrations

Goal: Land all DDL needed for the feature. No Go code touched in this phase. Dependencies: None. Exit criteria: Migrations up/down cleanly on a fresh database AND on a database with existing data; task db:generate runs clean; triggers verifiable via direct SQL.

Migration numbering note: The repo already contains migrations through 00000000000126_batch_document_outcomes_delete_actions. v3 therefore starts at 127 to avoid filename collisions. If another branch lands new migrations before this feature merges, renumber this block to the next free sequence and update the references below.

1.1 Migration 127 - schema_status_type enum + client_metadata_schemas

  • Create internal/database/migrations/00000000000127_create_client_metadata_schemas.up.sql with CREATE TYPE schema_status_type AS ENUM ('active', 'superseded', 'retired') -- test: migration applies cleanly on empty DB
  • Same migration creates client_metadata_schemas table per Section 4.2 (columns, PK, FK to clients, schema_def jsonb, version, status, timestamps, created_by) -- test: \d client_metadata_schemas shows all columns
  • Add CONSTRAINT uq_client_schema_name_version UNIQUE (client_id, name, version) -- test: direct SQL INSERT of two rows with same (client_id, name, version) fails
  • Add CONSTRAINT chk_schema_def_size CHECK (pg_column_size(schema_def) <= 70000) -- test: insert of a schema_def larger than 70000 bytes fails with CHECK violation
  • Add indexes idx_cms_client_id, idx_cms_client_name, idx_cms_client_status -- test: \di client_metadata_schemas shows the three indexes
  • Create 00000000000127_create_client_metadata_schemas.down.sql dropping the table and enum in correct order -- test: up then down leaves DB state byte-identical to pre-migration
  • Run task db:generate after migration lands -- test: no errors

1.2 Migration 128 - document_custom_metadata table + view

  • Create 00000000000128_create_document_custom_metadata.up.sql with document_custom_metadata table per Section 4.3 (columns, FKs, metadata jsonb, version column) -- test: migration applies cleanly
  • Add CONSTRAINT uq_doc_custom_metadata_version UNIQUE (document_id, version) -- test: direct SQL duplicate (document_id, version) INSERT fails
  • Add indexes idx_dcm_document_id, idx_dcm_schema_id, idx_dcm_doc_version_desc -- test: indexes present
  • Create current_document_custom_metadata view per Section 4.5 (DISTINCT ON (document_id) ... ORDER BY document_id, version DESC) -- test: insert 3 versions for same doc, SELECT * FROM current_document_custom_metadata WHERE document_id = X returns only version 3
  • Create corresponding down.sql (drop view then table) -- test: up/down round trip clean

1.3 Migration 129 - Add custom_schema_id to documents

  • Create 00000000000129_add_custom_schema_id_to_documents.up.sql -- ALTER TABLE documents ADD COLUMN custom_schema_id uuid NULL REFERENCES client_metadata_schemas(id) -- test: column present, nullable, FK enforced
  • Create idx_documents_custom_schema_id -- test: index present
  • Write down.sql that drops index then column -- test: round trip clean
  • Verify existing documents rows have custom_schema_id = NULL after migration -- test: SELECT COUNT(*) FROM documents WHERE custom_schema_id IS NOT NULL returns 0

1.4 Migration 130 - Invariant triggers

  • Create 00000000000130_add_schema_invariant_triggers.up.sql -- test: migration applies cleanly
  • Define trg_prevent_schema_reassignment() function per Section 4.7 Trigger 1 -- test: function exists in pg_proc
  • Attach trigger to documents BEFORE UPDATE OF custom_schema_id -- test: trigger listed in \d documents
  • Trigger 1 unit test: insert document, assign schema, write custom metadata, attempt UPDATE custom_schema_id -- expect RAISE EXCEPTION with "custom metadata already exists" message
  • Trigger 1 unit test: insert document, add legacy extraction row, attempt UPDATE custom_schema_id -- expect RAISE EXCEPTION with "legacy field extractions already exist" message
  • Trigger 1 unit test: insert document with no metadata and no extractions, UPDATE custom_schema_id from schema_a to schema_b -- expect success (pre-binding change allowed)
  • Define trg_validate_schema_client_match() function per Section 4.7 Trigger 2 -- test: function exists
  • Attach trigger to documents BEFORE INSERT OR UPDATE OF custom_schema_id -- test: trigger listed
  • Trigger 2 unit test: attempt to assign a schema owned by client A to a document owned by client B -- expect exception
  • Trigger 2 unit test: attempt to assign a non-existent schema ID -- expect "Schema % does not exist"
  • Trigger 2 unit test: same-client assignment -- expect success
  • Define trg_enforce_consistent_schema_id() function per Section 4.7 Trigger 3 -- test: function exists
  • Attach trigger to document_custom_metadata BEFORE INSERT -- test: trigger listed
  • Trigger 3 unit test: insert metadata v1 under schema_a, attempt direct SQL INSERT of metadata v2 under schema_b for same document -- expect exception
  • Trigger 3 unit test: insert metadata v1 and v2 under same schema -- expect success
  • Write down.sql dropping triggers and functions in correct order -- test: up/down round trip clean

1.5 Phase 1 integration

  • Run task generate -- test: clean
  • Run task test:unit:short -- test: all existing tests still pass (no regression)
  • Commit Phase 1 on a dedicated branch or commit cluster; journal entry recording migration numbers used

Phase 2 - SQLC queries + code generation

Goal: All DB access the feature needs is expressed as SQLC-generated Go. Dependencies: Phase 1. Exit criteria: task generate clean; generated *.sql.go files compile; every query listed below exists and has a typed wrapper.

2.1 New file internal/database/queries/customschemas.sql

Schema CRUD queries:

  • CreateClientMetadataSchema -- INSERT returning full row -- test: generated code compiles, query name appears in *.sql.go
  • GetClientMetadataSchema (by id) -- test: generated
  • ListClientMetadataSchemas (filters: client_id required, name optional, status optional with default active, pagination limit/offset) -- test: generated
  • GetLatestSchemaByName -- test: generated
  • GetSchemaDocumentCount (count of documents.custom_schema_id = $1) -- test: generated
  • GetSchemaMetadataRecordCount (count of document_custom_metadata.schema_id = $1) -- test: generated
  • SetSchemaStatus (update status, used for supersede and retire) -- test: generated
  • GetMaxSchemaVersion (for auto-increment on PUT) -- test: generated
  • LockSchemaVersionsForName (SELECT ... FOR UPDATE on all versions of a (client_id, name) pair) -- test: generated

Metadata CRUD queries:

  • CreateDocumentCustomMetadata -- test: generated
  • GetCurrentDocumentCustomMetadata (via view) -- test: generated
  • GetDocumentCustomMetadataByVersion -- test: generated
  • GetDocumentCustomMetadataHistory (with limit/offset) -- test: generated
  • LockDocumentCustomMetadataForVersion (SELECT ... FOR UPDATE on max version) -- test: generated

Document-binding queries:

  • SetDocumentCustomSchemaId -- test: generated
  • GetDocumentCustomSchemaId -- test: generated
  • BulkSetDocumentCustomSchemaIdInFolderTree (WITH RECURSIVE) -- test: generated; recursive CTE compiles
  • GetDocumentsWithMetadataInFolderTree -- test: generated
  • GetDocumentsWithLegacyExtractionsInFolderTree -- test: generated
  • DocumentHasLegacyExtractions (boolean, single doc) -- test: generated
  • CountDocumentsInFolderTree (for MaxBulkAssignDocuments check) -- test: generated

Reset-metadata support queries (v3):

  • LockDocumentForReset (SELECT id, "clientId", custom_schema_id FROM documents WHERE id = $1 FOR UPDATE) -- test: generated
  • GetDocumentSchemaBindingForReset (join documents -> client_metadata_schemas, returns (custom_schema_id, schema_name, schema_version) or all-null) -- test: generated
  • CountDocumentCustomMetadataVersions -- test: generated

2.2 Query additions for delete cascade (Phase 5b prep)

  • Add DeleteDocumentCustomMetadata to document.sql -- test: generated
  • Add NullifyDocumentCustomSchemaId to document.sql -- test: generated
  • Add DeleteClientMetadataSchemas to client.sql -- test: generated

2.3 Phase 2 integration

  • task generate clean -- test: no diff after re-run (idempotent)
  • Generated files compile: go build ./... -- test: clean
  • Commit Phase 2 with journal entry

Phase 3 - Schema validator component

Goal: A standalone internal/customschema/validator.go that can meta-validate JSON Schema definitions and validate metadata payloads against a compiled schema. Dependencies: JSON Schema library added to go.mod. Exit criteria: Validator unit tests cover all supported types in Section 6.3 and the explicit-additionalProperties rule.

3.1 Dependency

  • Add github.com/santhosh-tekuri/jsonschema/v6 to go.mod -- test: go mod tidy clean, build clean

3.2 Constants file

  • Create internal/customschema/constants.go with MaxSchemaDefinitionBytes = 65536, MaxMetadataPayloadBytes = 1048576, MaxBulkAssignDocuments = 10000 -- test: constants referenced by tests

3.3 Validator implementation

  • Create internal/customschema/validator.go with SchemaValidator struct -- test: compiles
  • Implement ValidateSchemaDefinition(schemaDef json.RawMessage) error -- test: valid JSON Schema draft 2020-12 accepted
  • Reject schema definitions that are not valid JSON -- test: TDD test with malformed JSON
  • Reject schema definitions that fail JSON Schema meta-validation -- test: TDD test with {"type": "nonsense"}
  • Reject schema definitions where root type is not "object" -- test: TDD test with {"type": "array"}
  • Reject schema definitions where root additionalProperties is not explicitly present -- test: TDD test with schema missing the key
  • Accept schema definitions where root additionalProperties is explicitly true -- test: TDD test
  • Accept schema definitions where root additionalProperties is explicitly false -- test: TDD test
  • Reject schema definitions larger than MaxSchemaDefinitionBytes -- test: TDD test with 64KB+1 payload
  • Implement ValidateMetadata(schemaDef, metadata json.RawMessage) ([]ValidationError, error) -- test: compiles
  • Conforming metadata returns empty error slice -- test: Appendix A.1 sample schema + conforming payload
  • Missing required field produces ValidationError with field pointer and message -- test: TDD
  • maxLength violation produces structured error -- test: TDD
  • minimum/maximum violation produces structured error -- test: TDD
  • pattern violation produces structured error -- test: TDD
  • enum violation produces structured error -- test: TDD
  • format: date / date-time / email / uri / uuid enforced -- test: one TDD test per format
  • additionalProperties: false rejects extra fields -- test: TDD
  • Nested object validation works (e.g., contact_info from A.1) -- test: TDD
  • Array items / minItems / maxItems enforced -- test: TDD
  • Metadata larger than MaxMetadataPayloadBytes rejected -- test: TDD with 1MB+1 payload

3.4 Phase 3 integration

  • go test ./internal/customschema/... green -- test: full package green
  • task test:race green for the package -- test: no races
  • Journal entry

Phase 4 - Service layer

Goal: internal/customschema/service.go exposes every operation the HTTP layer will call. Role-agnostic. Real DB in tests (testcontainers, no mocks per project rules). Dependencies: Phase 2 (SQLC), Phase 3 (validator). Exit criteria: Service integration tests cover every method and every documented error path.

4.1 Skeleton and models

  • Create internal/customschema/service.go with Service struct holding cfg serviceconfig.ConfigProvider and validator *SchemaValidator -- test: compiles
  • Create internal/customschema/models.go with Schema, SchemaSummary, CreateSchemaInput, UpdateSchemaInput, ListFilters, CustomMetadata, MetadataVersion, SetMetadataInput, BulkAssignResult, SkippedDocument, ResetMetadataResult -- test: compiles
  • Every *Input struct that historically carried CreatedBy string is renamed or documented so the service expects the actor to arrive as a separate function argument (e.g. Service.CreateSchema(ctx, input, actor string)). The service must never read CreatedBy off a request body model. -- test: compiles + unit test that asserts passing an input with a non-empty CreatedBy is ignored in favor of the actor argument (or, cleaner, the struct field is removed entirely)

4.1a Audit sink definition (new, required before 4.2 and 4.5 can log)

  • Decide sink location: either internal/customschema/audit.go (feature-local) or internal/audit/ (shared) -- test: decision recorded in journal
  • Define AuditRecord struct: Actor string, Action string (enum: schema.create|schema.update|schema.delete|schema.assign|metadata.write|metadata.reset), ResourceID uuid.UUID, ClientID uuid.UUID, Timestamp time.Time, Details map[string]any -- test: compiles
  • Define AuditSink interface with Record(ctx, AuditRecord) error -- test: compiles
  • Provide a default implementation (structured slog with audit=true attribute, or a DB-backed appender if Q decides) -- test: unit test verifies a recorded event lands in the sink
  • Wire the sink into customschema.Service via serviceconfig.ConfigProvider or a constructor arg -- test: compiles

4.2 Schema CRUD methods

  • CreateSchema(ctx, input) -- meta-validates, enforces 64KB, enforces explicit additionalProperties, enforces name unique within client for active schemas at version 1 -- test: happy path
  • CreateSchema rejects invalid JSON Schema -- test: TDD
  • CreateSchema rejects oversize definition -- test: TDD
  • CreateSchema rejects duplicate name for same client -- test: TDD
  • UpdateSchema(ctx, schemaID, input) -- uses LockSchemaVersionsForName -> GetMaxSchemaVersion -> INSERT new version -> SetSchemaStatus(old, 'superseded') inside a single tx -- test: happy path produces v2
  • UpdateSchema concurrency: two concurrent calls against same (client_id, name) produce distinct version numbers -- test: TDD with goroutines against testcontainer DB
  • UpdateSchema rejects when old version does not exist -- test: TDD
  • UpdateSchema rejects invalid schema definition (same validator rules as create) -- test: TDD
  • GetSchema(ctx, schemaID) -- test: returns full row including schema_def
  • GetSchema 404 when not found -- test: TDD
  • ListSchemas(ctx, clientID, filters) -- default returns only active -- test: TDD
  • ListSchemas filter by name -- test: TDD
  • ListSchemas filter by status -- test: TDD
  • ListSchemas includeAllVersions=true returns all versions -- test: TDD
  • ListSchemas pagination (limit, offset) -- test: TDD
  • ListSchemas populates documentCount and canDelete fields -- test: TDD
  • DeleteSchema(ctx, schemaID) -- sets status to retired -- test: happy path
  • DeleteSchema returns 409 if any document references the schema via custom_schema_id -- test: TDD
  • DeleteSchema returns 409 if any document_custom_metadata.schema_id row references it -- test: TDD

4.3 Metadata methods

  • SetDocumentMetadata(ctx, input) -- derives schemaId from documents.custom_schema_id, never trusts client -- test: happy path
  • SetDocumentMetadata rejects when document has no custom_schema_id -- test: TDD (400)
  • SetDocumentMetadata rejects when document has legacy extractions -- test: TDD (409)
  • SetDocumentMetadata validates metadata against schema via validator.ValidateMetadata -- test: TDD with conforming and non-conforming payloads
  • SetDocumentMetadata enforces MaxMetadataPayloadBytes -- test: TDD
  • SetDocumentMetadata serializes version assignment by first taking SELECT id FROM documents WHERE id = $1 FOR UPDATE on the parent row, then reading GetMaxDocumentMetadataVersion, then INSERTing version = max+1 -- test: TDD confirms the parent-row lock is acquired before the version read
  • SetDocumentMetadata concurrency: two concurrent writers against the same document that already has >=1 metadata row -- both succeed, produce distinct versions, no unique-violation -- test: TDD with goroutines against testcontainer DB
  • SetDocumentMetadata concurrency: two concurrent first writers against a document with zero metadata rows -- both succeed, produce version=1 and version=2 in some order, no unique-violation surfaces to the caller -- test: TDD with goroutines, proves the parent-row lock (not just FOR UPDATE on the non-existent max-version row) is what serializes the first write
  • SetDocumentMetadata retries once on UNIQUE (document_id, version) violation as a belt-and-suspenders fallback (should be unreachable once the parent lock is in place) -- test: TDD with an injected unique-violation on the first INSERT attempt
  • SetDocumentMetadata subsequent writes must use same schema (consistency with Trigger 3) -- test: TDD
  • GetCurrentMetadata(ctx, documentID) -- test: uses view, returns latest
  • GetMetadataByVersion(ctx, documentID, version) -- test: TDD
  • GetMetadataHistory(ctx, documentID) with pagination -- test: TDD

4.4 Document schema binding methods

  • AssignSchema(ctx, documentID, schemaID) -- test: happy path single-document assignment
  • AssignSchema rejects if schema and document belong to different clients (app-layer + trigger 2) -- test: TDD
  • AssignSchema rejects if schema status is not active -- test: TDD
  • AssignSchema rejects if document already has custom metadata (matches Trigger 1 behavior; expect 409) -- test: TDD
  • AssignSchema rejects if document has legacy extractions -- test: TDD
  • AssignSchema allows setting customSchemaId=nil to clear binding when no metadata exists -- test: TDD
  • AssignSchemaToFolder(ctx, folderID, schemaID, createdBy) -- test: walks subtree with WITH RECURSIVE and returns BulkAssignResult
  • Bulk assignment skips documents with different schema + existing metadata with SkippedDocument reason -- test: TDD
  • Bulk assignment skips documents with legacy extractions with reason -- test: TDD
  • Bulk assignment no-ops documents already on target schema with reason -- test: TDD
  • Bulk assignment reassigns documents with different schema but no metadata -- test: TDD
  • Bulk assignment returns 422 when subtree exceeds MaxBulkAssignDocuments -- test: TDD with 10001 docs (or a smaller constant override for tests)
  • Bulk assignment runs in single transaction -- test: verify partial-failure rolls back everything

4.5 Reset-metadata method (v3)

  • ResetDocumentMetadata(ctx, documentID, resetBy) -- single transaction, LockDocumentForReset, GetDocumentSchemaBindingForReset, CountDocumentCustomMetadataVersions, DeleteDocumentCustomMetadata, NullifyDocumentCustomSchemaId, audit log -- test: happy path wipe with schema bound + N metadata versions
  • Reset returns populated previous* fields when document had a schema -- test: TDD
  • Reset returns metadataVersionsDeleted matching actual deleted count -- test: TDD
  • Reset is idempotent: already-clean document returns metadataVersionsDeleted: 0, previousSchemaId: nil -- test: TDD
  • Reset rejects documents with legacy field extractions (409) -- test: TDD
  • Reset returns 404 on non-existent document -- test: TDD
  • Reset atomicity: inject failure between DELETE and UPDATE -- test: transaction rolls back, metadata rows still present
  • Reset + reassign + write: after reset, Trigger 1 still fires on second schema change under new metadata -- test: TDD regression
  • Reset + reassign + write: after reset, Trigger 3 still fires on mismatched-schema direct INSERT -- test: TDD regression
  • Reset writes audit log entry containing actor, documentID, previousSchemaId, metadataVersionsDeleted -- test: TDD reads the audit sink
  • Reset concurrency: two concurrent resets on same document serialize via FOR UPDATE -- test: TDD
  • Reset concurrency: reset vs AssignSchema on same document serialize -- test: TDD

4.6 Phase 4 integration

  • go test ./internal/customschema/... green -- test: full package green
  • task test:race green for package -- test: no races
  • task test:unit:short still green globally -- test: no cross-package regressions
  • Journal entry

Phase 5a - Legacy field extraction guard

Goal: The existing FieldExtractionService refuses to write to any document that has custom_schema_id IS NOT NULL, and the HTTP controller surfaces that refusal as 409 Conflict. Dependencies: Phase 2. Exit criteria: Existing field extraction tests still pass; new service-level test proves the guard rejects; new controller-level test proves the HTTP response is 409 (not 500).

5a.1 Service-layer guard

  • Locate the existing FieldExtractionService.CreateFieldExtraction entry point (currently internal/fieldextraction/service.go) -- test: grep and read code
  • Add a pre-check: read documents.custom_schema_id for target document -- test: happy path unchanged (null binding)
  • If custom_schema_id IS NOT NULL, return a typed sentinel error (e.g. ErrMutualExclusivityViolation) with message referencing mutual exclusivity -- test: TDD service test asserts errors.Is(err, ErrMutualExclusivityViolation)
  • Existing field extraction happy-path tests still green -- test: go test ./internal/fieldextraction/...

5a.2 Controller-layer 409 mapping (REQUIRED to deliver the documented response)

Why this sub-phase exists: today api/queryAPI/fieldextractions.go:CreateFieldExtraction maps every service error to http.StatusInternalServerError. Adding only the service pre-check is not enough -- without controller error mapping the new guard still surfaces as a 500. Phase 5a is not complete until the HTTP layer returns 409.

  • Update CreateFieldExtraction handler in api/queryAPI/fieldextractions.go to branch on the service sentinel error and return echo.NewHTTPError(http.StatusConflict, ...) with the mutual-exclusivity message -- test: handler unit test asserts status 409 and body message
  • Audit the other FieldExtractionService handler entry points in api/queryAPI/fieldextractions.go (Update, AppendValueTo*, etc.) and apply the same 409 mapping where those methods also run the guard -- test: TDD handler test per entry point
  • Preserve the existing mapping of non-sentinel service errors to 500 -- test: TDD handler test with a generic error path
  • End-to-end HTTP test: create document, assign custom schema, call POST /field-extractions, assert response status is 409 and JSON body carries the documented message -- test: integration against testcontainer DB
  • Journal entry

Phase 5b - Delete cascade updates

Goal: Hard-deleting a document or client works even when custom metadata and schemas are present. Dependencies: Phase 2. Exit criteria: Document and client delete cascades succeed with custom metadata present; no orphaned rows.

5b.1 Document delete cascade

  • Locate DeleteDocumentCascade implementation -- test: found
  • Insert step: DeleteDocumentCustomMetadata between legacy extractions delete and document row delete -- test: TDD
  • Insert step: NullifyDocumentCustomSchemaId before document row delete -- test: TDD
  • Document delete succeeds for a document with N metadata versions and a schema bound -- test: TDD integration
  • Document delete succeeds for a document with no custom metadata (regression) -- test: TDD
  • Post-delete: SELECT COUNT(*) FROM document_custom_metadata WHERE document_id = deleted_id returns 0 -- test: TDD
  • Post-delete: client_metadata_schemas rows referenced by the document are NOT deleted (only the binding is) -- test: TDD

5b.2 Client delete cascade

  • Locate client.HardDelete / deleteClientDependencies -- test: found
  • Insert step: DeleteClientMetadataSchemas after all documents are deleted, before client row delete -- test: TDD
  • Client hard-delete succeeds when client has schemas and documents with custom metadata -- test: TDD integration
  • Post-delete: SELECT COUNT(*) FROM client_metadata_schemas WHERE client_id = deleted_id returns 0 -- test: TDD
  • Post-delete: no orphaned document_custom_metadata rows -- test: TDD
  • Regression: client delete still works for clients with no custom metadata -- test: TDD
  • Journal entry

Phase 6 - OpenAPI spec updates

Goal: serviceAPIs/queryAPI.yaml reflects all new endpoints and schemas; code generation produces clean client/server stubs. Dependencies: None for spec authoring; must land before Phase 8 codegen. Exit criteria: task generate produces updated api/queryAPI/* and pkg/queryAPI/* without errors.

6.1 New tags

  • Add SuperAdminSchemaService tag with description explicitly stating the super_admin-only boundary -- test: spec lints
  • Add CustomMetadataService tag with description stating client_user + user_admin + super_admin access -- test: spec lints

6.2 Schemas

  • CustomSchemaRequest -- test: codegen produces Go type
  • CustomSchemaResponse -- test: codegen
  • CustomSchemaListResponse -- test: codegen
  • CustomMetadataRequest -- test: codegen
  • CustomMetadataResponse -- test: codegen
  • CustomMetadataHistoryResponse -- test: codegen
  • ValidationErrorResponse -- test: codegen
  • BulkSchemaAssignRequest -- test: codegen
  • BulkSchemaAssignResponse -- test: codegen
  • DocumentSchemaAssignRequest -- test: codegen
  • DocumentSchemaAssignResponse -- test: codegen
  • ResetDocumentMetadataResponse (v3) -- test: codegen
  • SchemaStatus enum (active/superseded/retired) -- test: codegen
  • Modify DocumentEnriched to add customSchemaId, customSchemaName, hasCustomMetadata, customMetadata -- test: codegen

6.3 Paths

  • POST /super-admin/custom-schemas -- test: codegen produces handler interface
  • GET /super-admin/custom-schemas (with query params) -- test: codegen
  • GET /super-admin/custom-schemas/{schemaId} -- test: codegen
  • PUT /super-admin/custom-schemas/{schemaId} -- test: codegen
  • DELETE /super-admin/custom-schemas/{schemaId} -- test: codegen
  • PATCH /super-admin/documents/{id}/schema -- test: codegen
  • POST /super-admin/folders/{folderId}/assign-schema -- test: codegen
  • POST /super-admin/documents/{id}/reset-metadata (v3) -- test: codegen
  • GET /custom-metadata -- test: codegen
  • POST /custom-metadata -- test: codegen
  • GET /custom-metadata/version -- test: codegen
  • GET /custom-metadata/history -- test: codegen
  • Modify GET /document/{id} to document new fields and the customMetadata query parameter -- test: codegen

6.4 Phase 6 integration

  • task generate clean -- test: no errors
  • Generated controllers compile -- test: go build ./...
  • Journal entry

Phase 7a - Permit.io policy file updates

Goal: permit_policies.yaml declares the new super-admin and custom-metadata resources and grants the correct role permissions. Dependencies: None. Exit criteria: YAML validates; setup tool dry-run shows the expected resource/role diff.

Accepted cross-tenant limitation: the authorization check performed by the middleware is role/action/resource-type only -- it does not pass a document or client object to Permit.io. Any caller with custom-metadata:* grants can therefore read or write any document's custom metadata if they know the UUID. This is unchanged from the rest of the API and is explicitly accepted for v3 because production is deployed per-client (one isolated stack per client, separate DB). The limitation only matters in shared non-prod environments. See Section 5.3 of the plan for the full rationale. Do not add application-level tenant checks to the custom-metadata service in v3 -- that is a separate, API-wide initiative.

  • Add super-admin resource with get, post, patch, delete actions -- test: YAML valid
  • Add custom-metadata resource with get, post actions -- test: YAML valid
  • Grant super_admin role: super-admin: [get, post, patch, delete], custom-metadata: [get, post] -- test: YAML valid
  • Confirm user_admin role NOT granted super-admin -- test: visual diff against v1 intent; test-assertion in Phase 11
  • Grant user_admin role: custom-metadata: [get, post] -- test: YAML valid
  • Grant auditor role: super-admin: [get], custom-metadata: [get] -- test: YAML valid
  • Grant client_user role: custom-metadata: [get, post] -- test: YAML valid
  • Add documentation-only policy_mappings entries for every /super-admin/* path including POST /super-admin/documents/{id}/reset-metadata -- test: YAML valid
  • Add documentation-only policy_mappings entries for every /custom-metadata/* path -- test: YAML valid
  • Resolve PUT-to-patch question per Section 5.3 footnote: either confirm middleware maps PUT -> patch OR add put action to super-admin resource and grant it. Record decision in the journal. -- test: decision logged
  • Journal entry

Phase 7b - Run Permit.io setup tool (dev / uat / prod)

Goal: New resources and grants provisioned in every environment BEFORE Phase 8 code is deployed to that environment. Dependencies: Phase 7a. Exit criteria: Setup tool run succeeds in each env; spot-check via permit.io admin UI or list_roles.sh matches YAML.

  • Run cmd/auth_related/permit.setup/run.tool.dev.sh -- test: exit code 0
  • Verify super-admin resource exists in dev Permit.io tenant -- test: list
  • Verify custom-metadata resource exists in dev -- test: list
  • Verify super_admin role has expected grants in dev -- test: role inspect
  • Verify user_admin role does NOT have super-admin grants in dev -- test: role inspect
  • Run run.tool.uat.sh after dev verified -- test: exit 0
  • Same verification pass on uat -- test: list + role inspect
  • Run run.tool.prod.sh after uat verified AND only with Q's explicit go-ahead -- test: exit 0, Q confirmation logged in journal
  • Same verification pass on prod -- test: list + role inspect
  • Journal entry with timestamps of each env run

Phase 8 - API controllers + codegen

Goal: HTTP handlers thin-wrap the service layer; every generated interface method is implemented. Dependencies: Phase 4, Phase 6. Exit criteria: Every new endpoint compiles and routes resolve; handler unit tests green.

8.1 Schema CRUD controller (api/queryAPI/customschemas.go)

  • File skeleton created with handler receivers wired to customschema.Service -- test: compiles
  • Actor extraction helper: every handler in this file derives actor identity via cognitoauth.GetUserSubject(c) (or the existing middleware helper) and passes it to the service as a separate argument; any createdBy on the request body is ignored -- test: handler unit test sets a JWT subject via middleware fixture, posts a body with a different createdBy, asserts the service receives the JWT subject
  • POST /super-admin/custom-schemas handler -- parse, validate, derive actor from JWT, call CreateSchema, map errors to 400/409/500 -- test: TDD handler test
  • GET /super-admin/custom-schemas handler with query params -- test: TDD
  • GET /super-admin/custom-schemas/{schemaId} handler -- test: TDD
  • PUT /super-admin/custom-schemas/{schemaId} handler -- derives actor from JWT -- test: TDD
  • DELETE /super-admin/custom-schemas/{schemaId} handler -- derives actor from JWT -- test: TDD including 409 path
  • PATCH /super-admin/documents/{id}/schema handler -- derives actor from JWT -- test: TDD
  • POST /super-admin/folders/{folderId}/assign-schema handler -- derives actor from JWT -- test: TDD (covered more deeply in Phase 10)
  • POST /super-admin/documents/{id}/reset-metadata handler (v3) -- derives actor from JWT -- test: TDD (covered more deeply in Phase 10b)

8.2 User-facing custom metadata controller (api/queryAPI/custommetadata.go)

  • File skeleton -- test: compiles
  • Actor extraction: POST /custom-metadata derives actor identity from the JWT and ignores any body-supplied createdBy -- test: TDD handler test asserts body createdBy does not override
  • POST /custom-metadata handler -- test: TDD
  • GET /custom-metadata handler -- test: TDD
  • GET /custom-metadata/version handler -- test: TDD
  • GET /custom-metadata/history handler -- test: TDD

8.3 Phase 8 integration

  • All new handler files compile -- test: go build ./...
  • Handler unit tests green -- test: go test ./api/queryAPI/...
  • Journal entry

Phase 9 - Document endpoint modifications

Goal: GET /document/{id} surfaces custom schema and metadata when requested. Dependencies: Phase 4, Phase 8. Exit criteria: Existing tests unchanged; new fields populated correctly for documents with and without custom schemas.

  • Extend DocumentEnriched builder to include customSchemaId, customSchemaName, hasCustomMetadata unconditionally -- test: TDD
  • Extend builder to include customMetadata when ?customMetadata=true -- test: TDD
  • Authorization: confirm GET /document/{id} still uses the existing document resource, not super-admin -- test: role matrix test in Phase 11
  • Document without any custom schema returns customSchemaId: null, hasCustomMetadata: false -- test: TDD
  • Document with schema bound but no metadata returns customSchemaId: <id>, hasCustomMetadata: false -- test: TDD
  • Document with schema + metadata returns populated fields -- test: TDD
  • PATCH /super-admin/documents/{id}/schema handler delegates to AssignSchema -- test: TDD
  • Assign-schema 409 paths surface correct error messages -- test: TDD
  • Journal entry

Phase 10 - Bulk folder schema assignment endpoint

Goal: POST /super-admin/folders/{folderId}/assign-schema behaves per Section 5.2 including all skip reasons and the 10K limit. Dependencies: Phase 4 (AssignSchemaToFolder), Phase 8. Exit criteria: Recursive walk correct, skip reasons complete, document count limit enforced.

  • Handler wired to AssignSchemaToFolder -- test: compiles
  • Recursive walk on a 3-deep folder tree produces correct documentsUpdated count -- test: TDD
  • Skip reason: "different schema + existing metadata" -- test: TDD
  • Skip reason: "already assigned to same schema (no-op)" -- test: TDD
  • Skip reason: "has legacy field extractions" -- test: TDD
  • Reassignment on document with different schema but no metadata -- test: TDD
  • Subtree > 10K documents returns 422 with the documented message -- test: TDD (constant override for test)
  • Empty folder returns 200 with 0 counts -- test: TDD
  • Deeply nested (10+ levels) tree walks correctly -- test: TDD
  • Single transaction: simulated mid-walk failure rolls back -- test: TDD
  • Journal entry

Phase 10b - Reset-metadata endpoint (v3)

Goal: POST /super-admin/documents/{id}/reset-metadata exposes Service.ResetDocumentMetadata via HTTP with correct auth, response shape, and error mapping. (Deep service-level tests already live in Phase 4.5; this phase validates the HTTP wiring.) Dependencies: Phase 4.5 (service method), Phase 5b (queries reused), Phase 8 (controller skeleton). Exit criteria: Handler + integration tests green; authorization matrix for the endpoint verified in Phase 11.

  • Handler parses document ID path parameter -- test: TDD
  • Handler extracts caller identity for resetBy field -- test: TDD
  • Handler calls Service.ResetDocumentMetadata -- test: TDD
  • Handler maps success to 200 with ResetDocumentMetadataResponse -- test: TDD
  • Handler maps "document not found" to 404 -- test: TDD
  • Handler maps "legacy extractions exist" to 409 with mutual-exclusivity message -- test: TDD
  • Handler maps unexpected DB errors to 500 -- test: TDD
  • End-to-end HTTP test: bind schema v1, POST metadata, PUT schema -> v2 created, reset-metadata, PATCH schema to v2, POST new metadata with added optional field -- test: integration test exercises the full upgrade flow
  • End-to-end HTTP test: reset on already-clean document returns 200 with metadataVersionsDeleted: 0 -- test: integration
  • Journal entry

Phase 11 - Integration and authorization tests

Goal: Every role's access to every new endpoint is verified. Every cross-cutting invariant has a test. task fullsuite:ci ends with All coverage checks passed!. Dependencies: Phase 10b and Phase 7b in each environment used for testing. Exit criteria: 80% minimum coverage across new packages; authz matrix 100% covered; no FAIL in fullsuite:ci output.

11.1 Role authorization matrix (every row = one test)

  • super_admin CAN POST /super-admin/custom-schemas -- test: integration
  • super_admin CAN GET /super-admin/custom-schemas -- test: integration
  • super_admin CAN GET /super-admin/custom-schemas/{id} -- test: integration
  • super_admin CAN PUT /super-admin/custom-schemas/{id} -- test: integration
  • super_admin CAN DELETE /super-admin/custom-schemas/{id} -- test: integration
  • super_admin CAN PATCH /super-admin/documents/{id}/schema -- test: integration
  • super_admin CAN POST /super-admin/folders/{folderId}/assign-schema -- test: integration
  • super_admin CAN POST /super-admin/documents/{id}/reset-metadata -- test: integration
  • user_admin gets 403 on every /super-admin/* path+method combination (8 tests) -- test: integration, one per route
  • user_admin CAN read + write /custom-metadata (no regression) -- test: integration
  • client_user gets 403 on every /super-admin/* path+method combination (8 tests) -- test: integration
  • client_user CAN read + write /custom-metadata -- test: integration
  • auditor CAN GET on /super-admin/custom-schemas and /super-admin/custom-schemas/{id} -- test: integration
  • auditor gets 403 on every /super-admin/* write path (POST/PUT/PATCH/DELETE) -- test: integration
  • auditor CAN GET /custom-metadata, CANNOT POST /custom-metadata -- test: integration
  • Every 403 test asserts DB state is unchanged on denial -- test: embedded in each

11.2 Mutual exclusivity

  • Legacy write blocked when document has custom schema -- test: integration
  • Schema assignment blocked when document has legacy extractions -- test: integration
  • Bulk folder assign skips legacy-extracting documents -- test: integration
  • Reset-metadata rejected on legacy-extracting document -- test: integration

11.3 Schema lifecycle

  • active -> superseded on PUT create new version -- test: integration
  • active -> retired on DELETE -- test: integration
  • Assignment rejected for non-active status -- test: integration
  • Schema version concurrency (concurrent PUTs) produces distinct versions -- test: integration

11.4 Delete cascade

  • Document hard-delete with custom metadata -- test: integration
  • Client hard-delete with schemas and metadata -- test: integration
  • No orphaned rows post-delete -- test: integration

11.5 Size and count limits

  • Metadata payload at 1MB boundary: accept 1048576, reject 1048577 -- test: integration
  • Schema definition at 64KB boundary: accept 65536, reject 65537 -- test: integration
  • Bulk folder > 10000 docs: 422 -- test: integration
  • Bulk folder exactly 10000 docs: succeeds -- test: integration

11.6 Trigger regression (defense-in-depth)

  • Direct SQL bypass of service layer: attempt UPDATE custom_schema_id on a document with metadata -- trigger fires
  • Direct SQL bypass: INSERT document_custom_metadata with mismatched schema_id -- trigger fires
  • Direct SQL bypass: assign cross-client schema -- trigger fires
  • After reset + reassign + new metadata, all three triggers still fire on repeat attempts -- test: integration

11.7 Backward compatibility

  • Existing /field-extractions endpoints unchanged for documents with no custom schema -- test: integration
  • Existing /admin/* endpoints unaffected by super-admin resource -- test: integration
  • DocumentEnriched for documents without custom schemas has null new fields, no regression on existing fields -- test: integration

11.8 Permit.io setup tool verification

  • Run setup tool against a disposable dev tenant and assert both resources exist with expected actions -- test: tool output + API inspection
  • Assert each role's grants match Section 5.3 exactly (including the negative case for user_admin) -- test: tool output

11.9 Coverage & full suite

  • internal/customschema/ >= 80% coverage -- test: go test -cover
  • api/queryAPI/customschemas.go + custommetadata.go >= 80% coverage -- test: go test -cover
  • task test:race green -- test: no race reports
  • task test:perf checked for regressions in existing hot paths -- test: review output
  • task fullsuite:ci ends with All coverage checks passed! and NO FAIL lines -- test: full run
  • Journal entry with test counts, coverage numbers, and any flaky test notes

Phase 12 - Documentation updates

Goal: Every doc that references endpoints, env vars, authorization, or the delete cascade is updated. Dependencies: Phase 11. Exit criteria: docs/ai.generated/ regenerated; manual review of key runbooks complete.

  • Update docs/ai.generated/README.md index if new doc files added -- test: review
  • Authorization matrix table published in a dedicated doc page -- test: review
  • Sample schemas from Appendix A published as copyable examples -- test: review
  • CustomMetadataService usage guide (end-user perspective) -- test: review
  • SuperAdminSchemaService usage guide (platform-operator perspective) -- test: review
  • Reset-metadata runbook: "How to upgrade a document to a newer schema version" with the full wipe -> reassign -> re-POST flow and a warning about metadata history loss -- test: review
  • Mutual exclusivity explanation (legacy vs custom) in the document processing guide -- test: review
  • Delete cascade documentation updated -- test: review
  • Permit.io deployment ordering note (Phase 7b before Phase 8) added to the deployment runbook -- test: review
  • Any new env vars? (none expected for v3; confirm.) -- test: review
  • Generate new API status report / architecture diagrams if the pipeline regenerates them -- test: review
  • Journal entry marking the feature shipped

Cross-phase reminders

  • Before starting any phase, re-read the relevant section of plans/mutable.metadata.plan.combo.v3.md.
  • TDD is mandatory per project rules: write the test, watch it fail, make it pass. Never write multiple tests before running any.
  • No mocks without Q's explicit permission. Use testcontainers for real Postgres.
  • Keep the journal current. When a checkbox flips to [x], add a timestamped line to ./journals/implement_mutableMetadata.md with the commit SHA and test command that proved it.
  • If any trigger, migration, or service invariant seems to conflict with the plan, STOP and ask Q before modifying it. The defense-in-depth design depends on the triggers being exactly as written.
  • task fullsuite:ci must be green at the end of every phase, not just the end of the project. Do not stack broken phases.
Reference in New Issue View Git Blame Copy Permalink