Merged in jmathison/v2-upload-batch (pull request #224)

Implement v2 upload batch cleanup * Implement v2 upload batch cleanup * Merge remote-tracking branch 'origin/main' into jmathison/v2-upload-batch * Address upload batch review feedback * Raise batch worker coverage * Fix batch cleanup review issues Approved-by: Jay Brown
2026-05-08 21:42:46 +00:00
parent 7b820b18c2
commit a080ca59d8
35 changed files with 2587 additions and 108 deletions
@@ -28,6 +28,7 @@ erDiagram
    batch_uploads ||--o{ documents : "contains"
    batch_uploads ||--o{ documentUploads : "tracks uploads"
    batch_uploads ||--o{ batch_document_outcomes : "tracks outcomes"
+    batch_uploads ||--o{ batch_folder_candidates : "tracks cleanup candidates"

    folders ||--o{ documents : "contains"
    folders ||--o{ folders : "has children"
@@ -487,6 +488,7 @@ CREATE INDEX idx_batch_uploads_status ON batch_uploads(status);
 - Temporal tracking with creation and completion timestamps
 - Storage metadata tracking (archive bucket/key and file size)
 - Storage consistency constraint ensures bucket and key are both set or both null
+- `folder_cleanup_completed_at` records when backend folder cleanup has finalized for the batch
 - Indexed by client_id and status for efficient queries
 - Note: Uses snake_case naming convention (unlike other tables)

@@ -530,6 +532,29 @@ CREATE INDEX idx_batch_doc_outcomes_document_id ON batch_document_outcomes(docum
 - Indexed by batch_id for efficient batch detail lookups and by document_id for pipeline updates
 - Added in migration 124

+### Batch Folder Candidates (`batch_folder_candidates`)
+Tracks folders touched by legacy ZIP batch extraction so the backend can remove empty auto-created folders after all file outcomes are terminal.
+
+```sql
+CREATE TABLE batch_folder_candidates (
+    batch_id uuid NOT NULL,
+    folder_id uuid NOT NULL,
+    path text NOT NULL,
+    existed_before boolean NOT NULL,
+    created_at timestamp NOT NULL DEFAULT NOW(),
+    PRIMARY KEY (batch_id, folder_id),
+    FOREIGN KEY (batch_id) REFERENCES batch_uploads(id) ON DELETE CASCADE,
+    FOREIGN KEY (folder_id) REFERENCES folders(id) ON DELETE CASCADE
+);
+```
+
+**Batch Folder Cleanup Features**:
+- Records each non-root folder segment touched by a batch upload
+- Preserves `existed_before=false` once any batch creates the folder, enabling idempotent retries
+- Cleanup finalizes only after terminal batch outcome evidence exists; zero-outcome failed batches stay cleanup-pending
+- Safe deletion never removes `/`, pre-existing folders, renamed folders, folders with documents/uploads/children/chatbot scopes, or folders still needed by another pending batch
+- Added in migration 132
+
 ### Folders (`folders`)
 Virtual folder hierarchy for document organization. Folders are database-only entities with no direct relationship to S3 storage locations.

@@ -91,6 +91,14 @@ Auth path customization:
 - `COGNITO_LOGOUT_PATH` (default `/logout`)
 - `HEALTH_PATH` (default `/health`)

+Local development bootstrap, active only when `DISABLE_AUTH=true`:
+- `LOCAL_DEV_CLIENT_ID` (optional) - Creates this client at queryAPI startup when it does not already exist.
+- `LOCAL_DEV_CLIENT_NAME` (optional) - Display name for `LOCAL_DEV_CLIENT_ID`; defaults to the client ID.
+- `LOCAL_DEV_EULA_VERSION` (optional) - Creates or updates this local EULA version and marks it current.
+- `LOCAL_DEV_EULA_TITLE` (optional) - Title for the local EULA; defaults to `AArete DoczyAI Terms of Use`.
+- `LOCAL_DEV_EULA_AUTO_ACCEPT_SUBJECT` (optional) - Cognito subject/user key to mark as accepted for the local EULA.
+- `LOCAL_DEV_EULA_AUTO_ACCEPT_EMAIL` (optional) - Email for the auto-accepted user; defaults to the subject value.
+
 ## Rate Limiting

 From `internal/serviceconfig/ratelimit/config.go`:
@@ -149,4 +157,3 @@ reference, including the grace-window floor.

 - Some deployment files still include legacy query/text environment variables because deprecated compatibility services remain in compose definitions.
 - For API/runtime behavior, prefer `serviceAPIs/queryAPI.yaml` plus `internal/serviceconfig/*` as source of truth.
-
@@ -446,6 +446,33 @@ kubectl get pods -l app=query-orchestration
 task health:check:all
 ```

+### Batch Folder Cleanup Operations
+
+Legacy ZIP batch uploads record auto-created folder candidates in `batch_folder_candidates`. Cleanup finalization is run by the batch worker after terminal batch status changes and by the QueryAPI background worker through `FinalizeReadyBatches`.
+
+Operational checks:
+
+```sql
+-- Batches waiting for folder cleanup
+SELECT id, client_id, status, total_documents, folder_cleanup_completed_at
+FROM batch_uploads
+WHERE folder_cleanup_completed_at IS NULL
+  AND id IN (SELECT batch_id FROM batch_folder_candidates);
+
+-- Candidate folders for one batch
+SELECT batch_id, folder_id, path, existed_before
+FROM batch_folder_candidates
+WHERE batch_id = '<batch-id>';
+
+-- Outcome evidence used by cleanup readiness
+SELECT outcome, COUNT(*)
+FROM batch_document_outcomes
+WHERE batch_id = '<batch-id>'
+GROUP BY outcome;
+```
+
+Cleanup should not complete for failed batches with `total_documents = 0` and no outcome rows. That state usually means outcome persistence failed before the worker could record a terminal per-file result. All-skipped archives are handled separately with a synthetic `failed_upload` outcome so the batch can fail visibly and cleanup can remove empty auto-created folders.
+
 ## Capacity Planning

 ### Resource Requirements
@@ -148,7 +148,7 @@ Each terminal outcome indicates a specific action for the caller:

 - **`failed_open`** / **`failed_read`** -- The file inside the ZIP was damaged or unreadable. Re-create the ZIP archive and resubmit.

- **`failed_s3_upload`** / **`failed_upload`** -- An infrastructure error occurred during processing. Retry the entire batch.
+- **`failed_s3_upload`** / **`failed_upload`** -- An infrastructure error occurred during processing. `failed_upload` is also used for ZIP archives that contain no processable files after directories, hidden metadata, and unsafe paths are skipped.

 - **`sync_skipped`** -- The client's sync configuration prevents processing. Contact an administrator.

@@ -181,3 +181,11 @@ ZIP extracted
 ```

 Non-terminal outcomes (`submitted`, `init_complete`, `sync_complete`) indicate the document is still being processed. Poll the batch details endpoint to see updates as files progress through each stage.
+
+## 7. Folder Cleanup Coupling
+
+Legacy ZIP batch uploads still create folders during extraction. For batch uploads, the backend records touched folder segments in `batch_folder_candidates` and finalizes cleanup when the batch has terminal outcome evidence.
+
+Accepted terminal outcomes (`clean_passed`, `sync_skipped`) keep their folder assignments. Rejected outcomes (`duplicate`, `init_duplicate`, `invalid_type`, `failed_open`, `failed_read`, `failed_s3_upload`, `failed_upload`, `clean_failed`) detach current-batch documents/uploads from auto-created candidate folders. Empty auto-created folders are then deleted only when they are not root, did not pre-exist, were not renamed, have no documents/uploads/children/chatbot scopes, and are not pending cleanup for another batch.
+
+If outcome persistence fails before any outcome row exists, cleanup remains pending instead of marking `folder_cleanup_completed_at`. This keeps failed zero-outcome batches visible for retry/diagnosis and prevents cleanup from closing before rejected artifacts can be detached.
@@ -89,6 +89,25 @@ This document describes current service behavior in the processing pipeline.

 **Output**: `{document_id}` to `DOCSYNC` for each client document.

+---
+
+### 6. batch folder cleanup
+**Purpose**: Remove empty folders auto-created by legacy ZIP batch extraction when all file outcomes are terminal.
+
+**Code locations**:
+- `internal/document/batch/foldercleanup/`
+- `internal/server/api/batch_worker.go`
+- `internal/database/queries/batch.sql`
+
+**Behavior**:
+- Locks the batch row before finalizing cleanup
+- Requires terminal batch status plus persisted outcome evidence
+- Detaches rejected current-batch documents and upload rows from candidate folders
+- Deletes only safe empty auto-created folders
+- Retries ready cleanup from the QueryAPI background worker
+
+**Output**: updates `batch_uploads.folder_cleanup_completed_at`; may null rejected `documents.folderId` and `documentUploads.folder_id`; may delete safe candidate folders.
+
 ## Deprecated Compatibility Services

 These services are retained for deployment compatibility and are currently health-only/no-op:
@@ -105,6 +124,7 @@ S3 Upload Event
  -> docInitRunner
  -> docSyncRunner
  -> docCleanRunner
+  -> batch folder cleanup (for legacy ZIP batches)
 ```

 ```text
@@ -113,4 +133,3 @@ Client Sync Trigger
  -> docSyncRunner
  -> docCleanRunner
 ```
-
@@ -368,6 +368,7 @@ s.cfg.GetQueueClient()
 | `models.go` | Generated Go types from database schema |
 | `client.sql.go` | Generated client query methods |
 | `document.sql.go` | Generated document query methods |
+| `batch.sql.go` | Generated batch, outcome, and folder cleanup query methods |
 | etc. | One file per SQL source file |

 **SQL Query Definition** (`internal/database/queries/client.sql`):
@@ -406,9 +407,13 @@ q.WithTx(tx)
 **Examples**:
 - `00000000000003_clients.up.sql` - Create clients table
 - `00000000000109_create_folders_table.up.sql` - Add folders feature
+- `00000000000124_batch_document_outcomes.up.sql` - Track per-file batch outcomes
+- `00000000000132_batch_folder_candidates.up.sql` - Track batch folder cleanup candidates

 **Migration Tool**: golang-migrate

+Batch cleanup queries live in `internal/database/queries/batch.sql` with the rest of the batch upload workflow. Keep raw production SQL there so SQLC regenerates typed methods for readiness checks, rejected artifact detachment, cleanup completion, and safe candidate folder deletion.
+
 ---

 ### 9. Configuration Layer (`internal/serviceconfig/`)