jacob-mathison/query-orchestration
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

Merged in feature/textExtractionsPart1 (pull request #192)

Browse Source 
all schema and rest apis for text extraction support

* in progress

* stage 8 complete

* phase 9 completed

* phase 9 complete

* ongoing - s3 path fix

* working

* optimize ci build

* e2e tests

* missing test
This commit is contained in:
Jay Brown
2025-11-26 19:23:42 +00:00
parent 2b43799f56
commit c45e1dd427
67 changed files with 16120 additions and 817 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ai.generated/03-api-documentation.md
+852 -403
View File
File diff suppressed because it is too large Load Diff
docs/ai.generated/04-data-architecture.md
+438 -10
View File
@@ -12,6 +12,7 @@ erDiagram
clients ||--o{ batch_uploads : "owns"
clients ||--|| clientCanSync : "has sync status"
clients ||--o{ documentUploads : "uploads"
clients ||--o{ folders : "organizes"
clients ||--o{ collectorVersions : "has versions"
clients ||--|| collectorActiveVersions : "has active version"
clients ||--o{ collectorQueries : "configures"
@@ -21,8 +22,14 @@ erDiagram
batch_uploads ||--o{ documents : "contains"
batch_uploads ||--o{ documentUploads : "tracks uploads"
folders ||--o{ documents : "contains"
folders ||--o{ folders : "has children"
folders ||--o{ documentUploads : "targets"
documents ||--o{ documentEntries : "has entries"
documents ||--o{ documentCleans : "has clean versions"
documents ||--o{ documentLabels : "has labels"
documents ||--o{ documentFieldExtractions : "has field extractions"
documentEntries }o--|| documents : "references"
@@ -34,6 +41,11 @@ erDiagram
documentTextExtractions ||--o{ documentTextExtractionEntries : "has entries"
documentTextExtractions ||--o{ results : "generates"
labels ||--o{ documentLabels : "applied to documents"
documentFieldExtractions ||--o{ documentFieldExtractionVersions : "has versions"
documentFieldExtractions ||--o{ documentFieldExtractionArrayFields : "has array fields"
queries ||--o{ queryVersions : "has versions"
queries ||--|| queryActiveVersions : "has active version"
queries ||--o{ requiredQueries : "requires other queries"
@@ -62,12 +74,23 @@ erDiagram
boolean canSync
}
folders {
uuid id PK
text path
uuid parentId FK
varchar clientId FK
timestamp createdAt
varchar createdBy
}
documents {
uuid id PK
varchar clientId FK
text hash
text filename
uuid batch_id FK
uuid folderId FK
text originalPath
}
batch_uploads {
@@ -83,6 +106,9 @@ erDiagram
jsonb failed_filenames
timestamp created_at
timestamp completed_at
text archive_bucket
text archive_key
bigint file_size_bytes
}
documentUploads {
@@ -94,6 +120,7 @@ erDiagram
timestamp createdAt
text filename
uuid batch_id FK
uuid folder_id FK
}
documentEntries {
@@ -135,6 +162,54 @@ erDiagram
bigint version
}
labels {
text label PK
text description
}
documentLabels {
uuid id PK
uuid documentId FK
text label FK
timestamp appliedAt
varchar appliedBy
}
documentFieldExtractions {
uuid id PK
uuid documentId FK
text fileName
text contractTitle
int aareteDerivedAmendmentNum
text clientName
text payerName
date aareteDerivedEffectiveDt
date aareteDerivedTerminationDt
boolean autoRenewalInd
timestamp createdAt
varchar createdBy
}
documentFieldExtractionVersions {
uuid id PK
uuid fieldExtractionId FK
bigint version
varchar createdBy
timestamp createdAt
}
documentFieldExtractionArrayFields {
uuid id PK
uuid fieldExtractionId FK
smallint arrayIndex
text exhibitTitle
text reimbProvTin
date reimbEffectiveDt
text aareteDerivedClaimTypeCd
numeric reimbPctRate
numeric reimbFeeRate
}
queries {
uuid queryId PK
queryType queryType
@@ -265,14 +340,18 @@ CREATE TABLE documents (
hash TEXT NOT NULL,
filename TEXT,
batch_id uuid,
folderId uuid,
originalPath TEXT,
UNIQUE(clientId, hash),
FOREIGN KEY (clientId) REFERENCES clients(clientId),
FOREIGN KEY (batch_id) REFERENCES batch_uploads(id)
FOREIGN KEY (batch_id) REFERENCES batch_uploads(id),
FOREIGN KEY (folderId) REFERENCES folders(id)
);
-- Indexes
CREATE INDEX idx_documents_filename ON documents(filename);
CREATE INDEX idx_documents_batch_id ON documents(batch_id);
CREATE INDEX idx_documents_folderid ON documents(folderId);
```
**Key Features**:
@@ -281,7 +360,9 @@ CREATE INDEX idx_documents_batch_id ON documents(batch_id);
- Hash-based deduplication within client scope (per client)
- Filename preservation for folder path support (added in migration 105)
- Optional batch association for grouped uploads (added in migration 103)
- Indexed on filename and batch_id for efficient queries
- Virtual folder organization via `folderId` (added in migration 111)
- Immutable `originalPath` preserving exact upload path (added in migration 111)
- Indexed on filename, batch_id, and folderId for efficient queries
### Batch Uploads (`batch_uploads`)
Tracks batch document upload operations with ZIP archive processing.
@@ -302,7 +383,14 @@ CREATE TABLE batch_uploads (
failed_filenames jsonb DEFAULT '[]'::jsonb,
created_at timestamp NOT NULL DEFAULT NOW(),
completed_at timestamp,
FOREIGN KEY (client_id) REFERENCES clients(clientId)
archive_bucket text,
archive_key text,
file_size_bytes bigint,
FOREIGN KEY (client_id) REFERENCES clients(clientId),
CONSTRAINT batch_storage_consistent CHECK (
(archive_bucket IS NULL AND archive_key IS NULL AND file_size_bytes IS NULL) OR
(archive_bucket IS NOT NULL AND archive_key IS NOT NULL)
)
);
-- Indexes
@@ -318,9 +406,97 @@ CREATE INDEX idx_batch_uploads_status ON batch_uploads(status);
- Failed filename tracking as JSONB array for flexible querying
- Original filename preservation for display
- Temporal tracking with creation and completion timestamps
- Storage metadata tracking (archive bucket/key and file size)
- Storage consistency constraint ensures bucket and key are both set or both null
- Indexed by client_id and status for efficient queries
- Note: Uses snake_case naming convention (unlike other tables)
### Folders (`folders`)
Virtual folder hierarchy for document organization. Folders are database-only entities with no direct relationship to S3 storage locations.
```sql
CREATE TABLE folders (
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
path text NOT NULL,
parentId uuid,
clientId varchar(255) NOT NULL,
createdAt timestamp NOT NULL DEFAULT NOW(),
createdBy varchar(255) NOT NULL,
FOREIGN KEY (parentId) REFERENCES folders(id),
FOREIGN KEY (clientId) REFERENCES clients(clientId),
UNIQUE(clientId, path),
CHECK (id != parentId)
);
-- Indexes
CREATE INDEX idx_folders_parentid ON folders(parentId);
CREATE INDEX idx_folders_clientid ON folders(clientId);
CREATE INDEX idx_folders_path ON folders(path);
```
**Key Features**:
- Self-referential hierarchy via `parentId` for nested folders
- Client isolation with unique constraint on (clientId, path)
- Self-reference prevention via check constraint (`id != parentId`)
- Path column stores virtual folder path (e.g., "/contracts/2025")
- Folders can be renamed without affecting S3 storage locations
- Audit tracking via `createdAt` and `createdBy`
- Note: Added in migration 109
### Labels System
#### Labels (`labels`)
Lookup table for document processing status labels.
```sql
CREATE TABLE labels (
label text PRIMARY KEY,
description text NOT NULL
);
-- Seeded initial values
INSERT INTO labels (label, description) VALUES
('Ingested', 'Document has been ingested into the system'),
('OCR_Processed', 'OCR text extraction completed'),
('GenAI_Processed', 'GenAI processing completed'),
('Doczy_AI_Completed', 'Doczy.AI processing completed'),
('Dashboard_Ready', 'Document ready for dashboard display');
```
**Key Features**:
- Predefined processing status labels
- Primary key on label text for direct referencing
- Description provides human-readable explanation
#### Document Labels (`documentLabels`)
Junction table tracking label application to documents.
```sql
CREATE TABLE documentLabels (
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
documentId uuid NOT NULL,
label text NOT NULL,
appliedAt timestamp NOT NULL DEFAULT NOW(),
appliedBy varchar(255) NOT NULL,
FOREIGN KEY (documentId) REFERENCES documents(id) ON DELETE CASCADE,
FOREIGN KEY (label) REFERENCES labels(label)
);
-- Indexes
CREATE INDEX idx_documentlabels_documentid ON documentLabels(documentId);
CREATE INDEX idx_documentlabels_label ON documentLabels(label);
CREATE INDEX idx_documentlabels_appliedat ON documentLabels(appliedAt);
CREATE INDEX idx_documentlabels_document_label_time ON documentLabels(documentId, label, appliedAt DESC);
```
**Key Features**:
- Tracks which labels are applied to which documents
- Audit tracking via `appliedAt` and `appliedBy`
- Cascade delete removes labels when document is deleted
- Composite index optimizes queries for most recent label application
- Multiple applications of same label are allowed (historical tracking)
- Note: Added in migration 110
### Queries (`queries`)
Defines data extraction logic with type-specific implementations.
@@ -350,6 +526,243 @@ CREATE TABLE results (
);
```
## Field Extractions System
The field extractions system stores structured data extracted from documents, supporting both single-value fields (1:1 relationship) and array fields (1:N relationship). This system was added in migrations 112-115.
### Field Extractions Architecture
```mermaid
graph TD
Document[documents] --> FE[documentFieldExtractions]
FE --> FEV[documentFieldExtractionVersions]
FE --> FEAF[documentFieldExtractionArrayFields]
FEV --> CFE[currentFieldExtractions View]
CFE --> CFEWAC[currentFieldExtractionsWithArrayCount View]
FEAF --> CFEWAC
```
### Document Field Extractions (`documentFieldExtractions`)
Stores single-value extracted fields from documents (19 fields total, 1:1 relationship with document).
```sql
CREATE TABLE documentFieldExtractions (
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
documentId uuid NOT NULL,
-- Document identification
fileName text,
contractTitle text,
aareteDerivedAmendmentNum int,
-- Party information
clientName text,
payerName text,
payerState text,
providerState text,
-- Tax identification numbers (stored as text to preserve leading zeros)
filenameTin text,
provGroupTin text,
provGroupNpi text,
provGroupNameFull text,
provOtherTin text,
provOtherNpi text,
provOtherNameFull text,
-- Contract dates
aareteDerivedEffectiveDt date,
aareteDerivedTerminationDt date,
-- Renewal information
autoRenewalInd boolean,
autoRenewalTerm text,
-- Metadata
createdAt timestamp NOT NULL DEFAULT NOW(),
createdBy varchar(255) NOT NULL,
FOREIGN KEY (documentId) REFERENCES documents(id)
);
-- Indexes
CREATE INDEX idx_documentfieldextractions_documentid
ON documentFieldExtractions(documentId);
CREATE INDEX idx_documentfieldextractions_createdby
ON documentFieldExtractions(createdBy);
CREATE INDEX idx_documentfieldextractions_filename
ON documentFieldExtractions(fileName);
```
**Single-Value Fields** (19 fields):
- **Document identification**: fileName, contractTitle, aareteDerivedAmendmentNum
- **Party information**: clientName, payerName, payerState, providerState
- **Tax IDs (as text for leading zeros)**: filenameTin, provGroupTin, provGroupNpi, provGroupNameFull, provOtherTin, provOtherNpi, provOtherNameFull
- **Contract dates**: aareteDerivedEffectiveDt, aareteDerivedTerminationDt
- **Renewal info**: autoRenewalInd, autoRenewalTerm
### Document Field Extraction Versions (`documentFieldExtractionVersions`)
Version tracking for field extractions, enabling historical queries.
```sql
CREATE TABLE documentFieldExtractionVersions (
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
fieldExtractionId uuid NOT NULL,
version bigint NOT NULL,
createdBy varchar(255) NOT NULL,
createdAt timestamp NOT NULL DEFAULT NOW(),
FOREIGN KEY (fieldExtractionId) REFERENCES documentFieldExtractions(id)
);
-- Indexes
CREATE INDEX idx_documentfieldextractionversions_fieldextractionid
ON documentFieldExtractionVersions(fieldExtractionId);
CREATE INDEX idx_documentfieldextractionversions_version
ON documentFieldExtractionVersions(version);
```
**Key Features**:
- Links field extractions to version numbers
- Audit tracking via createdBy and createdAt
- Enables temporal queries and rollback capability
### Document Field Extraction Array Fields (`documentFieldExtractionArrayFields`)
Stores array-value extracted fields (112 fields total, 1:N relationship with field extraction).
```sql
CREATE TABLE documentFieldExtractionArrayFields (
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
fieldExtractionId uuid NOT NULL,
arrayIndex smallint NOT NULL,
-- Array-value fields (112 fields, grouped by category)
-- Exhibit information
exhibitTitle text,
exhibitPage text,
-- Reimbursement provider information
reimbProvTin text,
reimbProvNpi text,
reimbProvName text,
reimbEffectiveDt date,
reimbTerminationDt date,
-- Claim and product codes
aareteDerivedClaimTypeCd text,
aareteDerivedProduct text,
aareteDerivedLob text,
aareteDerivedProgram text,
aareteDerivedNetwork text,
aareteDerivedProvType text,
-- Provider taxonomy and specialty
provTaxonomyCd text,
provTaxonomyCdDesc text,
provSpecialtyCd text,
provSpecialtyCdDesc text,
-- Service location
placeOfServiceCd text,
placeOfServiceCdDesc text,
billTypeCd text,
billTypeCdDesc text,
-- Reimbursement rates
reimbPctRate numeric(10,4),
reimbFeeRate numeric(12,2),
reimbConversionFactor numeric(12,4),
-- Grouper information
grouperType text,
grouperCd text,
grouperCdDesc text,
grouperPctRate numeric(10,4),
grouperBaseRate numeric(12,2),
-- ... (96 additional fields for outliers, facility adjustments, rate escalators, stop loss)
FOREIGN KEY (fieldExtractionId) REFERENCES documentFieldExtractions(id),
UNIQUE (fieldExtractionId, arrayIndex),
CHECK (arrayIndex >= 0)
);
-- Indexes
CREATE INDEX idx_documentfieldextractionarrayfields_fieldextractionid
ON documentFieldExtractionArrayFields(fieldExtractionId);
CREATE INDEX idx_documentfieldextractionarrayfields_id_index
ON documentFieldExtractionArrayFields(fieldExtractionId, arrayIndex);
```
**Array Field Categories** (112 fields total):
- **Exhibit info**: exhibitTitle, exhibitPage
- **Reimbursement provider**: reimbProvTin, reimbProvNpi, reimbProvName, dates
- **Claim/product codes**: claimTypeCd, product, LOB, program, network, provType
- **Provider taxonomy**: taxonomyCd, specialtyCd with descriptions
- **Service location**: placeOfServiceCd, billTypeCd with descriptions
- **Patient demographics**: patientAgeMin, patientAgeMax
- **Reimbursement terms**: reimbTerm, carveout info, payment logic
- **Rates**: reimbPctRate, reimbFeeRate, conversionFactor, thresholds
- **Fee schedules**: feeSchedule, feeScheduleVersion
- **Service codes**: CPT4, revenue, diagnosis, NDC codes with descriptions
- **Grouper info**: grouperType, grouperCd, rates, severity, transfer flags
- **Outlier terms**: thresholds, rates, exclusions
- **Facility adjustments**: DSH, IME, NTAP, UC, GME indicators and rates
- **Rate escalator**: escalator settings, max rate increases
- **Stop loss**: thresholds, maximums, daily rates
**Key Constraints**:
- Unique constraint on (fieldExtractionId, arrayIndex) prevents duplicate rows
- Check constraint ensures arrayIndex >= 0
### Field Extraction Views
#### Current Field Extractions (`currentFieldExtractions`)
Returns the newest field extraction per document.
```sql
CREATE VIEW currentFieldExtractions AS
SELECT DISTINCT ON (dfe.documentId)
dfe.id,
dfe.documentId,
dfe.fileName,
dfe.contractTitle,
-- ... all single-value fields ...
dfev.version,
dfev.createdBy,
dfev.createdAt
FROM documentFieldExtractions dfe
JOIN documentFieldExtractionVersions dfev
ON dfev.fieldExtractionId = dfe.id
ORDER BY dfe.documentId, dfev.id DESC;
```
**Key Features**:
- Uses `DISTINCT ON` for efficient latest-per-document queries
- Joins versions to include version metadata
- Orders by dfev.id DESC to get most recent version
#### Current Field Extractions With Array Count (`currentFieldExtractionsWithArrayCount`)
Extends currentFieldExtractions with count of array field rows.
```sql
CREATE VIEW currentFieldExtractionsWithArrayCount AS
SELECT
cfe.*,
COALESCE(array_counts.arraySize, 0) as arraySize
FROM currentFieldExtractions cfe
LEFT JOIN (
SELECT fieldExtractionId, COUNT(*) as arraySize
FROM documentFieldExtractionArrayFields
GROUP BY fieldExtractionId
) array_counts ON array_counts.fieldExtractionId = cfe.id;
```
**Key Features**:
- Includes all fields from currentFieldExtractions
- Adds arraySize column showing count of related array field rows
- Uses COALESCE for documents with no array fields (returns 0)
## Versioning System
The schema implements a comprehensive versioning system enabling temporal queries and configuration evolution.
@@ -418,21 +831,25 @@ CREATE TABLE documentUploads (
createdAt timestamp NOT NULL,
filename TEXT,
batch_id uuid,
FOREIGN KEY (clientId) REFERENCES clients(clientId)
folder_id uuid,
FOREIGN KEY (clientId) REFERENCES clients(clientId),
FOREIGN KEY (folder_id) REFERENCES folders(id)
);
-- Indexes
CREATE INDEX idx_documentuploads_filename ON documentUploads(filename);
CREATE INDEX idx_documentuploads_batch_id ON documentUploads(batch_id);
CREATE INDEX idx_documentuploads_clientid ON documentUploads(clientId);
CREATE INDEX idx_documentuploads_folder_id ON documentUploads(folder_id);
```
**Key Features**:
- Tracks initial S3 upload location (bucket/key)
- Part number for storage distribution
- Optional batch association (added in migration 107)
- Optional batch association (added in migration 107, note: no foreign key constraint on batch_id)
- Filename tracking (added in migration 106)
- Indexed for efficient lookups by client, filename, and batch
- Folder association for batch uploads (added in migration 116, has foreign key constraint)
- Indexed for efficient lookups by client, filename, batch, and folder
#### Document Entries (`documentEntries`)
Core document references after deduplication and validation.
@@ -1245,11 +1662,19 @@ Similar triggers exist for `collectorMinCleanVersions` and `collectorMinTextVers
### Recent Schema Changes
- **Migration 103**: Added `batch_uploads` table and `documents.batch_id`
- **Migration 104**: Added S3 storage columns to `batch_uploads` (later removed in favor of different approach)
- **Migration 104**: Added S3 storage columns to `batch_uploads` (`archive_bucket`, `archive_key`, `file_size_bytes`) with storage consistency constraint
- **Migration 105**: Added `documents.filename` column
- **Migration 106**: Added `documentUploads.filename` column
- **Migration 107**: Added `documentUploads.batch_id` column
- **Migration 107**: Added `documentUploads.batch_id` column (without foreign key constraint)
- **Migration 108**: Added index on `documentUploads.clientId`
- **Migration 109**: Added `folders` table for virtual folder hierarchy with self-referential parentId
- **Migration 110**: Added `labels` and `documentLabels` tables for document processing status tracking
- **Migration 111**: Added `documents.folderId` and `documents.originalPath` columns; added comments for S3 immutability
- **Migration 112**: Added `documentFieldExtractions` table with 19 single-value fields
- **Migration 113**: Added `documentFieldExtractionVersions` table for version tracking
- **Migration 114**: Added `documentFieldExtractionArrayFields` table with 112 array fields
- **Migration 115**: Added `currentFieldExtractions` and `currentFieldExtractionsWithArrayCount` views
- **Migration 116**: Added `documentUploads.folder_id` column with foreign key constraint
### Version Compatibility
- Backward-compatible schema changes prioritized
@@ -1301,14 +1726,17 @@ Views are layered for reusability:
### Table Count
- **Core entities**: 5 (clients, documents, batch_uploads, queries, results)
- **Organization tables**: 3 (folders, labels, documentLabels)
- **Field extraction tables**: 3 (documentFieldExtractions, documentFieldExtractionVersions, documentFieldExtractionArrayFields)
- **Supporting tables**: 16 (versions, entries, configs, dependencies, etc.)
- **Total tables**: 21
- **Total tables**: 27
### View Count
- **Query management**: 6 views
- **Collector management**: 7 views
- **Document processing**: 3 views
- **Total views**: 16+
- **Field extraction**: 2 views (currentFieldExtractions, currentFieldExtractionsWithArrayCount)
- **Total views**: 18+
### Function Count
- **Business logic**: 3 main functions