query-orchestration/docs/ai.generated/04-data-architecture.md

Data Architecture

Database Schema Overview

The system uses PostgreSQL 17.2 with a sophisticated schema designed for multi-tenant document processing, query orchestration, and comprehensive versioning. The schema is managed through database migrations and uses SQLC for type-safe Go integration.

Entity Relationship Diagram

erDiagram
    clients ||--o{ documents : "owns"
    clients ||--o{ batch_uploads : "owns"
    clients ||--|| clientCanSync : "has sync status"
    clients ||--o{ documentUploads : "uploads"
    clients ||--o{ folders : "organizes"
    clients ||--o{ collectorVersions : "has versions"
    clients ||--|| collectorActiveVersions : "has active version"
    clients ||--o{ collectorQueries : "configures"
    clients ||--o{ collectorMinCleanVersions : "sets min clean version"
    clients ||--o{ collectorMinTextVersions : "sets min text version"

    batch_uploads ||--o{ documents : "contains"
    batch_uploads ||--o{ documentUploads : "tracks uploads"

    folders ||--o{ documents : "contains"
    folders ||--o{ folders : "has children"
    folders ||--o{ documentUploads : "targets"

    documents ||--o{ documentEntries : "has entries"
    documents ||--o{ documentCleans : "has clean versions"
    documents ||--o{ documentLabels : "has labels"
    documents ||--o{ documentFieldExtractions : "has field extractions"

    documentEntries }o--|| documents : "references"

    documentCleans }o--|| documents : "cleans"
    documentCleans ||--o{ documentCleanEntries : "has entries"
    documentCleans ||--o{ documentTextExtractions : "produces text"

    documentTextExtractions }o--|| documentCleans : "from"
    documentTextExtractions ||--o{ documentTextExtractionEntries : "has entries"
    documentTextExtractions ||--o{ results : "generates"

    labels ||--o{ documentLabels : "applied to documents"

    documentFieldExtractions ||--o{ documentFieldExtractionVersions : "has versions"
    documentFieldExtractions ||--o{ documentFieldExtractionArrayFields : "has array fields"

    queries ||--o{ queryVersions : "has versions"
    queries ||--|| queryActiveVersions : "has active version"
    queries ||--o{ requiredQueries : "requires other queries"
    queries ||--o{ queryConfigs : "has configs"
    queries ||--o{ results : "produces"
    queries ||--o{ collectorQueries : "used by collectors"

    queryVersions ||--|| queryActiveVersions : "tracks active"

    results }o--|| documentTextExtractions : "from text"
    results }o--|| queries : "from query"
    results ||--o{ resultDependencies : "depends on"
    results ||--o{ resultDependencies : "required by"

    collectorVersions ||--|| collectorActiveVersions : "tracks active"
    collectorQueries }o--|| queries : "references"

    eulaVersions ||--o{ eulaAgreements : "has agreements"

    clients {
        varchar clientId PK
        text name UK
    }

    clientCanSync {
        uuid syncId PK
        varchar clientId FK
        boolean canSync
    }

    folders {
        uuid id PK
        text path
        uuid parentId FK
        varchar clientId FK
        timestamp createdAt
        varchar createdBy
    }

    documents {
        uuid id PK
        varchar clientId FK
        text hash
        text filename
        uuid batch_id FK
        uuid folderId FK
        text originalPath
    }

    batch_uploads {
        uuid id PK
        varchar client_id FK
        text original_filename
        integer total_documents
        integer processed_documents
        integer failed_documents
        integer invalid_type_documents
        batch_status status
        integer progress_percent
        jsonb failed_filenames
        timestamp created_at
        timestamp completed_at
        text archive_bucket
        text archive_key
        bigint file_size_bytes
    }

    documentUploads {
        uuid id PK
        text bucket
        text key
        varchar clientId FK
        smallint part
        timestamp createdAt
        text filename
        uuid batch_id FK
        uuid folder_id FK
    }

    documentEntries {
        uuid id PK
        uuid documentId FK
        text bucket
        text key
    }

    documentCleans {
        uuid id PK
        uuid documentId FK
        text bucket
        text key
        text hash
        cleanMimeType mimetype
        cleanFailType fail
    }

    documentCleanEntries {
        uuid id PK
        uuid cleanId FK
        bigint version
    }

    documentTextExtractions {
        uuid id PK
        uuid cleanId FK
        text bucket
        text key
        text hash
        timestamp createdAt
        smallint part
    }

    documentTextExtractionEntries {
        uuid id PK
        uuid textId FK
        bigint version
    }

    labels {
        text label PK
        text description
    }

    documentLabels {
        uuid id PK
        uuid documentId FK
        text label FK
        timestamp appliedAt
        varchar appliedBy
    }

    documentFieldExtractions {
        uuid id PK
        uuid documentId FK
        text fileName
        text contractTitle
        int aareteDerivedAmendmentNum
        text clientName
        text payerName
        date aareteDerivedEffectiveDt
        date aareteDerivedTerminationDt
        boolean autoRenewalInd
        timestamp createdAt
        varchar createdBy
    }

    documentFieldExtractionVersions {
        uuid id PK
        uuid fieldExtractionId FK
        bigint version
        varchar createdBy
        timestamp createdAt
    }

    documentFieldExtractionArrayFields {
        uuid id PK
        uuid fieldExtractionId FK
        smallint arrayIndex
        text exhibitTitle
        text reimbProvTin
        date reimbEffectiveDt
        text aareteDerivedClaimTypeCd
        numeric reimbPctRate
        numeric reimbFeeRate
    }

    queries {
        uuid queryId PK
        queryType queryType
    }

    queryVersions {
        uuid queryId FK
        int versionId "PK composite"
        timestamp addedAt
    }

    queryActiveVersions {
        uuid activeVersionEntryId PK
        uuid queryId FK
        int versionId FK
    }

    requiredQueries {
        uuid requiredQueryEntryId PK
        uuid queryId FK
        uuid requiredQueryId FK
        int addedVersion FK
        int removedVersion FK
    }

    queryConfigs {
        uuid configId PK
        uuid queryId FK
        jsonb config
        int addedVersion FK
        int removedVersion FK
    }

    results {
        uuid id PK
        uuid textEntryId FK
        uuid queryId FK
        text value
        int queryVersion FK
    }

    resultDependencies {
        uuid resultId FK
        uuid requiredResultId FK
    }

    collectorVersions {
        varchar clientId FK
        int id "PK composite"
        timestamp addedAt
    }

    collectorActiveVersions {
        uuid id PK
        varchar clientId FK
        int versionId FK
    }

    collectorQueries {
        uuid id PK
        varchar clientId FK
        varchar name
        uuid queryId FK
        int addedVersion FK
        int removedVersion FK
    }

    collectorMinCleanVersions {
        uuid id PK
        varchar clientId FK
        bigint versionId
        int addedVersion FK
        int removedVersion FK
    }

    collectorMinTextVersions {
        uuid id PK
        varchar clientId FK
        bigint versionId
        int addedVersion FK
        int removedVersion FK
    }

    eulaVersions {
        uuid id PK
        varchar version UK
        text title
        text content
        timestamptz effectiveDate
        timestamptz createdAt
        varchar createdBy
        boolean isCurrent
        timestamptz activatedAt
        varchar activatedBy
    }

    eulaAgreements {
        uuid id PK
        varchar cognitoSubjectId
        varchar userEmail
        uuid eulaVersionId FK
        timestamptz agreedAt
        varchar agreedFromIp
    }

Core Entities

Clients (clients)

Central entity representing organizations or tenants in the system.

CREATE TABLE clients (
    clientId varchar(255) PRIMARY KEY,
    name TEXT NOT NULL,
    UNIQUE (name)
);

Key Characteristics:

• String-based client identifiers for human readability
• Multi-tenant isolation through client-scoped data
• Central reference point for all client-owned entities
• Unique constraint on name prevents duplicate client names

Client Sync Status (clientCanSync)

Tracks whether a client is allowed to synchronize documents.

CREATE TABLE clientCanSync (
    syncId uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    clientId varchar(255) NOT NULL,
    canSync boolean NOT NULL,
    FOREIGN KEY (clientId) REFERENCES clients(clientId)
);

Key Features:

• Historical tracking of sync permission changes
• Multiple entries per client for audit trail
• Most recent entry determines current sync status

Documents (documents)

Represents uploaded files requiring processing.

CREATE TABLE documents (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    clientId varchar(255) NOT NULL,
    hash TEXT NOT NULL,
    filename TEXT,
    batch_id uuid,
    folderId uuid,
    originalPath TEXT,
    UNIQUE(clientId, hash),
    FOREIGN KEY (clientId) REFERENCES clients(clientId),
    FOREIGN KEY (batch_id) REFERENCES batch_uploads(id),
    FOREIGN KEY (folderId) REFERENCES folders(id)
);

-- Indexes
CREATE INDEX idx_documents_filename ON documents(filename);
CREATE INDEX idx_documents_batch_id ON documents(batch_id);
CREATE INDEX idx_documents_folderid ON documents(folderId);

Key Features:

• UUID v7 primary keys for time-ordered insertion
• Client isolation with foreign key constraints
• Hash-based deduplication within client scope (per client)
• Filename preservation for folder path support (added in migration 105)
• Optional batch association for grouped uploads (added in migration 103)
• Virtual folder organization via folderId (added in migration 111)
• Immutable originalPath preserving exact upload path (added in migration 111)
• Indexed on filename, batch_id, and folderId for efficient queries

Batch Uploads (batch_uploads)

Tracks batch document upload operations with ZIP archive processing.

CREATE TYPE batch_status AS ENUM ('processing', 'completed', 'failed', 'cancelled');

CREATE TABLE batch_uploads (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    client_id varchar(255) NOT NULL,
    original_filename TEXT NOT NULL,
    total_documents integer NOT NULL,
    processed_documents integer NOT NULL DEFAULT 0,
    failed_documents integer NOT NULL DEFAULT 0,
    invalid_type_documents integer NOT NULL DEFAULT 0,
    status batch_status NOT NULL DEFAULT 'processing',
    progress_percent integer NOT NULL DEFAULT 0,
    failed_filenames jsonb DEFAULT '[]'::jsonb,
    created_at timestamp NOT NULL DEFAULT NOW(),
    completed_at timestamp,
    archive_bucket text,
    archive_key text,
    file_size_bytes bigint,
    FOREIGN KEY (client_id) REFERENCES clients(clientId),
    CONSTRAINT batch_storage_consistent CHECK (
        (archive_bucket IS NULL AND archive_key IS NULL AND file_size_bytes IS NULL) OR
        (archive_bucket IS NOT NULL AND archive_key IS NOT NULL)
    )
);

-- Indexes
CREATE INDEX idx_batch_uploads_client_id ON batch_uploads(client_id);
CREATE INDEX idx_batch_uploads_status ON batch_uploads(status);

Batch Processing Features:

• UUID v7 for time-ordered batch tracking
• Comprehensive status tracking with ENUM types (batch_status)
• Progress counters for processed/failed/invalid documents
• Progress percentage for UI updates
• Failed filename tracking as JSONB array for flexible querying
• Original filename preservation for display
• Temporal tracking with creation and completion timestamps
• Storage metadata tracking (archive bucket/key and file size)
• Storage consistency constraint ensures bucket and key are both set or both null
• Indexed by client_id and status for efficient queries
• Note: Uses snake_case naming convention (unlike other tables)

Folders (folders)

Virtual folder hierarchy for document organization. Folders are database-only entities with no direct relationship to S3 storage locations.

CREATE TABLE folders (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    path text NOT NULL,
    parentId uuid,
    clientId varchar(255) NOT NULL,
    createdAt timestamp NOT NULL DEFAULT NOW(),
    createdBy varchar(255) NOT NULL,
    FOREIGN KEY (parentId) REFERENCES folders(id),
    FOREIGN KEY (clientId) REFERENCES clients(clientId),
    UNIQUE(clientId, path),
    CHECK (id != parentId)
);

-- Indexes
CREATE INDEX idx_folders_parentid ON folders(parentId);
CREATE INDEX idx_folders_clientid ON folders(clientId);
CREATE INDEX idx_folders_path ON folders(path);

Key Features:

• Self-referential hierarchy via parentId for nested folders
• Client isolation with unique constraint on (clientId, path)
• Self-reference prevention via check constraint (id != parentId)
• Path column stores virtual folder path (e.g., "/contracts/2025")
• Folders can be renamed without affecting S3 storage locations
• Audit tracking via createdAt and createdBy
• Note: Added in migration 109

Labels System

Labels (labels)

Lookup table for document processing status labels.

CREATE TABLE labels (
    label text PRIMARY KEY,
    description text NOT NULL
);

-- Seeded initial values
INSERT INTO labels (label, description) VALUES
    ('Ingested', 'Document has been ingested into the system'),
    ('OCR_Processed', 'OCR text extraction completed'),
    ('GenAI_Processed', 'GenAI processing completed'),
    ('Doczy_AI_Completed', 'Doczy.AI processing completed'),
    ('Dashboard_Ready', 'Document ready for dashboard display');

Key Features:

• Predefined processing status labels
• Primary key on label text for direct referencing
• Description provides human-readable explanation

Document Labels (documentLabels)

Junction table tracking label application to documents.

CREATE TABLE documentLabels (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    documentId uuid NOT NULL,
    label text NOT NULL,
    appliedAt timestamp NOT NULL DEFAULT NOW(),
    appliedBy varchar(255) NOT NULL,
    FOREIGN KEY (documentId) REFERENCES documents(id) ON DELETE CASCADE,
    FOREIGN KEY (label) REFERENCES labels(label)
);

-- Indexes
CREATE INDEX idx_documentlabels_documentid ON documentLabels(documentId);
CREATE INDEX idx_documentlabels_label ON documentLabels(label);
CREATE INDEX idx_documentlabels_appliedat ON documentLabels(appliedAt);
CREATE INDEX idx_documentlabels_document_label_time ON documentLabels(documentId, label, appliedAt DESC);

Key Features:

• Tracks which labels are applied to which documents
• Audit tracking via appliedAt and appliedBy
• Cascade delete removes labels when document is deleted
• Composite index optimizes queries for most recent label application
• Multiple applications of same label are allowed (historical tracking)
• Note: Added in migration 110

EULA System

The EULA (End User License Agreement) system tracks EULA versions and user consent. Added in migration 121.

EULA Versions (eulaVersions)

Stores EULA version records with content and activation tracking.

CREATE TABLE eulaVersions (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    version VARCHAR(50) NOT NULL UNIQUE,
    title TEXT NOT NULL,
    content TEXT NOT NULL,
    effectiveDate TIMESTAMPTZ NOT NULL,
    createdAt TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    createdBy VARCHAR(320) NOT NULL,
    isCurrent BOOLEAN NOT NULL DEFAULT FALSE,
    activatedAt TIMESTAMPTZ,
    activatedBy VARCHAR(320)
);

-- Partial unique index enforces only ONE row can have isCurrent=true at any time
CREATE UNIQUE INDEX idx_eulaversions_one_current ON eulaVersions(isCurrent) WHERE isCurrent = TRUE;

-- Indexes for efficient queries
CREATE INDEX idx_eulaversions_effectivedate ON eulaVersions(effectiveDate DESC);
CREATE INDEX idx_eulaversions_iscurrent ON eulaVersions(isCurrent);

Key Features:

• UUID v7 primary keys for time-ordered insertion
• Unique version string constraint prevents duplicate versions
• Content stored in Markdown format for flexible rendering
• Partial unique index ensures only one version can be isCurrent=true at any time
• Activation tracking with timestamp and admin email
• Effective date supports scheduling future EULA versions

EULA Agreements (eulaAgreements)

Tracks user consent with audit metadata.

CREATE TABLE eulaAgreements (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    cognitoSubjectId VARCHAR(256) NOT NULL,
    userEmail VARCHAR(320) NOT NULL,
    eulaVersionId uuid NOT NULL REFERENCES eulaVersions(id) ON DELETE RESTRICT,
    agreedAt TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    agreedFromIp VARCHAR(45) NOT NULL,
    UNIQUE(cognitoSubjectId, eulaVersionId)
);

-- Indexes for efficient queries
CREATE INDEX idx_eulaagreements_subjectid ON eulaAgreements(cognitoSubjectId);
CREATE INDEX idx_eulaagreements_versionid ON eulaAgreements(eulaVersionId);
CREATE INDEX idx_eulaagreements_version_time ON eulaAgreements(eulaVersionId, agreedAt DESC);
CREATE INDEX idx_eulaagreements_user_time ON eulaAgreements(cognitoSubjectId, agreedAt DESC);

Key Features:

• Links users (by Cognito subject ID) to EULA versions they've agreed to
• Unique constraint on (cognitoSubjectId, eulaVersionId) prevents duplicate agreements
• Stores user email at time of agreement for audit purposes
• IP address tracking for legal compliance
• ON DELETE RESTRICT prevents deleting EULA versions that have agreements
• Composite indexes optimize common query patterns (agreements by version, user history)

Queries (queries)

Defines data extraction logic with type-specific implementations.

CREATE TABLE queries (
    queryId uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    queryType querytype NOT NULL
);

CREATE TYPE querytype AS ENUM ('context_full', 'json_extractor');

Query Types:

• context_full: Returns complete document context
• json_extractor: Extracts specific JSON paths from structured data

Results (results)

Stores extracted data from document processing.

CREATE TABLE results (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    textEntryId uuid NOT NULL REFERENCES documentTextExtractions(id),
    queryId uuid NOT NULL REFERENCES queries(queryId),
    value text NOT NULL,
    queryVersion int4 NOT NULL
);

Field Extractions System

The field extractions system stores structured data extracted from documents, supporting both single-value fields (1:1 relationship) and array fields (1:N relationship). This system was added in migrations 112-115.

Field Extractions Architecture

graph TD
    Document[documents] --> FE[documentFieldExtractions]
    FE --> FEV[documentFieldExtractionVersions]
    FE --> FEAF[documentFieldExtractionArrayFields]
    FEV --> CFE[currentFieldExtractions View]
    CFE --> CFEWAC[currentFieldExtractionsWithArrayCount View]
    FEAF --> CFEWAC

Document Field Extractions (documentFieldExtractions)

Stores single-value extracted fields from documents (19 fields total, 1:1 relationship with document).

CREATE TABLE documentFieldExtractions (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    documentId uuid NOT NULL,

    -- Document identification
    fileName text,
    contractTitle text,
    aareteDerivedAmendmentNum int,

    -- Party information
    clientName text,
    payerName text,
    payerState text,
    providerState text,

    -- Tax identification numbers (stored as text to preserve leading zeros)
    filenameTin text,
    provGroupTin text,
    provGroupNpi text,
    provGroupNameFull text,
    provOtherTin text,
    provOtherNpi text,
    provOtherNameFull text,

    -- Contract dates
    aareteDerivedEffectiveDt date,
    aareteDerivedTerminationDt date,

    -- Renewal information
    autoRenewalInd boolean,
    autoRenewalTerm text,

    -- Metadata
    createdAt timestamp NOT NULL DEFAULT NOW(),
    createdBy varchar(255) NOT NULL,

    FOREIGN KEY (documentId) REFERENCES documents(id)
);

-- Indexes
CREATE INDEX idx_documentfieldextractions_documentid
    ON documentFieldExtractions(documentId);
CREATE INDEX idx_documentfieldextractions_createdby
    ON documentFieldExtractions(createdBy);
CREATE INDEX idx_documentfieldextractions_filename
    ON documentFieldExtractions(fileName);

Single-Value Fields (19 fields):

• Document identification: fileName, contractTitle, aareteDerivedAmendmentNum
• Party information: clientName, payerName, payerState, providerState
• Tax IDs (as text for leading zeros): filenameTin, provGroupTin, provGroupNpi, provGroupNameFull, provOtherTin, provOtherNpi, provOtherNameFull
• Contract dates: aareteDerivedEffectiveDt, aareteDerivedTerminationDt
• Renewal info: autoRenewalInd, autoRenewalTerm

Document Field Extraction Versions (documentFieldExtractionVersions)

Version tracking for field extractions, enabling historical queries.

CREATE TABLE documentFieldExtractionVersions (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    fieldExtractionId uuid NOT NULL,
    version bigint NOT NULL,
    createdBy varchar(255) NOT NULL,
    createdAt timestamp NOT NULL DEFAULT NOW(),

    FOREIGN KEY (fieldExtractionId) REFERENCES documentFieldExtractions(id)
);

-- Indexes
CREATE INDEX idx_documentfieldextractionversions_fieldextractionid
    ON documentFieldExtractionVersions(fieldExtractionId);
CREATE INDEX idx_documentfieldextractionversions_version
    ON documentFieldExtractionVersions(version);

Key Features:

• Links field extractions to version numbers
• Audit tracking via createdBy and createdAt
• Enables temporal queries and rollback capability

Document Field Extraction Array Fields (documentFieldExtractionArrayFields)

Stores array-value extracted fields (112 fields total, 1:N relationship with field extraction).

CREATE TABLE documentFieldExtractionArrayFields (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    fieldExtractionId uuid NOT NULL,
    arrayIndex smallint NOT NULL,

    -- Array-value fields (112 fields, grouped by category)
    -- Exhibit information
    exhibitTitle text,
    exhibitPage text,

    -- Reimbursement provider information
    reimbProvTin text,
    reimbProvNpi text,
    reimbProvName text,
    reimbEffectiveDt date,
    reimbTerminationDt date,

    -- Claim and product codes
    aareteDerivedClaimTypeCd text,
    aareteDerivedProduct text,
    aareteDerivedLob text,
    aareteDerivedProgram text,
    aareteDerivedNetwork text,
    aareteDerivedProvType text,

    -- Provider taxonomy and specialty
    provTaxonomyCd text,
    provTaxonomyCdDesc text,
    provSpecialtyCd text,
    provSpecialtyCdDesc text,

    -- Service location
    placeOfServiceCd text,
    placeOfServiceCdDesc text,
    billTypeCd text,
    billTypeCdDesc text,

    -- Reimbursement rates
    reimbPctRate numeric(10,4),
    reimbFeeRate numeric(12,2),
    reimbConversionFactor numeric(12,4),

    -- Grouper information
    grouperType text,
    grouperCd text,
    grouperCdDesc text,
    grouperPctRate numeric(10,4),
    grouperBaseRate numeric(12,2),

    -- ... (96 additional fields for outliers, facility adjustments, rate escalators, stop loss)

    FOREIGN KEY (fieldExtractionId) REFERENCES documentFieldExtractions(id),
    UNIQUE (fieldExtractionId, arrayIndex),
    CHECK (arrayIndex >= 0)
);

-- Indexes
CREATE INDEX idx_documentfieldextractionarrayfields_fieldextractionid
    ON documentFieldExtractionArrayFields(fieldExtractionId);
CREATE INDEX idx_documentfieldextractionarrayfields_id_index
    ON documentFieldExtractionArrayFields(fieldExtractionId, arrayIndex);

Array Field Categories (112 fields total):

• Exhibit info: exhibitTitle, exhibitPage
• Reimbursement provider: reimbProvTin, reimbProvNpi, reimbProvName, dates
• Claim/product codes: claimTypeCd, product, LOB, program, network, provType
• Provider taxonomy: taxonomyCd, specialtyCd with descriptions
• Service location: placeOfServiceCd, billTypeCd with descriptions
• Patient demographics: patientAgeMin, patientAgeMax
• Reimbursement terms: reimbTerm, carveout info, payment logic
• Rates: reimbPctRate, reimbFeeRate, conversionFactor, thresholds
• Fee schedules: feeSchedule, feeScheduleVersion
• Service codes: CPT4, revenue, diagnosis, NDC codes with descriptions
• Grouper info: grouperType, grouperCd, rates, severity, transfer flags
• Outlier terms: thresholds, rates, exclusions
• Facility adjustments: DSH, IME, NTAP, UC, GME indicators and rates
• Rate escalator: escalator settings, max rate increases
• Stop loss: thresholds, maximums, daily rates

Key Constraints:

• Unique constraint on (fieldExtractionId, arrayIndex) prevents duplicate rows
• Check constraint ensures arrayIndex >= 0

Field Extraction Views

Current Field Extractions (currentFieldExtractions)

Returns the newest field extraction per document.

CREATE VIEW currentFieldExtractions AS
SELECT DISTINCT ON (dfe.documentId)
    dfe.id,
    dfe.documentId,
    dfe.fileName,
    dfe.contractTitle,
    -- ... all single-value fields ...
    dfev.version,
    dfev.createdBy,
    dfev.createdAt
FROM documentFieldExtractions dfe
JOIN documentFieldExtractionVersions dfev
    ON dfev.fieldExtractionId = dfe.id
ORDER BY dfe.documentId, dfev.id DESC;

Key Features:

• Uses DISTINCT ON for efficient latest-per-document queries
• Joins versions to include version metadata
• Orders by dfev.id DESC to get most recent version

Current Field Extractions With Array Count (currentFieldExtractionsWithArrayCount)

Extends currentFieldExtractions with count of array field rows.

CREATE VIEW currentFieldExtractionsWithArrayCount AS
SELECT
    cfe.*,
    COALESCE(array_counts.arraySize, 0) as arraySize
FROM currentFieldExtractions cfe
LEFT JOIN (
    SELECT fieldExtractionId, COUNT(*) as arraySize
    FROM documentFieldExtractionArrayFields
    GROUP BY fieldExtractionId
) array_counts ON array_counts.fieldExtractionId = cfe.id;

Key Features:

• Includes all fields from currentFieldExtractions
• Adds arraySize column showing count of related array field rows
• Uses COALESCE for documents with no array fields (returns 0)

Versioning System

The schema implements a comprehensive versioning system enabling temporal queries and configuration evolution.

Version Management Pattern

-- Generic versioning pattern
CREATE TABLE entity_versions (
    entityId uuid NOT NULL,
    versionId int4 NOT NULL,
    addedVersion int4 NOT NULL,
    removedVersion int4,
    -- entity-specific fields
    PRIMARY KEY (entityId, versionId)
);

CREATE TABLE entity_active_versions (
    entityId uuid PRIMARY KEY,
    versionId int4 NOT NULL
);

Temporal Validity Functions

-- Check if version is active at specific point
-- Note: Parameter order is _version, _addedVersion, _removedVersion
CREATE OR REPLACE FUNCTION isInVersion(
    _version int,
    _addedVersion int,
    _removedVersion int
)
RETURNS boolean AS $$
BEGIN
    RETURN _version IS NULL OR (
        _version >= _addedVersion AND
        (_removedVersion IS NULL OR _version < _removedVersion)
    );
END;
$$ LANGUAGE plpgsql;

Key Points:

• Returns true if _version is NULL (allowing queries without version constraints)
• Returns true if version is within the valid range [_addedVersion, _removedVersion)
• Used throughout views for temporal queries

Version-Enabled Entities

• Query Versions: queryVersions, queryActiveVersions
• Collector Versions: collectorVersions, collectorActiveVersions
• Processing Versions: Track minimum required versions for document stages

Document Processing Pipeline

Document Lifecycle Tables

Document Uploads (documentUploads)

Initial upload tracking for audit and debugging.

CREATE TABLE documentUploads (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    bucket TEXT NOT NULL,
    key TEXT NOT NULL,
    clientId varchar(255) NOT NULL,
    part unsignedsmallint NOT NULL,
    createdAt timestamp NOT NULL,
    filename TEXT,
    batch_id uuid,
    folder_id uuid,
    FOREIGN KEY (clientId) REFERENCES clients(clientId),
    FOREIGN KEY (folder_id) REFERENCES folders(id)
);

-- Indexes
CREATE INDEX idx_documentuploads_filename ON documentUploads(filename);
CREATE INDEX idx_documentuploads_batch_id ON documentUploads(batch_id);
CREATE INDEX idx_documentuploads_clientid ON documentUploads(clientId);
CREATE INDEX idx_documentuploads_folder_id ON documentUploads(folder_id);

Key Features:

• Tracks initial S3 upload location (bucket/key)
• Part number for storage distribution
• Optional batch association (added in migration 107, note: no foreign key constraint on batch_id)
• Filename tracking (added in migration 106)
• Folder association for batch uploads (added in migration 116, has foreign key constraint)
• Indexed for efficient lookups by client, filename, batch, and folder

Document Entries (documentEntries)

Core document references after deduplication and validation.

CREATE TABLE documentEntries (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    documentId uuid NOT NULL,
    bucket TEXT NOT NULL,
    key TEXT NOT NULL,
    FOREIGN KEY (documentId) REFERENCES documents(id)
);

Purpose:

• Links documents to their S3 storage locations after initial processing
• Multiple entries possible per document for version tracking

Document Cleans (documentCleans)

PDF validation and format processing records.

CREATE TYPE cleanFailType AS ENUM (
    'invalid_mimetype',
    'invalid_read',
    'invalid_read_pages',
    'zero_page_count',
    'large_file',
    'small_dimensions',
    'large_dimensions',
    'small_dpi',
    'large_dpi'
);

CREATE TYPE cleanMimeType AS ENUM ('application/pdf');

CREATE TABLE documentCleans (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    documentId uuid NOT NULL,
    bucket TEXT,
    key TEXT,
    hash TEXT,
    mimetype cleanMimeType,
    fail cleanFailType,
    FOREIGN KEY (documentId) REFERENCES documents(id),
    CONSTRAINT bucket_and_key_together CHECK (
        (bucket IS NULL) = (key IS NULL) AND
        (bucket IS NULL) = (mimetype IS NULL) AND
        (bucket IS NULL) = (hash IS NULL)
    ),
    CONSTRAINT location_xor_fail CHECK (
        (bucket IS NOT NULL) != (fail IS NOT NULL)
    )
);

Validation Logic:

• Either succeeds (has bucket/key/mimetype/hash) OR fails (has fail type)
• Cannot have both success and failure indicators
• Tracks specific failure reasons for troubleshooting

Document Clean Entries (documentCleanEntries)

Version tracking for clean operations.

CREATE TABLE documentCleanEntries (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    cleanId uuid NOT NULL,
    version bigint NOT NULL,
    FOREIGN KEY (cleanId) REFERENCES documentCleans(id)
);

Purpose:

• Enables version-based filtering of clean results
• Used by currentCleanEntries view to get latest valid cleans

Document Text Extractions (documentTextExtractions)

OCR and text extraction results from AWS Textract.

CREATE TABLE documentTextExtractions (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    cleanId uuid NOT NULL,
    bucket TEXT NOT NULL,
    key TEXT NOT NULL,
    hash TEXT NOT NULL,
    createdAt timestamp NOT NULL,
    part unsignedsmallint NOT NULL,
    FOREIGN KEY (cleanId) REFERENCES documentCleans(id)
);

Key Features:

• Links to successful clean operation
• Stores S3 location of extracted text
• Hash for content verification
• Part number for storage distribution

Document Text Extraction Entries (documentTextExtractionEntries)

Version tracking for text extraction operations.

CREATE TABLE documentTextExtractionEntries (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    textId uuid NOT NULL,
    version bigint NOT NULL,
    FOREIGN KEY (textId) REFERENCES documentTextExtractions(id)
);

Purpose:

• Enables version-based filtering of text extraction results
• Used by currentTextEntries view to get latest valid extractions

Processing Dependencies

link to rendered version here

graph LR
    Upload[documentUploads] --> Entry[documentEntries]
    Entry --> Clean[documentCleans]
    Clean --> Text[documentTextExtractions]
    Text --> Results[results]

Query Dependency System

Dependency Management

-- Query-to-query dependencies
CREATE TABLE requiredQueries (
    queryId uuid NOT NULL REFERENCES queries(queryId),
    requiredQueryId uuid NOT NULL REFERENCES queries(queryId),
    versionId int4 NOT NULL,
    addedVersion int4 NOT NULL,
    removedVersion int4,
    CHECK (queryId != requiredQueryId)
);

-- Result-to-result dependencies
CREATE TABLE resultDependencies (
    resultId uuid NOT NULL REFERENCES results(id),
    requiredResultId uuid NOT NULL REFERENCES results(id),
    CHECK (resultId != requiredResultId)
);

Dependency Resolution Views

-- Recursive view for transitive dependencies
CREATE VIEW queryActiveDependencies AS
WITH RECURSIVE deps AS (
    -- Direct dependencies
    SELECT DISTINCT queryId, requiredQueryId, 1 as depth
    FROM requiredQueries rq
    JOIN queryActiveVersions qav ON rq.queryId = qav.queryId AND rq.versionId = qav.versionId
    WHERE rq.removedVersion IS NULL
    
    UNION
    
    -- Transitive dependencies
    SELECT d.queryId, rq.requiredQueryId, d.depth + 1
    FROM deps d
    JOIN requiredQueries rq ON d.requiredQueryId = rq.queryId
    JOIN queryActiveVersions qav ON rq.queryId = qav.queryId AND rq.versionId = qav.versionId
    WHERE rq.removedVersion IS NULL AND d.depth < 10 -- Cycle prevention
)
SELECT * FROM deps;

Collector Configuration System

Client-Query Mapping

-- Maps named queries to specific query implementations for clients
CREATE TABLE collectorQueries (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    clientId varchar(255) NOT NULL,
    name varchar(255) NOT NULL,
    queryId uuid NOT NULL,
    addedVersion int NOT NULL,
    removedVersion int,
    FOREIGN KEY (queryId) REFERENCES queries(queryId),
    FOREIGN KEY (clientId) REFERENCES clients(clientId),
    FOREIGN KEY (clientId, addedVersion) REFERENCES collectorVersions(clientId, id),
    FOREIGN KEY (clientId, removedVersion) REFERENCES collectorVersions(clientId, id),
    UNIQUE (clientId, name, removedVersion)
);

Key Features:

• Maps human-readable query names to query implementations per client
• Version tracking with addedVersion/removedVersion
• Unique constraint on (clientId, name, removedVersion) allows name reuse across versions

Version Thresholds

-- Minimum clean processing version required
CREATE TABLE collectorMinCleanVersions (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    clientId varchar(255) NOT NULL,
    versionId bigint NOT NULL,
    addedVersion int NOT NULL,
    removedVersion int,
    FOREIGN KEY (clientId, addedVersion) REFERENCES collectorVersions(clientId, id),
    FOREIGN KEY (clientId, removedVersion) REFERENCES collectorVersions(clientId, id),
    FOREIGN KEY (clientId) REFERENCES clients(clientId)
);

-- Minimum text extraction version required
CREATE TABLE collectorMinTextVersions (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    clientId varchar(255) NOT NULL,
    versionId bigint NOT NULL,
    addedVersion int NOT NULL,
    removedVersion int,
    FOREIGN KEY (clientId, addedVersion) REFERENCES collectorVersions(clientId, id),
    FOREIGN KEY (clientId, removedVersion) REFERENCES collectorVersions(clientId, id),
    FOREIGN KEY (clientId) REFERENCES clients(clientId)
);

Purpose:

• Controls which document processing versions are valid for a client
• Allows clients to ignore outdated processing results
• Version-tracked to support temporal queries

Complex Dependency Views

-- Builds complete query dependency tree per client
CREATE VIEW collectorQueryDependencyTree AS
SELECT DISTINCT 
    cq.clientId,
    cq.name as queryName,
    qad.queryId,
    qad.requiredQueryId,
    qad.depth
FROM collectorQueries cq
JOIN queryActiveDependencies qad ON cq.queryId = qad.queryId
WHERE cq.removedVersion IS NULL;

Database Views

The system uses extensive views to simplify complex queries and encapsulate business logic. Views are organized by domain.

Query Management Views

Query Current Active Versions (queryCurrentActiveVersions)

Resolves the current active version for each query, handling cases where no active version is set.

CREATE VIEW queryCurrentActiveVersions AS
SELECT DISTINCT
    q.queryId,
    COALESCE(
        (FIRST_VALUE(av.versionId) OVER (PARTITION BY q.queryId ORDER BY av.activeVersionEntryId DESC)),
        0
    )::int AS activeVersion
FROM queries AS q
LEFT JOIN queryActiveVersions AS av ON av.queryId = q.queryId;

Purpose: Provides latest active version per query, defaulting to 0 if none set.

Query Latest Versions (queryLatestVersions)

Tracks the highest version number for each query.

CREATE VIEW queryLatestVersions AS
SELECT
    q.queryId,
    COALESCE(MAX(v.versionId), 0)::int AS latestVersion
FROM queries AS q
LEFT JOIN queryVersions AS v ON v.queryId = q.queryId
GROUP BY q.queryId;

Query Current Configs (queryCurrentConfigs)

Returns the active configuration for each query based on current version.

CREATE VIEW queryCurrentConfigs AS
SELECT av.queryId, c.config
FROM queryCurrentActiveVersions AS av
LEFT JOIN queryConfigs AS c ON av.queryId = c.queryId
    AND isInVersion(av.activeVersion, c.addedVersion, c.removedVersion);

Query Current Required IDs (queryCurrentRequiredIds)

Lists direct dependencies for each query at their current active version.

CREATE VIEW queryCurrentRequiredIds AS
SELECT DISTINCT av.queryId, r.requiredQueryId
FROM queryCurrentActiveVersions AS av
LEFT JOIN requiredQueries AS r ON av.queryId = r.queryId
    AND isInVersion(av.activeVersion, r.addedVersion, r.removedVersion);

Query Current Required IDs Aggregated (queryCurrentRequiredIdsAGG)

Aggregates required query IDs into arrays for easier consumption.

CREATE VIEW queryCurrentRequiredIdsAGG AS
SELECT queryId,
    COALESCE(
        ARRAY_AGG(DISTINCT requiredQueryId)
        FILTER (WHERE requiredQueryId != '00000000-0000-0000-0000-000000000000')::uuid[],
        ARRAY[]::uuid[]
    )::uuid[] AS requiredIds
FROM queryCurrentRequiredIds
GROUP BY queryId;

Full Active Queries (fullActiveQueries)

Complete query information with versions, configs, and dependencies.

CREATE VIEW fullActiveQueries AS
SELECT DISTINCT
    q.queryId,
    q.queryType,
    av.activeVersion,
    lv.latestVersion,
    c.config,
    r.requiredIds
FROM queries AS q
JOIN queryCurrentActiveVersions AS av ON q.queryId = av.queryId
JOIN queryLatestVersions AS lv ON lv.queryId = q.queryId
JOIN queryCurrentConfigs AS c ON q.queryId = c.queryId
JOIN queryCurrentRequiredIdsAGG AS r ON q.queryId = r.queryId;

Purpose: Single view providing all query metadata needed for execution.

Query Active Dependencies (queryActiveDependencies)

Recursive view computing transitive query dependencies with cycle detection.

CREATE VIEW queryActiveDependencies AS
WITH RECURSIVE queryActiveDependencies(queryId, requiredQueryId, path, cycle) AS (
    SELECT
        queryId,
        requiredQueryId,
        ARRAY[queryId, requiredQueryId]::uuid[] AS path,
        false AS cycle
    FROM queryCurrentRequiredIds

    UNION ALL

    SELECT
        q.queryId,
        qd.requiredQueryId,
        path || qd.requiredQueryId,
        qd.requiredQueryId = ANY(path) AS cycle
    FROM queryCurrentRequiredIds AS q
    JOIN queryActiveDependencies AS qd ON q.queryId = qd.requiredQueryId
    WHERE NOT qd.cycle
)
SELECT DISTINCT queryId AS id, requiredQueryId
FROM queryActiveDependencies
WHERE NOT cycle;

Key Features:

• Computes full dependency tree (not just direct dependencies)
• Detects and prevents infinite loops from circular dependencies
• Used for query execution ordering

Collector Management Views

Collector Current Active Versions (collectorCurrentActiveVersions)

Resolves active collector version per client.

CREATE VIEW collectorCurrentActiveVersions AS
SELECT
    c.clientId,
    COALESCE(v.versionId, 0)::int AS activeVersion
FROM clients c
LEFT JOIN (
    SELECT DISTINCT ON (clientId)
        clientId,
        versionId
    FROM collectorActiveVersions
    ORDER BY clientId, id DESC
) v ON c.clientId = v.clientId;

Collector Latest Versions (collectorLatestVersions)

Tracks highest version number per collector.

CREATE VIEW collectorLatestVersions AS
SELECT
    j.clientId,
    COALESCE(MAX(v.id), 0)::int AS latestVersion
FROM clients AS j
LEFT JOIN collectorVersions AS v ON v.clientId = j.clientId
GROUP BY j.clientId;

Current Collector Min Text/Clean Versions

Views tracking minimum processing versions required per client.

CREATE VIEW currentCollectorMinTextVersions AS
SELECT DISTINCT
    av.clientId,
    COALESCE(ctv.versionId, 0) AS minTextVersion
FROM collectorCurrentActiveVersions AS av
LEFT JOIN collectorMinTextVersions AS ctv ON av.clientId = ctv.clientId
    AND isInVersion(av.activeVersion, ctv.addedVersion, ctv.removedVersion);

CREATE VIEW currentCollectorMinCleanVersions AS
SELECT DISTINCT
    av.clientId,
    COALESCE(ctv.versionId, 0) AS minCleanVersion
FROM collectorCurrentActiveVersions AS av
LEFT JOIN collectorMinCleanVersions AS ctv ON av.clientId = ctv.clientId
    AND isInVersion(av.activeVersion, ctv.addedVersion, ctv.removedVersion);

Current Collector Queries (currentCollectorQueries)

Active query mappings per client.

CREATE VIEW currentCollectorQueries AS
SELECT DISTINCT
    av.clientId,
    q.name,
    q.queryId
FROM collectorCurrentActiveVersions AS av
LEFT JOIN collectorQueries AS q ON av.clientId = q.clientId
    AND isInVersion(av.activeVersion, q.addedVersion, q.removedVersion);

Full Active Collectors (fullActiveCollectors)

Complete collector configuration with all metadata.

CREATE VIEW fullActiveCollectors AS
SELECT DISTINCT
    av.clientId,
    ccv.minCleanVersion,
    ctv.minTextVersion,
    av.activeVersion,
    lv.latestVersion,
    q.fields
FROM collectorCurrentActiveVersions AS av
JOIN collectorLatestVersions AS lv ON lv.clientId = av.clientId
JOIN currentCollectorMinCleanVersions AS ccv ON av.clientId = ccv.clientId
JOIN currentCollectorMinTextVersions AS ctv ON av.clientId = ctv.clientId
JOIN currentCollectorQueriesJSONAGG AS q ON av.clientId = q.clientId;

Collector Query Dependency Tree (collectorQueryDependencyTree)

Recursive view computing all queries (direct and transitive) needed by a collector.

CREATE VIEW collectorQueryDependencyTree AS
WITH RECURSIVE collectorQueryDependencyTree AS (
    SELECT cq.clientId, cq.queryId, ri.requiredIds
    FROM currentCollectorQueries AS cq
    JOIN queryCurrentRequiredIdsAGG AS ri ON cq.queryId = ri.queryId

    UNION ALL

    SELECT acq.clientId, q.queryId, q.requiredIds
    FROM queryCurrentRequiredIdsAGG AS q
    JOIN collectorQueryDependencyTree AS acq ON q.queryId = ANY(acq.requiredIds)
)
SELECT DISTINCT
    ct.clientId,
    ct.queryId,
    q.queryType,
    av.activeVersion AS queryVersion,
    ct.requiredIds
FROM collectorQueryDependencyTree AS ct
JOIN queryCurrentActiveVersions AS av ON ct.queryId = av.queryId
JOIN queries AS q ON q.queryId = ct.queryId;

Purpose: Used to determine all queries that must be executed for a client's configuration.

Document Processing Views

Current Clean Entries (currentCleanEntries)

Latest valid clean entry per document meeting minimum version requirements.

CREATE VIEW currentCleanEntries AS
WITH RankedExtractions AS (
    SELECT
        dc.id, dc.documentId, dc.bucket, dc.key, dc.mimetype,
        dc.fail, dc.hash, dce.version, d.clientId,
        ROW_NUMBER() OVER (PARTITION BY dc.documentId ORDER BY dc.id DESC, dce.id DESC) AS row_num
    FROM documents d
    JOIN currentCollectorMinCleanVersions ccv ON d.clientId = ccv.clientId
    JOIN documentCleans dc ON dc.documentId = d.id
    JOIN documentCleanEntries dce ON dc.id = dce.cleanId
        AND dce.version >= ccv.minCleanVersion
)
SELECT id, documentId, bucket, key, version, hash, mimetype, fail, clientId
FROM RankedExtractions
WHERE row_num = 1;

Key Features:

• Filters by collector's minimum clean version requirement
• Returns most recent clean entry per document
• Includes both successful and failed cleans

Current Text Entries (currentTextEntries)

Latest valid text extraction per document meeting minimum version requirements.

CREATE VIEW currentTextEntries AS
WITH RankedExtractions AS (
    SELECT
        extract.id, cc.documentId, extract.bucket, extract.key,
        extract.hash, extract.cleanId, dtee.version AS extractionVersion,
        ROW_NUMBER() OVER (PARTITION BY cc.documentId ORDER BY dtee.id DESC) AS row_num
    FROM documents d
    JOIN currentCleanEntries cc ON cc.documentId = d.id
    JOIN currentCollectorMinTextVersions ctv ON ctv.clientId = d.clientId
    JOIN documentTextExtractions extract ON extract.cleanId = cc.id
    JOIN documentTextExtractionEntries dtee ON dtee.textId = extract.id
        AND dtee.version >= ctv.minTextVersion
)
SELECT id, documentId, bucket, key, hash, cleanId, extractionVersion
FROM RankedExtractions
WHERE row_num = 1;

Purpose: Used to determine which text extractions are valid for query processing.

Current Client Can Sync (currentClientCanSync)

Latest sync permission status per client.

CREATE VIEW currentClientCanSync AS
WITH RankedExtractions AS (
    SELECT
        ccs.clientId,
        ccs.canSync,
        ROW_NUMBER() OVER (PARTITION BY ccs.clientId ORDER BY ccs.syncId DESC) AS row_num
    FROM clients c
    JOIN clientCanSync ccs ON c.clientId = ccs.clientId
)
SELECT
    c.clientId,
    COALESCE(r.canSync, false) AS canSync
FROM clients c
LEFT JOIN RankedExtractions r ON r.clientId = c.clientId AND r.row_num = 1;

Full Clients (fullClients)

Complete client information with sync status.

CREATE VIEW fullClients AS
SELECT c.clientId, c.name, cs.canSync
FROM clients AS c
JOIN currentClientCanSync AS cs ON cs.clientId = c.clientId;

Advanced Database Features

Business Logic Functions

Document Processing Functions

listDocumentIDs: Paginated document listing with total count.

CREATE OR REPLACE FUNCTION listDocumentIDs(
    _clientId varchar(255),
    _batchSize INTEGER,
    _offset INTEGER
)
RETURNS TABLE (id uuid, totalCount BIGINT) AS $$
DECLARE
    totalCount BIGINT := 0;
BEGIN
    SELECT COUNT(*)::BIGINT INTO totalCount FROM documents WHERE clientId = _clientId;

    RETURN QUERY
        SELECT d.id, totalCount
        FROM documents AS d
        WHERE d.clientId = _clientId
        ORDER BY d.id
        LIMIT _batchSize
        OFFSET _offset;
END;
$$ LANGUAGE plpgsql;

Key Features:

• Returns both document IDs and total count in single query
• Supports pagination via batch size and offset
• Orders by document ID for consistent paging

collectorQueryDependencyTreeByClient: Optimized function for getting query dependencies per client.

CREATE OR REPLACE FUNCTION collectorQueryDependencyTreeByClient(
    _clientId varchar(255)
)
RETURNS TABLE (
    clientId varchar(255),
    queryId uuid,
    queryType queryType,
    queryVersion int,
    requiredIds uuid[]
) AS $$
BEGIN
    RETURN QUERY
    WITH clientQueries AS (
        SELECT q.clientId, q.queryId
        FROM currentCollectorQueries AS q
        WHERE q.clientId = _clientId
    ),
    dependencyTree AS (
        WITH RECURSIVE collectorQueryDependencyTree AS (
            SELECT cq.clientId, cq.queryId, ri.requiredIds
            FROM clientQueries AS cq
            JOIN queryCurrentRequiredIdsAGG AS ri ON cq.queryId = ri.queryId

            UNION ALL

            SELECT acq.clientId, q.queryId, q.requiredIds
            FROM queryCurrentRequiredIdsAGG AS q
            JOIN collectorQueryDependencyTree AS acq ON q.queryId = ANY(acq.requiredIds)
        )
        SELECT DISTINCT
            ct.clientId, ct.queryId, q.queryType,
            av.activeVersion AS queryVersion, ct.requiredIds
        FROM collectorQueryDependencyTree AS ct
        JOIN queryCurrentActiveVersions AS av ON ct.queryId = av.queryId
        JOIN queries AS q ON q.queryId = ct.queryId
    )
    SELECT t.clientId, t.queryId, t.queryType, t.queryVersion, t.requiredIds
    FROM dependencyTree AS t
    WHERE t.clientID IS NOT NULL;
END;
$$ LANGUAGE plpgsql;

Purpose: More efficient than view for single-client queries.

Result Validation Functions

listValidDocumentResults: Complex dependency resolution for query results.

CREATE OR REPLACE FUNCTION listValidDocumentResults(doc_id uuid)
RETURNS TABLE (documentId uuid, queryId uuid, resultId uuid, value text) AS $$
DECLARE
    count_before INT;
    count_after INT;
    iterations INT := 0;
BEGIN
    -- Creates temporary tables and iteratively resolves dependencies
    -- Full implementation in migration 00000000000102_document_views.up.sql:103-211
END;
$$ LANGUAGE plpgsql;

Key Features:

• Validates that all query dependencies have been satisfied
• Uses temporary tables for efficient processing
• Iteratively builds valid result set
• Checks both query dependencies and result dependencies
• Returns only results where all prerequisites are met

Database Triggers

The schema uses several triggers to automate versioning and configuration management:

Query Version Auto-Increment

CREATE OR REPLACE FUNCTION setQueryVersionNumber()
RETURNS TRIGGER AS $$
BEGIN
    SELECT COALESCE(MAX(versionId), 0) + 1
    INTO NEW.versionId
    FROM queryVersions
    WHERE queryId = NEW.queryId;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER setQueryVersionNumberTrigger
BEFORE INSERT ON queryVersions
FOR EACH ROW
EXECUTE FUNCTION setQueryVersionNumber();

Collector Version Auto-Increment

CREATE OR REPLACE FUNCTION setCollectorVersionNumber()
RETURNS TRIGGER AS $$
BEGIN
    SELECT COALESCE(MAX(id), 0) + 1
    INTO NEW.id
    FROM collectorVersions
    WHERE clientId = NEW.clientId;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER setCollectorVersionNumberTrigger
BEFORE INSERT ON collectorVersions
FOR EACH ROW
EXECUTE FUNCTION setCollectorVersionNumber();

Query Config Version Management

CREATE OR REPLACE FUNCTION removeQueryConfig()
RETURNS TRIGGER AS $$
BEGIN
    UPDATE queryConfigs
        SET removedVersion = NEW.addedVersion
        WHERE queryId = NEW.queryId AND removedVersion IS NULL;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER removeQueryConfigTrigger
BEFORE INSERT ON queryConfigs
FOR EACH ROW
EXECUTE FUNCTION removeQueryConfig();

Purpose: Automatically closes out previous configs when new ones are added.

Collector Min Version Management

Similar triggers exist for collectorMinCleanVersions and collectorMinTextVersions to automatically set removedVersion when new minimum versions are added.

Data Integrity and Constraints

Referential Integrity

• All foreign key relationships enforced at database level
• No cascade deletes (explicit deletion required for safety)
• Unique constraints prevent duplicate business entities
• Composite foreign keys ensure version referential integrity

Data Validation Constraints

Enum Types

• queryType: 'context_full', 'json_extractor'
• cleanMimeType: 'application/pdf'
• cleanFailType: Various failure reasons (invalid_mimetype, invalid_read, etc.)
• batch_status: 'processing', 'completed', 'failed', 'cancelled'

Check Constraints

• requiredQueries: Self-reference prevention (queryId != requiredQueryId)
• resultDependencies: Self-reference prevention (resultId != requiredResultId)
• documentCleans: XOR constraint ensures either success OR failure, not both
  • bucket_and_key_together: Bucket, key, mimetype, hash must all be present or all null
  • location_xor_fail: Cannot have both location (success) and fail (failure) indicators

Custom Domain Types

• unsignedsmallint: SMALLINT constrained to non-negative values
• Used for part numbers in storage distribution

Audit and Traceability

• UUID v7 generation provides time-ordered insertion
• Version tracking enables complete audit trails
• Temporal validity functions support point-in-time queries
• Triggers automatically manage version transitions
• All critical tables have UUID primary keys for global uniqueness

Scalability Considerations

Indexing Strategy

• Primary keys automatically indexed (UUID v7)
• Foreign keys automatically indexed
• Composite indexes on frequent query patterns
• Unique constraints create secondary indexes

Partitioning Opportunities

• Time-based partitioning using UUID v7 timestamp component
• Client-based partitioning for multi-tenant isolation
• Archive strategies for old versions

Performance Optimizations

• Window functions for latest version queries
• Recursive CTEs with depth limits for dependency resolution
• Pagination support in all list functions
• Efficient temporal queries using isInVersion() function

Migration Strategy

Database Evolution

• Sequential numbered migrations for version control (00000000000001, 00000000000002, etc.)
• Separate UP/DOWN migrations for rollback capability
• Extension management for UUID generation and other features
• View-based abstractions for query flexibility
• Migration files located in: internal/database/migrations/

Migration Numbering Convention

• 00000000000001-00000000000099: Core tables (extensions, queries, clients, collectors, documents, results)
• 00000000000100-00000000000199: Views and functions
• 00000000000200+: Schema modifications and new features

Recent Schema Changes

• Migration 103: Added batch_uploads table and documents.batch_id
• Migration 104: Added S3 storage columns to batch_uploads (archive_bucket, archive_key, file_size_bytes) with storage consistency constraint
• Migration 105: Added documents.filename column
• Migration 106: Added documentUploads.filename column
• Migration 107: Added documentUploads.batch_id column (without foreign key constraint)
• Migration 108: Added index on documentUploads.clientId
• Migration 109: Added folders table for virtual folder hierarchy with self-referential parentId
• Migration 110: Added labels and documentLabels tables for document processing status tracking
• Migration 111: Added documents.folderId and documents.originalPath columns; added comments for S3 immutability
• Migration 112: Added documentFieldExtractions table with 19 single-value fields
• Migration 113: Added documentFieldExtractionVersions table for version tracking
• Migration 114: Added documentFieldExtractionArrayFields table with 112 array fields
• Migration 115: Added currentFieldExtractions and currentFieldExtractionsWithArrayCount views
• Migration 116: Added documentUploads.folder_id column with foreign key constraint
• Migration 121: Added eulaVersions and eulaAgreements tables for EULA management and consent tracking

Version Compatibility

• Backward-compatible schema changes prioritized
• Graceful handling of version mismatches via views
• New columns added as nullable to avoid breaking existing code
• Indexes added separately from table creation for safety

Key Schema Patterns

Naming Conventions

• Tables: Generally use camelCase (e.g., clientId, documentId)
• Exception: batch_uploads uses snake_case (client_id, original_filename)
• Views: camelCase naming (e.g., currentCleanEntries, fullActiveQueries)
• Functions: camelCase naming (e.g., listDocumentIDs, isInVersion)

Common Data Types

• IDs: uuid with DEFAULT uuid_generate_v7() for time-ordered insertion
• Client references: varchar(255) for clientId
• Text fields: TEXT type (not varchar) for unlimited length
• Timestamps: timestamp without timezone
• Versions: int or bigint depending on expected range
• Part numbers: unsignedsmallint (0-32767)
• JSON data: jsonb for queryable JSON storage

Versioning Pattern

Most versioned entities follow this pattern:

1. Core table with basic info (e.g., queries)
2. Versions table with composite PK (e.g., queryVersions)
3. Active version tracking table (e.g., queryActiveVersions)
4. Related data with addedVersion/removedVersion (e.g., queryConfigs)
5. Views to compute current state (e.g., queryCurrentActiveVersions)
6. Triggers to auto-increment versions

View Pattern

Views are layered for reusability:

1. Base views: Compute current active versions
2. Aggregation views: Roll up related data (e.g., queryCurrentRequiredIdsAGG)
3. Full views: Join everything together (e.g., fullActiveQueries)
4. Recursive views: Compute transitive relationships (e.g., queryActiveDependencies)

Performance Considerations

• Views use DISTINCT liberally to handle version overlap
• Window functions (ROW_NUMBER, FIRST_VALUE) for latest record selection
• Recursive CTEs include cycle detection
• Indexes created on foreign keys and frequently filtered columns
• Pagination functions return total count to avoid separate COUNT query

Schema Statistics

Table Count

• Core entities: 5 (clients, documents, batch_uploads, queries, results)
• Organization tables: 3 (folders, labels, documentLabels)
• Field extraction tables: 3 (documentFieldExtractions, documentFieldExtractionVersions, documentFieldExtractionArrayFields)
• EULA tables: 2 (eulaVersions, eulaAgreements)
• Supporting tables: 16 (versions, entries, configs, dependencies, etc.)
• Total tables: 29

View Count

• Query management: 6 views
• Collector management: 7 views
• Document processing: 3 views
• Field extraction: 2 views (currentFieldExtractions, currentFieldExtractionsWithArrayCount)
• Total views: 18+

Function Count

• Business logic: 3 main functions
• Trigger functions: 5 functions
• Utility functions: 2 (uuid_generate_v7, isInVersion)
• Total functions: 10+

Enum Types

• queryType: 2 values
• cleanMimeType: 1 value
• cleanFailType: 9 values
• batch_status: 4 values
• Total enums: 4 types, 16 values