query-orchestration/docs/ai.generated/04-data-architecture.md

Data Architecture

Database Schema Overview

The system uses PostgreSQL 17.2 with a sophisticated schema designed for multi-tenant document processing, query orchestration, and comprehensive versioning. The schema is managed through database migrations and uses SQLC for type-safe Go integration.

Core Entities

Clients (clients)

Central entity representing organizations or tenants in the system.

CREATE TABLE clients (
    clientId varchar(256) PRIMARY KEY,
    name varchar(256) NOT NULL
);

Key Characteristics:

• String-based client identifiers for human readability
• Multi-tenant isolation through client-scoped data
• Central reference point for all client-owned entities

Documents (documents)

Represents uploaded files requiring processing.

CREATE TABLE documents (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    clientId varchar(256) NOT NULL REFERENCES clients(clientId),
    hash varchar(256) NOT NULL,
    filename varchar(512),
    UNIQUE(clientId, hash)
);

Key Features:

• UUID v7 primary keys for time-ordered insertion
• Client isolation with foreign key constraints
• Hash-based deduplication within client scope
• Filename preservation for folder path support
• Supports document processing pipeline

Batch Uploads (batch_uploads)

Tracks batch document upload operations with ZIP archive processing.

CREATE TYPE batchstatus AS ENUM ('processing', 'completed', 'failed', 'cancelled');

CREATE TABLE batch_uploads (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    clientId varchar(256) NOT NULL REFERENCES clients(clientId),
    status batchstatus NOT NULL DEFAULT 'processing',
    total_documents integer NOT NULL DEFAULT 0,
    processed_documents integer NOT NULL DEFAULT 0,
    failed_documents integer NOT NULL DEFAULT 0,
    failed_filenames text[],
    s3_bucket varchar(256),
    s3_key varchar(512),
    created_at timestamp NOT NULL DEFAULT NOW(),
    completed_at timestamp
);

Batch Processing Features:

• UUID v7 for time-ordered batch tracking
• Comprehensive status tracking with ENUM types
• Progress counters for processed/failed documents
• Failed filename tracking for error reporting
• S3 storage metadata for archive management
• Temporal tracking with creation and completion timestamps

Queries (queries)

Defines data extraction logic with type-specific implementations.

CREATE TABLE queries (
    queryId uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    queryType querytype NOT NULL
);

CREATE TYPE querytype AS ENUM ('context_full', 'json_extractor');

Query Types:

• context_full: Returns complete document context
• json_extractor: Extracts specific JSON paths from structured data

Results (results)

Stores extracted data from document processing.

CREATE TABLE results (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    textEntryId uuid NOT NULL REFERENCES documentTextExtractions(id),
    queryId uuid NOT NULL REFERENCES queries(queryId),
    value text NOT NULL,
    queryVersion int4 NOT NULL
);

Versioning System

The schema implements a comprehensive versioning system enabling temporal queries and configuration evolution.

Version Management Pattern

-- Generic versioning pattern
CREATE TABLE entity_versions (
    entityId uuid NOT NULL,
    versionId int4 NOT NULL,
    addedVersion int4 NOT NULL,
    removedVersion int4,
    -- entity-specific fields
    PRIMARY KEY (entityId, versionId)
);

CREATE TABLE entity_active_versions (
    entityId uuid PRIMARY KEY,
    versionId int4 NOT NULL
);

Temporal Validity Functions

-- Check if version is active at specific point
CREATE OR REPLACE FUNCTION isInVersion(addedVersion int4, removedVersion int4, targetVersion int4)
RETURNS boolean AS $$
BEGIN
    RETURN addedVersion <= targetVersion AND (removedVersion IS NULL OR removedVersion > targetVersion);
END;
$$ LANGUAGE plpgsql IMMUTABLE;

Version-Enabled Entities

• Query Versions: queryVersions, queryActiveVersions
• Collector Versions: collectorVersions, collectorActiveVersions
• Processing Versions: Track minimum required versions for document stages

Document Processing Pipeline

Document Lifecycle Tables

Document Uploads (documentUploads)

Initial upload tracking for audit and debugging.

Document Entries (documentEntries)

Core document references after deduplication and validation.

Document Cleans (documentCleans)

PDF validation and format processing records.

Document Text Extractions (documentTextExtractions)

OCR and text extraction results from AWS Textract.

Processing Dependencies

link to rendered version here

graph LR
    Upload[documentUploads] --> Entry[documentEntries]
    Entry --> Clean[documentCleans]
    Clean --> Text[documentTextExtractions]
    Text --> Results[results]

Query Dependency System

Dependency Management

-- Query-to-query dependencies
CREATE TABLE requiredQueries (
    queryId uuid NOT NULL REFERENCES queries(queryId),
    requiredQueryId uuid NOT NULL REFERENCES queries(queryId),
    versionId int4 NOT NULL,
    addedVersion int4 NOT NULL,
    removedVersion int4,
    CHECK (queryId != requiredQueryId)
);

-- Result-to-result dependencies
CREATE TABLE resultDependencies (
    resultId uuid NOT NULL REFERENCES results(id),
    requiredResultId uuid NOT NULL REFERENCES results(id),
    CHECK (resultId != requiredResultId)
);

Dependency Resolution Views

-- Recursive view for transitive dependencies
CREATE VIEW queryActiveDependencies AS
WITH RECURSIVE deps AS (
    -- Direct dependencies
    SELECT DISTINCT queryId, requiredQueryId, 1 as depth
    FROM requiredQueries rq
    JOIN queryActiveVersions qav ON rq.queryId = qav.queryId AND rq.versionId = qav.versionId
    WHERE rq.removedVersion IS NULL
    
    UNION
    
    -- Transitive dependencies
    SELECT d.queryId, rq.requiredQueryId, d.depth + 1
    FROM deps d
    JOIN requiredQueries rq ON d.requiredQueryId = rq.queryId
    JOIN queryActiveVersions qav ON rq.queryId = qav.queryId AND rq.versionId = qav.versionId
    WHERE rq.removedVersion IS NULL AND d.depth < 10 -- Cycle prevention
)
SELECT * FROM deps;

Collector Configuration System

Client-Query Mapping

-- Maps named queries to specific query implementations for clients
CREATE TABLE collectorQueries (
    clientId varchar(256) NOT NULL REFERENCES clients(clientId),
    name varchar(256) NOT NULL,
    queryId uuid NOT NULL REFERENCES queries(queryId),
    versionId int4 NOT NULL,
    addedVersion int4 NOT NULL,
    removedVersion int4
);

Version Thresholds

-- Minimum processing versions required
CREATE TABLE collectorMinCleanVersions (
    clientId varchar(256) PRIMARY KEY REFERENCES clients(clientId),
    version int4 NOT NULL
);

CREATE TABLE collectorMinTextVersions (
    clientId varchar(256) PRIMARY KEY REFERENCES clients(clientId), 
    version int4 NOT NULL
);

Complex Dependency Views

-- Builds complete query dependency tree per client
CREATE VIEW collectorQueryDependencyTree AS
SELECT DISTINCT 
    cq.clientId,
    cq.name as queryName,
    qad.queryId,
    qad.requiredQueryId,
    qad.depth
FROM collectorQueries cq
JOIN queryActiveDependencies qad ON cq.queryId = qad.queryId
WHERE cq.removedVersion IS NULL;

Advanced Database Features

Business Logic Functions

Document Processing Functions

-- List documents ready for processing
CREATE OR REPLACE FUNCTION listDocumentIDs(clientId varchar(256), limit_val int4, offset_val int4)
RETURNS TABLE(documentId uuid) AS $$
BEGIN
    RETURN QUERY
    SELECT de.documentId
    FROM documentEntries de
    WHERE de.clientId = $1
    ORDER BY de.documentId
    LIMIT limit_val OFFSET offset_val;
END;
$$ LANGUAGE plpgsql;

Result Validation Functions

-- Complex validation for result dependencies
CREATE OR REPLACE FUNCTION listValidDocumentResults(
    clientId varchar(256),
    documentId uuid,
    queryId uuid
)
RETURNS TABLE(resultId uuid, value text) AS $$
-- Complex logic for dependency resolution
$$ LANGUAGE plpgsql;

Materialized Views and Performance

-- Current clean entries above minimum thresholds
CREATE VIEW currentCleanEntries AS
SELECT DISTINCT ON (dc.documentId) 
    dc.*
FROM documentCleans dc
JOIN collectorMinCleanVersions cmcv ON dc.clientId = cmcv.clientId
WHERE dc.version >= cmcv.version
ORDER BY dc.documentId, dc.version DESC;

Data Integrity and Constraints

Referential Integrity

• All foreign key relationships enforced at database level
• Cascade delete policies for cleanup operations
• Unique constraints prevent duplicate business entities

Data Validation

• Enum types for controlled vocabularies (querytype, mimetype)
• Check constraints for business rules (no self-dependencies)
• NOT NULL constraints for required fields

Audit and Traceability

• UUID v7 generation provides insertion ordering
• Version tracking enables complete audit trails
• Temporal validity supports point-in-time queries

Scalability Considerations

Indexing Strategy

• Primary keys automatically indexed (UUID v7)
• Foreign keys automatically indexed
• Composite indexes on frequent query patterns
• Unique constraints create secondary indexes

Partitioning Opportunities

• Time-based partitioning using UUID v7 timestamp component
• Client-based partitioning for multi-tenant isolation
• Archive strategies for old versions

Performance Optimizations

• Window functions for latest version queries
• Recursive CTEs with depth limits for dependency resolution
• Pagination support in all list functions
• Efficient temporal queries using isInVersion() function

Migration Strategy

Database Evolution

• Sequential numbered migrations for version control
• Separate UP/DOWN migrations for rollback capability
• Extension management for UUID generation and other features
• View-based abstractions for query flexibility

Version Compatibility

• Backward-compatible schema changes
• Graceful handling of version mismatches
• Data migration scripts for major version upgrades