query-orchestration/docs/ai.generated/04-data-architecture.md

# Data Architecture

## Database Schema Overview

The system uses PostgreSQL 17.2 with a sophisticated schema designed for multi-tenant document processing, query orchestration, and comprehensive versioning. The schema is managed through database migrations and uses SQLC for type-safe Go integration.

## Core Entities

### Clients (`clients`)
Central entity representing organizations or tenants in the system.

```sql
CREATE TABLE clients (
    clientId varchar(256) PRIMARY KEY,
    name varchar(256) NOT NULL
);
```

**Key Characteristics**:
- String-based client identifiers for human readability
- Multi-tenant isolation through client-scoped data
- Central reference point for all client-owned entities

### Documents (`documents`)
Represents uploaded files requiring processing.

```sql
CREATE TABLE documents (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    clientId varchar(256) NOT NULL REFERENCES clients(clientId),
    hash varchar(256) NOT NULL,
    filename varchar(512),
    UNIQUE(clientId, hash)
);
```

**Key Features**:
- UUID v7 primary keys for time-ordered insertion
- Client isolation with foreign key constraints
- Hash-based deduplication within client scope
- Filename preservation for folder path support
- Supports document processing pipeline

### Batch Uploads (`batch_uploads`)
Tracks batch document upload operations with ZIP archive processing.

```sql
CREATE TYPE batchstatus AS ENUM ('processing', 'completed', 'failed', 'cancelled');

CREATE TABLE batch_uploads (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    clientId varchar(256) NOT NULL REFERENCES clients(clientId),
    status batchstatus NOT NULL DEFAULT 'processing',
    total_documents integer NOT NULL DEFAULT 0,
    processed_documents integer NOT NULL DEFAULT 0,
    failed_documents integer NOT NULL DEFAULT 0,
    failed_filenames text[],
    s3_bucket varchar(256),
    s3_key varchar(512),
    created_at timestamp NOT NULL DEFAULT NOW(),
    completed_at timestamp
);
```

**Batch Processing Features**:
- UUID v7 for time-ordered batch tracking
- Comprehensive status tracking with ENUM types
- Progress counters for processed/failed documents
- Failed filename tracking for error reporting
- S3 storage metadata for archive management
- Temporal tracking with creation and completion timestamps

### Queries (`queries`)
Defines data extraction logic with type-specific implementations.

```sql
CREATE TABLE queries (
    queryId uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    queryType querytype NOT NULL
);

CREATE TYPE querytype AS ENUM ('context_full', 'json_extractor');
```

**Query Types**:
- **`context_full`**: Returns complete document context
- **`json_extractor`**: Extracts specific JSON paths from structured data

### Results (`results`)
Stores extracted data from document processing.

```sql
CREATE TABLE results (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
    textEntryId uuid NOT NULL REFERENCES documentTextExtractions(id),
    queryId uuid NOT NULL REFERENCES queries(queryId),
    value text NOT NULL,
    queryVersion int4 NOT NULL
);
```

## Versioning System

The schema implements a comprehensive versioning system enabling temporal queries and configuration evolution.

### Version Management Pattern
```sql
-- Generic versioning pattern
CREATE TABLE entity_versions (
    entityId uuid NOT NULL,
    versionId int4 NOT NULL,
    addedVersion int4 NOT NULL,
    removedVersion int4,
    -- entity-specific fields
    PRIMARY KEY (entityId, versionId)
);

CREATE TABLE entity_active_versions (
    entityId uuid PRIMARY KEY,
    versionId int4 NOT NULL
);
```

### Temporal Validity Functions
```sql
-- Check if version is active at specific point
CREATE OR REPLACE FUNCTION isInVersion(addedVersion int4, removedVersion int4, targetVersion int4)
RETURNS boolean AS $$
BEGIN
    RETURN addedVersion <= targetVersion AND (removedVersion IS NULL OR removedVersion > targetVersion);
END;
$$ LANGUAGE plpgsql IMMUTABLE;
```

### Version-Enabled Entities
- **Query Versions**: `queryVersions`, `queryActiveVersions`
- **Collector Versions**: `collectorVersions`, `collectorActiveVersions`
- **Processing Versions**: Track minimum required versions for document stages

## Document Processing Pipeline

### Document Lifecycle Tables

#### Document Uploads (`documentUploads`)
Initial upload tracking for audit and debugging.

#### Document Entries (`documentEntries`)
Core document references after deduplication and validation.

#### Document Cleans (`documentCleans`)
PDF validation and format processing records.

#### Document Text Extractions (`documentTextExtractions`)
OCR and text extraction results from AWS Textract.

### Processing Dependencies
[link to rendered version here](https://mermaid.live/edit#pako:eNo9jz0KwzAMRq9iNDcX6NClZOsU2inNYGy1Mfgn2AokhNy9tkytRTx9TyAdoIJGuAr4RrnM4jG8vcj1WmyQetRBrQ49VUyT6Lqb6D3FvUWFDKap7nHG1t2i9M1i-ksMLD1xo-YU6DeKUpEJTS5TdgdMq6U0xtonuAhwGJ00Ot9_AM3o-BONH5kNOM8fK3VMWw==)

```mermaid
graph LR
    Upload[documentUploads] --> Entry[documentEntries]
    Entry --> Clean[documentCleans]
    Clean --> Text[documentTextExtractions]
    Text --> Results[results]
```

## Query Dependency System

### Dependency Management
```sql
-- Query-to-query dependencies
CREATE TABLE requiredQueries (
    queryId uuid NOT NULL REFERENCES queries(queryId),
    requiredQueryId uuid NOT NULL REFERENCES queries(queryId),
    versionId int4 NOT NULL,
    addedVersion int4 NOT NULL,
    removedVersion int4,
    CHECK (queryId != requiredQueryId)
);

-- Result-to-result dependencies
CREATE TABLE resultDependencies (
    resultId uuid NOT NULL REFERENCES results(id),
    requiredResultId uuid NOT NULL REFERENCES results(id),
    CHECK (resultId != requiredResultId)
);
```

### Dependency Resolution Views
```sql
-- Recursive view for transitive dependencies
CREATE VIEW queryActiveDependencies AS
WITH RECURSIVE deps AS (
    -- Direct dependencies
    SELECT DISTINCT queryId, requiredQueryId, 1 as depth
    FROM requiredQueries rq
    JOIN queryActiveVersions qav ON rq.queryId = qav.queryId AND rq.versionId = qav.versionId
    WHERE rq.removedVersion IS NULL
    
    UNION
    
    -- Transitive dependencies
    SELECT d.queryId, rq.requiredQueryId, d.depth + 1
    FROM deps d
    JOIN requiredQueries rq ON d.requiredQueryId = rq.queryId
    JOIN queryActiveVersions qav ON rq.queryId = qav.queryId AND rq.versionId = qav.versionId
    WHERE rq.removedVersion IS NULL AND d.depth < 10 -- Cycle prevention
)
SELECT * FROM deps;
```

## Collector Configuration System

### Client-Query Mapping
```sql
-- Maps named queries to specific query implementations for clients
CREATE TABLE collectorQueries (
    clientId varchar(256) NOT NULL REFERENCES clients(clientId),
    name varchar(256) NOT NULL,
    queryId uuid NOT NULL REFERENCES queries(queryId),
    versionId int4 NOT NULL,
    addedVersion int4 NOT NULL,
    removedVersion int4
);
```

### Version Thresholds
```sql
-- Minimum processing versions required
CREATE TABLE collectorMinCleanVersions (
    clientId varchar(256) PRIMARY KEY REFERENCES clients(clientId),
    version int4 NOT NULL
);

CREATE TABLE collectorMinTextVersions (
    clientId varchar(256) PRIMARY KEY REFERENCES clients(clientId), 
    version int4 NOT NULL
);
```

### Complex Dependency Views
```sql
-- Builds complete query dependency tree per client
CREATE VIEW collectorQueryDependencyTree AS
SELECT DISTINCT 
    cq.clientId,
    cq.name as queryName,
    qad.queryId,
    qad.requiredQueryId,
    qad.depth
FROM collectorQueries cq
JOIN queryActiveDependencies qad ON cq.queryId = qad.queryId
WHERE cq.removedVersion IS NULL;
```

## Advanced Database Features

### Business Logic Functions

#### Document Processing Functions
```sql
-- List documents ready for processing
CREATE OR REPLACE FUNCTION listDocumentIDs(clientId varchar(256), limit_val int4, offset_val int4)
RETURNS TABLE(documentId uuid) AS $$
BEGIN
    RETURN QUERY
    SELECT de.documentId
    FROM documentEntries de
    WHERE de.clientId = $1
    ORDER BY de.documentId
    LIMIT limit_val OFFSET offset_val;
END;
$$ LANGUAGE plpgsql;
```

#### Result Validation Functions
```sql
-- Complex validation for result dependencies
CREATE OR REPLACE FUNCTION listValidDocumentResults(
    clientId varchar(256),
    documentId uuid,
    queryId uuid
)
RETURNS TABLE(resultId uuid, value text) AS $$
-- Complex logic for dependency resolution
$$ LANGUAGE plpgsql;
```

### Materialized Views and Performance
```sql
-- Current clean entries above minimum thresholds
CREATE VIEW currentCleanEntries AS
SELECT DISTINCT ON (dc.documentId) 
    dc.*
FROM documentCleans dc
JOIN collectorMinCleanVersions cmcv ON dc.clientId = cmcv.clientId
WHERE dc.version >= cmcv.version
ORDER BY dc.documentId, dc.version DESC;
```

## Data Integrity and Constraints

### Referential Integrity
- All foreign key relationships enforced at database level
- Cascade delete policies for cleanup operations
- Unique constraints prevent duplicate business entities

### Data Validation
- Enum types for controlled vocabularies (`querytype`, `mimetype`)
- Check constraints for business rules (no self-dependencies)
- NOT NULL constraints for required fields

### Audit and Traceability
- UUID v7 generation provides insertion ordering
- Version tracking enables complete audit trails
- Temporal validity supports point-in-time queries

## Scalability Considerations

### Indexing Strategy
- Primary keys automatically indexed (UUID v7)
- Foreign keys automatically indexed
- Composite indexes on frequent query patterns
- Unique constraints create secondary indexes

### Partitioning Opportunities
- Time-based partitioning using UUID v7 timestamp component
- Client-based partitioning for multi-tenant isolation
- Archive strategies for old versions

### Performance Optimizations
- Window functions for latest version queries
- Recursive CTEs with depth limits for dependency resolution
- Pagination support in all list functions
- Efficient temporal queries using `isInVersion()` function

## Migration Strategy

### Database Evolution
- Sequential numbered migrations for version control
- Separate UP/DOWN migrations for rollback capability
- Extension management for UUID generation and other features
- View-based abstractions for query flexibility

### Version Compatibility
- Backward-compatible schema changes
- Graceful handling of version mismatches
- Data migration scripts for major version upgrades