317 lines
9.2 KiB
Markdown
317 lines
9.2 KiB
Markdown
|
# Data Architecture
|
||
|
|
|
||
|
|
## Database Schema Overview
|
||
|
|
|
||
|
|
The system uses PostgreSQL 17.2 with a sophisticated schema designed for multi-tenant document processing, query orchestration, and comprehensive versioning. The schema is managed through database migrations and uses SQLC for type-safe Go integration.
|
||
|
|
|
||
|
|
## Core Entities
|
||
|
|
|
||
|
|
### Clients (`clients`)
|
||
|
|
Central entity representing organizations or tenants in the system.
|
||
|
|
|
||
|
|
```sql
|
||
|
|
CREATE TABLE clients (
|
||
|
|
clientId varchar(256) PRIMARY KEY,
|
||
|
|
name varchar(256) NOT NULL
|
||
|
|
);
|
||
|
|
```
|
||
|
|
|
||
|
|
**Key Characteristics**:
|
||
|
|
- String-based client identifiers for human readability
|
||
|
|
- Multi-tenant isolation through client-scoped data
|
||
|
|
- Central reference point for all client-owned entities
|
||
|
|
|
||
|
|
### Documents (`documents`)
|
||
|
|
Represents uploaded files requiring processing.
|
||
|
|
|
||
|
|
```sql
|
||
|
|
CREATE TABLE documents (
|
||
|
|
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
|
||
|
|
clientId varchar(256) NOT NULL REFERENCES clients(clientId),
|
||
|
|
hash varchar(256) NOT NULL,
|
||
|
|
UNIQUE(clientId, hash)
|
||
|
|
);
|
||
|
|
```
|
||
|
|
|
||
|
|
**Key Features**:
|
||
|
|
- UUID v7 primary keys for time-ordered insertion
|
||
|
|
- Client isolation with foreign key constraints
|
||
|
|
- Hash-based deduplication within client scope
|
||
|
|
- Supports document processing pipeline
|
||
|
|
|
||
|
|
### Queries (`queries`)
|
||
|
|
Defines data extraction logic with type-specific implementations.
|
||
|
|
|
||
|
|
```sql
|
||
|
|
CREATE TABLE queries (
|
||
|
|
queryId uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
|
||
|
|
queryType querytype NOT NULL
|
||
|
|
);
|
||
|
|
|
||
|
|
CREATE TYPE querytype AS ENUM ('context_full', 'json_extractor');
|
||
|
|
```
|
||
|
|
|
||
|
|
**Query Types**:
|
||
|
|
- **`context_full`**: Returns complete document context
|
||
|
|
- **`json_extractor`**: Extracts specific JSON paths from structured data
|
||
|
|
|
||
|
|
### Results (`results`)
|
||
|
|
Stores extracted data from document processing.
|
||
|
|
|
||
|
|
```sql
|
||
|
|
CREATE TABLE results (
|
||
|
|
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
|
||
|
|
textEntryId uuid NOT NULL REFERENCES documentTextExtractions(id),
|
||
|
|
queryId uuid NOT NULL REFERENCES queries(queryId),
|
||
|
|
value text NOT NULL,
|
||
|
|
queryVersion int4 NOT NULL
|
||
|
|
);
|
||
|
|
```
|
||
|
|
|
||
|
|
## Versioning System
|
||
|
|
|
||
|
|
The schema implements a comprehensive versioning system enabling temporal queries and configuration evolution.
|
||
|
|
|
||
|
|
### Version Management Pattern
|
||
|
|
```sql
|
||
|
|
-- Generic versioning pattern
|
||
|
|
CREATE TABLE entity_versions (
|
||
|
|
entityId uuid NOT NULL,
|
||
|
|
versionId int4 NOT NULL,
|
||
|
|
addedVersion int4 NOT NULL,
|
||
|
|
removedVersion int4,
|
||
|
|
-- entity-specific fields
|
||
|
|
PRIMARY KEY (entityId, versionId)
|
||
|
|
);
|
||
|
|
|
||
|
|
CREATE TABLE entity_active_versions (
|
||
|
|
entityId uuid PRIMARY KEY,
|
||
|
|
versionId int4 NOT NULL
|
||
|
|
);
|
||
|
|
```
|
||
|
|
|
||
|
|
### Temporal Validity Functions
|
||
|
|
```sql
|
||
|
|
-- Check if version is active at specific point
|
||
|
|
CREATE OR REPLACE FUNCTION isInVersion(addedVersion int4, removedVersion int4, targetVersion int4)
|
||
|
|
RETURNS boolean AS $$
|
||
|
|
BEGIN
|
||
|
|
RETURN addedVersion <= targetVersion AND (removedVersion IS NULL OR removedVersion > targetVersion);
|
||
|
|
END;
|
||
|
|
$$ LANGUAGE plpgsql IMMUTABLE;
|
||
|
|
```
|
||
|
|
|
||
|
|
### Version-Enabled Entities
|
||
|
|
- **Query Versions**: `queryVersions`, `queryActiveVersions`
|
||
|
|
- **Collector Versions**: `collectorVersions`, `collectorActiveVersions`
|
||
|
|
- **Processing Versions**: Track minimum required versions for document stages
|
||
|
|
|
||
|
|
## Document Processing Pipeline
|
||
|
|
|
||
|
|
### Document Lifecycle Tables
|
||
|
|
|
||
|
|
#### Document Uploads (`documentUploads`)
|
||
|
|
Initial upload tracking for audit and debugging.
|
||
|
|
|
||
|
|
#### Document Entries (`documentEntries`)
|
||
|
|
Core document references after deduplication and validation.
|
||
|
|
|
||
|
|
#### Document Cleans (`documentCleans`)
|
||
|
|
PDF validation and format processing records.
|
||
|
|
|
||
|
|
#### Document Text Extractions (`documentTextExtractions`)
|
||
|
|
OCR and text extraction results from AWS Textract.
|
||
|
|
|
||
|
|
### Processing Dependencies
|
||
|
|
[link to rendered version here](https://mermaid.live/edit#pako:eNo9jz0KwzAMRq9iNDcX6NClZOsU2inNYGy1Mfgn2AokhNy9tkytRTx9TyAdoIJGuAr4RrnM4jG8vcj1WmyQetRBrQ49VUyT6Lqb6D3FvUWFDKap7nHG1t2i9M1i-ksMLD1xo-YU6DeKUpEJTS5TdgdMq6U0xtonuAhwGJ00Ot9_AM3o-BONH5kNOM8fK3VMWw==)
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
graph LR
|
||
|
|
Upload[documentUploads] --> Entry[documentEntries]
|
||
|
|
Entry --> Clean[documentCleans]
|
||
|
|
Clean --> Text[documentTextExtractions]
|
||
|
|
Text --> Results[results]
|
||
|
|
```
|
||
|
|
|
||
|
|
## Query Dependency System
|
||
|
|
|
||
|
|
### Dependency Management
|
||
|
|
```sql
|
||
|
|
-- Query-to-query dependencies
|
||
|
|
CREATE TABLE requiredQueries (
|
||
|
|
queryId uuid NOT NULL REFERENCES queries(queryId),
|
||
|
|
requiredQueryId uuid NOT NULL REFERENCES queries(queryId),
|
||
|
|
versionId int4 NOT NULL,
|
||
|
|
addedVersion int4 NOT NULL,
|
||
|
|
removedVersion int4,
|
||
|
|
CHECK (queryId != requiredQueryId)
|
||
|
|
);
|
||
|
|
|
||
|
|
-- Result-to-result dependencies
|
||
|
|
CREATE TABLE resultDependencies (
|
||
|
|
resultId uuid NOT NULL REFERENCES results(id),
|
||
|
|
requiredResultId uuid NOT NULL REFERENCES results(id),
|
||
|
|
CHECK (resultId != requiredResultId)
|
||
|
|
);
|
||
|
|
```
|
||
|
|
|
||
|
|
### Dependency Resolution Views
|
||
|
|
```sql
|
||
|
|
-- Recursive view for transitive dependencies
|
||
|
|
CREATE VIEW queryActiveDependencies AS
|
||
|
|
WITH RECURSIVE deps AS (
|
||
|
|
-- Direct dependencies
|
||
|
|
SELECT DISTINCT queryId, requiredQueryId, 1 as depth
|
||
|
|
FROM requiredQueries rq
|
||
|
|
JOIN queryActiveVersions qav ON rq.queryId = qav.queryId AND rq.versionId = qav.versionId
|
||
|
|
WHERE rq.removedVersion IS NULL
|
||
|
|
|
||
|
|
UNION
|
||
|
|
|
||
|
|
-- Transitive dependencies
|
||
|
|
SELECT d.queryId, rq.requiredQueryId, d.depth + 1
|
||
|
|
FROM deps d
|
||
|
|
JOIN requiredQueries rq ON d.requiredQueryId = rq.queryId
|
||
|
|
JOIN queryActiveVersions qav ON rq.queryId = qav.queryId AND rq.versionId = qav.versionId
|
||
|
|
WHERE rq.removedVersion IS NULL AND d.depth < 10 -- Cycle prevention
|
||
|
|
)
|
||
|
|
SELECT * FROM deps;
|
||
|
|
```
|
||
|
|
|
||
|
|
## Collector Configuration System
|
||
|
|
|
||
|
|
### Client-Query Mapping
|
||
|
|
```sql
|
||
|
|
-- Maps named queries to specific query implementations for clients
|
||
|
|
CREATE TABLE collectorQueries (
|
||
|
|
clientId varchar(256) NOT NULL REFERENCES clients(clientId),
|
||
|
|
name varchar(256) NOT NULL,
|
||
|
|
queryId uuid NOT NULL REFERENCES queries(queryId),
|
||
|
|
versionId int4 NOT NULL,
|
||
|
|
addedVersion int4 NOT NULL,
|
||
|
|
removedVersion int4
|
||
|
|
);
|
||
|
|
```
|
||
|
|
|
||
|
|
### Version Thresholds
|
||
|
|
```sql
|
||
|
|
-- Minimum processing versions required
|
||
|
|
CREATE TABLE collectorMinCleanVersions (
|
||
|
|
clientId varchar(256) PRIMARY KEY REFERENCES clients(clientId),
|
||
|
|
version int4 NOT NULL
|
||
|
|
);
|
||
|
|
|
||
|
|
CREATE TABLE collectorMinTextVersions (
|
||
|
|
clientId varchar(256) PRIMARY KEY REFERENCES clients(clientId),
|
||
|
|
version int4 NOT NULL
|
||
|
|
);
|
||
|
|
```
|
||
|
|
|
||
|
|
### Complex Dependency Views
|
||
|
|
```sql
|
||
|
|
-- Builds complete query dependency tree per client
|
||
|
|
CREATE VIEW collectorQueryDependencyTree AS
|
||
|
|
SELECT DISTINCT
|
||
|
|
cq.clientId,
|
||
|
|
cq.name as queryName,
|
||
|
|
qad.queryId,
|
||
|
|
qad.requiredQueryId,
|
||
|
|
qad.depth
|
||
|
|
FROM collectorQueries cq
|
||
|
|
JOIN queryActiveDependencies qad ON cq.queryId = qad.queryId
|
||
|
|
WHERE cq.removedVersion IS NULL;
|
||
|
|
```
|
||
|
|
|
||
|
|
## Advanced Database Features
|
||
|
|
|
||
|
|
### Business Logic Functions
|
||
|
|
|
||
|
|
#### Document Processing Functions
|
||
|
|
```sql
|
||
|
|
-- List documents ready for processing
|
||
|
|
CREATE OR REPLACE FUNCTION listDocumentIDs(clientId varchar(256), limit_val int4, offset_val int4)
|
||
|
|
RETURNS TABLE(documentId uuid) AS $$
|
||
|
|
BEGIN
|
||
|
|
RETURN QUERY
|
||
|
|
SELECT de.documentId
|
||
|
|
FROM documentEntries de
|
||
|
|
WHERE de.clientId = $1
|
||
|
|
ORDER BY de.documentId
|
||
|
|
LIMIT limit_val OFFSET offset_val;
|
||
|
|
END;
|
||
|
|
$$ LANGUAGE plpgsql;
|
||
|
|
```
|
||
|
|
|
||
|
|
#### Result Validation Functions
|
||
|
|
```sql
|
||
|
|
-- Complex validation for result dependencies
|
||
|
|
CREATE OR REPLACE FUNCTION listValidDocumentResults(
|
||
|
|
clientId varchar(256),
|
||
|
|
documentId uuid,
|
||
|
|
queryId uuid
|
||
|
|
)
|
||
|
|
RETURNS TABLE(resultId uuid, value text) AS $$
|
||
|
|
-- Complex logic for dependency resolution
|
||
|
|
$$ LANGUAGE plpgsql;
|
||
|
|
```
|
||
|
|
|
||
|
|
### Materialized Views and Performance
|
||
|
|
```sql
|
||
|
|
-- Current clean entries above minimum thresholds
|
||
|
|
CREATE VIEW currentCleanEntries AS
|
||
|
|
SELECT DISTINCT ON (dc.documentId)
|
||
|
|
dc.*
|
||
|
|
FROM documentCleans dc
|
||
|
|
JOIN collectorMinCleanVersions cmcv ON dc.clientId = cmcv.clientId
|
||
|
|
WHERE dc.version >= cmcv.version
|
||
|
|
ORDER BY dc.documentId, dc.version DESC;
|
||
|
|
```
|
||
|
|
|
||
|
|
## Data Integrity and Constraints
|
||
|
|
|
||
|
|
### Referential Integrity
|
||
|
|
- All foreign key relationships enforced at database level
|
||
|
|
- Cascade delete policies for cleanup operations
|
||
|
|
- Unique constraints prevent duplicate business entities
|
||
|
|
|
||
|
|
### Data Validation
|
||
|
|
- Enum types for controlled vocabularies (`querytype`, `mimetype`)
|
||
|
|
- Check constraints for business rules (no self-dependencies)
|
||
|
|
- NOT NULL constraints for required fields
|
||
|
|
|
||
|
|
### Audit and Traceability
|
||
|
|
- UUID v7 generation provides insertion ordering
|
||
|
|
- Version tracking enables complete audit trails
|
||
|
|
- Temporal validity supports point-in-time queries
|
||
|
|
|
||
|
|
## Scalability Considerations
|
||
|
|
|
||
|
|
### Indexing Strategy
|
||
|
|
- Primary keys automatically indexed (UUID v7)
|
||
|
|
- Foreign keys automatically indexed
|
||
|
|
- Composite indexes on frequent query patterns
|
||
|
|
- Unique constraints create secondary indexes
|
||
|
|
|
||
|
|
### Partitioning Opportunities
|
||
|
|
- Time-based partitioning using UUID v7 timestamp component
|
||
|
|
- Client-based partitioning for multi-tenant isolation
|
||
|
|
- Archive strategies for old versions
|
||
|
|
|
||
|
|
### Performance Optimizations
|
||
|
|
- Window functions for latest version queries
|
||
|
|
- Recursive CTEs with depth limits for dependency resolution
|
||
|
|
- Pagination support in all list functions
|
||
|
|
- Efficient temporal queries using `isInVersion()` function
|
||
|
|
|
||
|
|
## Migration Strategy
|
||
|
|
|
||
|
|
### Database Evolution
|
||
|
|
- Sequential numbered migrations for version control
|
||
|
|
- Separate UP/DOWN migrations for rollback capability
|
||
|
|
- Extension management for UUID generation and other features
|
||
|
|
- View-based abstractions for query flexibility
|
||
|
|
|
||
|
|
### Version Compatibility
|
||
|
|
- Backward-compatible schema changes
|
||
|
|
- Graceful handling of version mismatches
|
||
|
|
- Data migration scripts for major version upgrades
|