jacob-mathison/query-orchestration
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

Merged in feature/remove-query (pull request #201)

Browse Source 
remove query from codebase part 1

* remove query

* fix localstack run
This commit is contained in:
Jay Brown
2026-01-14 17:59:04 +00:00
parent ebf47c6013
commit 0ddae4f91e
167 changed files with 1059 additions and 20641 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ai.generated/01-system-overview.md
+8 -14
View File
@@ -1,16 +1,16 @@
# System Overview
## DoczyAI Query Orchestration Platform
## DoczyAI Document Processing Platform
### Purpose and Domain
DoczyAI is a cloud-native document processing and query orchestration platform designed to extract structured information from documents (primarily PDFs) using configurable, hierarchical queries. The system serves as a multi-tenant platform where organizations can upload documents, define custom data extraction rules, and receive structured results for business intelligence and analysis.
DoczyAI is a cloud-native document processing platform designed to extract structured information from documents (primarily PDFs). The system serves as a multi-tenant platform where organizations can upload documents, perform text extraction, and organize documents with folders and labels for business intelligence and analysis.
### Core Business Capabilities
- **Multi-tenant Document Processing**: Isolated processing environments per client
- **Batch Document Processing**: ZIP archive uploads for processing multiple documents simultaneously
- **Configurable Query System**: Custom data extraction rules with dependency management
- **Text Extraction Pipeline**: Automated OCR and text extraction from documents
- **Automated Processing Pipeline**: Seamless flow from document upload to structured output
- **Version Management**: Complete versioning for queries and configurations
- **Version Management**: Complete versioning for configurations
- **Export and Integration**: Structured data export capabilities for downstream systems
- **Enhanced Observability**: Comprehensive metrics and monitoring with Prometheus integration
@@ -30,7 +30,7 @@ DoczyAI is a cloud-native document processing and query orchestration platform d
### System Architecture Pattern
The system follows an **event-driven microservices architecture** with the following characteristics:
- **12 distinct services**: 1 HTTP API + 8 queue-based runners + 3 utility services
- **8 distinct services**: 1 HTTP API + 6 queue-based runners + 1 utility service
- **Asynchronous communication**: SQS queues between services with batch processing capabilities
- **Background Task Framework**: Comprehensive background task execution with periodic scheduling
- **State management**: PostgreSQL as central data store with enhanced schema for batch operations
@@ -56,13 +56,10 @@ graph TB
DI --> DS[docSyncRunner<br/>8083]
DS --> DC[docCleanRunner<br/>8084]
DC --> DT[docTextRunner<br/>8085]
DT --> QS[querySyncRunner<br/>8087]
QS --> QR[queryRunner<br/>8088]
end
subgraph "Synchronization Services"
CS[clientSyncRunner<br/>8089]
QVS[queryVersionSyncRunner<br/>8090]
end
subgraph "Infrastructure Services"
@@ -87,11 +84,8 @@ graph TB
DS --> DB
DC --> S3
DT --> TEXTRACT
QS --> DB
QR --> DB
CS --> SQS
QVS --> SQS
All_Services --> PROM
```
@@ -125,7 +119,7 @@ graph TB
2. **AWS SQS**: Inter-service messaging and event processing
3. **AWS Textract**: OCR and text extraction from documents
4. **AWS Cognito**: User authentication and session management
5. **PostgreSQL**: Central data persistence and query processing
5. **PostgreSQL**: Central data persistence
### API Integration
- **REST API**: OpenAPI 3.0.3 specification with Swagger UI
docs/ai.generated/02-service-architecture.md
+12 -46
View File
@@ -12,7 +12,8 @@
- Client management and authentication
- Document upload and metadata management (single and batch)
- Batch upload processing with ZIP archive support
- Query definition and testing
- Folder and label management
- Field extraction configuration
- Export operations and status monitoring
- Background task coordination and monitoring
- User interface and API documentation
@@ -57,25 +58,9 @@
- **Port**: 8085
- **Type**: Queue-based processor
- **Input**: `{document_id}` from DOCTEXT queue
- **Output**: Events to QUERYSYNC queue
- **Output**: Text stored in database, processing complete
- **Dependencies**: AWS Textract, S3 ObjectStore
#### querySyncRunner - Query Dependency Resolution
- **Purpose**: Identifies executable queries for documents
- **Port**: 8087
- **Type**: Queue-based processor
- **Input**: `{document_id}` from QUERYSYNC queue
- **Output**: Events to QUERY queue
- **Logic**: Finds queries with satisfied dependencies
#### queryRunner - Query Execution Engine
- **Purpose**: Executes queries and stores results
- **Port**: 8088
- **Type**: Queue-based processor
- **Input**: `{document_id, query_id}` from QUERY queue
- **Output**: Events to downstream QUERY queue
- **Responsibilities**: Query execution and result persistence
### Synchronization Services
#### clientSyncRunner - Client-Level Synchronization
@@ -85,13 +70,6 @@
- **Input**: `{client_id}` from CLIENTSYNC queue
- **Output**: Events to DOCSYNC queue for each client document
#### queryVersionSyncRunner - Query Change Propagation
- **Purpose**: Propagates query updates to affected clients
- **Port**: 8090
- **Type**: Queue-based processor
- **Input**: `{query_id}` from QUERYVERSIONSYNC queue
- **Output**: Events to CLIENTSYNC queue for affected clients
### Utility Services
#### healthcheck - Health Monitoring
@@ -134,10 +112,8 @@ sequenceDiagram
participant Sync as docSyncRunner
participant Clean as docCleanRunner
participant Text as docTextRunner
participant QSync as querySyncRunner
participant Query as queryRunner
participant DB as Database
Client->>S3: Upload Document
S3->>Store: S3 Event Notification
Store->>Init: DOCINIT Queue
@@ -149,29 +125,22 @@ sequenceDiagram
Clean->>Text: DOCTEXT Queue
Text->>S3: Read Document
Text->>AWS: Textract Processing
Text->>QSync: QUERYSYNC Queue
QSync->>DB: Find Executable Queries
QSync->>Query: QUERY Queue
Query->>DB: Execute & Store Results
Text->>DB: Store Extracted Text
```
### Query Version Management Flow
[link to rendered version here](https://mermaid.live/edit#pako:eNp9kctqwzAQRX9l0KqF9ge8CBTZC0Nx_GgMgW6ENG4GLFmVZEII-ffKj_SVUi2EGJ0z3GHOTA4KWQLM4_uIRmJK4s0J_WogHitcIElWmABPSpP5o1zmIDxE2Z3i-xao2uYTaNF5GkxzMrIejUF3i_OZlj2hCf9xJVnsyeBEp4McNX4rLvhyz7kfN5sYLoGdVSIgVFMYSLEjQyEGWskyj1zMm0C1y-p9m9VNvi2afcEnY1zbRiBiPFL8Oc-Kl69_uLPoQHQdyoBqHeJ-sfgkXfMlkG75b2_BQa3DrN5V-WHXaN0g0Xs4UjhAgcdlJPYATKPTglTc6ZmFA-p5uwo7MfaBXS4faYamhw==)
### Client Synchronization Flow
```mermaid
sequenceDiagram
participant Admin
participant API as queryAPI
participant QVS as queryVersionSyncRunner
participant CS as clientSyncRunner
participant Pipeline as Document Pipeline
Admin->>API: Update Query Definition
API->>QVS: QUERYVERSIONSYNC Queue
QVS->>CS: CLIENTSYNC Queue (per affected client)
Admin->>API: Update Client Configuration
API->>CS: CLIENTSYNC Queue
CS->>Pipeline: DOCSYNC Queue (per client document)
Pipeline->>Pipeline: Reprocess with New Query
Pipeline->>Pipeline: Reprocess Documents
```
## Communication Patterns
@@ -191,10 +160,7 @@ All services communicate through **AWS SQS queues** with the following patterns:
| DOCSYNC | docInitRunner, clientSyncRunner | docSyncRunner | `{document_id}` |
| DOCCLEAN | docSyncRunner | docCleanRunner | `{document_id}` |
| DOCTEXT | docCleanRunner | docTextRunner | `{document_id}` |
| QUERYSYNC | docTextRunner | querySyncRunner | `{document_id}` |
| QUERY | querySyncRunner, queryRunner | queryRunner | `{document_id, query_id}` |
| CLIENTSYNC | queryVersionSyncRunner | clientSyncRunner | `{client_id}` |
| QUERYVERSIONSYNC | queryAPI | queryVersionSyncRunner | `{query_id}` |
| CLIENTSYNC | queryAPI | clientSyncRunner | `{client_id}` |
| BATCH_PROCESSING | queryAPI (batch worker) | Internal batch processor | `{batch_id, files[]}` |
### Database Access Patterns
@@ -208,7 +174,7 @@ All services communicate through **AWS SQS queues** with the following patterns:
### Horizontal Scaling Strategy
- **API Service**: Load balancer with multiple instances
- **Queue Processors**: Auto-scaling based on queue depth
- **Database**: Read replicas for query scaling
- **Database**: Read replicas for scaling
- **Storage**: S3 scales automatically
### Performance Optimizations
docs/ai.generated/README.md
+9 -9
View File
@@ -1,8 +1,8 @@
# DoczyAI Query Orchestration Platform Documentation
# DoczyAI Document Processing Platform Documentation
## Overview
This documentation provides comprehensive coverage of the DoczyAI Query Orchestration Platform - a cloud-native document processing and query orchestration system designed to extract structured information from documents using configurable, hierarchical queries.
This documentation provides comprehensive coverage of the DoczyAI Document Processing Platform - a cloud-native document processing system designed to extract structured information from documents.
## Documentation Structure
@@ -15,7 +15,7 @@ This documentation provides comprehensive coverage of the DoczyAI Query Orchestr
- Deployment architecture and scalability
2. **[Service Architecture](02-service-architecture.md)**
- Detailed service catalog (12 services)
- Detailed service catalog (8 services)
- Document processing pipeline flow
- Inter-service communication patterns
- Queue architecture and message formats
@@ -28,8 +28,8 @@ This documentation provides comprehensive coverage of the DoczyAI Query Orchestr
4. **[Data Architecture](04-data-architecture.md)**
- Database schema and entity relationships
- Versioning system and temporal queries
- Query dependency management
- Versioning system
- Document organization (folders and labels)
- Performance optimization strategies
### 🛠️ Developer Documentation
@@ -82,9 +82,9 @@ This documentation provides comprehensive coverage of the DoczyAI Query Orchestr
### 🏗️ Architecture
- **Event-driven microservices** with SQS-based communication
- **12 distinct services**: 1 HTTP API + 8 runners + 3 utilities
- **8 distinct services**: 1 HTTP API + 6 runners + 1 utility
- **Multi-tenant design** with client isolation
- **Comprehensive versioning** for queries and configurations
- **Comprehensive versioning** for configurations
### 🔧 Technology Stack
- **Backend**: Go 1.24.0 with Echo framework
@@ -95,10 +95,10 @@ This documentation provides comprehensive coverage of the DoczyAI Query Orchestr
### 📊 Key Capabilities
- **Document Processing**: PDF upload, cleaning, and text extraction
- **Query Orchestration**: Configurable data extraction with dependencies
- **Text Extraction**: Automated OCR with AWS Textract
- **Multi-tenant Support**: Isolated client environments
- **Document Organization**: Folders and labels for document management
- **Export Operations**: Structured data export in multiple formats
- **Version Management**: Full query and configuration versioning
## Documentation Tools
docs/ai.generated/query-system-explanation.md
-140
View File
@@ -1,140 +0,0 @@
# Query System Explanation
## Overview
The query system in this application is designed to extract structured data from processed documents. Queries are configurable data extraction rules that can depend on each other, creating a pipeline of data transformations.
## Query Types
The system currently supports two query types:
### 1. CONTEXT_FULL Query
- **Purpose**: Originally intended to provide the full document text/context
- **Current Implementation**: Returns a hardcoded JSON placeholder `{"keyone":"valueone","keytwo":"valuetwo"}`
- **No Dependencies**: This query type doesn't require any other queries to run first
- **Note**: The current implementation appears to be a mock/placeholder - in production, this would likely return the actual extracted text from the document
### 2. JSON_EXTRACTOR Query
- **Purpose**: Extracts specific values from JSON data using path expressions
- **Dependencies**: Requires exactly one other query result as input (typically a CONTEXT_FULL query)
- **Configuration**: Takes a JSON config with a `path` field specifying what to extract
- **Uses gjson**: Leverages the `tidwall/gjson` library for JSON path extraction
## How Queries Work
### Query Definition Structure
```go
type Query {
ID uuid
Type queryType // CONTEXT_FULL or JSON_EXTRACTOR
ActiveVersion int32 // Current version being used
RequiredQueryIDs []uuid // Dependencies - other queries that must run first
Config string // JSON configuration (e.g., {"path":"keyone"})
}
```
### Example Query Chain
1. **First Query (CONTEXT_FULL)**:
- Type: `CONTEXT_FULL`
- Config: None needed
- Dependencies: None
- Returns: `{"keyone":"valueone","keytwo":"valuetwo"}`
2. **Second Query (JSON_EXTRACTOR)**:
- Type: `JSON_EXTRACTOR`
- Config: `{"path":"keyone"}`
- Dependencies: [First Query ID]
- Input: Result from First Query
- Returns: `"valueone"`
3. **Third Query (JSON_EXTRACTOR)** - could extract from second:
- Type: `JSON_EXTRACTOR`
- Config: `{"path":"some.nested.path"}`
- Dependencies: [Second Query ID]
- Would extract from the result of the second query
## Query Execution Flow
### 1. querySyncRunner Phase
- After document text extraction completes, querySyncRunner is triggered
- It queries the database for all queries that:
- Are configured for this client (via collector configuration)
- Have NO dependencies, OR
- Have all dependencies already satisfied (results exist in database)
- For each eligible query, sends a message to the QUERY queue
### 2. queryRunner Phase
- Receives {document_id, query_id} from QUERY queue
- Retrieves the query definition and active version
- If query has dependencies:
- Fetches results from all required queries
- Validates all dependencies are satisfied
- Executes the appropriate processor based on query type:
- CONTEXT_FULL: Returns the mock JSON (should return document text)
- JSON_EXTRACTOR: Extracts specified path from dependency result
- Stores the result in database with version tracking
- **Triggers dependent queries**: Finds all queries that depend on this one and sends them to QUERY queue
### 3. Recursive Processing
- As each query completes, it triggers its dependents
- This creates a cascade effect through the dependency graph
- Continues until all queries in the chain have been executed
## Collector Configuration
Collectors define which queries should run for each client:
```sql
-- collectorQueries table maps queries to clients
CREATE TABLE collectorQueries (
clientId varchar(255), -- Which client
name varchar(255), -- Named reference (e.g., "JSON_QUERY")
queryId uuid, -- Which query to run
...
)
```
When documents are processed, only queries configured in the client's collector will be executed.
## Current Implementation Status
### What's Working
- Query dependency resolution and scheduling
- Recursive execution of dependent queries
- Version tracking for queries and results
- JSON path extraction from query results
- Client-specific query configuration via collectors
### What Appears Incomplete
- **CONTEXT_FULL Implementation**: Currently returns mock data instead of actual document text
- The system stores extracted text in S3 (via docTextRunner) but CONTEXT_FULL doesn't retrieve it
- There's infrastructure for text storage (`documentTextExtractions` table) but it's not connected to the query processor
## Real-World Use Case
In a production scenario, this system would work like:
1. **Document uploaded** → Text extracted via OCR
2. **Base Query** (CONTEXT_FULL) → Returns full document text as JSON
3. **Field Extraction Queries** (JSON_EXTRACTOR) → Extract specific fields like:
- Customer name
- Invoice number
- Date
- Amount
4. **Derived Queries** → Further process extracted data:
- Calculate totals
- Validate formats
- Cross-reference with other data
## Key Design Benefits
1. **Flexibility**: New extraction rules can be added without code changes
2. **Reusability**: Queries can be shared across clients or documents
3. **Dependency Management**: Complex multi-step extractions are handled automatically
4. **Versioning**: Changes to queries don't break existing results
5. **Client Isolation**: Each client can have different extraction rules for the same document types
## Summary
The query system is a sophisticated, dependency-aware data extraction pipeline. While the current implementation has placeholder logic for the base CONTEXT_FULL query, the infrastructure for chaining queries, managing dependencies, and extracting structured data from documents is fully functional. The system allows for complex, multi-step data extraction workflows to be defined declaratively and executed automatically as documents are processed.