Merged in feature/remove-query (pull request #201)
remove query from codebase part 1 * remove query * fix localstack run
This commit is contained in:
Jay Brown
+16
-32
@@ -56,7 +56,7 @@ If eligible, send an event to the DOCCLEAN queue.
|
||||
|
||||
If the document text is not already extracted according to the collector standards, trigger the textract job for text detection.
|
||||
|
||||
Otherwise, add an event to the QUERYSYNC queue.
|
||||
This is the terminal step in the document processing pipeline. Query processing has been removed - pipeline ends after text extraction.
|
||||
|
||||
## healthcheck
|
||||
|
||||
@@ -86,49 +86,25 @@ The entities in question:
|
||||
|
||||
- Client - A client is a single client of the Doczy project, all configurations and clients for a client will be in reference to the respective client.
|
||||
|
||||
- Collector - A collector specifies the required queries for export for a given client.
|
||||
- Collector - A collector specifies document processing settings for a given client.
|
||||
|
||||
- Sample - A sample is a subset of documents to be used for testing. When it is active a client will only process documents in the sample.
|
||||
|
||||
- Query - A query is a block of processing of data which takes an input and produces an output.
|
||||
|
||||
- Export - An export is a manually started process which outputs the results for a client to a given location.
|
||||
|
||||
Note: Query functionality has been removed from this API. See remove_query_plan.md for details.
|
||||
|
||||
## queryRunner
|
||||
|
||||
**Listens For**: Body Containing Document ID and Query ID
|
||||
|
||||
**Action**.
|
||||
|
||||
Run the query for the given document and store the result.
|
||||
|
||||
Events with the document id and downstream query ids are pushed to the QUERY queue.
|
||||
DEPRECATED: This runner is deprecated and no longer has any functionality. It is left in place until deployment code is updated accordingly. See remove_query_plan.md for details on the query removal.
|
||||
|
||||
## querySyncRunner
|
||||
|
||||
**Listens For**: Body Containing Document ID
|
||||
|
||||
**Action**.
|
||||
|
||||
Find the queries with no dependencies that do not have a valid result for a document.
|
||||
|
||||
The reasoning for only extracting these is to start the query extraction process only for necessary queries, and the query runner will manage updating downstream dependencies.
|
||||
|
||||
There are 2 situations that cause a query to show up in this list:
|
||||
|
||||
The query has no required dependencies.
|
||||
|
||||
All required dependencies have a valid result.
|
||||
|
||||
An event with the document id is pushed to the DOCTEXT queue.
|
||||
DEPRECATED: This runner is deprecated and no longer has any functionality. It is left in place until deployment code is updated accordingly. See remove_query_plan.md for details on the query removal.
|
||||
|
||||
## queryVersionSyncRunner
|
||||
|
||||
**Listens For**: Body Containing Query ID
|
||||
|
||||
**Action**.
|
||||
|
||||
Adds an event for each client into the CLIENTSYNC queue for the given query, i.e. each client that contains the query in its dependency tree.
|
||||
DEPRECATED: This runner is deprecated and no longer has any functionality. It is left in place until deployment code is updated accordingly. See remove_query_plan.md for details on the query removal.
|
||||
|
||||
## storeEventRunner
|
||||
|
||||
@@ -144,7 +120,13 @@ It supports the document initialisation and document text processing stages. Eve
|
||||
|
||||
## cognito.auth.harness
|
||||
|
||||
## permit.harness
|
||||
## permit.setup
|
||||
|
||||
setup_permit.go - CLI tool to configure Permit.io roles and resources This tool reads a YAML configuration file and creates resources, actions, and roles in a Permit.io project/environment using the Permit.io REST API.
|
||||
|
||||
Usage:
|
||||
|
||||
|
||||
|
||||
## user.creation.tool
|
||||
|
||||
@@ -160,4 +142,6 @@ cognito.go handles AWS Cognito User Pool operations for user identity management
|
||||
|
||||
permit.go handles Permit.io API operations for user management and role-based access control (RBAC). This file provides functions to create, delete, search, and manage users in Permit.io, as well as assign/unassign roles and validate role definitions. It integrates with Cognito user data to sync user information between the two systems.
|
||||
|
||||
## permit.harness
|
||||
|
||||
Generated by Doczy
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
# System Overview
|
||||
|
||||
## DoczyAI Query Orchestration Platform
|
||||
## DoczyAI Document Processing Platform
|
||||
|
||||
### Purpose and Domain
|
||||
DoczyAI is a cloud-native document processing and query orchestration platform designed to extract structured information from documents (primarily PDFs) using configurable, hierarchical queries. The system serves as a multi-tenant platform where organizations can upload documents, define custom data extraction rules, and receive structured results for business intelligence and analysis.
|
||||
DoczyAI is a cloud-native document processing platform designed to extract structured information from documents (primarily PDFs). The system serves as a multi-tenant platform where organizations can upload documents, perform text extraction, and organize documents with folders and labels for business intelligence and analysis.
|
||||
|
||||
### Core Business Capabilities
|
||||
- **Multi-tenant Document Processing**: Isolated processing environments per client
|
||||
- **Batch Document Processing**: ZIP archive uploads for processing multiple documents simultaneously
|
||||
- **Configurable Query System**: Custom data extraction rules with dependency management
|
||||
- **Text Extraction Pipeline**: Automated OCR and text extraction from documents
|
||||
- **Automated Processing Pipeline**: Seamless flow from document upload to structured output
|
||||
- **Version Management**: Complete versioning for queries and configurations
|
||||
- **Version Management**: Complete versioning for configurations
|
||||
- **Export and Integration**: Structured data export capabilities for downstream systems
|
||||
- **Enhanced Observability**: Comprehensive metrics and monitoring with Prometheus integration
|
||||
|
||||
@@ -30,7 +30,7 @@ DoczyAI is a cloud-native document processing and query orchestration platform d
|
||||
|
||||
### System Architecture Pattern
|
||||
The system follows an **event-driven microservices architecture** with the following characteristics:
|
||||
- **12 distinct services**: 1 HTTP API + 8 queue-based runners + 3 utility services
|
||||
- **8 distinct services**: 1 HTTP API + 6 queue-based runners + 1 utility service
|
||||
- **Asynchronous communication**: SQS queues between services with batch processing capabilities
|
||||
- **Background Task Framework**: Comprehensive background task execution with periodic scheduling
|
||||
- **State management**: PostgreSQL as central data store with enhanced schema for batch operations
|
||||
@@ -56,13 +56,10 @@ graph TB
|
||||
DI --> DS[docSyncRunner<br/>8083]
|
||||
DS --> DC[docCleanRunner<br/>8084]
|
||||
DC --> DT[docTextRunner<br/>8085]
|
||||
DT --> QS[querySyncRunner<br/>8087]
|
||||
QS --> QR[queryRunner<br/>8088]
|
||||
end
|
||||
|
||||
|
||||
subgraph "Synchronization Services"
|
||||
CS[clientSyncRunner<br/>8089]
|
||||
QVS[queryVersionSyncRunner<br/>8090]
|
||||
end
|
||||
|
||||
subgraph "Infrastructure Services"
|
||||
@@ -87,11 +84,8 @@ graph TB
|
||||
DS --> DB
|
||||
DC --> S3
|
||||
DT --> TEXTRACT
|
||||
QS --> DB
|
||||
QR --> DB
|
||||
|
||||
|
||||
CS --> SQS
|
||||
QVS --> SQS
|
||||
|
||||
All_Services --> PROM
|
||||
```
|
||||
@@ -125,7 +119,7 @@ graph TB
|
||||
2. **AWS SQS**: Inter-service messaging and event processing
|
||||
3. **AWS Textract**: OCR and text extraction from documents
|
||||
4. **AWS Cognito**: User authentication and session management
|
||||
5. **PostgreSQL**: Central data persistence and query processing
|
||||
5. **PostgreSQL**: Central data persistence
|
||||
|
||||
### API Integration
|
||||
- **REST API**: OpenAPI 3.0.3 specification with Swagger UI
|
||||
|
||||
@@ -12,7 +12,8 @@
|
||||
- Client management and authentication
|
||||
- Document upload and metadata management (single and batch)
|
||||
- Batch upload processing with ZIP archive support
|
||||
- Query definition and testing
|
||||
- Folder and label management
|
||||
- Field extraction configuration
|
||||
- Export operations and status monitoring
|
||||
- Background task coordination and monitoring
|
||||
- User interface and API documentation
|
||||
@@ -57,25 +58,9 @@
|
||||
- **Port**: 8085
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id}` from DOCTEXT queue
|
||||
- **Output**: Events to QUERYSYNC queue
|
||||
- **Output**: Text stored in database, processing complete
|
||||
- **Dependencies**: AWS Textract, S3 ObjectStore
|
||||
|
||||
#### querySyncRunner - Query Dependency Resolution
|
||||
- **Purpose**: Identifies executable queries for documents
|
||||
- **Port**: 8087
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id}` from QUERYSYNC queue
|
||||
- **Output**: Events to QUERY queue
|
||||
- **Logic**: Finds queries with satisfied dependencies
|
||||
|
||||
#### queryRunner - Query Execution Engine
|
||||
- **Purpose**: Executes queries and stores results
|
||||
- **Port**: 8088
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{document_id, query_id}` from QUERY queue
|
||||
- **Output**: Events to downstream QUERY queue
|
||||
- **Responsibilities**: Query execution and result persistence
|
||||
|
||||
### Synchronization Services
|
||||
|
||||
#### clientSyncRunner - Client-Level Synchronization
|
||||
@@ -85,13 +70,6 @@
|
||||
- **Input**: `{client_id}` from CLIENTSYNC queue
|
||||
- **Output**: Events to DOCSYNC queue for each client document
|
||||
|
||||
#### queryVersionSyncRunner - Query Change Propagation
|
||||
- **Purpose**: Propagates query updates to affected clients
|
||||
- **Port**: 8090
|
||||
- **Type**: Queue-based processor
|
||||
- **Input**: `{query_id}` from QUERYVERSIONSYNC queue
|
||||
- **Output**: Events to CLIENTSYNC queue for affected clients
|
||||
|
||||
### Utility Services
|
||||
|
||||
#### healthcheck - Health Monitoring
|
||||
@@ -134,10 +112,8 @@ sequenceDiagram
|
||||
participant Sync as docSyncRunner
|
||||
participant Clean as docCleanRunner
|
||||
participant Text as docTextRunner
|
||||
participant QSync as querySyncRunner
|
||||
participant Query as queryRunner
|
||||
participant DB as Database
|
||||
|
||||
|
||||
Client->>S3: Upload Document
|
||||
S3->>Store: S3 Event Notification
|
||||
Store->>Init: DOCINIT Queue
|
||||
@@ -149,29 +125,22 @@ sequenceDiagram
|
||||
Clean->>Text: DOCTEXT Queue
|
||||
Text->>S3: Read Document
|
||||
Text->>AWS: Textract Processing
|
||||
Text->>QSync: QUERYSYNC Queue
|
||||
QSync->>DB: Find Executable Queries
|
||||
QSync->>Query: QUERY Queue
|
||||
Query->>DB: Execute & Store Results
|
||||
Text->>DB: Store Extracted Text
|
||||
```
|
||||
|
||||
### Query Version Management Flow
|
||||
|
||||
[link to rendered version here](https://mermaid.live/edit#pako:eNp9kctqwzAQRX9l0KqF9ge8CBTZC0Nx_GgMgW6ENG4GLFmVZEII-ffKj_SVUi2EGJ0z3GHOTA4KWQLM4_uIRmJK4s0J_WogHitcIElWmABPSpP5o1zmIDxE2Z3i-xao2uYTaNF5GkxzMrIejUF3i_OZlj2hCf9xJVnsyeBEp4McNX4rLvhyz7kfN5sYLoGdVSIgVFMYSLEjQyEGWskyj1zMm0C1y-p9m9VNvi2afcEnY1zbRiBiPFL8Oc-Kl69_uLPoQHQdyoBqHeJ-sfgkXfMlkG75b2_BQa3DrN5V-WHXaN0g0Xs4UjhAgcdlJPYATKPTglTc6ZmFA-p5uwo7MfaBXS4faYamhw==)
|
||||
### Client Synchronization Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Admin
|
||||
participant API as queryAPI
|
||||
participant QVS as queryVersionSyncRunner
|
||||
participant CS as clientSyncRunner
|
||||
participant Pipeline as Document Pipeline
|
||||
|
||||
Admin->>API: Update Query Definition
|
||||
API->>QVS: QUERYVERSIONSYNC Queue
|
||||
QVS->>CS: CLIENTSYNC Queue (per affected client)
|
||||
|
||||
Admin->>API: Update Client Configuration
|
||||
API->>CS: CLIENTSYNC Queue
|
||||
CS->>Pipeline: DOCSYNC Queue (per client document)
|
||||
Pipeline->>Pipeline: Reprocess with New Query
|
||||
Pipeline->>Pipeline: Reprocess Documents
|
||||
```
|
||||
|
||||
## Communication Patterns
|
||||
@@ -191,10 +160,7 @@ All services communicate through **AWS SQS queues** with the following patterns:
|
||||
| DOCSYNC | docInitRunner, clientSyncRunner | docSyncRunner | `{document_id}` |
|
||||
| DOCCLEAN | docSyncRunner | docCleanRunner | `{document_id}` |
|
||||
| DOCTEXT | docCleanRunner | docTextRunner | `{document_id}` |
|
||||
| QUERYSYNC | docTextRunner | querySyncRunner | `{document_id}` |
|
||||
| QUERY | querySyncRunner, queryRunner | queryRunner | `{document_id, query_id}` |
|
||||
| CLIENTSYNC | queryVersionSyncRunner | clientSyncRunner | `{client_id}` |
|
||||
| QUERYVERSIONSYNC | queryAPI | queryVersionSyncRunner | `{query_id}` |
|
||||
| CLIENTSYNC | queryAPI | clientSyncRunner | `{client_id}` |
|
||||
| BATCH_PROCESSING | queryAPI (batch worker) | Internal batch processor | `{batch_id, files[]}` |
|
||||
|
||||
### Database Access Patterns
|
||||
@@ -208,7 +174,7 @@ All services communicate through **AWS SQS queues** with the following patterns:
|
||||
### Horizontal Scaling Strategy
|
||||
- **API Service**: Load balancer with multiple instances
|
||||
- **Queue Processors**: Auto-scaling based on queue depth
|
||||
- **Database**: Read replicas for query scaling
|
||||
- **Database**: Read replicas for scaling
|
||||
- **Storage**: S3 scales automatically
|
||||
|
||||
### Performance Optimizations
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# DoczyAI Query Orchestration Platform Documentation
|
||||
# DoczyAI Document Processing Platform Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
This documentation provides comprehensive coverage of the DoczyAI Query Orchestration Platform - a cloud-native document processing and query orchestration system designed to extract structured information from documents using configurable, hierarchical queries.
|
||||
This documentation provides comprehensive coverage of the DoczyAI Document Processing Platform - a cloud-native document processing system designed to extract structured information from documents.
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
@@ -15,7 +15,7 @@ This documentation provides comprehensive coverage of the DoczyAI Query Orchestr
|
||||
- Deployment architecture and scalability
|
||||
|
||||
2. **[Service Architecture](02-service-architecture.md)**
|
||||
- Detailed service catalog (12 services)
|
||||
- Detailed service catalog (8 services)
|
||||
- Document processing pipeline flow
|
||||
- Inter-service communication patterns
|
||||
- Queue architecture and message formats
|
||||
@@ -28,8 +28,8 @@ This documentation provides comprehensive coverage of the DoczyAI Query Orchestr
|
||||
|
||||
4. **[Data Architecture](04-data-architecture.md)**
|
||||
- Database schema and entity relationships
|
||||
- Versioning system and temporal queries
|
||||
- Query dependency management
|
||||
- Versioning system
|
||||
- Document organization (folders and labels)
|
||||
- Performance optimization strategies
|
||||
|
||||
### 🛠️ Developer Documentation
|
||||
@@ -82,9 +82,9 @@ This documentation provides comprehensive coverage of the DoczyAI Query Orchestr
|
||||
|
||||
### 🏗️ Architecture
|
||||
- **Event-driven microservices** with SQS-based communication
|
||||
- **12 distinct services**: 1 HTTP API + 8 runners + 3 utilities
|
||||
- **8 distinct services**: 1 HTTP API + 6 runners + 1 utility
|
||||
- **Multi-tenant design** with client isolation
|
||||
- **Comprehensive versioning** for queries and configurations
|
||||
- **Comprehensive versioning** for configurations
|
||||
|
||||
### 🔧 Technology Stack
|
||||
- **Backend**: Go 1.24.0 with Echo framework
|
||||
@@ -95,10 +95,10 @@ This documentation provides comprehensive coverage of the DoczyAI Query Orchestr
|
||||
|
||||
### 📊 Key Capabilities
|
||||
- **Document Processing**: PDF upload, cleaning, and text extraction
|
||||
- **Query Orchestration**: Configurable data extraction with dependencies
|
||||
- **Text Extraction**: Automated OCR with AWS Textract
|
||||
- **Multi-tenant Support**: Isolated client environments
|
||||
- **Document Organization**: Folders and labels for document management
|
||||
- **Export Operations**: Structured data export in multiple formats
|
||||
- **Version Management**: Full query and configuration versioning
|
||||
|
||||
## Documentation Tools
|
||||
|
||||
|
||||
@@ -1,140 +0,0 @@
|
||||
# Query System Explanation
|
||||
|
||||
## Overview
|
||||
|
||||
The query system in this application is designed to extract structured data from processed documents. Queries are configurable data extraction rules that can depend on each other, creating a pipeline of data transformations.
|
||||
|
||||
## Query Types
|
||||
|
||||
The system currently supports two query types:
|
||||
|
||||
### 1. CONTEXT_FULL Query
|
||||
- **Purpose**: Originally intended to provide the full document text/context
|
||||
- **Current Implementation**: Returns a hardcoded JSON placeholder `{"keyone":"valueone","keytwo":"valuetwo"}`
|
||||
- **No Dependencies**: This query type doesn't require any other queries to run first
|
||||
- **Note**: The current implementation appears to be a mock/placeholder - in production, this would likely return the actual extracted text from the document
|
||||
|
||||
### 2. JSON_EXTRACTOR Query
|
||||
- **Purpose**: Extracts specific values from JSON data using path expressions
|
||||
- **Dependencies**: Requires exactly one other query result as input (typically a CONTEXT_FULL query)
|
||||
- **Configuration**: Takes a JSON config with a `path` field specifying what to extract
|
||||
- **Uses gjson**: Leverages the `tidwall/gjson` library for JSON path extraction
|
||||
|
||||
## How Queries Work
|
||||
|
||||
### Query Definition Structure
|
||||
```go
|
||||
type Query {
|
||||
ID uuid
|
||||
Type queryType // CONTEXT_FULL or JSON_EXTRACTOR
|
||||
ActiveVersion int32 // Current version being used
|
||||
RequiredQueryIDs []uuid // Dependencies - other queries that must run first
|
||||
Config string // JSON configuration (e.g., {"path":"keyone"})
|
||||
}
|
||||
```
|
||||
|
||||
### Example Query Chain
|
||||
|
||||
1. **First Query (CONTEXT_FULL)**:
|
||||
- Type: `CONTEXT_FULL`
|
||||
- Config: None needed
|
||||
- Dependencies: None
|
||||
- Returns: `{"keyone":"valueone","keytwo":"valuetwo"}`
|
||||
|
||||
2. **Second Query (JSON_EXTRACTOR)**:
|
||||
- Type: `JSON_EXTRACTOR`
|
||||
- Config: `{"path":"keyone"}`
|
||||
- Dependencies: [First Query ID]
|
||||
- Input: Result from First Query
|
||||
- Returns: `"valueone"`
|
||||
|
||||
3. **Third Query (JSON_EXTRACTOR)** - could extract from second:
|
||||
- Type: `JSON_EXTRACTOR`
|
||||
- Config: `{"path":"some.nested.path"}`
|
||||
- Dependencies: [Second Query ID]
|
||||
- Would extract from the result of the second query
|
||||
|
||||
## Query Execution Flow
|
||||
|
||||
### 1. querySyncRunner Phase
|
||||
- After document text extraction completes, querySyncRunner is triggered
|
||||
- It queries the database for all queries that:
|
||||
- Are configured for this client (via collector configuration)
|
||||
- Have NO dependencies, OR
|
||||
- Have all dependencies already satisfied (results exist in database)
|
||||
- For each eligible query, sends a message to the QUERY queue
|
||||
|
||||
### 2. queryRunner Phase
|
||||
- Receives {document_id, query_id} from QUERY queue
|
||||
- Retrieves the query definition and active version
|
||||
- If query has dependencies:
|
||||
- Fetches results from all required queries
|
||||
- Validates all dependencies are satisfied
|
||||
- Executes the appropriate processor based on query type:
|
||||
- CONTEXT_FULL: Returns the mock JSON (should return document text)
|
||||
- JSON_EXTRACTOR: Extracts specified path from dependency result
|
||||
- Stores the result in database with version tracking
|
||||
- **Triggers dependent queries**: Finds all queries that depend on this one and sends them to QUERY queue
|
||||
|
||||
### 3. Recursive Processing
|
||||
- As each query completes, it triggers its dependents
|
||||
- This creates a cascade effect through the dependency graph
|
||||
- Continues until all queries in the chain have been executed
|
||||
|
||||
## Collector Configuration
|
||||
|
||||
Collectors define which queries should run for each client:
|
||||
|
||||
```sql
|
||||
-- collectorQueries table maps queries to clients
|
||||
CREATE TABLE collectorQueries (
|
||||
clientId varchar(255), -- Which client
|
||||
name varchar(255), -- Named reference (e.g., "JSON_QUERY")
|
||||
queryId uuid, -- Which query to run
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
When documents are processed, only queries configured in the client's collector will be executed.
|
||||
|
||||
## Current Implementation Status
|
||||
|
||||
### What's Working
|
||||
- Query dependency resolution and scheduling
|
||||
- Recursive execution of dependent queries
|
||||
- Version tracking for queries and results
|
||||
- JSON path extraction from query results
|
||||
- Client-specific query configuration via collectors
|
||||
|
||||
### What Appears Incomplete
|
||||
- **CONTEXT_FULL Implementation**: Currently returns mock data instead of actual document text
|
||||
- The system stores extracted text in S3 (via docTextRunner) but CONTEXT_FULL doesn't retrieve it
|
||||
- There's infrastructure for text storage (`documentTextExtractions` table) but it's not connected to the query processor
|
||||
|
||||
## Real-World Use Case
|
||||
|
||||
In a production scenario, this system would work like:
|
||||
|
||||
1. **Document uploaded** → Text extracted via OCR
|
||||
2. **Base Query** (CONTEXT_FULL) → Returns full document text as JSON
|
||||
3. **Field Extraction Queries** (JSON_EXTRACTOR) → Extract specific fields like:
|
||||
- Customer name
|
||||
- Invoice number
|
||||
- Date
|
||||
- Amount
|
||||
4. **Derived Queries** → Further process extracted data:
|
||||
- Calculate totals
|
||||
- Validate formats
|
||||
- Cross-reference with other data
|
||||
|
||||
## Key Design Benefits
|
||||
|
||||
1. **Flexibility**: New extraction rules can be added without code changes
|
||||
2. **Reusability**: Queries can be shared across clients or documents
|
||||
3. **Dependency Management**: Complex multi-step extractions are handled automatically
|
||||
4. **Versioning**: Changes to queries don't break existing results
|
||||
5. **Client Isolation**: Each client can have different extraction rules for the same document types
|
||||
|
||||
## Summary
|
||||
|
||||
The query system is a sophisticated, dependency-aware data extraction pipeline. While the current implementation has placeholder logic for the base CONTEXT_FULL query, the infrastructure for chaining queries, managing dependencies, and extracting structured data from documents is fully functional. The system allows for complex, multi-step data extraction workflows to be defined declaratively and executed automatically as documents are processed.
|
||||
Reference in New Issue
Block a user