query-orchestration/docs/ai.generated/query-system-explanation.md

Query System Explanation

Overview

The query system in this application is designed to extract structured data from processed documents. Queries are configurable data extraction rules that can depend on each other, creating a pipeline of data transformations.

Query Types

The system currently supports two query types:

1. CONTEXT_FULL Query

• Purpose: Originally intended to provide the full document text/context
• Current Implementation: Returns a hardcoded JSON placeholder {"keyone":"valueone","keytwo":"valuetwo"}
• No Dependencies: This query type doesn't require any other queries to run first
• Note: The current implementation appears to be a mock/placeholder - in production, this would likely return the actual extracted text from the document

2. JSON_EXTRACTOR Query

• Purpose: Extracts specific values from JSON data using path expressions
• Dependencies: Requires exactly one other query result as input (typically a CONTEXT_FULL query)
• Configuration: Takes a JSON config with a path field specifying what to extract
• Uses gjson: Leverages the tidwall/gjson library for JSON path extraction

How Queries Work

Query Definition Structure

type Query {
    ID               uuid
    Type             queryType  // CONTEXT_FULL or JSON_EXTRACTOR
    ActiveVersion    int32      // Current version being used
    RequiredQueryIDs []uuid     // Dependencies - other queries that must run first
    Config           string     // JSON configuration (e.g., {"path":"keyone"})
}

Example Query Chain

1. First Query (CONTEXT_FULL):

   • Type: CONTEXT_FULL
   • Config: None needed
   • Dependencies: None
   • Returns: {"keyone":"valueone","keytwo":"valuetwo"}

2. Second Query (JSON_EXTRACTOR):

   • Type: JSON_EXTRACTOR
   • Config: {"path":"keyone"}
   • Dependencies: [First Query ID]
   • Input: Result from First Query
   • Returns: "valueone"

3. Third Query (JSON_EXTRACTOR) - could extract from second:

   • Type: JSON_EXTRACTOR
   • Config: {"path":"some.nested.path"}
   • Dependencies: [Second Query ID]
   • Would extract from the result of the second query

Query Execution Flow

1. querySyncRunner Phase

• After document text extraction completes, querySyncRunner is triggered
• It queries the database for all queries that:
  • Are configured for this client (via collector configuration)
  • Have NO dependencies, OR
  • Have all dependencies already satisfied (results exist in database)
• For each eligible query, sends a message to the QUERY queue

2. queryRunner Phase

• Receives {document_id, query_id} from QUERY queue
• Retrieves the query definition and active version
• If query has dependencies:
  • Fetches results from all required queries
  • Validates all dependencies are satisfied
• Executes the appropriate processor based on query type:
  • CONTEXT_FULL: Returns the mock JSON (should return document text)
  • JSON_EXTRACTOR: Extracts specified path from dependency result
• Stores the result in database with version tracking
• Triggers dependent queries: Finds all queries that depend on this one and sends them to QUERY queue

3. Recursive Processing

• As each query completes, it triggers its dependents
• This creates a cascade effect through the dependency graph
• Continues until all queries in the chain have been executed

Collector Configuration

Collectors define which queries should run for each client:

-- collectorQueries table maps queries to clients
CREATE TABLE collectorQueries (
    clientId varchar(255),  -- Which client
    name varchar(255),      -- Named reference (e.g., "JSON_QUERY")
    queryId uuid,          -- Which query to run
    ...
)

When documents are processed, only queries configured in the client's collector will be executed.

Current Implementation Status

What's Working

• Query dependency resolution and scheduling
• Recursive execution of dependent queries
• Version tracking for queries and results
• JSON path extraction from query results
• Client-specific query configuration via collectors

What Appears Incomplete

• CONTEXT_FULL Implementation: Currently returns mock data instead of actual document text
• The system stores extracted text in S3 (via docTextRunner) but CONTEXT_FULL doesn't retrieve it
• There's infrastructure for text storage (documentTextExtractions table) but it's not connected to the query processor

Real-World Use Case

In a production scenario, this system would work like:

1. Document uploaded → Text extracted via OCR
2. Base Query (CONTEXT_FULL) → Returns full document text as JSON
3. Field Extraction Queries (JSON_EXTRACTOR) → Extract specific fields like:
   • Customer name
   • Invoice number
   • Date
   • Amount
4. Derived Queries → Further process extracted data:
   • Calculate totals
   • Validate formats
   • Cross-reference with other data

Key Design Benefits

1. Flexibility: New extraction rules can be added without code changes
2. Reusability: Queries can be shared across clients or documents
3. Dependency Management: Complex multi-step extractions are handled automatically
4. Versioning: Changes to queries don't break existing results
5. Client Isolation: Each client can have different extraction rules for the same document types

Summary

The query system is a sophisticated, dependency-aware data extraction pipeline. While the current implementation has placeholder logic for the base CONTEXT_FULL query, the infrastructure for chaining queries, managing dependencies, and extracting structured data from documents is fully functional. The system allows for complex, multi-step data extraction workflows to be defined declaratively and executed automatically as documents are processed.