package docs
This commit is contained in:
jay brown
@@ -1,5 +1,11 @@
|
||||
# Cognito Auth
|
||||
|
||||
## Package Documentation
|
||||
|
||||
For comprehensive package-level documentation, see [README.md](./README.md).
|
||||
|
||||
# Cognito Auth
|
||||
|
||||
A reusable Go package for AWS Cognito authentication and Permit.io authorization with Echo framework using PKCE flow.
|
||||
|
||||
## Features
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
# Package database
|
||||
|
||||
## Overview
|
||||
|
||||
Package database provides the data access layer for the query orchestration platform. It manages PostgreSQL database connections, schema migrations, and generated query functions using SQLC. The package implements a repository pattern with type-safe database operations and comprehensive transaction support.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
#### Database Schema
|
||||
The database schema supports document processing and query orchestration with the following key entities:
|
||||
- **Clients**: Customer configurations with associated document sets
|
||||
- **Documents**: Uploaded files with processing state tracking
|
||||
- **Queries**: Data extraction and processing definitions with versioning
|
||||
- **Collectors**: Query collection specifications per client
|
||||
- **Results**: Processed query outputs linked to documents
|
||||
- **Text Extractions**: OCR results from document processing
|
||||
|
||||
#### Migration System
|
||||
- Uses `golang-migrate` for schema version management
|
||||
- Migrations stored in `migrations/` directory with up/down SQL files
|
||||
- Automatic migration execution on startup
|
||||
- Support for extensions, tables, views, and functions
|
||||
|
||||
#### Query Generation
|
||||
- SQLC generates type-safe Go code from SQL queries
|
||||
- Query definitions in `queries/` directory
|
||||
- Generated code in `repository/` subdirectory
|
||||
- Automatic struct mapping and null handling
|
||||
|
||||
### Key Files
|
||||
|
||||
- **migrations.go**: Database migration management and execution
|
||||
- **queries/*.sql**: SQLC query definitions for each entity
|
||||
- **repository/db.go**: Database connection interface and transaction support
|
||||
- **repository/*.sql.go**: Generated query functions from SQLC
|
||||
- **repository/models.go**: Generated data models and type definitions
|
||||
- **database.md**: Schema documentation with Mermaid diagram
|
||||
- **diagram.mmd**: Visual database schema representation
|
||||
|
||||
## Database Schema Design
|
||||
|
||||
### Tables
|
||||
|
||||
#### Core Tables
|
||||
- **clients**: Customer configurations and metadata
|
||||
- **documents**: Document upload tracking and processing status
|
||||
- **documentCleans**: Cleaned document content storage
|
||||
- **documentTextExtractions**: OCR text extraction results
|
||||
- **queries**: Query definitions with JSON configurations
|
||||
- **queryVersions**: Query version history and dependency tracking
|
||||
- **results**: Query execution results per document
|
||||
- **collectors**: Client-specific query collections
|
||||
- **collectorVersions**: Collector version history
|
||||
|
||||
### Views
|
||||
|
||||
#### Query Views
|
||||
- **fullActiveQueries**: Complete active query information with dependencies
|
||||
- **queryActiveVersions**: Current active version for each query
|
||||
- **queryActiveDependencies**: Recursive dependency resolution
|
||||
- **activeQueryNameTypeMap**: Mapping of query names to types
|
||||
|
||||
#### Collector Views
|
||||
- **fullActiveCollectorVersions**: Active collector configurations
|
||||
- **collectorActiveVersions**: Current version for each collector
|
||||
- **collectorQueryDependencyTree**: Full dependency graph for collectors
|
||||
|
||||
#### Document Views
|
||||
- **currentTextEntries**: Latest text extraction for each document
|
||||
- **latestDocumentTextResults**: Most recent results per document
|
||||
|
||||
### Custom Functions
|
||||
- **uuid_generate_v7()**: Generates time-ordered UUIDs for better indexing
|
||||
- **json_matches_schema()**: Validates JSON against schema definitions
|
||||
|
||||
## Usage
|
||||
|
||||
### Database Connection
|
||||
|
||||
```go
|
||||
// Create database configuration
|
||||
dbConfig := &serviceconfig.DBConfig{
|
||||
DBHost: "localhost",
|
||||
DBPort: "5432",
|
||||
DBName: "querydb",
|
||||
DBUser: "dbuser",
|
||||
DBPassword: "dbpass",
|
||||
}
|
||||
|
||||
// Get connection pool
|
||||
pool, err := dbConfig.GetDBPool(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer pool.Close()
|
||||
|
||||
// Create repository queries instance
|
||||
queries := repository.New(pool)
|
||||
```
|
||||
|
||||
### Transaction Management
|
||||
|
||||
```go
|
||||
// Execute operations in a transaction
|
||||
err := dbConfig.ExecuteDBTransaction(ctx, func(ctx context.Context, qtx *repository.Queries) error {
|
||||
// All operations here run in a single transaction
|
||||
client, err := qtx.CreateClient(ctx, repository.CreateClientParams{
|
||||
ID: clientID,
|
||||
Name: clientName,
|
||||
})
|
||||
if err != nil {
|
||||
return err // Transaction will rollback
|
||||
}
|
||||
|
||||
// Additional operations...
|
||||
return nil // Transaction will commit
|
||||
})
|
||||
```
|
||||
|
||||
### Query Examples
|
||||
|
||||
```go
|
||||
// Get a document by ID
|
||||
document, err := queries.GetDocument(ctx, documentID)
|
||||
|
||||
// List documents for a client
|
||||
documents, err := queries.ListDocumentsByClient(ctx, repository.ListDocumentsByClientParams{
|
||||
ClientID: clientID,
|
||||
Limit: 50,
|
||||
Offset: 0,
|
||||
})
|
||||
|
||||
// Create a query result
|
||||
result, err := queries.CreateResult(ctx, repository.CreateResultParams{
|
||||
TextEntryID: textEntryID,
|
||||
QueryID: queryID,
|
||||
Result: resultJSON,
|
||||
})
|
||||
```
|
||||
|
||||
## Migration Management
|
||||
|
||||
### Creating Migrations
|
||||
|
||||
```bash
|
||||
# Create a new migration
|
||||
migrate create -ext sql -dir internal/database/migrations -seq add_new_feature
|
||||
|
||||
# This creates:
|
||||
# - 000000XX_add_new_feature.up.sql
|
||||
# - 000000XX_add_new_feature.down.sql
|
||||
```
|
||||
|
||||
### Migration Conventions
|
||||
- Use sequential numbering with zero padding
|
||||
- Provide complete rollback logic in down migrations
|
||||
- Test both up and down migrations
|
||||
- Include data migrations where necessary
|
||||
|
||||
### Migration Execution
|
||||
Migrations run automatically on application startup via:
|
||||
```go
|
||||
migrator := database.NewMigrator(dbURL)
|
||||
err := migrator.Up()
|
||||
```
|
||||
|
||||
## SQLC Integration
|
||||
|
||||
### Query Definition Format
|
||||
|
||||
```sql
|
||||
-- name: GetDocument :one
|
||||
SELECT * FROM documents
|
||||
WHERE id = $1;
|
||||
|
||||
-- name: ListDocumentsByClient :many
|
||||
SELECT * FROM documents
|
||||
WHERE clientId = $1
|
||||
ORDER BY created DESC
|
||||
LIMIT $2 OFFSET $3;
|
||||
|
||||
-- name: UpdateDocumentStatus :exec
|
||||
UPDATE documents
|
||||
SET status = $2, updated = CURRENT_TIMESTAMP
|
||||
WHERE id = $1;
|
||||
```
|
||||
|
||||
### Generated Code
|
||||
SQLC generates:
|
||||
- Type-safe query functions
|
||||
- Struct definitions matching table schemas
|
||||
- Null handling with sql.NullString, sql.NullTime, etc.
|
||||
- Parameter validation
|
||||
|
||||
## Testing
|
||||
|
||||
### Test Database Setup
|
||||
Tests use testcontainers to spin up PostgreSQL instances:
|
||||
```go
|
||||
container := test.NewDatabaseContainer(t)
|
||||
dbConfig := container.DBConfig(t)
|
||||
```
|
||||
|
||||
### Repository Testing
|
||||
- Each repository file has corresponding test coverage
|
||||
- Tests validate CRUD operations
|
||||
- Transaction rollback testing
|
||||
- Concurrent access validation
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Connection Pooling
|
||||
- Uses pgx connection pooling
|
||||
- Configurable pool size and timeouts
|
||||
- Health check pings with timeout
|
||||
- Automatic connection recovery
|
||||
|
||||
### Query Optimization
|
||||
- Views provide pre-computed joins
|
||||
- Recursive CTEs for dependency resolution
|
||||
- Proper use of LIMIT/OFFSET for pagination
|
||||
- JSON indexing for configuration queries
|
||||
|
||||
## Data Types
|
||||
|
||||
### Custom Types
|
||||
- **UUID**: Used for all primary keys
|
||||
- **JSONB**: Query configurations and results
|
||||
- **Timestamptz**: All timestamp fields timezone-aware
|
||||
- **Text**: Unbounded string storage
|
||||
- **Custom Enums**: Via CHECK constraints
|
||||
|
||||
### Null Handling
|
||||
- Explicit NULL/NOT NULL constraints
|
||||
- Go sql.Null* types for nullable fields
|
||||
- Default values where appropriate
|
||||
|
||||
## Security
|
||||
|
||||
### Access Control
|
||||
- Database credentials via environment variables
|
||||
- Connection string sanitization
|
||||
- Parameterized queries prevent SQL injection
|
||||
- No dynamic SQL construction
|
||||
|
||||
### Data Protection
|
||||
- Sensitive data fields identified
|
||||
- Audit fields (created, updated) on all tables
|
||||
- Logical deletion support where needed
|
||||
@@ -0,0 +1,217 @@
|
||||
# Package query
|
||||
|
||||
## Overview
|
||||
|
||||
Package query implements the core query processing engine for the document orchestration platform. It provides a sophisticated dependency-driven execution model where queries can extract data from documents and depend on results from other queries. The package supports multiple query types, versioning, and asynchronous processing through AWS SQS queues.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Query Processing Model
|
||||
|
||||
The system implements a directed acyclic graph (DAG) of query dependencies:
|
||||
1. **Independent Queries**: Process document text directly
|
||||
2. **Dependent Queries**: Use results from other queries as input
|
||||
3. **Version Management**: Each query maintains version history
|
||||
4. **Result Caching**: Processed results stored for reuse
|
||||
|
||||
### Query Types
|
||||
|
||||
#### JsonExtractor
|
||||
Extracts values from JSON documents using JSONPath expressions:
|
||||
- Supports nested path extraction
|
||||
- Handles arrays and complex objects
|
||||
- Returns extracted values as strings
|
||||
- Configurable default values for missing paths
|
||||
|
||||
#### ContextFull
|
||||
Processes complete document context:
|
||||
- Access to full document text
|
||||
- Can reference other query results
|
||||
- Supports complex transformations
|
||||
- Used for advanced extraction logic
|
||||
|
||||
### Core Components
|
||||
|
||||
#### Service Layer (`service.go`)
|
||||
Central service orchestrating all query operations:
|
||||
- Query creation and updates
|
||||
- Version management
|
||||
- Result processing coordination
|
||||
- Dependency resolution
|
||||
|
||||
#### Query Management
|
||||
- **create.go**: Query creation with validation
|
||||
- **update.go**: Query updates with version tracking
|
||||
- **get.go**: Query retrieval operations
|
||||
- **list.go**: Query listing with filtering
|
||||
- **normalize.go**: Configuration normalization
|
||||
- **parse.go**: Query parsing and validation
|
||||
|
||||
#### Result Processing (`result/`)
|
||||
- **processor/**: Query type-specific processors
|
||||
- **get.go**: Result retrieval
|
||||
- **process.go**: Result computation logic
|
||||
- **set/**: Result storage operations
|
||||
- **sync/**: Result synchronization
|
||||
|
||||
#### Synchronization
|
||||
- **sync/**: Query dependency synchronization
|
||||
- **versionsync/**: Version update propagation
|
||||
- **test/**: Query testing utilities
|
||||
|
||||
## Query Lifecycle
|
||||
|
||||
### 1. Query Creation
|
||||
```go
|
||||
// Create a new JSON extractor query
|
||||
query, err := queryService.Create(ctx, &CreateRequest{
|
||||
ClientID: clientID,
|
||||
Name: "extract_invoice_number",
|
||||
Type: "JsonExtractor",
|
||||
Description: "Extracts invoice number from document",
|
||||
Config: map[string]interface{}{
|
||||
"path": "$.invoice.number",
|
||||
"default": "N/A",
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
### 2. Query Dependencies
|
||||
```go
|
||||
// Create dependent query
|
||||
dependentQuery, err := queryService.Create(ctx, &CreateRequest{
|
||||
ClientID: clientID,
|
||||
Name: "calculate_total",
|
||||
Type: "ContextFull",
|
||||
DependsOn: []string{"extract_line_items"},
|
||||
Description: "Calculates total from line items",
|
||||
Config: config,
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Version Management
|
||||
- Each update creates a new version
|
||||
- Active version tracked separately
|
||||
- Previous versions retained for history
|
||||
- Dependency chains updated on version changes
|
||||
|
||||
### 4. Result Processing
|
||||
```go
|
||||
// Process query for a document
|
||||
result, err := resultProcessor.Process(ctx, &ProcessRequest{
|
||||
DocumentID: documentID,
|
||||
QueryID: queryID,
|
||||
})
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Query Configuration Structure
|
||||
|
||||
#### JsonExtractor Config
|
||||
```json
|
||||
{
|
||||
"path": "$.data.field",
|
||||
"default": "default_value",
|
||||
"type": "string"
|
||||
}
|
||||
```
|
||||
|
||||
#### ContextFull Config
|
||||
```json
|
||||
{
|
||||
"template": "Process {{.Document}} with {{.Dependencies.other_query}}",
|
||||
"format": "json",
|
||||
"validations": []
|
||||
}
|
||||
```
|
||||
|
||||
### Service Configuration
|
||||
- Database connection for query storage
|
||||
- Queue configuration for async processing
|
||||
- Logging configuration
|
||||
- Thread pool for parallel processing
|
||||
|
||||
## Query Processing Flow
|
||||
|
||||
### Synchronous Processing
|
||||
1. **Validation**: Check query configuration
|
||||
2. **Dependency Resolution**: Ensure dependencies are satisfied
|
||||
3. **Execution**: Run query processor
|
||||
4. **Storage**: Save results to database
|
||||
5. **Notification**: Trigger dependent queries
|
||||
|
||||
### Asynchronous Processing
|
||||
1. **Queue Message**: Receive processing request
|
||||
2. **Lock Acquisition**: Prevent concurrent processing
|
||||
3. **Processing**: Execute query logic
|
||||
4. **Result Storage**: Persist results
|
||||
5. **Dependency Trigger**: Queue dependent queries
|
||||
|
||||
## Dependency Management
|
||||
|
||||
### Dependency Resolution
|
||||
- Recursive dependency checking
|
||||
- Circular dependency prevention
|
||||
- Missing dependency handling
|
||||
- Version-aware dependency tracking
|
||||
|
||||
### Dependency Types
|
||||
- **Hard Dependencies**: Must complete before processing
|
||||
- **Soft Dependencies**: Optional, use if available
|
||||
- **Version Dependencies**: Specific version requirements
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Types
|
||||
- **Validation Errors**: Invalid configuration or input
|
||||
- **Dependency Errors**: Missing or failed dependencies
|
||||
- **Processing Errors**: Query execution failures
|
||||
- **Storage Errors**: Database operation failures
|
||||
|
||||
### Error Recovery
|
||||
- Automatic retry for transient failures
|
||||
- Dead letter queue for persistent failures
|
||||
- Error context preservation
|
||||
- Detailed error logging
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Testing
|
||||
- Mock-based testing for isolated components
|
||||
- Configuration validation tests
|
||||
- Dependency resolution tests
|
||||
- Error scenario coverage
|
||||
|
||||
### Integration Testing
|
||||
- Full query lifecycle tests
|
||||
- Dependency chain testing
|
||||
- Concurrent processing validation
|
||||
- Performance benchmarks
|
||||
|
||||
### Test Utilities (`test/`)
|
||||
- Query creation helpers
|
||||
- Result validation utilities
|
||||
- Mock data generators
|
||||
- Test scenario builders
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Caching Strategy
|
||||
- Result caching for repeated queries
|
||||
- Dependency result reuse
|
||||
- Configuration caching
|
||||
- Connection pooling
|
||||
|
||||
### Parallel Processing
|
||||
- Concurrent query execution where possible
|
||||
- Thread pool management
|
||||
- Resource throttling
|
||||
- Queue-based load distribution
|
||||
|
||||
### Database Optimization
|
||||
- Indexed query lookups
|
||||
- Batch result storage
|
||||
- Efficient dependency queries
|
||||
- View-based aggregations
|
||||
|
||||
@@ -1,3 +1,9 @@
|
||||
# Service Configuration
|
||||
|
||||
## Package Documentation
|
||||
|
||||
For comprehensive package-level documentation, see [README.md](./README.md).
|
||||
|
||||
# Service Configuration
|
||||
This directory contains the service configuration for the DoczyAI project.
|
||||
Common code that is shared by all of the services in the project.
|
||||
|
||||
Reference in New Issue
Block a user