query-orchestration/internal/database/README.md

# Package database

## Overview

Package database provides the data access layer for the query orchestration platform. It manages PostgreSQL database connections, schema migrations, and generated query functions using SQLC. The package implements a repository pattern with type-safe database operations and comprehensive transaction support.

## Architecture

### Core Components

#### Database Schema
The database schema supports document processing and query orchestration with the following key entities:
- **Clients**: Customer configurations with associated document sets
- **Documents**: Uploaded files with processing state tracking
- **Queries**: Data extraction and processing definitions with versioning
- **Collectors**: Query collection specifications per client
- **Results**: Processed query outputs linked to documents
- **Text Extractions**: OCR results from document processing

#### Migration System
- Uses `golang-migrate` for schema version management
- Migrations stored in `migrations/` directory with up/down SQL files
- Automatic migration execution on startup
- Support for extensions, tables, views, and functions

#### Query Generation
- SQLC generates type-safe Go code from SQL queries
- Query definitions in `queries/` directory
- Generated code in `repository/` subdirectory
- Automatic struct mapping and null handling

### Key Files

- **migrations.go**: Database migration management and execution
- **queries/*.sql**: SQLC query definitions for each entity
- **repository/db.go**: Database connection interface and transaction support
- **repository/*.sql.go**: Generated query functions from SQLC
- **repository/models.go**: Generated data models and type definitions
- **database.md**: Schema documentation with Mermaid diagram
- **diagram.mmd**: Visual database schema representation

## Database Schema Design

### Tables

#### Core Tables
- **clients**: Customer configurations and metadata
- **documents**: Document upload tracking and processing status
- **documentCleans**: Cleaned document content storage
- **documentTextExtractions**: OCR text extraction results
- **queries**: Query definitions with JSON configurations
- **queryVersions**: Query version history and dependency tracking
- **results**: Query execution results per document
- **collectors**: Client-specific query collections
- **collectorVersions**: Collector version history

### Views

#### Query Views
- **fullActiveQueries**: Complete active query information with dependencies
- **queryActiveVersions**: Current active version for each query
- **queryActiveDependencies**: Recursive dependency resolution
- **activeQueryNameTypeMap**: Mapping of query names to types

#### Collector Views
- **fullActiveCollectorVersions**: Active collector configurations
- **collectorActiveVersions**: Current version for each collector
- **collectorQueryDependencyTree**: Full dependency graph for collectors

#### Document Views
- **currentTextEntries**: Latest text extraction for each document
- **latestDocumentTextResults**: Most recent results per document

### Custom Functions
- **uuid_generate_v7()**: Generates time-ordered UUIDs for better indexing
- **json_matches_schema()**: Validates JSON against schema definitions

## Usage

### Database Connection

```go
// Create database configuration
dbConfig := &serviceconfig.DBConfig{
    DBHost:     "localhost",
    DBPort:     "5432",
    DBName:     "querydb",
    DBUser:     "dbuser",
    DBPassword: "dbpass",
}

// Get connection pool
pool, err := dbConfig.GetDBPool(ctx)
if err != nil {
    return err
}
defer pool.Close()

// Create repository queries instance
queries := repository.New(pool)
```

### Transaction Management

```go
// Execute operations in a transaction
err := dbConfig.ExecuteDBTransaction(ctx, func(ctx context.Context, qtx *repository.Queries) error {
    // All operations here run in a single transaction
    client, err := qtx.CreateClient(ctx, repository.CreateClientParams{
        ID:   clientID,
        Name: clientName,
    })
    if err != nil {
        return err // Transaction will rollback
    }
    
    // Additional operations...
    return nil // Transaction will commit
})
```

### Query Examples

```go
// Get a document by ID
document, err := queries.GetDocument(ctx, documentID)

// List documents for a client
documents, err := queries.ListDocumentsByClient(ctx, repository.ListDocumentsByClientParams{
    ClientID: clientID,
    Limit:    50,
    Offset:   0,
})

// Create a query result
result, err := queries.CreateResult(ctx, repository.CreateResultParams{
    TextEntryID: textEntryID,
    QueryID:     queryID,
    Result:      resultJSON,
})
```

## Migration Management

### Creating Migrations

```bash
# Create a new migration
migrate create -ext sql -dir internal/database/migrations -seq add_new_feature

# This creates:
# - 000000XX_add_new_feature.up.sql
# - 000000XX_add_new_feature.down.sql
```

### Migration Conventions
- Use sequential numbering with zero padding
- Provide complete rollback logic in down migrations
- Test both up and down migrations
- Include data migrations where necessary

### Migration Execution
Migrations run automatically on application startup via:
```go
migrator := database.NewMigrator(dbURL)
err := migrator.Up()
```

## SQLC Integration

### Query Definition Format

```sql
-- name: GetDocument :one
SELECT * FROM documents
WHERE id = $1;

-- name: ListDocumentsByClient :many
SELECT * FROM documents
WHERE clientId = $1
ORDER BY created DESC
LIMIT $2 OFFSET $3;

-- name: UpdateDocumentStatus :exec
UPDATE documents
SET status = $2, updated = CURRENT_TIMESTAMP
WHERE id = $1;
```

### Generated Code
SQLC generates:
- Type-safe query functions
- Struct definitions matching table schemas
- Null handling with sql.NullString, sql.NullTime, etc.
- Parameter validation

## Testing

### Test Database Setup
Tests use testcontainers to spin up PostgreSQL instances:
```go
container := test.NewDatabaseContainer(t)
dbConfig := container.DBConfig(t)
```

### Repository Testing
- Each repository file has corresponding test coverage
- Tests validate CRUD operations
- Transaction rollback testing
- Concurrent access validation

## Performance Considerations

### Connection Pooling
- Uses pgx connection pooling
- Configurable pool size and timeouts
- Health check pings with timeout
- Automatic connection recovery

### Query Optimization
- Views provide pre-computed joins
- Recursive CTEs for dependency resolution
- Proper use of LIMIT/OFFSET for pagination
- JSON indexing for configuration queries

## Data Types

### Custom Types
- **UUID**: Used for all primary keys
- **JSONB**: Query configurations and results
- **Timestamptz**: All timestamp fields timezone-aware
- **Text**: Unbounded string storage
- **Custom Enums**: Via CHECK constraints

### Null Handling
- Explicit NULL/NOT NULL constraints
- Go sql.Null* types for nullable fields
- Default values where appropriate

## Security

### Access Control
- Database credentials via environment variables
- Connection string sanitization
- Parameterized queries prevent SQL injection
- No dynamic SQL construction

### Data Protection
- Sensitive data fields identified
- Audit fields (created, updated) on all tables
- Logical deletion support where needed