package docs

2025-07-23 09:47:01 -07:00
parent 00a46eb629
commit fa41884e84
4 changed files with 480 additions and 0 deletions
@@ -0,0 +1,251 @@
+# Package database
+
+## Overview
+
+Package database provides the data access layer for the query orchestration platform. It manages PostgreSQL database connections, schema migrations, and generated query functions using SQLC. The package implements a repository pattern with type-safe database operations and comprehensive transaction support.
+
+## Architecture
+
+### Core Components
+
+#### Database Schema
+The database schema supports document processing and query orchestration with the following key entities:
+- **Clients**: Customer configurations with associated document sets
+- **Documents**: Uploaded files with processing state tracking
+- **Queries**: Data extraction and processing definitions with versioning
+- **Collectors**: Query collection specifications per client
+- **Results**: Processed query outputs linked to documents
+- **Text Extractions**: OCR results from document processing
+
+#### Migration System
+- Uses `golang-migrate` for schema version management
+- Migrations stored in `migrations/` directory with up/down SQL files
+- Automatic migration execution on startup
+- Support for extensions, tables, views, and functions
+
+#### Query Generation
+- SQLC generates type-safe Go code from SQL queries
+- Query definitions in `queries/` directory
+- Generated code in `repository/` subdirectory
+- Automatic struct mapping and null handling
+
+### Key Files
+
+- **migrations.go**: Database migration management and execution
+- **queries/*.sql**: SQLC query definitions for each entity
+- **repository/db.go**: Database connection interface and transaction support
+- **repository/*.sql.go**: Generated query functions from SQLC
+- **repository/models.go**: Generated data models and type definitions
+- **database.md**: Schema documentation with Mermaid diagram
+- **diagram.mmd**: Visual database schema representation
+
+## Database Schema Design
+
+### Tables
+
+#### Core Tables
+- **clients**: Customer configurations and metadata
+- **documents**: Document upload tracking and processing status
+- **documentCleans**: Cleaned document content storage
+- **documentTextExtractions**: OCR text extraction results
+- **queries**: Query definitions with JSON configurations
+- **queryVersions**: Query version history and dependency tracking
+- **results**: Query execution results per document
+- **collectors**: Client-specific query collections
+- **collectorVersions**: Collector version history
+
+### Views
+
+#### Query Views
+- **fullActiveQueries**: Complete active query information with dependencies
+- **queryActiveVersions**: Current active version for each query
+- **queryActiveDependencies**: Recursive dependency resolution
+- **activeQueryNameTypeMap**: Mapping of query names to types
+
+#### Collector Views
+- **fullActiveCollectorVersions**: Active collector configurations
+- **collectorActiveVersions**: Current version for each collector
+- **collectorQueryDependencyTree**: Full dependency graph for collectors
+
+#### Document Views
+- **currentTextEntries**: Latest text extraction for each document
+- **latestDocumentTextResults**: Most recent results per document
+
+### Custom Functions
+- **uuid_generate_v7()**: Generates time-ordered UUIDs for better indexing
+- **json_matches_schema()**: Validates JSON against schema definitions
+
+## Usage
+
+### Database Connection
+
+```go
+// Create database configuration
+dbConfig := &serviceconfig.DBConfig{
+    DBHost:     "localhost",
+    DBPort:     "5432",
+    DBName:     "querydb",
+    DBUser:     "dbuser",
+    DBPassword: "dbpass",
+}
+
+// Get connection pool
+pool, err := dbConfig.GetDBPool(ctx)
+if err != nil {
+    return err
+}
+defer pool.Close()
+
+// Create repository queries instance
+queries := repository.New(pool)
+```
+
+### Transaction Management
+
+```go
+// Execute operations in a transaction
+err := dbConfig.ExecuteDBTransaction(ctx, func(ctx context.Context, qtx *repository.Queries) error {
+    // All operations here run in a single transaction
+    client, err := qtx.CreateClient(ctx, repository.CreateClientParams{
+        ID:   clientID,
+        Name: clientName,
+    })
+    if err != nil {
+        return err // Transaction will rollback
+    }
+    
+    // Additional operations...
+    return nil // Transaction will commit
+})
+```
+
+### Query Examples
+
+```go
+// Get a document by ID
+document, err := queries.GetDocument(ctx, documentID)
+
+// List documents for a client
+documents, err := queries.ListDocumentsByClient(ctx, repository.ListDocumentsByClientParams{
+    ClientID: clientID,
+    Limit:    50,
+    Offset:   0,
+})
+
+// Create a query result
+result, err := queries.CreateResult(ctx, repository.CreateResultParams{
+    TextEntryID: textEntryID,
+    QueryID:     queryID,
+    Result:      resultJSON,
+})
+```
+
+## Migration Management
+
+### Creating Migrations
+
+```bash
+# Create a new migration
+migrate create -ext sql -dir internal/database/migrations -seq add_new_feature
+
+# This creates:
+# - 000000XX_add_new_feature.up.sql
+# - 000000XX_add_new_feature.down.sql
+```
+
+### Migration Conventions
+- Use sequential numbering with zero padding
+- Provide complete rollback logic in down migrations
+- Test both up and down migrations
+- Include data migrations where necessary
+
+### Migration Execution
+Migrations run automatically on application startup via:
+```go
+migrator := database.NewMigrator(dbURL)
+err := migrator.Up()
+```
+
+## SQLC Integration
+
+### Query Definition Format
+
+```sql
+-- name: GetDocument :one
+SELECT * FROM documents
+WHERE id = $1;
+
+-- name: ListDocumentsByClient :many
+SELECT * FROM documents
+WHERE clientId = $1
+ORDER BY created DESC
+LIMIT $2 OFFSET $3;
+
+-- name: UpdateDocumentStatus :exec
+UPDATE documents
+SET status = $2, updated = CURRENT_TIMESTAMP
+WHERE id = $1;
+```
+
+### Generated Code
+SQLC generates:
+- Type-safe query functions
+- Struct definitions matching table schemas
+- Null handling with sql.NullString, sql.NullTime, etc.
+- Parameter validation
+
+## Testing
+
+### Test Database Setup
+Tests use testcontainers to spin up PostgreSQL instances:
+```go
+container := test.NewDatabaseContainer(t)
+dbConfig := container.DBConfig(t)
+```
+
+### Repository Testing
+- Each repository file has corresponding test coverage
+- Tests validate CRUD operations
+- Transaction rollback testing
+- Concurrent access validation
+
+## Performance Considerations
+
+### Connection Pooling
+- Uses pgx connection pooling
+- Configurable pool size and timeouts
+- Health check pings with timeout
+- Automatic connection recovery
+
+### Query Optimization
+- Views provide pre-computed joins
+- Recursive CTEs for dependency resolution
+- Proper use of LIMIT/OFFSET for pagination
+- JSON indexing for configuration queries
+
+## Data Types
+
+### Custom Types
+- **UUID**: Used for all primary keys
+- **JSONB**: Query configurations and results
+- **Timestamptz**: All timestamp fields timezone-aware
+- **Text**: Unbounded string storage
+- **Custom Enums**: Via CHECK constraints
+
+### Null Handling
+- Explicit NULL/NOT NULL constraints
+- Go sql.Null* types for nullable fields
+- Default values where appropriate
+
+## Security
+
+### Access Control
+- Database credentials via environment variables
+- Connection string sanitization
+- Parameterized queries prevent SQL injection
+- No dynamic SQL construction
+
+### Data Protection
+- Sensitive data fields identified
+- Audit fields (created, updated) on all tables
+- Logical deletion support where needed