query-orchestration/docs/ai.generated/05-getting-started.md

Getting Started Guide

Prerequisites

Required Tools

• Devbox: Provides reproducible development environment with Go 1.24.3
  • Ubuntu/MacOS: curl -fsSL https://get.jetify.com/devbox | bash
  • NixOS: devbox (already available)
• Docker: Container runtime for services and testing
  • Install from: https://docs.docker.com/engine/install/
• Git: Version control (usually pre-installed)
• AWS CLI: Required for Textract credentials and S3 access (installed via devbox)

System Requirements

• OS: Linux, macOS, or Windows with WSL2
• RAM: 8GB minimum, 16GB recommended
• Disk: 10GB free space for development environment
• Network: Internet access for downloading dependencies

Docker Group Setup (Linux)

Ensure your user can run Docker commands without sudo:

sudo groupadd docker
sudo usermod -aG docker $USER
# Log out and back in for changes to take effect

Local Development Setup

1. Environment Initialization

# Clone repository
git clone <repository-url>
cd query-orchestration.2

# Create environment file for custom variables
touch .env

# Enter development environment (installs all tools automatically)
devbox shell

2. Verify Installation

# Check Go version
go version  # Should show 1.24.0

# Check Docker
docker --version

# Check Task runner
task --version  # Should show 3.39.2+

# List available tasks
task --list

3. Full System Validation

Run the complete test suite to ensure everything works:

task fullsuite

This command will:

1. Generate latest API specifications
2. Run linting commands
3. Execute unit tests
4. Run end-to-end integration tests
5. Validate coverage thresholds (80% total, 60% function)

Development Workflow

Common Development Tasks

Running Services Locally

# Start all services with local infrastructure
task dev:up

# Start specific service
task dev:api      # Just the API service
task dev:runners  # Just the queue runners

API Development

# Generate API code from OpenAPI spec
task openapi:generate

# Start API server with hot reload
task dev:api

# View API documentation
open http://localhost:8080/swagger/index.html

Database Operations

# Run database migrations
task db:migrate

# Reset database (development only)
task db:reset

# Generate database models from SQL
task db:generate

Testing

# Run unit tests
task test:unit

# Run integration tests
task test:integration

# Run tests with coverage
task test:coverage

# Run specific test package
go test ./internal/client/...

Code Quality

# Run linters
task lint

# Format code
task format

# Generate mocks for testing
task generate:mocks

Development Environment Configuration

Environment Variables Hierarchy

1. devbox.json "env" section (highest priority)
2. devbox.json "init_hook" exports
3. .env file variables (lowest priority)

Key Development Variables

Create a .env file with:

# Database (optional - defaults provided)
PGUSER=postgres
PGPASSWORD=password
PGDATABASE=queryorchestration

# AWS (uses LocalStack in development)
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=test
AWS_SECRET_ACCESS_KEY=test

# Authentication (disable for local development)
DISABLE_AUTH=true

# Logging
LOG_LEVEL=DEBUG

Service Ports (Local Development)

• queryAPI: 8080 (main API)
• storeEventRunner: 8081
• docInitRunner: 8082
• docSyncRunner: 8083
• docCleanRunner: 8084
• docTextRunner: 8085
• querySyncRunner: 8087
• queryRunner: 8088
• clientSyncRunner: 8089
• queryVersionSyncRunner: 8090
• Prometheus: 9091
• PostgreSQL: 5432
• LocalStack: 4566

Common Development Workflows

1. API Endpoint Development

# 1. Modify OpenAPI specification
vim serviceAPIs/queryAPI.yaml

# 2. Regenerate API code
task openapi:generate

# 3. Implement controller logic
vim api/queryAPI/controllers.go

# 4. Run API server
task dev:api

# 5. Test endpoint
curl -X GET http://localhost:8080/health

2. Database Schema Changes

# 1. Create new migration files
vim internal/database/migrations/000000000000X_feature.up.sql
vim internal/database/migrations/000000000000X_feature.down.sql

# 2. Add queries to SQL files
vim internal/database/queries/feature.sql

# 3. Regenerate database models
task db:generate

# 4. Run migrations
task db:migrate

# 5. Update Go service code
vim internal/feature/service.go

3. New Service Development

# 1. Implement controller interface
vim api/newRunner/runner.go

# 2. Create main command
vim cmd/newRunner/main.go

# 3. Add service configuration
vim internal/serviceconfig/newrunner/config.go

# 4. Update docker-compose
vim deployments/compose.local.yaml

# 5. Run tests
task test:integration

4. Adding New Queue

# 1. Add queue configuration
vim internal/serviceconfig/queue/newqueue/config.go

# 2. Update producer services
vim internal/service/producer.go

# 3. Update consumer services  
vim api/consumerRunner/runner.go

# 4. Update environment variables
vim deployments/compose.local.yaml

Testing Guidelines

Unit Testing

• Location: *_test.go files alongside source code
• Pattern: Table-driven tests preferred
• Mocking: Use generated mocks for external dependencies
• Coverage: Minimum 60% function coverage required

Example unit test:

func TestService_CreateClient(t *testing.T) {
    tests := []struct {
        name     string
        input    CreateClientRequest
        expected Client
        wantErr  bool
    }{
        {
            name: "valid client creation",
            input: CreateClientRequest{
                ClientID: "test-client",
                Name:     "Test Client",
            },
            expected: Client{
                ClientID: "test-client", 
                Name:     "Test Client",
            },
            wantErr: false,
        },
    }
    
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            // Test implementation
        })
    }
}

Integration Testing

• Location: test/ directory and *_test.go files
• Infrastructure: Uses testcontainers for real dependencies
• Isolation: Each test gets clean environment
• Coverage: Minimum 80% total coverage required

End-to-End Testing

• Environment: Full service deployment via Docker Compose
• Scope: Complete workflows from API to database
• Authentication: Uses test Cognito setup or disabled auth

Debugging and Troubleshooting

Common Issues

Docker Permission Errors

# Fix Docker permissions (Linux)
sudo groupadd docker
sudo usermod -aG docker $USER
# Log out and log back in

Database Connection Issues

# Check PostgreSQL status
task db:status

# Reset database
task db:reset

# Check connection manually
psql -h localhost -p 5432 -U postgres -d queryorchestration

Service Startup Issues

# Check service logs
docker-compose -f deployments/compose.local.yaml logs [service-name]

# Check environment variables
task env:check

# Verify ports aren't in use
lsof -i :8080

Code Generation Issues

# Clean generated files
task clean

# Regenerate everything
task generate:all

# Check for syntax errors in source files
task lint

Development Tools

Debugging

• Delve: Go debugger integrated with devbox
• IDE Integration: VS Code and GoLand support provided
• Logging: Structured logging with slog at DEBUG level

Monitoring

• Prometheus: Metrics at http://localhost:9091
• Swagger UI: API docs at http://localhost:8080/swagger/
• Health Checks: http://localhost:8080/health

Database Tools

• psql: Command-line PostgreSQL client
• Database browser: Connect to localhost:5432
• Migration status: task db:status

Performance Profiling

# CPU profiling
go test -cpuprofile=cpu.prof -bench=.

# Memory profiling  
go test -memprofile=mem.prof -bench=.

# Race detection
go test -race ./...

Next Steps

After completing the setup:

1. Explore the API: Visit http://localhost:8080/swagger/
2. Review Architecture: Read the service architecture documentation
3. Check Data Model: Examine database schema documentation
4. Run Examples: Try the metrics example service
5. Write Tests: Create tests for new features