query-orchestration/docs/ai.generated/05-getting-started.md

# Getting Started Guide

## Prerequisites

### Required Tools
- **Devbox**: Provides reproducible development environment
  - Ubuntu/MacOS: `curl -fsSL https://get.jetify.com/devbox | bash`
  - NixOS: `devbox` (already available)
- **Docker**: Container runtime for services and testing
  - Install from: https://docs.docker.com/engine/install/
- **Git**: Version control (usually pre-installed)

### System Requirements
- **OS**: Linux, macOS, or Windows with WSL2
- **RAM**: 8GB minimum, 16GB recommended
- **Disk**: 10GB free space for development environment
- **Network**: Internet access for downloading dependencies

### Docker Group Setup (Linux)
Ensure your user can run Docker commands without sudo:
```bash
sudo groupadd docker
sudo usermod -aG docker $USER
# Log out and back in for changes to take effect
```

## Local Development Setup

### 1. Environment Initialization
```bash
# Clone repository
git clone <repository-url>
cd query-orchestration.2

# Create environment file for custom variables
touch .env

# Enter development environment (installs all tools automatically)
devbox shell
```

### 2. Verify Installation
```bash
# Check Go version
go version  # Should show 1.24.0

# Check Docker
docker --version

# Check Task runner
task --version  # Should show 3.39.2+

# List available tasks
task --list
```

### 3. Full System Validation
Run the complete test suite to ensure everything works:
```bash
task fullsuite
```

This command will:
1. Generate latest API specifications
2. Run linting commands
3. Execute unit tests
4. Run end-to-end integration tests
5. Validate coverage thresholds (80% total, 60% function)

## Development Workflow

### Common Development Tasks

#### Running Services Locally
```bash
# Start all services with local infrastructure
task dev:up

# Start specific service
task dev:api      # Just the API service
task dev:runners  # Just the queue runners
```

#### API Development
```bash
# Generate API code from OpenAPI spec
task openapi:generate

# Start API server with hot reload
task dev:api

# View API documentation
open http://localhost:8080/swagger/index.html
```

#### Database Operations
```bash
# Run database migrations
task db:migrate

# Reset database (development only)
task db:reset

# Generate database models from SQL
task db:generate
```

#### Testing
```bash
# Run unit tests
task test:unit

# Run integration tests
task test:integration

# Run tests with coverage
task test:coverage

# Run specific test package
go test ./internal/client/...
```

#### Code Quality
```bash
# Run linters
task lint

# Format code
task format

# Generate mocks for testing
task generate:mocks
```

### Development Environment Configuration

#### Environment Variables Hierarchy
1. **devbox.json "env"** section (highest priority)
2. **devbox.json "init_hook"** exports
3. **.env file** variables (lowest priority)

#### Key Development Variables
Create a `.env` file with:
```bash
# Database (optional - defaults provided)
PGUSER=postgres
PGPASSWORD=password
PGDATABASE=queryorchestration

# AWS (uses LocalStack in development)
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=test
AWS_SECRET_ACCESS_KEY=test

# Authentication (disable for local development)
DISABLE_AUTH=true

# Logging
LOG_LEVEL=DEBUG
```

#### Service Ports (Local Development)
- **queryAPI**: 8080 (main API)
- **storeEventRunner**: 8081
- **docInitRunner**: 8082
- **docSyncRunner**: 8083
- **docCleanRunner**: 8084
- **docTextRunner**: 8085
- **querySyncRunner**: 8087
- **queryRunner**: 8088
- **clientSyncRunner**: 8089
- **queryVersionSyncRunner**: 8090
- **Prometheus**: 9091
- **PostgreSQL**: 5432
- **LocalStack**: 4566

## Common Development Workflows

### 1. API Endpoint Development
```bash
# 1. Modify OpenAPI specification
vim serviceAPIs/queryAPI.yaml

# 2. Regenerate API code
task openapi:generate

# 3. Implement controller logic
vim api/queryAPI/controllers.go

# 4. Run API server
task dev:api

# 5. Test endpoint
curl -X GET http://localhost:8080/health
```

### 2. Database Schema Changes
```bash
# 1. Create new migration files
vim internal/database/migrations/000000000000X_feature.up.sql
vim internal/database/migrations/000000000000X_feature.down.sql

# 2. Add queries to SQL files
vim internal/database/queries/feature.sql

# 3. Regenerate database models
task db:generate

# 4. Run migrations
task db:migrate

# 5. Update Go service code
vim internal/feature/service.go
```

### 3. New Service Development
```bash
# 1. Implement controller interface
vim api/newRunner/runner.go

# 2. Create main command
vim cmd/newRunner/main.go

# 3. Add service configuration
vim internal/serviceconfig/newrunner/config.go

# 4. Update docker-compose
vim deployments/compose.local.yaml

# 5. Run tests
task test:integration
```

### 4. Adding New Queue
```bash
# 1. Add queue configuration
vim internal/serviceconfig/queue/newqueue/config.go

# 2. Update producer services
vim internal/service/producer.go

# 3. Update consumer services  
vim api/consumerRunner/runner.go

# 4. Update environment variables
vim deployments/compose.local.yaml
```

## Testing Guidelines

### Unit Testing
- **Location**: `*_test.go` files alongside source code
- **Pattern**: Table-driven tests preferred
- **Mocking**: Use generated mocks for external dependencies
- **Coverage**: Minimum 60% function coverage required

Example unit test:
```go
func TestService_CreateClient(t *testing.T) {
    tests := []struct {
        name     string
        input    CreateClientRequest
        expected Client
        wantErr  bool
    }{
        {
            name: "valid client creation",
            input: CreateClientRequest{
                ClientID: "test-client",
                Name:     "Test Client",
            },
            expected: Client{
                ClientID: "test-client", 
                Name:     "Test Client",
            },
            wantErr: false,
        },
    }
    
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            // Test implementation
        })
    }
}
```

### Integration Testing
- **Location**: `test/` directory and `*_test.go` files
- **Infrastructure**: Uses testcontainers for real dependencies
- **Isolation**: Each test gets clean environment
- **Coverage**: Minimum 80% total coverage required

### End-to-End Testing
- **Environment**: Full service deployment via Docker Compose
- **Scope**: Complete workflows from API to database
- **Authentication**: Uses test Cognito setup or disabled auth

## Debugging and Troubleshooting

### Common Issues

#### Docker Permission Errors
```bash
# Fix Docker permissions (Linux)
sudo groupadd docker
sudo usermod -aG docker $USER
# Log out and log back in
```

#### Database Connection Issues
```bash
# Check PostgreSQL status
task db:status

# Reset database
task db:reset

# Check connection manually
psql -h localhost -p 5432 -U postgres -d queryorchestration
```

#### Service Startup Issues
```bash
# Check service logs
docker-compose -f deployments/compose.local.yaml logs [service-name]

# Check environment variables
task env:check

# Verify ports aren't in use
lsof -i :8080
```

#### Code Generation Issues
```bash
# Clean generated files
task clean

# Regenerate everything
task generate:all

# Check for syntax errors in source files
task lint
```

### Development Tools

#### Debugging
- **Delve**: Go debugger integrated with devbox
- **IDE Integration**: VS Code and GoLand support provided
- **Logging**: Structured logging with slog at DEBUG level

#### Monitoring
- **Prometheus**: Metrics at http://localhost:9091
- **Swagger UI**: API docs at http://localhost:8080/swagger/
- **Health Checks**: http://localhost:8080/health

#### Database Tools
- **psql**: Command-line PostgreSQL client
- **Database browser**: Connect to localhost:5432
- **Migration status**: `task db:status`

### Performance Profiling
```bash
# CPU profiling
go test -cpuprofile=cpu.prof -bench=.

# Memory profiling  
go test -memprofile=mem.prof -bench=.

# Race detection
go test -race ./...
```

## Next Steps

After completing the setup:
1. **Explore the API**: Visit http://localhost:8080/swagger/
2. **Review Architecture**: Read the service architecture documentation
3. **Check Data Model**: Examine database schema documentation
4. **Run Examples**: Try the metrics example service
5. **Write Tests**: Create tests for new features