query-orchestration/testing.md

# Testing Documentation

This document describes all available test targets in the Task build system and their specific purposes.

## Primary Test Suites

### `task test:functional`
**Default comprehensive test suite with intelligent incremental testing**
- **Purpose**: Main test runner that adapts behavior based on context
- **Unique Features**:
  - **Incremental Mode** (default): Only tests packages affected by changes since `origin/main`
  - **Coverage Merging**: Maintains cumulative coverage across incremental runs
  - **Full Fallback**: Automatically runs full suite if no baseline exists or git branch issues
- **Coverage**: 80% total threshold, 60% function threshold
- **Parallelism**: Minimal settings to avoid resource contention (GOMAXPROCS=2, -p 1, -parallel 2)
- **Use Case**: Development workflow, pre-commit checks

### `task test:functional:full`
**Complete test suite with coverage reset**
- **Purpose**: Run all tests from scratch, ignoring previous coverage
- **Unique Features**: 
  - Deletes existing coverage file before running
  - Forces full test execution regardless of git changes
- **Use Case**: Clean slate testing, debugging coverage issues, release validation

### `task fullsuite:ci`
**CI/CD optimized complete workflow**
- **Purpose**: Full build, lint, and test pipeline for continuous integration
- **Unique Features**:
  - Forces `INCREMENTAL=false` to ensure all tests run
  - Includes generation, building, linting, and testing
  - No shortcuts or incremental optimizations
- **Use Case**: CI/CD pipelines, release builds, comprehensive validation

## Specialized Test Types

### `task test:race`
**Race condition detection**
- **Purpose**: Detect data races and concurrency issues
- **Unique Features**: Uses Go's `-race` flag to detect race conditions
- **Use Case**: Debugging concurrency issues, ensuring thread safety

### `task test:unit:short`
**Fast unit tests only**
- **Purpose**: Quick feedback loop for development
- **Unique Features**:
  - Uses `-short` flag to skip long-running tests
  - Focuses on `./internal/...` and `./api/...` packages only
  - Generates mocks automatically before running
- **Use Case**: Rapid development feedback, pre-commit quick checks

### `task test:permitio`
**Permit.io integration tests**
- **Purpose**: Run tests that require local Permit.io PDP service
- **Unique Features**:
  - Uses `-tags=permitio` to include only tagged integration tests
  - Requires `PERMIT_IO_API_KEY` environment variable
  - Requires local PDP running on port 7766
  - Provides clear error messages about requirements
- **Use Case**: Testing authorization integration, local development with full Permit.io stack

### `task test:bench`
**Benchmark testing**
- **Purpose**: Performance testing and optimization
- **Unique Features**:
  - Runs only benchmark functions (`-run ^Benchmark`)
  - Uses `benchstat` for statistical analysis of results
  - Outputs performance metrics to `out/bench.out`
- **Use Case**: Performance optimization, regression testing for speed

## Performance Analysis

### `task test:perf`
**Performance profiling and slowest test identification**
- **Purpose**: Identify performance bottlenecks in the test suite
- **Unique Features**:
  - Reports top 5 slowest packages
  - Reports top 5 slowest individual tests
  - Deep-dives into slowest package for detailed analysis
  - Uses JSON output for precise timing analysis
- **Use Case**: Test suite optimization, CI performance tuning

### `task test:mem`
**Memory usage analysis**
- **Purpose**: Monitor memory consumption across test packages
- **Unique Features**:
  - Runs tests with memory profiling for each package
  - Ranks packages by memory usage (top 5)
  - Provides total memory consumption summary
  - Individual timeout protection (120s per package)
- **Use Case**: Memory optimization, resource planning, leak detection

### `task test:timeline`
**Visual test execution timeline**
- **Purpose**: Visualize test execution flow and timing
- **Unique Features**:
  - Uses JSON output piped to visual timeline generator
  - Shows parallel execution patterns
  - Helps identify bottlenecks and optimization opportunities
- **Use Case**: Understanding test execution patterns, optimization planning

## Test Infrastructure

### `task test:mocks:generate`
**Mock generation for testing**
- **Purpose**: Generate test mocks from interfaces
- **Unique Features**:
  - Uses mockery tool to auto-generate mocks
  - Clears existing mocks before regeneration
  - Runs only once per task execution cycle
- **Use Case**: Maintaining test isolation, dependency mocking

### `task test:wait`
**Test environment preparation**
- **Purpose**: Ensure clean test environment before running tests
- **Unique Features**:
  - Waits for testcontainers cleanup (Ryuk containers)
  - Prunes Docker containers and volumes
  - Prevents resource conflicts between test runs
- **Use Case**: Test environment hygiene, CI stability

### `task test:prune`
**Docker resource cleanup**
- **Purpose**: Clean up Docker resources left by tests
- **Unique Features**:
  - Force removes unused containers and volumes
  - Prevents disk space issues from accumulated test artifacts
- **Use Case**: Resource management, CI environment maintenance

## Visualization and Analysis

### `task test:graph`
**Resource usage visualization during testing**
- **Purpose**: Monitor system resources during test execution
- **Unique Features**:
  - Uses `psrecord` to track CPU/memory during `test:functional`
  - Generates visual plots of resource usage
  - Includes child processes in monitoring
- **Use Case**: Performance analysis, resource optimization

### `task fullsuite:graph`
**Complete build and test resource monitoring**
- **Purpose**: Monitor resources during entire build and test cycle
- **Unique Features**: Same as `test:graph` but for the complete `fullsuite` workflow
- **Use Case**: End-to-end performance analysis, CI optimization

## Test Execution Patterns

### Incremental Testing Logic
The `test:functional` target implements sophisticated incremental testing:

1. **Full Mode Triggers**:
   - `INCREMENTAL=false` explicitly set
   - No existing coverage file found
   - Cannot find `origin/main` branch reference

2. **Incremental Mode**:
   - Compares against `origin/main` branch
   - Only tests packages with changed Go files
   - Merges coverage with existing baseline
   - Skips testing if no Go files changed

3. **Coverage Merging**:
   - Preserves existing coverage for unchanged packages
   - Adds new coverage for changed packages
   - Maintains comprehensive coverage metrics

### Parallelism Settings
All test targets use conservative parallelism to avoid resource contention:
- **GOMAXPROCS**: 2 (fixed minimal value)
- **Package Parallelism**: 1 (single package at a time)
- **Test Parallelism**: 2 (minimal test parallelism)

This ensures stable execution in resource-constrained environments like CI.

## Usage Recommendations

- **Development**: Use `task test:functional` for normal development workflow
- **Pre-commit**: Use `task test:unit:short` for quick feedback
- **CI/CD**: Use `task fullsuite:ci` for comprehensive validation
- **Performance**: Use `task test:perf` and `task test:mem` for optimization
- **Debugging**: Use `task test:race` for concurrency issues
- **Clean Slate**: Use `task test:functional:full` when coverage gets corrupted
- **Permit.io Integration**: Use `task test:permitio` for testing authorization features with local PDP

## Integration with Build Tags

Some tests may use build tags to control execution:
- Tests requiring external services (like Permit.io PDP) may be tagged
- Use `-tags=tagname` with `go test` commands to include tagged tests
- CI typically excludes integration tests that require external dependencies