query-orchestration/testing.md

Testing Documentation

This document describes all available test targets in the Task build system and their specific purposes.

Primary Test Suites

task test:functional

Default comprehensive test suite with intelligent incremental testing

• Purpose: Main test runner that adapts behavior based on context
• Unique Features:
  • Incremental Mode (default): Only tests packages affected by changes since origin/main
  • Coverage Merging: Maintains cumulative coverage across incremental runs
  • Full Fallback: Automatically runs full suite if no baseline exists or git branch issues
• Coverage: 80% total threshold, 60% function threshold
• Parallelism: Minimal settings to avoid resource contention (GOMAXPROCS=2, -p 1, -parallel 2)
• Use Case: Development workflow, pre-commit checks

task test:functional:full

Complete test suite with coverage reset

• Purpose: Run all tests from scratch, ignoring previous coverage
• Unique Features:
  • Deletes existing coverage file before running
  • Forces full test execution regardless of git changes
• Use Case: Clean slate testing, debugging coverage issues, release validation

task fullsuite:ci

CI/CD optimized complete workflow

• Purpose: Full build, lint, and test pipeline for continuous integration
• Unique Features:
  • Forces INCREMENTAL=false to ensure all tests run
  • Includes generation, building, linting, and testing
  • No shortcuts or incremental optimizations
• Use Case: CI/CD pipelines, release builds, comprehensive validation

Specialized Test Types

task test:race

Race condition detection

• Purpose: Detect data races and concurrency issues
• Unique Features: Uses Go's -race flag to detect race conditions
• Use Case: Debugging concurrency issues, ensuring thread safety

task test:unit:short

Fast unit tests only

• Purpose: Quick feedback loop for development
• Unique Features:
  • Uses -short flag to skip long-running tests
  • Focuses on ./internal/... and ./api/... packages only
  • Generates mocks automatically before running
• Use Case: Rapid development feedback, pre-commit quick checks

task test:permitio

Permit.io integration tests

• Purpose: Run tests that require local Permit.io PDP service
• Unique Features:
  • Uses -tags=permitio to include only tagged integration tests
  • Requires PERMIT_IO_API_KEY environment variable
  • Requires local PDP running on port 7766
  • Provides clear error messages about requirements
• Use Case: Testing authorization integration, local development with full Permit.io stack

task test:bench

Benchmark testing

• Purpose: Performance testing and optimization
• Unique Features:
  • Runs only benchmark functions (-run ^Benchmark)
  • Uses benchstat for statistical analysis of results
  • Outputs performance metrics to out/bench.out
• Use Case: Performance optimization, regression testing for speed

Performance Analysis

task test:perf

Performance profiling and slowest test identification

• Purpose: Identify performance bottlenecks in the test suite
• Unique Features:
  • Reports top 5 slowest packages
  • Reports top 5 slowest individual tests
  • Deep-dives into slowest package for detailed analysis
  • Uses JSON output for precise timing analysis
• Use Case: Test suite optimization, CI performance tuning

task test:mem

Memory usage analysis

• Purpose: Monitor memory consumption across test packages
• Unique Features:
  • Runs tests with memory profiling for each package
  • Ranks packages by memory usage (top 5)
  • Provides total memory consumption summary
  • Individual timeout protection (120s per package)
• Use Case: Memory optimization, resource planning, leak detection

task test:timeline

Visual test execution timeline

• Purpose: Visualize test execution flow and timing
• Unique Features:
  • Uses JSON output piped to visual timeline generator
  • Shows parallel execution patterns
  • Helps identify bottlenecks and optimization opportunities
• Use Case: Understanding test execution patterns, optimization planning

Test Infrastructure

task test:mocks:generate

Mock generation for testing

• Purpose: Generate test mocks from interfaces
• Unique Features:
  • Uses mockery tool to auto-generate mocks
  • Clears existing mocks before regeneration
  • Runs only once per task execution cycle
• Use Case: Maintaining test isolation, dependency mocking

task test:wait

Test environment preparation

• Purpose: Ensure clean test environment before running tests
• Unique Features:
  • Waits for testcontainers cleanup (Ryuk containers)
  • Prunes Docker containers and volumes
  • Prevents resource conflicts between test runs
• Use Case: Test environment hygiene, CI stability

task test:prune

Docker resource cleanup

• Purpose: Clean up Docker resources left by tests
• Unique Features:
  • Force removes unused containers and volumes
  • Prevents disk space issues from accumulated test artifacts
• Use Case: Resource management, CI environment maintenance

Visualization and Analysis

task test:graph

Resource usage visualization during testing

• Purpose: Monitor system resources during test execution
• Unique Features:
  • Uses psrecord to track CPU/memory during test:functional
  • Generates visual plots of resource usage
  • Includes child processes in monitoring
• Use Case: Performance analysis, resource optimization

task fullsuite:graph

Complete build and test resource monitoring

• Purpose: Monitor resources during entire build and test cycle
• Unique Features: Same as test:graph but for the complete fullsuite workflow
• Use Case: End-to-end performance analysis, CI optimization

Test Execution Patterns

Incremental Testing Logic

The test:functional target implements sophisticated incremental testing:

1. Full Mode Triggers:

   • INCREMENTAL=false explicitly set
   • No existing coverage file found
   • Cannot find origin/main branch reference

2. Incremental Mode:

   • Compares against origin/main branch
   • Only tests packages with changed Go files
   • Merges coverage with existing baseline
   • Skips testing if no Go files changed

3. Coverage Merging:

   • Preserves existing coverage for unchanged packages
   • Adds new coverage for changed packages
   • Maintains comprehensive coverage metrics

Parallelism Settings

All test targets use conservative parallelism to avoid resource contention:

• GOMAXPROCS: 2 (fixed minimal value)
• Package Parallelism: 1 (single package at a time)
• Test Parallelism: 2 (minimal test parallelism)

This ensures stable execution in resource-constrained environments like CI.

Usage Recommendations

• Development: Use task test:functional for normal development workflow
• Pre-commit: Use task test:unit:short for quick feedback
• CI/CD: Use task fullsuite:ci for comprehensive validation
• Performance: Use task test:perf and task test:mem for optimization
• Debugging: Use task test:race for concurrency issues
• Clean Slate: Use task test:functional:full when coverage gets corrupted
• Permit.io Integration: Use task test:permitio for testing authorization features with local PDP

Integration with Build Tags

Some tests may use build tags to control execution:

• Tests requiring external services (like Permit.io PDP) may be tagged
• Use -tags=tagname with go test commands to include tagged tests
• CI typically excludes integration tests that require external dependencies