# Getting Started Guide ## Prerequisites ### Required Tools - **Devbox**: Provides reproducible development environment - Ubuntu/MacOS: `curl -fsSL https://get.jetify.com/devbox | bash` - NixOS: `devbox` (already available) - **Docker**: Container runtime for services and testing - Install from: https://docs.docker.com/engine/install/ - **Git**: Version control (usually pre-installed) ### System Requirements - **OS**: Linux, macOS, or Windows with WSL2 - **RAM**: 8GB minimum, 16GB recommended - **Disk**: 10GB free space for development environment - **Network**: Internet access for downloading dependencies ### Docker Group Setup (Linux) Ensure your user can run Docker commands without sudo: ```bash sudo groupadd docker sudo usermod -aG docker $USER # Log out and back in for changes to take effect ``` ## Local Development Setup ### 1. Environment Initialization ```bash # Clone repository git clone cd query-orchestration.2 # Create environment file for custom variables touch .env # Enter development environment (installs all tools automatically) devbox shell ``` ### 2. Verify Installation ```bash # Check Go version go version # Should show 1.24.0 # Check Docker docker --version # Check Task runner task --version # Should show 3.39.2+ # List available tasks task --list ``` ### 3. Full System Validation Run the complete test suite to ensure everything works: ```bash task fullsuite ``` This command will: 1. Generate latest API specifications 2. Run linting commands 3. Execute unit tests 4. Run end-to-end integration tests 5. Validate coverage thresholds (80% total, 60% function) ## Development Workflow ### Common Development Tasks #### Running Services Locally ```bash # Start all services with local infrastructure task dev:up # Start specific service task dev:api # Just the API service task dev:runners # Just the queue runners ``` #### API Development ```bash # Generate API code from OpenAPI spec task openapi:generate # Start API server with hot reload task dev:api # View API documentation open http://localhost:8080/swagger/index.html ``` #### Database Operations ```bash # Run database migrations task db:migrate # Reset database (development only) task db:reset # Generate database models from SQL task db:generate ``` #### Testing ```bash # Run unit tests task test:unit # Run integration tests task test:integration # Run tests with coverage task test:coverage # Run specific test package go test ./internal/client/... ``` #### Code Quality ```bash # Run linters task lint # Format code task format # Generate mocks for testing task generate:mocks ``` ### Development Environment Configuration #### Environment Variables Hierarchy 1. **devbox.json "env"** section (highest priority) 2. **devbox.json "init_hook"** exports 3. **.env file** variables (lowest priority) #### Key Development Variables Create a `.env` file with: ```bash # Database (optional - defaults provided) PGUSER=postgres PGPASSWORD=password PGDATABASE=queryorchestration # AWS (uses LocalStack in development) AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test # Authentication (disable for local development) DISABLE_AUTH=true # Logging LOG_LEVEL=DEBUG ``` #### Service Ports (Local Development) - **queryAPI**: 8080 (main API) - **storeEventRunner**: 8081 - **docInitRunner**: 8082 - **docSyncRunner**: 8083 - **docCleanRunner**: 8084 - **docTextRunner**: 8085 - **querySyncRunner**: 8087 - **queryRunner**: 8088 - **clientSyncRunner**: 8089 - **queryVersionSyncRunner**: 8090 - **Prometheus**: 9091 - **PostgreSQL**: 5432 - **LocalStack**: 4566 ## Common Development Workflows ### 1. API Endpoint Development ```bash # 1. Modify OpenAPI specification vim serviceAPIs/queryAPI.yaml # 2. Regenerate API code task openapi:generate # 3. Implement controller logic vim api/queryAPI/controllers.go # 4. Run API server task dev:api # 5. Test endpoint curl -X GET http://localhost:8080/health ``` ### 2. Database Schema Changes ```bash # 1. Create new migration files vim internal/database/migrations/000000000000X_feature.up.sql vim internal/database/migrations/000000000000X_feature.down.sql # 2. Add queries to SQL files vim internal/database/queries/feature.sql # 3. Regenerate database models task db:generate # 4. Run migrations task db:migrate # 5. Update Go service code vim internal/feature/service.go ``` ### 3. New Service Development ```bash # 1. Implement controller interface vim api/newRunner/runner.go # 2. Create main command vim cmd/newRunner/main.go # 3. Add service configuration vim internal/serviceconfig/newrunner/config.go # 4. Update docker-compose vim deployments/compose.local.yaml # 5. Run tests task test:integration ``` ### 4. Adding New Queue ```bash # 1. Add queue configuration vim internal/serviceconfig/queue/newqueue/config.go # 2. Update producer services vim internal/service/producer.go # 3. Update consumer services vim api/consumerRunner/runner.go # 4. Update environment variables vim deployments/compose.local.yaml ``` ## Testing Guidelines ### Unit Testing - **Location**: `*_test.go` files alongside source code - **Pattern**: Table-driven tests preferred - **Mocking**: Use generated mocks for external dependencies - **Coverage**: Minimum 60% function coverage required Example unit test: ```go func TestService_CreateClient(t *testing.T) { tests := []struct { name string input CreateClientRequest expected Client wantErr bool }{ { name: "valid client creation", input: CreateClientRequest{ ClientID: "test-client", Name: "Test Client", }, expected: Client{ ClientID: "test-client", Name: "Test Client", }, wantErr: false, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { // Test implementation }) } } ``` ### Integration Testing - **Location**: `test/` directory and `*_test.go` files - **Infrastructure**: Uses testcontainers for real dependencies - **Isolation**: Each test gets clean environment - **Coverage**: Minimum 80% total coverage required ### End-to-End Testing - **Environment**: Full service deployment via Docker Compose - **Scope**: Complete workflows from API to database - **Authentication**: Uses test Cognito setup or disabled auth ## Debugging and Troubleshooting ### Common Issues #### Docker Permission Errors ```bash # Fix Docker permissions (Linux) sudo groupadd docker sudo usermod -aG docker $USER # Log out and log back in ``` #### Database Connection Issues ```bash # Check PostgreSQL status task db:status # Reset database task db:reset # Check connection manually psql -h localhost -p 5432 -U postgres -d queryorchestration ``` #### Service Startup Issues ```bash # Check service logs docker-compose -f deployments/compose.local.yaml logs [service-name] # Check environment variables task env:check # Verify ports aren't in use lsof -i :8080 ``` #### Code Generation Issues ```bash # Clean generated files task clean # Regenerate everything task generate:all # Check for syntax errors in source files task lint ``` ### Development Tools #### Debugging - **Delve**: Go debugger integrated with devbox - **IDE Integration**: VS Code and GoLand support provided - **Logging**: Structured logging with slog at DEBUG level #### Monitoring - **Prometheus**: Metrics at http://localhost:9091 - **Swagger UI**: API docs at http://localhost:8080/swagger/ - **Health Checks**: http://localhost:8080/health #### Database Tools - **psql**: Command-line PostgreSQL client - **Database browser**: Connect to localhost:5432 - **Migration status**: `task db:status` ### Performance Profiling ```bash # CPU profiling go test -cpuprofile=cpu.prof -bench=. # Memory profiling go test -memprofile=mem.prof -bench=. # Race detection go test -race ./... ``` ## Next Steps After completing the setup: 1. **Explore the API**: Visit http://localhost:8080/swagger/ 2. **Review Architecture**: Read the service architecture documentation 3. **Check Data Model**: Examine database schema documentation 4. **Run Examples**: Try the metrics example service 5. **Write Tests**: Create tests for new features