query-orchestration/CLAUDE.MD

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## General rules  DO NOT VIOLATE
NEVER create mock data or simplified components unless explicitly told to do so
NEVER replace existing complex components with simplified versions - always fix the actual problem or the feature that was requested.
ALWAYS work with the existing codebase - do not create new simplified alternatives
ALWAYS find and fix the root cause of issues instead of creating workarounds
When debugging issues, focus on fixing the existing implementation, not replacing it
When something doesn't work, debug and fix it - don't start over with a simple version
Don’t build things I am not specifically asking for unless its required to build the thing I am asking for.
When asked to make unit tests avoid mocks and use the actual thing that is being tested when reasonable.
For commands that read such as grep or cat there is no need to ask for permission.

## Essential Commands

### Development Setup
- `devbox shell` - Enter development environment (Nix-based)
- `touch .env` - Create environment variables file
- `task fullsuite` - Complete development workflow (generate, build, lint, test)

### Core Development Tasks
- `task generate` - Generate all code (DB queries via SQLC, OpenAPI clients/servers, mocks, docs)
- `task build` - Build Docker image
- `task lint` - Run all linting (Go, YAML, JSON, Docker, OpenAPI)
- `task go:lint -- --fix` - Fix automatically fixable linting issues in Go code
- `task test:functional` - Run full test suite with 80% coverage threshold
- `task precommit` - Pre-commit checks for CI/CD readiness

### Testing Commands
- `task test:unit:short` - Quick unit tests
- `task test:race` - Race condition detection
- `task test:perf` - Performance analysis with slowest test reporting
- `task test:mem` - Memory usage profiling
- `task test:bench` - Benchmark execution

## Architecture Overview

### System Design
This is a **document processing and query orchestration platform** with microservices architecture using:
- **Event-driven processing** via AWS SQS queues
- **RESTful API** (queryAPI on port 8080) for external interactions
- **Runner services** (ports 8081-8090) as queue consumers
- **PostgreSQL** with migration-driven schema management
- **AWS services** (S3, SQS, Textract, Cognito) for cloud functionality

### Document Processing Pipeline
1. **Upload** → S3 storage → storeEventRunner
2. **Init** → Validation/deduplication → docInitRunner
3. **Sync** → Client eligibility → docSyncRunner
4. **Clean** → Content processing → docCleanRunner
5. **Text Extraction** → AWS Textract OCR → docTextRunner
6. **Query Processing** → Custom queries → queryRunner
7. **Export** → Result generation

### Core Entities
- **Client**: Customer configurations with document sets and permissions
- **Collector**: Query collection specifications per client
- **Query**: Data processing blocks with dependency chains
- **Document**: Uploaded files with processing state tracking
- **Export**: Manual result output processes

### Service Architecture
- **queryAPI** (8080): Main REST API with OpenAPI-generated handlers
- **Runners** (8081-8090): Queue-based processing services implementing `Controller` interface
  - Each runner consumes from specific SQS queues
  - Stateless processing with database persistence
  - Prometheus metrics integration

## Development Patterns

### Code Generation
All code generation happens via `task generate`:
- **SQLC**: Database queries from `internal/database/queries/*.sql`
- **OpenAPI**: API clients/servers from `serviceAPIs/*.yaml`
- **Mockery**: Test mocks for interfaces
- **gomarkdoc**: Documentation generation

### Testing Strategy
- **Testcontainers**: Docker-based integration testing for database/queue interactions
- **80% coverage threshold** (60% function coverage) enforced
- **Private test files**: Use `*private_test.go` for internal testing
- **Mock interfaces**: Auto-generated in `mocks/` directory

### Project Structure
Follows Go Standard Project Layout:
- `cmd/`: Application entry points (main.go files)
- `internal/`: Private application code
- `api/`: Generated API server implementations
- `pkg/`: Generated API client libraries
- `mocks/`: Auto-generated test mocks

### Naming Conventions
- **Service**: Commands deployed as APIs (e.g., queryAPI)
- **Runner**: Commands deployed as queue consumers (e.g., docTextRunner)
- **API naming**: `<functionality>API` format
- **Runner naming**: `<functionality>Runner` format

### Environment Configuration
Environment variable hierarchy (first overrides subsequent):
1. `devbox.json` "env" section
2. `devbox.json` "init_hook" exports
3. `.env` file variables

### Authentication & Authorization
- **AWS Cognito**: JWT-based authentication
- **Permit.io**: RBAC integration for fine-grained permissions
- Middleware handles token validation and permission checking

## Key Dependencies

### Core Technologies
- **Go 1.24**: Primary language
- **PostgreSQL**: Database with pgx driver
- **Echo v4**: HTTP framework with middleware
- **Testcontainers**: Integration testing
- **AWS SDK v2**: Cloud service integration

### Code Quality Tools
- **golangci-lint**: Go linting
- **yamllint**: YAML validation
- **Task**: Build automation
- **devbox**: Development environment

## Quick Start for New Features

### Adding New API Endpoint
1. Modify `serviceAPIs/queryAPI.yaml` OpenAPI specification
2. Run `task generate` to regenerate API code
3. Implement handlers in `api/queryAPI/`
4. Add tests following existing patterns

### Adding New Queue Consumer
1. Create `api/<runnerName>/runner.go` implementing `Controller` interface
2. Create `cmd/<runnerName>/main.go` entry point
3. Add queue configuration in `internal/serviceconfig/queue/`
4. Add to docker-compose and deployment configurations

### Database Changes
1. Add migration files in `internal/database/migrations/`
2. Add queries in `internal/database/queries/`
3. Run `task db:generate` to regenerate Go code
4. Update repository layer as needed