query-orchestration/CLAUDE.MD

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## linting all edits after every change

Always run `task lint` once you think you are done with edits to verify your changes have not caused other issues.
When running `task lint` output should always be clean. 0 errors.  
Warnings about variable not set are ok.  
Lint output output must contain ‘Linting passed, A perfect score! well done!’ or you must fix an issue to resume.

Use `task go:lint -- --fix` to fix go (file not formmatted) types of errors from lint. Always run this before running `task fullsuite:ci`

## Always Active

- Follow `.claude/skills/journaling.md` - log decisions, completed work, and fixes to `./journals/`

## Essential Commands

### Development Setup

- `devbox shell` - Enter development environment (Nix-based) I will start you already in the devbox shell for the project if there is one.
- `touch .env` - Create environment variables file
- `task fullsuite:ci` - Complete development workflow (generate, build, lint, test) This must pass clean or its not working. If its output has the word `FAIL` then investigate what the issue is before calling the change complete.
- when running fullsuite:ci make sure that you have stopped the containers that run from a previous `task compose:refresh' if this applies. (use task compose:down to clean those up first)
- If you need to run the full integration tests that require login, first ask Q to get you authenticated to aws. These tests need to be run when we change something with cognito access for example.

### Core Development Tasks

- `task generate` - Generate all code (DB queries via SQLC, OpenAPI clients/servers, mocks, docs)
- `task build` - Build Docker image
  - `task lint` - Run all linting (Go, YAML, JSON, Docker, OpenAPI)
- `task go:lint -- --fix` - Fix automatically fixable linting issues in Go code
- `task test:functional` - Run full test suite with 80% coverage threshold
- `task precommit` - Pre-commit checks for CI/CD readiness

### Testing Commands

- `task test:unit:short` - Quick unit tests
- `task test:race` - Race condition detection
- `task test:perf` - Performance analysis with slowest test reporting
- `task test:mem` - Memory usage profiling
- `task test:bench` - Benchmark execution
- full suite of tests is `task fullsuit:ci` and it must have this statement at the end of its output 'All coverage checks passed!'
- Never use mocks unless its too difficult to use the real asset and then you must get Q permission first to mock.

## Architecture Overview

### System Design

This is a **document processing platform** with microservices architecture using:

- **Event-driven processing** via AWS SQS queues
- **RESTful API** (queryAPI on port 8080) for external interactions
- **Runner services** (ports 8081-8089) as queue consumers
- **PostgreSQL** with migration-driven schema management
- **AWS services** (S3, SQS, Textract, Cognito) for cloud functionality

### Document Processing Pipeline

1. **Upload** → S3 storage → storeEventRunner
2. **Init** → Validation/deduplication → docInitRunner
3. **Sync** → Client eligibility → docSyncRunner
4. **Clean** → Content processing → docCleanRunner

### Core Entities

- **Client**: Customer configurations with document sets and permissions
- **Collector**: Processing configuration specifications per client
- **Document**: Uploaded files with processing state tracking
- **Folder/Label**: Document organization structures
- **Export**: Manual result output processes

### Service Architecture

- **queryAPI** (8080): Main REST API with OpenAPI-generated handlers
- **Runners** (8081-8090): Queue-based processing services implementing `Controller` interface
  - Each runner consumes from specific SQS queues
  - Stateless processing with database persistence
  - Prometheus metrics integration

## Development Patterns

### Code Generation

All code generation happens via `task generate`:

- **SQLC**: Database queries from `internal/database/queries/*.sql`
- **OpenAPI**: API clients/servers from `serviceAPIs/*.yaml`
- **gomarkdoc**: Documentation generation

### Testing Strategy

- **Testcontainers**: Docker-based integration testing for database/queue interactions
- **80% coverage threshold** (60% function coverage) enforced
- **Private test files**: Use `*private_test.go` for internal testing

### Project Structure

Follows Go Standard Project Layout:

- `cmd/`: Application entry points (main.go files)
- `internal/`: Private application code
- `api/`: Generated API server implementations
- `pkg/`: Generated API client libraries
- `mocks/`: Auto-generated test mocks

### Naming Conventions

- **Service**: Commands deployed as APIs (e.g., queryAPI)
- **Runner**: Commands deployed as queue consumers (e.g., docTextRunner)
- **API naming**: `<functionality>API` format
- **Runner naming**: `<functionality>Runner` format

### Environment Configuration

Environment variable hierarchy (first overrides subsequent):

1. `devbox.json` "env" section
2. `devbox.json` "init_hook" exports
3. `.env` file variables

### Authentication & Authorization

- **AWS Cognito**: JWT-based authentication
- **Permit.io**: RBAC integration for fine-grained permissions
- Middleware handles token validation and permission checking

## Key Dependencies

### Core Technologies

- **Go 1.24**: Primary language
- **PostgreSQL**: Database with pgx driver
- **Echo v4**: HTTP framework with middleware
- **Testcontainers**: Integration testing
- **AWS SDK v2**: Cloud service integration

### Code Quality Tools

- **golangci-lint**: Go linting
- **yamllint**: YAML validation
- **Task**: Build automation
- **devbox**: Development environment

## Quick Start for New Features

### Adding New API Endpoint

1. Modify `serviceAPIs/queryAPI.yaml` OpenAPI specification
2. Run `task generate` to regenerate API code
3. Implement handlers in `api/queryAPI/`
4. Add tests following existing patterns

### Database Changes

1. Add migration files in `internal/database/migrations/`
2. Add queries in `internal/database/queries/`
3. Run `task db:generate` to regenerate Go code
4. Update repository layer as needed

### documenting all changes

Any feature change must not be considered complete until any related docs have been updated. for example did we add a new ENV variable. Then update the docs for showing how to setup and configure the service. New rest endpoint? Make sure all docs are updated including ./docs/ai.generated