query-orchestration/CLAUDE.MD

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

linting all edits after every change

Always run task lint once you think you are done with edits to verify your changes have not caused other issues. When running task lint output should always be clean. 0 errors.
Warnings about variable not set are ok.
Lint output output must contain ‘Linting passed, A perfect score! well done!’ or you must fix an issue to resume.

Use task go:lint -- --fix to fix go (file not formmatted) types of errors from lint. Always run this before running task fullsuite:ci

Always Active

• Follow .claude/skills/journaling.md - log decisions, completed work, and fixes to ./journals/

Essential Commands

Development Setup

• devbox shell - Enter development environment (Nix-based) I will start you already in the devbox shell for the project if there is one.
• touch .env - Create environment variables file
• task fullsuite:ci - Complete development workflow (generate, build, lint, test) This must pass clean or its not working. If its output has the word FAIL then investigate what the issue is before calling the change complete.
• when running fullsuite:ci make sure that you have stopped the containers that run from a previous `task compose:refresh' if this applies. (use task compose:down to clean those up first)
• If you need to run the full integration tests that require login, first ask Q to get you authenticated to aws. These tests need to be run when we change something with cognito access for example.

Core Development Tasks

• task generate - Generate all code (DB queries via SQLC, OpenAPI clients/servers, mocks, docs)
• task build - Build Docker image
  • task lint - Run all linting (Go, YAML, JSON, Docker, OpenAPI)
• task go:lint -- --fix - Fix automatically fixable linting issues in Go code
• task test:functional - Run full test suite with 80% coverage threshold
• task precommit - Pre-commit checks for CI/CD readiness

Testing Commands

• task test:unit:short - Quick unit tests
• task test:race - Race condition detection
• task test:perf - Performance analysis with slowest test reporting
• task test:mem - Memory usage profiling
• task test:bench - Benchmark execution
• full suite of tests is task fullsuit:ci and it must have this statement at the end of its output 'All coverage checks passed!'
• Never use mocks unless its too difficult to use the real asset and then you must get Q permission first to mock.

Architecture Overview

System Design

This is a document processing platform with microservices architecture using:

• Event-driven processing via AWS SQS queues
• RESTful API (queryAPI on port 8080) for external interactions
• Runner services (ports 8081-8089) as queue consumers
• PostgreSQL with migration-driven schema management
• AWS services (S3, SQS, Textract, Cognito) for cloud functionality

Document Processing Pipeline

1. Upload → S3 storage → storeEventRunner
2. Init → Validation/deduplication → docInitRunner
3. Sync → Client eligibility → docSyncRunner
4. Clean → Content processing → docCleanRunner

Core Entities

• Client: Customer configurations with document sets and permissions
• Collector: Processing configuration specifications per client
• Document: Uploaded files with processing state tracking
• Folder/Label: Document organization structures
• Export: Manual result output processes

Service Architecture

• queryAPI (8080): Main REST API with OpenAPI-generated handlers
• Runners (8081-8090): Queue-based processing services implementing Controller interface
  • Each runner consumes from specific SQS queues
  • Stateless processing with database persistence
  • Prometheus metrics integration

Development Patterns

Code Generation

All code generation happens via task generate:

• SQLC: Database queries from internal/database/queries/*.sql
• OpenAPI: API clients/servers from serviceAPIs/*.yaml
• gomarkdoc: Documentation generation

Testing Strategy

• Testcontainers: Docker-based integration testing for database/queue interactions
• 80% coverage threshold (60% function coverage) enforced
• Private test files: Use *private_test.go for internal testing

Project Structure

Follows Go Standard Project Layout:

• cmd/: Application entry points (main.go files)
• internal/: Private application code
• api/: Generated API server implementations
• pkg/: Generated API client libraries
• mocks/: Auto-generated test mocks

Naming Conventions

• Service: Commands deployed as APIs (e.g., queryAPI)
• Runner: Commands deployed as queue consumers (e.g., docTextRunner)
• API naming: <functionality>API format
• Runner naming: <functionality>Runner format

Environment Configuration

Environment variable hierarchy (first overrides subsequent):

1. devbox.json "env" section
2. devbox.json "init_hook" exports
3. .env file variables

Authentication & Authorization

• AWS Cognito: JWT-based authentication
• Permit.io: RBAC integration for fine-grained permissions
• Middleware handles token validation and permission checking

Key Dependencies

Core Technologies

• Go 1.24: Primary language
• PostgreSQL: Database with pgx driver
• Echo v4: HTTP framework with middleware
• Testcontainers: Integration testing
• AWS SDK v2: Cloud service integration

Code Quality Tools

• golangci-lint: Go linting
• yamllint: YAML validation
• Task: Build automation
• devbox: Development environment

Quick Start for New Features

Adding New API Endpoint

1. Modify serviceAPIs/queryAPI.yaml OpenAPI specification
2. Run task generate to regenerate API code
3. Implement handlers in api/queryAPI/
4. Add tests following existing patterns

Database Changes

1. Add migration files in internal/database/migrations/
2. Add queries in internal/database/queries/
3. Run task db:generate to regenerate Go code
4. Update repository layer as needed

documenting all changes

Any feature change must not be considered complete until any related docs have been updated. for example did we add a new ENV variable. Then update the docs for showing how to setup and configure the service. New rest endpoint? Make sure all docs are updated including ./docs/ai.generated