query-orchestration/README.md

# Query Orchestration

AWS ambient version
This repository contains the DoczyAI code.

Using the following project as a baseline: `https://github.com/golang-standards/project-layout`.

## Documentation

- [Queue Pipeline Flow](./queue_flow.md) - Detailed documentation of the SQS queue-based processing pipeline
- [Deployment Environment Variables](./deployment_env_variables.md) - Configuration guide for environment variables across all services

## Installation and Usage

- Install devbox: `https://www.jetify.com/docs/devbox/installing_devbox`
  - Ubuntu/MacOS: `curl -fsSL https://get.jetify.com/devbox | bash`
  - NixOS: `devbox`
- Install docker: `https://docs.docker.com/engine/install/`
- (IF NECESSARY) Ensure user in docker group and docker group is in sudo group

  ```sh
    sudo groupadd docker
    sudo usermod -aG docker $USER
  ```

- Run `touch .env` to create `.env` file for custom environment variables
- Run `devbox shell` to enter the environment
- Run `task fullsuite:ci` to run all tests that ensure the current state

To find new packages: `https://search.nixos.org/packages`

For regular usage, please use `task` scripts in order run the most appropriate configuration.

## Available Task Commands

### Core Development Workflow

- **`task fullsuite:ci`** - Complete CI workflow: generate, build, lint, test with coverage
- **`task fullsuite`** - Full development suite (similar to CI but for local development)
- **`task fullsuite:graph`** - Generate dependency graph visualization
- **`task precommit`** - Pre-commit checks for CI/CD readiness

### Code Generation

- **`task generate`** - Generate all code (DB queries via SQLC, OpenAPI clients/servers, mocks, docs)
- **`task go:generate`** - Generate Go-specific code
- **`task openapi:generate`** - Generate OpenAPI clients and servers from specs
- **`task db:generate`** - Generate database queries from SQL files
- **`task docs:generate`** - Generate project documentation
- **`task docs:ai:generate`** - Generate full system docs using AI ($$)
- **`task test:mocks:generate`** - Generate test mocks

### Build Commands

- **`task build`** - Build the project
- **`task docker:build`** - Build Docker images
- **`task docker:build:debug`** - Build Docker images with debug support

### Docker Compose Management

#### Build Commands

- **`task compose:build`** - Build Docker images using local compose file
- **`task compose:build:test`** - Build Docker images using test compose file
- **`task compose:build:generate`** - Build Docker images using generate compose file

#### Start/Up Commands

- **`task compose:up`** - Full local development startup with AWS resource initialization
- **`task compose:up:test`** - Start test environment containers
- **`task compose:up:generate`** - Start containers for code generation
- **`task compose:up:aws`** - Start AWS-specific environment

#### Stop/Down Commands

- **`task compose:down`** - Stop and remove local containers
- **`task compose:down:test`** - Stop and remove test containers
- **`task compose:down:aws`** - Stop and remove AWS containers

#### Cleanup Commands

- **`task compose:clean`** - Stop containers and remove volumes (local)
- **`task compose:clean:test`** - Stop containers and remove volumes (test)
- **`task compose:clean:aws`** - Stop containers and remove volumes (AWS)

#### Other Compose Commands

- **`task compose:refresh`** - Full restart: rebuild everything and start fresh
- **`task compose:rebuild`** - Force complete rebuild of all containers (use when build cache causes issues)
- **`task compose:lint`** - Validate syntax of all compose files

### Testing Commands

- **`task test:functional`** - Run functional tests with 80% coverage threshold
- **`task test:functional:full`** - Run all tests and reset coverage tracking
- **`task test:unit:short`** - Quick unit tests
- **`task test:race`** - Race condition detection tests
- **`task test:perf`** - Performance analysis with slowest test reporting
- **`task test:mem`** - Memory usage profiling tests
- **`task test:bench`** - Benchmark execution
- **`task test:permitio`** - Run Permit.io integration tests (requires local PDP)
- **`task test:graph`** - Generate test dependency graphs
- **`task test:timeline`** - Show test execution timeline
- **`task test:prune`** - Clean up test artifacts
- **`task test:wait`** - Wait for test dependencies

### Linting Commands

- **`task lint`** - Run all linting (Go, YAML, JSON, Docker, OpenAPI)
- **`task go:lint`** - Run Go-specific linting
- **`task go:lint -- --fix`** - Fix automatically fixable Go linting issues
- **`task docker:lint`** - Lint Docker files
- **`task openapi:lint`** - Lint OpenAPI specifications
- **`task yaml:lint`** - Lint YAML files
- **`task json:lint`** - Lint JSON files
- **`task db:lint`** - Lint database-related files

### Database Commands

- **`task db:mig:create`** - Create new database migration
- **`task db:generate`** - Generate Go code from SQL queries

### AWS Commands

- **`task aws:login`** - Login to AWS services
- **`task aws:textract:credentials`** - Set up AWS Textract credentials

### Utility Commands

- **`task deps:tidy`** - Tidy Go module dependencies

## Environment Variables

### Configuration Hierarchy

When using environment variables, there is a hierarchy that will be respected. (First will override the next)

1. Within the devbox.json file, a variable added to "env"
2. Within the devbox.json file, a variable exported in "init_hook"
3. A variable added to .env

### Required Environment Variables

The following environment variables are required for the application to function properly:

#### Core Application

```bash
LOG_LEVEL=INFO                          # Logging level: DEBUG|INFO|WARN|ERROR
```

#### Database (Required)

```bash
PGHOST=localhost                        # Database host
PGPORT=5432                            # Database port
PGDATABASE=query_orchestration         # Database name
PGUSER=postgres                        # Database username
PGPASSWORD=your_password               # Database password
```

#### AWS Configuration (Required)

```bash
AWS_REGION=us-east-1                   # AWS region
AWS_ACCESS_KEY_ID=your_access_key      # AWS access key ID
AWS_SECRET_ACCESS_KEY=your_secret_key  # AWS secret access key
AWS_SESSION_TOKEN=your_session_token   # AWS session token (if using temporary credentials)
```

#### S3 Storage (Required)

```bash
BUCKET=your-s3-bucket-name             # S3 bucket for document storage
```

#### SQS Queue URLs (All Required)

```bash
QUEUE_URL=https://sqs.region.amazonaws.com/account/queue-name          # Main query processing queue
DOCUMENT_INIT_URL=https://sqs.region.amazonaws.com/account/doc-init    # Document initialization queue
DOCUMENT_SYNC_URL=https://sqs.region.amazonaws.com/account/doc-sync    # Document synchronization queue
DOCUMENT_CLEAN_URL=https://sqs.region.amazonaws.com/account/doc-clean  # Document cleaning queue
DOCUMENT_TEXT_URL=https://sqs.region.amazonaws.com/account/doc-text    # Document text extraction queue
QUERY_SYNC_URL=https://sqs.region.amazonaws.com/account/query-sync     # Query synchronization queue
STORE_EVENT_URL=https://sqs.region.amazonaws.com/account/store-event   # Store event queue
CLIENT_SYNC_URL=https://sqs.region.amazonaws.com/account/client-sync   # Client synchronization queue
QUERY_VERSION_SYNC_URL=https://sqs.region.amazonaws.com/account/query-version-sync  # Query version sync queue
```

#### Authentication (Required for Production)

```bash
COGNITO_USER_POOL_ID=us-east-1_xxxxxxx # AWS Cognito User Pool ID
COGNITO_CLIENT_ID=your_client_id       # AWS Cognito App Client ID
COGNITO_CLIENT_SECRET=your_secret      # AWS Cognito App Client Secret
COGNITO_DOMAIN=your-cognito-domain     # AWS Cognito domain name
```

#### Authorization (Required for RBAC)

```bash
PERMIT_IO_API_KEY=permit_key_xxxxxxx     # Permit.io API key for RBAC
PERMIT_IO_PROJECT_ID=default             # Permit.io project ID (default: "default")
PERMIT_IO_ENV_ID=dev                     # Permit.io environment ID (e.g., dev, staging, production)
PERMIT_IO_TENANT=default                 # Permit.io tenant identifier (default: "default")
PERMIT_IO_PDP_URL=http://localhost:7766  # Permit.io Policy Decision Point URL - can use cloud version optionally
```

### Optional Environment Variables

#### Development & Testing

```bash
DEBUG=true                             # Enable debug mode (development only)
DISABLE_AUTH=true                      # Bypass authentication (development/testing only)
DB_NOSSL=true                         # Disable SSL for database connections (development only)
COGNITO_SUPPRESS_EMAILS=true          # Suppress welcome emails to avoid AWS 50/day limit (testing only)
```

#### AWS LocalStack (Development)

```bash
AWS_ENDPOINT_URL=http://localhost:4566 # Override AWS endpoints for LocalStack
AWS_S3_USE_PATH_STYLE=true            # Use path-style S3 URLs for LocalStack
```

#### Server Configuration

```bash
PORT=8080                             # HTTP server port (default: 8080)
HTTP_HOST=0.0.0.0                     # HTTP server host (default: 0.0.0.0)
```

#### Observability

```bash
ENABLE_OTEL=false                     # Enable OpenTelemetry (default: false)
```

### Example .env File for Development

```bash
# Development overrides
LOG_LEVEL=DEBUG
DEBUG=true
DISABLE_AUTH=true
DB_NOSSL=true

# Local services
PGHOST=localhost
PGPORT=5432
PGDATABASE=query_orchestration
PGUSER=postgres
PGPASSWORD=pass

# LocalStack for AWS services
AWS_ENDPOINT_URL=http://localhost:4566
AWS_S3_USE_PATH_STYLE=true
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=test
AWS_SECRET_ACCESS_KEY=test

# S3 bucket
BUCKET=test-bucket

# LocalStack SQS queues
QUEUE_URL=http://localhost:4566/queue/us-east-1/000000000000/query_runner
DOCUMENT_INIT_URL=http://localhost:4566/queue/us-east-1/000000000000/document_init
DOCUMENT_SYNC_URL=http://localhost:4566/queue/us-east-1/000000000000/document_sync
DOCUMENT_CLEAN_URL=http://localhost:4566/queue/us-east-1/000000000000/document_clean
DOCUMENT_TEXT_URL=http://localhost:4566/queue/us-east-1/000000000000/document_text
QUERY_SYNC_URL=http://localhost:4566/queue/us-east-1/000000000000/query_sync
STORE_EVENT_URL=http://localhost:4566/queue/us-east-1/000000000000/store_event
CLIENT_SYNC_URL=http://localhost:4566/queue/us-east-1/000000000000/client_sync
QUERY_VERSION_SYNC_URL=http://localhost:4566/queue/us-east-1/000000000000/query_version_sync

# Development auth (optional)
PERMIT_IO_API_KEY=test-key
PERMIT_IO_PROJECT_ID=default
PERMIT_IO_ENV_ID=dev
PERMIT_IO_TENANT=default
PERMIT_IO_PDP_URL=http://localhost:7766
```

## Testing

For testing endtoends such as queries against a database, or interactions with an SQS queue, this repository employs the use of docker testcontainers.

`https://testcontainers.com/`

This enables tests to be fully self-contained, thus allowing allowing tests to be reliable and replicable.

### Execution

The simplest way of ensuring the validity of the full project is by running `task fullsuite` when in the devbox shell.

This will:

1. Generate the latest version of the specs.
2. Run the linting commands.
3. Run the unit tests.
4. Run the endtoend tests.

## Naming Conventions

**Service** - This term is used internally in order to refer to commands that get deployed as APIs.
**Runner** - This term is used internally in order to refer to commands that get deployed as consumers of a queue.

### Creating a new API

1. Add a file named `<api_name>.yml`. The api name is important, as it will be used as a reference throughout the codebase. The api name must be in the format `<functionality>API`. E.g. queryAPI, ClientAPI
2. Run `task openapi:generate`. This will generate the following:
   a. A location to implement the server-side functions in `./api/<api_name>/`, as well as generated functions, models and swagger docs.
   b. A location with the client-side code in `./pkg/<api_name>/`
3. Implement the controllers in `./api/<api_name>/`
4. Create a command in `./cmd/<api_name>/main.go` which creates a new instance from `./internal/api`

### Creating a new Runner

1. Implement the controller in `./api/<runner_name>/` which implements the `Controller` interface in `./internal/queue`. The runner name must be in the format `<functionality>Runner`. E.g. queryRunner, csvExportRunner
2. Create a command in `./cmd/<runner_name>/main.go` which creates a new instance from `./internal/queue`

## Debugging in Containers

This project supports debugging Go applications running inside Docker containers using [Delve](https://github.com/go-delve/delve), the Go debugger.

### Building Debug Images

The project includes two build tasks for different use cases:

- `task docker:build` - Build the regular production image
- `task docker:build:debug` - Build the debug image with Delve

The debug image is built using `build/Dockerfile.debug`, which:

- Installs Delve debugger with static linking for Alpine compatibility
- Builds Go binaries with debug symbols (`-gcflags="all=-N -l"`)
- Exposes port 2345 for debugger connections

### Container Configuration for Debugging

To enable debugging for a service in `deployments/compose.local.yaml`, modify the service configuration:

```yaml
query_api:
  image: queryorchestration:debug # Use debug image instead of latest
  command:
    [
      "/dlv",
      "exec",
      "./queryAPI",
      "--listen=:2345",
      "--headless",
      "--api-version=2",
      "--accept-multiclient",
      "--continue",
    ]
  ports:
    - "8080:8080" # Application port
    - "2345:2345" # Debugger port
  expose:
    - 8080
    - 2345
```

Key changes from production configuration:

1. **Image**: Use `queryorchestration:debug` instead of `queryorchestration:latest`
2. **Command**: Replace direct binary execution with Delve wrapper
3. **Ports**: Expose port 2345 for debugger connections
4. **Delve flags**:
   - `--listen=:2345`: Listen on port 2345 for debugger connections
   - `--headless`: Run without terminal interface
   - `--accept-multiclient`: Allow multiple debugger connections
   - `--continue`: Start the application immediately (don't wait for debugger)

### Connecting a Debugger

#### GoLand/IntelliJ IDEA

1. Go to **Run → Edit Configurations...**
2. Click **"+"** → **"Go Remote"**
3. Configure:
   - **Name**: `Docker Debug`
   - **Host**: `localhost`
   - **Port**: `2345`
4. Click **Apply** and **OK**
5. Select the configuration and click **Debug** (Shift+F9)
6. Set breakpoints in your code and make HTTP requests to trigger them

#### VS Code

1. Add to `.vscode/launch.json`:

```json
{
  "name": "Connect to Docker",
  "type": "go",
  "request": "attach",
  "mode": "remote",
  "remotePath": "/app",
  "port": 2345,
  "host": "localhost"
}
```

2. Set breakpoints and start debugging

The debugger will connect to the running container and pause execution when breakpoints are hit, allowing you to inspect variables, step through code, and debug issues in the containerized environment.

## S3 Storage Format

The system uses a structured naming convention for S3 objects that embeds metadata for parsing, recovery, and organization.

### Object Key Structure

S3 object keys follow this format:

```
{client_id}/{location}/{date}/{part}/{filename}
```

#### Example

```
test_client_1757026403/import/20250904/0/2025-09-04T225324Z~test_client_1757026403~import~56df8aa9-873c-48ea-a0e4-bc4f6b8ad69e~019916ef-46ab-73f7-b97b-5709df5c3985
```

### Components

#### Directory Path (Prefix)

- **Client ID**: Unique identifier for the client (e.g., `test_client_1757026403`)
- **Location**: Processing stage - `import`, `text`, or `export`
- **Date**: Date in `YYYYMMDD` format (e.g., `20250904`)
- **Part**: Partition number for storage distribution (e.g., `0`)

#### Filename

The filename uses tilde (`~`) as a delimiter and contains:

- **Timestamp**: Full ISO timestamp (e.g., `2025-09-04T225324Z`)
- **Client ID**: Repeated for self-contained metadata
- **Location**: Repeated for self-contained metadata
- **Entity ID**: UUID for the specific document (e.g., `56df8aa9-873c-48ea-a0e4-bc4f6b8ad69e`)
- **Batch ID**: Optional UUID for batch uploads (e.g., `019916ef-46ab-73f7-b97b-5709df5c3985`)
- **File Extension**: Optional file type extension (e.g., `.pdf`)

### Location Types

- **import**: Initial document upload location
- **text**: Text extraction results storage
- **export**: Export results storage

### Partitioning Strategy

The part number (`0-n`) is used to distribute files across multiple directories, preventing performance issues with too many objects in a single S3 prefix. Export location does not use partitioning.

### Metadata Encoding

The filename contains all necessary metadata for document recovery and processing:

- Self-contained recovery capability from filename alone
- Support for both path-based and filename-based queries
- Human-readable format for debugging and troubleshooting