query-orchestration/README.md

# Query Orchestration

This repository contains the DoczyAI code.

Using the following project as a baseline: `https://github.com/golang-standards/project-layout`.

## Installation and Usage

- Install devbox: `https://www.jetify.com/docs/devbox/installing_devbox`
  - Ubuntu/MacOS: `curl -fsSL https://get.jetify.com/devbox | bash`
  - NixOS: `devbox`
- Install docker: `https://docs.docker.com/engine/install/`
- (IF NECESSARY) Ensure user in docker group and docker group is in sudo group

  ```sh
    sudo groupadd docker
    sudo usermod -aG docker $USER
  ```

- Run `touch .env` to create `.env` file for custom environment variables
- Run `devbox shell` to enter the environment
- Run `task fullsuite:ci` to run all tests that ensure the current state

To find new packages: `https://search.nixos.org/packages`

For regular usage, please use `task` scripts in order run the most appropriate configuration.

## Environment Variables

When using environment variables, there is a hiearchy that will be respected. (First will override the next)

1. Within the devbox.json file, a variable added to "env"
2. Within the devbox.json file, a variable exported in "init_hook"
3. A variable added to .env

## Testing

For testing endtoends such as queries against a database, or interactions with an SQS queue, this repository employs the use of docker testcontainers.

`https://testcontainers.com/`

This enables tests to be fully self-contained, thus allowing allowing tests to be reliable and replicable.

### Execution

The simplest way of ensuring the validity of the full project is by running `task fullsuite` when in the devbox shell.

This will:

1. Generate the latest version of the specs.
2. Run the linting commands.
3. Run the unit tests.
4. Run the endtoend tests.

## Naming Conventions

**Service** - This term is used internally in order to refer to commands that get deployed as APIs.
**Runner** - This term is used internally in order to refer to commands that get deployed as consumers of a queue.

### Creating a new API

1. Add a file named `<api_name>.yml`. The api name is important, as it will be used as a reference throughout the codebase. The api name must be in the format `<functionality>API`. E.g. queryAPI, ClientAPI
2. Run `task openapi:generate`. This will generate the following:
  a. A location to implement the server-side functions in `./api/<api_name>/`, as well as generated functions, models and swagger docs.
  b. A location with the client-side code in `./pkg/<api_name>/`
3. Implement the controllers in `./api/<api_name>/`
4. Create a command in `./cmd/<api_name>/main.go` which creates a new instance from `./internal/api`

### Creating a new Runner

1. Implement the controller in `./api/<runner_name>/` which implements the `Controller` interface in `./internal/queue`. The runner name must be in the format `<functionality>Runner`. E.g. queryRunner, csvExportRunner
2. Create a command in `./cmd/<runner_name>/main.go` which creates a new instance from `./internal/queue`

## Debugging in Containers

This project supports debugging Go applications running inside Docker containers using [Delve](https://github.com/go-delve/delve), the Go debugger.

### Building Debug Images

The project includes two build tasks for different use cases:

- `task docker:build` - Build the regular production image
- `task docker:build:debug` - Build the debug image with Delve

The debug image is built using `build/Dockerfile.debug`, which:
- Installs Delve debugger with static linking for Alpine compatibility
- Builds Go binaries with debug symbols (`-gcflags="all=-N -l"`)
- Exposes port 2345 for debugger connections

### Container Configuration for Debugging

To enable debugging for a service in `deployments/compose.local.yaml`, modify the service configuration:

```yaml
query_api:
  image: queryorchestration:debug  # Use debug image instead of latest
  command: ["/dlv", "exec", "./queryAPI", "--listen=:2345", "--headless", "--api-version=2", "--accept-multiclient", "--continue"]
  ports:
    - "8080:8080"  # Application port
    - "2345:2345"  # Debugger port
  expose:
    - 8080
    - 2345
```

Key changes from production configuration:
1. **Image**: Use `queryorchestration:debug` instead of `queryorchestration:latest`
2. **Command**: Replace direct binary execution with Delve wrapper
3. **Ports**: Expose port 2345 for debugger connections
4. **Delve flags**:
   - `--listen=:2345`: Listen on port 2345 for debugger connections
   - `--headless`: Run without terminal interface
   - `--accept-multiclient`: Allow multiple debugger connections
   - `--continue`: Start the application immediately (don't wait for debugger)

### Connecting a Debugger

#### GoLand/IntelliJ IDEA
1. Go to **Run → Edit Configurations...**
2. Click **"+"** → **"Go Remote"**
3. Configure:
   - **Name**: `Docker Debug`
   - **Host**: `localhost`
   - **Port**: `2345`
4. Click **Apply** and **OK**
5. Select the configuration and click **Debug** (Shift+F9)
6. Set breakpoints in your code and make HTTP requests to trigger them

#### VS Code
1. Add to `.vscode/launch.json`:
```json
{
  "name": "Connect to Docker",
  "type": "go",
  "request": "attach",
  "mode": "remote",
  "remotePath": "/app",
  "port": 2345,
  "host": "localhost"
}
```
2. Set breakpoints and start debugging

The debugger will connect to the running container and pause execution when breakpoints are hit, allowing you to inspect variables, step through code, and debug issues in the containerized environment.