query-orchestration/README.md

# Query Orchestration

This repository contains the DoczyAI code.

Using the following project as a baseline: `https://github.com/golang-standards/project-layout`.

## Installation and Usage

- Install devbox: `https://www.jetify.com/docs/devbox/installing_devbox`
  - Ubuntu/MacOS: `curl -fsSL https://get.jetify.com/devbox | bash`
  - NixOS: `devbox`
- Install docker: `https://docs.docker.com/engine/install/`
- (IF NECESSARY) Ensure user in docker group and docker group is in sudo group

  ```sh
    sudo groupadd docker
    sudo usermod -aG docker $USER
  ```

- Run `touch .env` to create `.env` file for custom environment variables
- Run `devbox shell` to enter the environment
- Run `task fullsuite` to run all tests that ensure the current state

To find new packages: `https://search.nixos.org/packages`

For regular usage, please use `task` scripts in order run the most appropriate configuration.

## Environment Variables

When using environment variables, there is a hiearchy that will be respected. (First will override the next)

1. Within the devbox.json file, a variable added to "env"
2. Within the devbox.json file, a variable exported in "init_hook"
3. A variable added to .env

## Testing

For testing endtoends such as queries against a database, or interactions with an SQS queue, this repository employs the use of docker testcontainers.

`https://testcontainers.com/`

This enables tests to be fully self-contained, thus allowing allowing tests to be reliable and replicable.

### Execution

The simplest way of ensuring the validity of the full project is by running `task fullsuite` when in the devbox shell.

This will:

1. Generate the latest version of the specs.
2. Run the linting commands.
3. Run the unit tests.
4. Run the endtoend tests.

## Naming Conventions

**Service** - This term is used internally in order to refer to commands that get deployed as APIs.
**Runner** - This term is used internally in order to refer to commands that get deployed as consumers of a queue.

### Creating a new API

1. Add a file named `<api_name>.yml`. The api name is important, as it will be used as a reference throughout the codebase. The api name must be in the format `<functionality>API`. E.g. queryAPI, ClientAPI
2. Run `task openapi:generate`. This will generate the following:
  a. A location to implement the server-side functions in `./api/<api_name>/`, as well as generated functions, models and swagger docs.
  b. A location with the client-side code in `./pkg/<api_name>/`
3. Implement the controllers in `./api/<api_name>/`
4. Create a command in `./cmd/<api_name>/main.go` which creates a new instance from `./internal/api`

### Creating a new Runner

1. Implement the controller in `./api/<runner_name>/` which implements the `Controller` interface in `./internal/queue`. The runner name must be in the format `<functionality>Runner`. E.g. queryRunner, csvExportRunner
2. Create a command in `./cmd/<runner_name>/main.go` which creates a new instance from `./internal/queue`