Michael McGuinness
c1e9d93037
Document Sync and Job Sync * docsyncfunc * startedQueryWork * morefixes * jobsyncsvcstart * save * jobsynctext * save * save * docsync * shorttest * jobsync * jobsyncupdateoncollectorupdate * passlivetest * collectortest * lint
70 lines
3.0 KiB
Markdown
70 lines
3.0 KiB
Markdown
# Query Orchestration
|
|
|
|
This repository contains the DoczyAI code.
|
|
|
|
Using the following project as a baseline: https://github.com/golang-standards/project-layout.
|
|
|
|
## Installation and Usage
|
|
|
|
- Install devbox: `https://www.jetify.com/docs/devbox/installing_devbox`
|
|
- Ubuntu/MacOS: `curl -fsSL https://get.jetify.com/devbox | bash`
|
|
- Install docker: `https://docs.docker.com/engine/install/`
|
|
- (IF NECESSARY) Ensure user in docker group and docker group is in sudo group
|
|
```
|
|
sudo groupadd docker
|
|
sudo usermod -aG docker $USER
|
|
```
|
|
- Run `touch .env` to create `.env` file for custom environment variables
|
|
- Run `devbox shell` to enter the environment
|
|
- Run `task fullsuite` to run all tests that ensure the current state
|
|
|
|
To find new packages: `https://search.nixos.org/packages`
|
|
|
|
For regular usage, please use `task` scripts in order run the most appropriate configuration.
|
|
|
|
## Environment Variables
|
|
|
|
When using environment variables, there is a hiearchy that will be respected. (First will override the next)
|
|
|
|
1. Within the devbox.json file, a variable added to "env"
|
|
2. Within the devbox.json file, a variable exported in "init_hook"
|
|
3. A variable added to .env
|
|
|
|
## Testing
|
|
|
|
For testing endtoends such as queries against a database, or interactions with an SQS queue, this repository employs the use of docker testcontainers.
|
|
|
|
`https://testcontainers.com/`
|
|
|
|
This enables tests to be fully self-contained, thus allowing allowing tests to be reliable and replicable.
|
|
|
|
### Execution
|
|
|
|
The simplest way of ensuring the validity of the full project is by running `task fullsuite` when in the devbox shell.
|
|
|
|
This will:
|
|
|
|
1. Generate the latest version of the specs.
|
|
2. Run the linting commands.
|
|
3. Run the unit tests.
|
|
4. Run the endtoend tests.
|
|
|
|
## Naming Conventions
|
|
|
|
**Service** - This term is used internally in order to refer to commands that get deployed as APIs.
|
|
**Runner** - This term is used internally in order to refer to commands that get deployed as consumers of a queue.
|
|
|
|
### Creating a new Service
|
|
|
|
1. Add a file named `<service_name>.yml`. The service name is important, as it will be used as a reference throughout the codebase. The service name must be in the format `<functionality>Service`. E.g. queryService, ClientService
|
|
2. Run `task openapi:generate`. This will generate the following:
|
|
- A location to implement the server-side functions in `./api/<service_name>/`, as well as generated functions, models and swagger docs.
|
|
- A location with the client-side code in `./pkg/<service_name>/`
|
|
3. Implement the controllers in `./api/<service_name>/`
|
|
4. Create a command in `./cmd/<service_name>/main.go` which creates a new instance from `./internal/api`
|
|
|
|
### Creating a new Runner
|
|
|
|
1. Implement the controller in `./api/<runner_name>/` which implements the `Controller` interface in `./internal/queue`. The runner name must be in the format `<functionality>Runner`. E.g. queryRunner, csvExportRunner
|
|
2. Create a command in `./cmd/<runner_name>/main.go` which creates a new instance from `./internal/queue`
|