query-orchestration/docs/README.md

# Query Orchestration

## clientSyncRunner

**Listens For**: Body Containing Client ID

**Action**.

Adds an event for each document into the DOCSYNC queue for the given Client.

## docCleanRunner

Listens For: Body Containing Document ID.

Action: If the document is not already clean according to the collector standards, a clean is triggered. Note: Text extraction has been removed. The document pipeline ends after cleaning. See remove_texttract_and_mocks_plan.md for details.

## docInitRunner

**Listens For**: S3 Event notifications - limited to PUT notifications.

**Action**.

Checks if the uploaded document is a duplicate.

Currently whether the document content is a duplicate or not is determines using the document ETag, and check if it already exists for the client.

Find reference entity.

If NOT a duplicate - Create a document reference entity.

Create a document entry to log the successful upload of a document.

Add an event to the DOC SYNC queue.

## docSyncRunner

**Listens For**: Body Containing Document ID.

**Action**.

Checks if the document is eligible for syncing.

By validating: the client can_sync parameter.

If eligible, send an event to the DOCCLEAN queue.

## docTextRunner

DEPRECATED: This runner is deprecated and no longer has any functionality. It is left in place until deployment code is updated accordingly. See remove_texttract_and_mocks_plan.md for details.

## healthcheck

cmd/healthcheck/main.go

## metricsExample\_test

This main.go serves as an example of how to use the custom Prometheus metrics package.

It demonstrates:

1. Setting up metrics with proper context handling

2. Configuring different types of metrics (Counter, Gauge, Histogram, Summary)

3. Recording metrics safely in a concurrent environment

4. Implementing graceful shutdown

5. Proper error handling

## queryAPI

This API currently manages all user entities to be managed. This allows users of the service to change required entities and processes.

The entities in question:

- Client - A client is a single client of the Doczy project, all configurations and clients for a client will be in reference to the respective client.

- Collector - A collector specifies document processing settings for a given client.

- Sample - A sample is a subset of documents to be used for testing. When it is active a client will only process documents in the sample.

- Export - An export is a manually started process which outputs the results for a client to a given location.

Note: Query functionality has been removed from this API. See remove_query_plan.md for details.

## queryRunner

DEPRECATED: This runner is deprecated and no longer has any functionality. It is left in place until deployment code is updated accordingly. See remove_query_plan.md for details on the query removal.

## querySyncRunner

DEPRECATED: This runner is deprecated and no longer has any functionality. It is left in place until deployment code is updated accordingly. See remove_query_plan.md for details on the query removal.

## queryVersionSyncRunner

DEPRECATED: This runner is deprecated and no longer has any functionality. It is left in place until deployment code is updated accordingly. See remove_query_plan.md for details on the query removal.

## storeEventRunner

**Listens For**: S3 Event

**Action**.

It will identify the type of event and the location of said event, and forward the task accordingly.

It supports the document initialisation and document text processing stages. Events may be pushed to the DOCINIT queue.

## cognito.auth.harness

## permit.setup

setup_permit.go - CLI tool to configure Permit.io roles and resources This tool reads a YAML configuration file and creates resources, actions, and roles in a Permit.io project/environment using the Permit.io REST API.

Usage:



## user.creation.tool

audit.go provides comprehensive audit trail logging functionality for compliance and monitoring. This file now uses the `log/slog` structured logging package introduced in Go 1.21. A single environment variable LOG_FORMAT controls the output format:



The public API (type AuditLogger and all its methods) is unchanged, so no call‑sites (e.g. cognito-permit-sync.go) need to be modified.

cognito-permit-sync.go is the main entry point and orchestration logic for the user creation tool. This file coordinates the synchronization of users between AWS Cognito and Permit.io systems, handling CSV parsing, user creation/deletion/disable/enable operations, role validation, configuration management, and command-line interface. It serves as the primary workflow controller.

cognito.go handles AWS Cognito User Pool operations for user identity management. This file provides functions to create, delete, enable, disable, and retrieve users from AWS Cognito User Pools, including credential validation, user attribute management, and integration with AWS SDK v2 for identity provider operations.

permit.go handles Permit.io API operations for user management and role-based access control (RBAC). This file provides functions to create, delete, search, and manage users in Permit.io, as well as assign/unassign roles and validate role definitions. It integrates with Cognito user data to sync user information between the two systems.

## permit.harness

Generated by Doczy