Jay Brown
7638fd3a90
Implement global and per ip rate limiting * all tests passing * test fix * Enhances test environment for rate limiting Updates the test environment to better support rate limiting tests. - Increases rate limits in test container to avoid interference with legitimate test traffic. - Increases the polling interval for client status checks to reduce load on the rate limiter. - Adds logic to retry status checks if rate limiting is encountered. * Merge branch 'main' of bitbucket.org:aarete/query-orchestration into feature/rate-limiting * build fix * test fix
Query Orchestration
clientSyncRunner
Listens For: Body Containing Client ID
Action.
Adds an event for each document into the DOCSYNC queue for the given Client.
docCleanRunner
Listens For: Body Containing Document ID.
Action.
If the document is not already clean according to the collector standards, a clean is triggered.
An event with the document id is pushed to the DOCTEXT queue.
docInitRunner
Listens For: S3 Event notifications - limited to PUT notifications.
Action.
Checks if the uploaded document is a duplicate.
Currently whether the document content is a duplicate or not is determines using the document ETag, and check if it already exists for the client.
Find reference entity.
If NOT a duplicate - Create a document reference entity.
Create a document entry to log the successful upload of a document.
Add an event to the DOC SYNC queue.
docSyncRunner
Listens For: Body Containing Document ID.
Action.
Checks if the document is eligible for syncing.
By validating: the client can_sync parameter.
If eligible, send an event to the DOCCLEAN queue.
docTextRunner
Listens For: Body Containing Document ID
Action.
If the document text is not already extracted according to the collector standards, trigger the textract job for text detection.
Otherwise, add an event to the QUERYSYNC queue.
healthcheck
cmd/healthcheck/main.go
metricsExample_test
This main.go serves as an example of how to use the custom Prometheus metrics package.
It demonstrates:
-
Setting up metrics with proper context handling
-
Configuring different types of metrics (Counter, Gauge, Histogram, Summary)
-
Recording metrics safely in a concurrent environment
-
Implementing graceful shutdown
-
Proper error handling
queryAPI
This API currently manages all user entities to be managed. This allows users of the service to change required entities and processes.
The entities in question:
-
Client - A client is a single client of the Doczy project, all configurations and clients for a client will be in reference to the respective client.
-
Collector - A collector specifies the required queries for export for a given client.
-
Sample - A sample is a subset of documents to be used for testing. When it is active a client will only process documents in the sample.
-
Query - A query is a block of processing of data which takes an input and produces an output.
-
Export - An export is a manually started process which outputs the results for a client to a given location.
queryRunner
Listens For: Body Containing Document ID and Query ID
Action.
Run the query for the given document and store the result.
Events with the document id and downstream query ids are pushed to the QUERY queue.
querySyncRunner
Listens For: Body Containing Document ID
Action.
Find the queries with no dependencies that do not have a valid result for a document.
The reasoning for only extracting these is to start the query extraction process only for necessary queries, and the query runner will manage updating downstream dependencies.
There are 2 situations that cause a query to show up in this list:
The query has no required dependencies.
All required dependencies have a valid result.
An event with the document id is pushed to the DOCTEXT queue.
queryVersionSyncRunner
Listens For: Body Containing Query ID
Action.
Adds an event for each client into the CLIENTSYNC queue for the given query, i.e. each client that contains the query in its dependency tree.
storeEventRunner
Listens For: S3 Event
Action.
It will identify the type of event and the location of said event, and forward the task accordingly.
It supports the document initialisation and document text processing stages. Events may be pushed to the DOCINIT queue.
textExtractor
cognito.auth.harness
permit.harness
user.creation.tool
audit.go provides comprehensive audit trail logging functionality for compliance and monitoring. This file now uses the log/slog structured logging package introduced in Go 1.21. A single environment variable LOG_FORMAT controls the output format:
The public API (type AuditLogger and all its methods) is unchanged, so no call‑sites (e.g. cognito-permit-sync.go) need to be modified.
cognito-permit-sync.go is the main entry point and orchestration logic for the user creation tool. This file coordinates the synchronization of users between AWS Cognito and Permit.io systems, handling CSV parsing, user creation/deletion/disable/enable operations, role validation, configuration management, and command-line interface. It serves as the primary workflow controller.
cognito.go handles AWS Cognito User Pool operations for user identity management. This file provides functions to create, delete, enable, disable, and retrieve users from AWS Cognito User Pools, including credential validation, user attribute management, and integration with AWS SDK v2 for identity provider operations.
permit.go handles Permit.io API operations for user management and role-based access control (RBAC). This file provides functions to create, delete, search, and manage users in Permit.io, as well as assign/unassign roles and validate role definitions. It integrates with Cognito user data to sync user information between the two systems.
Generated by Doczy