Michael McGuinness
b8302344b4
Test * tests * tar * go * tmp * test * go * go * go * config * goup * versioning * order * cleanup * git * smallopts * flags * lessisgood * go * order * ordermatters * order * cc * test
144 lines
3.8 KiB
Markdown
144 lines
3.8 KiB
Markdown
# Query Orchestration
|
|
|
|
## clientSyncRunner
|
|
|
|
**Listens For**: Body Containing Client ID
|
|
|
|
**Action**.
|
|
|
|
Adds an event for each document into the DOCSYNC queue for the given Client.
|
|
|
|
## docCleanRunner
|
|
|
|
**Listens For**: Body Containing Document ID.
|
|
|
|
**Action**.
|
|
|
|
If the document is not already clean according to the collector standards, a clean is triggered.
|
|
|
|
An event with the document id is pushed to the DOCTEXT queue.
|
|
|
|
## docInitRunner
|
|
|
|
**Listens For**: S3 Event notifications - limited to PUT notifications.
|
|
|
|
**Action**.
|
|
|
|
Checks if the uploaded document is a duplicate.
|
|
|
|
Currently whether the document content is a duplicate or not is determines using the document ETag, and check if it already exists for the client.
|
|
|
|
Find reference entity.
|
|
|
|
If NOT a duplicate - Create a document reference entity.
|
|
|
|
Create a document entry to log the successful upload of a document.
|
|
|
|
Add an event to the DOC SYNC queue.
|
|
|
|
## docSyncRunner
|
|
|
|
**Listens For**: Body Containing Document ID.
|
|
|
|
**Action**.
|
|
|
|
Checks if the document is eligible for syncing.
|
|
|
|
By validating: the client can_sync parameter.
|
|
|
|
If eligible, send an event to the DOCCLEAN queue.
|
|
|
|
## docTextRunner
|
|
|
|
**Listens For**: Body Containing Document ID
|
|
|
|
**Action**.
|
|
|
|
If the document text is not already extracted according to the collector standards, trigger the textract job for text detection.
|
|
|
|
Otherwise, add an event to the QUERYSYNC queue.
|
|
|
|
## metricsExample\_test
|
|
|
|
This main.go serves as an example of how to use the custom Prometheus metrics package.
|
|
|
|
It demonstrates:
|
|
|
|
1. Setting up metrics with proper context handling
|
|
|
|
2. Configuring different types of metrics (Counter, Gauge, Histogram, Summary)
|
|
|
|
3. Recording metrics safely in a concurrent environment
|
|
|
|
4. Implementing graceful shutdown
|
|
|
|
5. Proper error handling
|
|
|
|
## queryAPI
|
|
|
|
This API currently manages all user entities to be managed. This allows users of the service to change required entities and processes.
|
|
|
|
The entities in question:
|
|
|
|
- Client - A client is a single client of the Doczy project, all configurations and clients for a client will be in reference to the respective client.
|
|
|
|
- Collector - A collector specifies the required queries for export for a given client.
|
|
|
|
- Sample - A sample is a subset of documents to be used for testing. When it is active a client will only process documents in the sample.
|
|
|
|
- Query - A query is a block of processing of data which takes an input and produces an output.
|
|
|
|
- Export - An export is a manually started process which outputs the results for a client to a given location.
|
|
|
|
## queryRunner
|
|
|
|
**Listens For**: Body Containing Document ID and Query ID
|
|
|
|
**Action**.
|
|
|
|
Run the query for the given document and store the result.
|
|
|
|
Events with the document id and downstream query ids are pushed to the QUERY queue.
|
|
|
|
## querySyncRunner
|
|
|
|
**Listens For**: Body Containing Document ID
|
|
|
|
**Action**.
|
|
|
|
Find the queries with no dependencies that do not have a valid result for a document.
|
|
|
|
The reasoning for only extracting these is to start the query extraction process only for necessary queries, and the query runner will manage updating downstream dependencies.
|
|
|
|
There are 2 situations that cause a query to show up in this list:
|
|
|
|
The query has no required dependencies.
|
|
|
|
All required dependencies have a valid result.
|
|
|
|
An event with the document id is pushed to the DOCTEXT queue.
|
|
|
|
## queryVersionSyncRunner
|
|
|
|
**Listens For**: Body Containing Query ID
|
|
|
|
**Action**.
|
|
|
|
Adds an event for each client into the CLIENTSYNC queue for the given query, i.e. each client that contains the query in its dependency tree.
|
|
|
|
## storeEventRunner
|
|
|
|
**Listens For**: S3 Event
|
|
|
|
**Action**.
|
|
|
|
It will identify the type of event and the location of said event, and forward the task accordingly.
|
|
|
|
It supports the document initialisation and document text processing stages. Events may be pushed to the DOCINIT queue.
|
|
|
|
## textExtractor
|
|
|
|
## cognito.auth.harness
|
|
|
|
Generated by Doczy
|