Michael McGuinness
fee71e7740
Feature/postprocessing * tests * passtest * fixshorttests * mosttests * improvingbasedockerfile * testspeeds * testing * host * canparallel * clean * passfullsuite * singlepagemax * test * findfeatures * findstables * tbls * tablestoo * tablestoo * lateraltests * tableloc * cleanup * inlinetable * childids * cleanup * tests
3.8 KiB
Query Orchestration
clientSyncRunner
Listens For: Body Containing Client ID
Action.
Adds an event for each document into the DOCSYNC queue for the given Client.
docCleanRunner
Listens For: Body Containing Document ID.
Action.
If the document is not already clean according to the collector standards, a clean is triggered.
An event with the document id is pushed to the DOCTEXT queue.
docInitRunner
Listens For: S3 Event notifications - limited to PUT notifications.
Action.
Checks if the uploaded document is a duplicate.
Currently whether the document content is a duplicate or not is determines using the document ETag, and check if it already exists for the client.
Find reference entity.
If NOT a duplicate - Create a document reference entity.
Create a document entry to log the successful upload of a document.
Add an event to the DOC SYNC queue.
docSyncRunner
Listens For: Body Containing Document ID.
Action.
Checks if the document is eligible for syncing.
By validating: the client can_sync parameter.
If eligible, send an event to the DOCCLEAN queue.
docTextRunner
Listens For: Body Containing Document ID
Action.
If the document text is not already extracted according to the collector standards, trigger the textract job for text detection.
Otherwise, add an event to the QUERYSYNC queue.
metricsExample_test
This main.go serves as an example of how to use the custom Prometheus metrics package.
It demonstrates:
-
Setting up metrics with proper context handling
-
Configuring different types of metrics (Counter, Gauge, Histogram, Summary)
-
Recording metrics safely in a concurrent environment
-
Implementing graceful shutdown
-
Proper error handling
queryAPI
This API currently manages all user entities to be managed. This allows users of the service to change required entities and processes.
The entities in question:
-
Client - A client is a single client of the Doczy project, all configurations and clients for a client will be in reference to the respective client.
-
Collector - A collector specifies the required queries for export for a given client.
-
Sample - A sample is a subset of documents to be used for testing. When it is active a client will only process documents in the sample.
-
Query - A query is a block of processing of data which takes an input and produces an output.
-
Export - An export is a manually started process which outputs the results for a client to a given location.
queryRunner
Listens For: Body Containing Document ID and Query ID
Action.
Run the query for the given document and store the result.
Events with the document id and downstream query ids are pushed to the QUERY queue.
querySyncRunner
Listens For: Body Containing Document ID
Action.
Find the queries with no dependencies that do not have a valid result for a document.
The reasoning for only extracting these is to start the query extraction process only for necessary queries, and the query runner will manage updating downstream dependencies.
There are 2 situations that cause a query to show up in this list:
The query has no required dependencies.
All required dependencies have a valid result.
An event with the document id is pushed to the DOCTEXT queue.
queryVersionSyncRunner
Listens For: Body Containing Query ID
Action.
Adds an event for each client into the CLIENTSYNC queue for the given query, i.e. each client that contains the query in its dependency tree.
storeEventRunner
Listens For: S3 Event
Action.
It will identify the type of event and the location of said event, and forward the task accordingly.
It supports the document initialisation and document text processing stages. Events may be pushed to the DOCINIT queue.
Generated by Doczy