Merged in feature/removejob (pull request #95)
Remove Job * removejob * rmjob * sync * cleanup * precommit * startslow * startslow * startslow * openapi * clean * test * scripts * littlecleanercmds * mermaid
This commit is contained in:
+17
-19
@@ -1,12 +1,20 @@
|
||||
# Query Orchestration
|
||||
|
||||
## clientSyncRunner
|
||||
|
||||
**Listens For**: Body Containing Client ID
|
||||
|
||||
**Action**.
|
||||
|
||||
Adds an event for each document into the DOCSYNC queue for the given Client.
|
||||
|
||||
## docCleanRunner
|
||||
|
||||
**Listens For**: Body Containing Document ID.
|
||||
|
||||
**Action**.
|
||||
|
||||
If the document is not already clean according to the job collector standards, a clean is triggered.
|
||||
If the document is not already clean according to the collector standards, a clean is triggered.
|
||||
|
||||
An event with the document id is pushed to the DOCTEXT queue.
|
||||
|
||||
@@ -18,7 +26,7 @@ An event with the document id is pushed to the DOCTEXT queue.
|
||||
|
||||
Checks if the uploaded document is a duplicate.
|
||||
|
||||
Currently whether the document content is a duplicate or not is determines using the document ETag, and check if it already exists for the job.
|
||||
Currently whether the document content is a duplicate or not is determines using the document ETag, and check if it already exists for the client.
|
||||
|
||||
Find reference entity.
|
||||
|
||||
@@ -36,7 +44,7 @@ Add an event to the DOC SYNC queue.
|
||||
|
||||
Checks if the document is eligible for syncing.
|
||||
|
||||
By validating: the client can_sync parameter, the job can_sync parameter.
|
||||
By validating: the client can_sync parameter.
|
||||
|
||||
If eligible, send an event to the DOCCLEAN queue.
|
||||
|
||||
@@ -46,18 +54,10 @@ If eligible, send an event to the DOCCLEAN queue.
|
||||
|
||||
**Action**.
|
||||
|
||||
If the document text is not already extracted according to the job collector standards, the text is extracted.
|
||||
If the document text is not already extracted according to the collector standards, the text is extracted.
|
||||
|
||||
An event with the document id is pushed to the QUERYSYNC queue.
|
||||
|
||||
## jobSyncRunner
|
||||
|
||||
**Listens For**: Body Containing Job ID
|
||||
|
||||
**Action**.
|
||||
|
||||
Adds an event for each document into the DOCSYNC queue for the given Job.
|
||||
|
||||
## metricsExample\_test
|
||||
|
||||
This main.go serves as an example of how to use the custom Prometheus metrics package.
|
||||
@@ -90,17 +90,15 @@ This API currently manages all user entities to be managed. This allows users of
|
||||
|
||||
The entities in question:
|
||||
|
||||
- Client - A client is a single client of the Doczy project, all configurations and jobs for a client will be in reference to the respective client.
|
||||
- Client - A client is a single client of the Doczy project, all configurations and clients for a client will be in reference to the respective client.
|
||||
|
||||
- Job - A job is a single instance of requirements for a client. This maintains a set of queries and processes in relation to itself.
|
||||
- Collector - A collector specifies the required queries for export for a given client.
|
||||
|
||||
- Collector - A collector specifies the required queries for export for a given job.
|
||||
|
||||
- Sample - A sample is a subset of documents to be used for testing. When it is active a job will only process documents in the sample.
|
||||
- Sample - A sample is a subset of documents to be used for testing. When it is active a client will only process documents in the sample.
|
||||
|
||||
- Query - A query is a block of processing of data which takes an input and produces an output.
|
||||
|
||||
- Export - An export is a manually started process which outputs the results for a job to a given location.
|
||||
- Export - An export is a manually started process which outputs the results for a client to a given location.
|
||||
|
||||
## querySyncRunner
|
||||
|
||||
@@ -126,6 +124,6 @@ An event with the document id is pushed to the DOCTEXT queue.
|
||||
|
||||
**Action**.
|
||||
|
||||
Adds an event for each job into the JOBSYNC queue for the given query, i.e. each job that contains the query in its dependency tree.
|
||||
Adds an event for each client into the CLIENTSYNC queue for the given query, i.e. each client that contains the query in its dependency tree.
|
||||
|
||||
Generated by Doczy
|
||||
|
||||
Reference in New Issue
Block a user