Merged in feature/removejob (pull request #95)

Remove Job

* removejob

* rmjob

* sync

* cleanup

* precommit

* startslow

* startslow

* startslow

* openapi

* clean

* test

* scripts

* littlecleanercmds

* mermaid
This commit is contained in:
Michael McGuinness
2025-03-10 11:03:00 +00:00
parent 7d78e65d0c
commit 62886dbba8
148 changed files with 2088 additions and 3828 deletions
+17 -19
View File
@@ -1,12 +1,20 @@
# Query Orchestration
## clientSyncRunner
**Listens For**: Body Containing Client ID
**Action**.
Adds an event for each document into the DOCSYNC queue for the given Client.
## docCleanRunner
**Listens For**: Body Containing Document ID.
**Action**.
If the document is not already clean according to the job collector standards, a clean is triggered.
If the document is not already clean according to the collector standards, a clean is triggered.
An event with the document id is pushed to the DOCTEXT queue.
@@ -18,7 +26,7 @@ An event with the document id is pushed to the DOCTEXT queue.
Checks if the uploaded document is a duplicate.
Currently whether the document content is a duplicate or not is determines using the document ETag, and check if it already exists for the job.
Currently whether the document content is a duplicate or not is determines using the document ETag, and check if it already exists for the client.
Find reference entity.
@@ -36,7 +44,7 @@ Add an event to the DOC SYNC queue.
Checks if the document is eligible for syncing.
By validating: the client can_sync parameter, the job can_sync parameter.
By validating: the client can_sync parameter.
If eligible, send an event to the DOCCLEAN queue.
@@ -46,18 +54,10 @@ If eligible, send an event to the DOCCLEAN queue.
**Action**.
If the document text is not already extracted according to the job collector standards, the text is extracted.
If the document text is not already extracted according to the collector standards, the text is extracted.
An event with the document id is pushed to the QUERYSYNC queue.
## jobSyncRunner
**Listens For**: Body Containing Job ID
**Action**.
Adds an event for each document into the DOCSYNC queue for the given Job.
## metricsExample\_test
This main.go serves as an example of how to use the custom Prometheus metrics package.
@@ -90,17 +90,15 @@ This API currently manages all user entities to be managed. This allows users of
The entities in question:
- Client - A client is a single client of the Doczy project, all configurations and jobs for a client will be in reference to the respective client.
- Client - A client is a single client of the Doczy project, all configurations and clients for a client will be in reference to the respective client.
- Job - A job is a single instance of requirements for a client. This maintains a set of queries and processes in relation to itself.
- Collector - A collector specifies the required queries for export for a given client.
- Collector - A collector specifies the required queries for export for a given job.
- Sample - A sample is a subset of documents to be used for testing. When it is active a job will only process documents in the sample.
- Sample - A sample is a subset of documents to be used for testing. When it is active a client will only process documents in the sample.
- Query - A query is a block of processing of data which takes an input and produces an output.
- Export - An export is a manually started process which outputs the results for a job to a given location.
- Export - An export is a manually started process which outputs the results for a client to a given location.
## querySyncRunner
@@ -126,6 +124,6 @@ An event with the document id is pushed to the DOCTEXT queue.
**Action**.
Adds an event for each job into the JOBSYNC queue for the given query, i.e. each job that contains the query in its dependency tree.
Adds an event for each client into the CLIENTSYNC queue for the given query, i.e. each client that contains the query in its dependency tree.
Generated by Doczy