query-orchestration/plans/mutable.metadata.requirments.md

Persona

You are an expert db architect and software engineer.

Context

We are going to extend the schema for the text extraction records in this application that we currently expose so that each client can have their own collections of document schemas instead of everyone being tied into the same static 120 or so field extraction fields.

For example. Say client A (Acme corp) has 200 custom fields that they want to associate with documents that are in the aircraft engineering class of document. Each field has a name and data type plus constraints like max length or precision for floating point values. Then acme might also have 100 custom fields that it needs to track for documents that are for auto engineering documents.

Requirements

A SUPER ADMIN should be able to curate ALL collectionions of schemas for their documents. (for field extraction records on their documents) (FieldExtractionService)

So the client should be able to submit a new schema (which will get assigned an id by the system) The system will automatically reject the schema if it does not conform to jsonschema or whatever we are using for the schema. (json schema is assumed unless we come up with something better)

Then once a schema is created they should be able to assign a document to use that schema id. Then when they set the text extraction record for that document the metadata submitted must conform to the schema that the document is currently using. A document's schema id must remain the same for the lifetime of the document onse text extraction data (metadata) has been assigned. There can be a way to reset and clear the metadata on a document if we need to assigned an new schema and metadata.

Only the super admin should be able to list the schemas they have registered and curate them. (delete ones that are not used) or update ones that need changes. Note that when a schema is updated it will be assigned a new schema id.

document metadata needs to be extended so that each document has an optional (can be nill) custom schema id.

Task

You will thoroughly study the current API design (code and docs) expecially the FieldExtractionService to come up with a suggested design for how the new API should look and how the new DB scheam that holds all of this metadata (and now custom schemas) will be structured. It is acceptable for a document to store custom metadata in json blobs in the database if that simplifies the organization.

You will write the new design plan to (plan path here)

You will not change any code in this repository or on this server as part of your research. This is analysis and design only. and the only file you will write is the plan specified above.

When you are done with the design and the plan is written to disk I will review.