jacob-mathison/query-orchestration
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity
Files
a080ca59d8565bd5ccb84ed246dc11e69e876c31
query-orchestration/docs/ai.generated/10-appendix.md
T

523 lines
17 KiB
Markdown
Raw Normal View History

Merged in feature/local-pdp (pull request #208)
2026-02-12 23:46:53 +00:00
# Appendix
## Table of Contents
1. [S3 Storage Guide](#s3-storage-guide)
- [Overview](#overview)
- [Bucket Configuration](#bucket-configuration)
- [BucketKey Path Format](#bucketkey-path-format)
- [Path Segment Reference](#path-segment-reference)
- [Location Types](#location-types)
- [Filename Segment Reference](#filename-segment-reference)
- [Client ID Constraints](#client-id-constraints)
- [Directory Partitioning](#directory-partitioning)
- [S3 Object Metadata](#s3-object-metadata)
- [Batch Upload Paths](#batch-upload-paths)
- [Concrete Examples](#concrete-examples)
- [Parsing a Key Back Into Its Components](#parsing-a-key-back-into-its-components)
- [Guide: Watching an S3 Bucket for Client Document Imports](#guide-watching-an-s3-bucket-for-client-document-imports)
---
## S3 Storage Guide
### Overview
Every document uploaded through the platform is stored in AWS S3 using a structured key (path) that encodes enough metadata to identify and recover the document without a database lookup. The key contains the client ID, processing stage, date, partition number, timestamp, entity ID, and optionally a batch ID and file extension.
The implementation lives in:
| File | Purpose |
|------|---------|
| `internal/serviceconfig/objectstore/bucketkey.go` | `BucketKey` struct, `String()`, `ParseBucketKey()` |
| `internal/serviceconfig/objectstore/bucketkey_test.go` | Unit tests with concrete path examples |
| `internal/serviceconfig/objectstore/config.go` | Bucket config, partition logic (`GetDirectoryPart`) |
| `internal/document/upload/get.go` | Upload flow that constructs keys |
| `internal/document/batch/storage/service.go` | Batch (ZIP) upload storage |
### Bucket Configuration
The bucket name is configured via environment variable:
| Variable | Required | Description |
|----------|----------|-------------|
| `BUCKET` | Yes | S3 bucket name |
| `AWS_ENDPOINT_URL_S3` | No | Override for local dev (default: `http://localhost:4566` via localstack) |
| `AWS_S3_USE_PATH_STYLE` | No | Path-style addressing (default: `false`) |
| `AWS_REGION` | No | AWS region (default: `us-east-1`) |
### BucketKey Path Format
A full S3 object key follows this structure:
```
{ClientID}/{Location}/{Date}/{Part}/{Timestamp}~{ClientID}~{Location}~{EntityID}[~{BatchID}][.FileType]
|_______________ prefix _________________| |______________________ filename __________________________|
```
For the `export` location, the `Part` segment is omitted:
```
{ClientID}/export/{Date}/{Timestamp}~{ClientID}~export~{EntityID}[~{BatchID}][.FileType]
```
### Path Segment Reference
The **prefix** portion determines the directory structure in S3:
| Segment | Format | Example | Description |
|---------|--------|---------|-------------|
| ClientID | `[a-zA-Z0-9_#-]+` | `acme_corp` | Client identifier. Same value repeated in the filename. |
| Location | `import`, `text`, or `export` | `import` | Processing stage (see Location Types below). |
| Date | `YYYYMMDD` | `20250331` | Date the document was uploaded, in UTC. |
| Part | Integer (0-based) | `0` | Partition shard within a date directory. Omitted for `export`. |
### Location Types
| Location | Meaning | Has Part? |
|----------|---------|-----------|
| `import` | Original uploaded document | Yes |
| `text` | Text extraction output | Yes |
| `export` | Exported results (CSV, etc.) | No |
### Filename Segment Reference
The **filename** portion is delimited by the tilde character (`~`). It comes in two variants depending on how the document was uploaded:
Merged in feature/permit-policy-cleanup (pull request #210)
2026-02-19 20:22:59 +00:00
**4-segment filename (single upload via `POST /client/{id}/document`):**
Merged in feature/local-pdp (pull request #208)
2026-02-12 23:46:53 +00:00
```
{Timestamp}~{ClientID}~{Location}~{EntityID}[.FileType]
```
Merged in feature/permit-policy-cleanup (pull request #210)
2026-02-19 20:22:59 +00:00
**5-segment filename (batch upload via `POST /client/{id}/document/batch`):**
Merged in feature/local-pdp (pull request #208)
2026-02-12 23:46:53 +00:00
```
{Timestamp}~{ClientID}~{Location}~{EntityID}~{BatchID}[.FileType]
```
The only difference is the 5th `~`-delimited segment containing the BatchID. Both formats may optionally end with a `.{FileType}` extension, though in practice the current upload code does not set FileType for `import` documents -- the file extension is typically absent.
| Segment | Position | Format | Example | Description |
|---------|----------|--------|---------|-------------|
| Timestamp | 1 (always) | `YYYY-MM-DDTHHMMSSZ` (ISO 8601 UTC) | `2025-03-31T040816Z` | Exact creation time. Uses Go format `2006-01-02T150405Z`. |
| ClientID | 2 (always) | `[a-zA-Z0-9_#-]+` | `acme_corp` | Repeated from prefix for self-describing keys. |
| Location | 3 (always) | `import`, `text`, or `export` | `import` | Repeated from prefix for self-describing keys. |
| EntityID | 4 (always) | UUID | `b745910f-f529-43c5-87e1-25629d46ef40` | The upload ID (for `import`) or document/entity ID. |
| BatchID | 5 (optional) | UUID | `12345678-1234-5678-9012-123456789012` | Present only for documents uploaded as part of a batch ZIP. |
| FileType | suffix | Extension without dot | `pdf` | Appended after a `.` separator on the last segment. Often absent for imports. |
**Real-world examples:**
Single upload (4 segments, no extension):
```
2026-02-07T002850Z~demo-data-loader-1770424122~import~438b605f-1b80-4c9f-bfb9-3f6bfbf853d8
```
Batch upload (5 segments, no extension):
```
2026-02-06T183937Z~int-test-1770403169~import~13234476-2881-4b22-924d-e3ebaf63e7c0~019c3440-a4c1-7da0-a3da-3c97f6287a24
```
### Client ID Constraints
Client IDs must match the regex pattern:
```
[a-zA-Z0-9_#-]+
```
Valid characters: letters, numbers, underscore (`_`), hash (`#`), hyphen (`-`).
The tilde (`~`) character is explicitly banned from client IDs because it serves as the delimiter between segments in the filename portion of the key. See `internal/client/create.go`:
```
// NOTE: Make sure the char ~ never makes way in as a valid client id character
// Reference bucket key filename
const CLIENT_ID_REGEX = `[a-zA-Z0-9\_#-]+`
```
### Directory Partitioning
Within each `{ClientID}/{Location}/{Date}/` prefix, documents are distributed across numbered partitions to avoid placing too many objects in a single virtual directory.
- The default partition limit is **1000 objects** per partition.
- Partition numbering starts at `0`.
- When partition `N` reaches 1000 objects for a given date, new uploads go to partition `N+1`.
- The current part and count are tracked in the database via `GetDocumentUploadCurrentPart`.
Logic from `ObjectStoreConfig.GetDirectoryPart()`:
```
MaxDirSize = 1000 (default)
if count >= MaxDirSize:
return part + 1
else:
return part
```
Example progression for a busy client on a single day:
```
acme_corp/import/20250331/0/... (objects 1-1000)
acme_corp/import/20250331/1/... (objects 1001-2000)
acme_corp/import/20250331/2/... (objects 2001-3000)
```
The `export` location does not use partitions. Export keys skip the part segment entirely.
### S3 Object Metadata
In addition to the structured key, each uploaded object carries S3 metadata headers:
| Metadata Key | Value | Example |
|-------------|-------|---------|
| `original-path` | Full path from the upload request (includes folders) | `contracts/2025/Q1/agreement.pdf` |
| `original-filename` | Just the filename | `agreement.pdf` |
| `Content-Type` | MIME type detected from file extension | `application/pdf` |
These metadata values preserve the user's original file naming, which is not encoded in the S3 key itself.
### Batch Upload Paths
ZIP files uploaded via the batch endpoint (`POST /client/{id}/document/batch`) use a separate, simpler path scheme:
```
batches/{ClientID}/{YYYY}/{MM}/{DD}/{HH}/{BatchID}-{OriginalFilename}
```
Example:
```
batches/acme_corp/2025/03/15/14/550e8400-e29b-41d4-a716-446655440000-uploads.zip
```
| Segment | Description |
|---------|-------------|
| `batches/` | Fixed prefix for all batch uploads |
| ClientID | Client identifier |
| `YYYY/MM/DD/HH` | Timestamp directory (year/month/day/hour) |
| BatchID | UUID assigned to the batch |
| OriginalFilename | The user's original ZIP filename |
Once the batch ZIP is extracted, individual documents from the ZIP are stored using the standard `BucketKey` format with the `import` location and the `BatchID` field populated.
### Concrete Examples
**1. Single PDF upload for client `acme_corp`:**
```
s3://my-bucket/acme_corp/import/20250331/0/2025-03-31T040816Z~acme_corp~import~b745910f-f529-43c5-87e1-25629d46ef40.pdf
```
Decoded:
- Client: `acme_corp`
- Stage: `import` (original upload)
- Date: March 31, 2025
- Partition: `0`
- Created at: 2025-03-31 04:08:16 UTC
- Upload/Entity ID: `b745910f-f529-43c5-87e1-25629d46ef40`
- File type: `pdf`
**2. Document from a batch upload:**
```
s3://my-bucket/acme_corp/import/20250331/0/2025-03-31T040816Z~acme_corp~import~b745910f-f529-43c5-87e1-25629d46ef40~12345678-1234-5678-9012-123456789012.pdf
```
Same as above, plus:
- Batch ID: `12345678-1234-5678-9012-123456789012`
**3. Text extraction output:**
```
s3://my-bucket/acme_corp/text/20250331/3/2025-03-31T040816Z~acme_corp~text~b745910f-f529-43c5-87e1-25629d46ef40
```
- Stage: `text` (extraction result)
- Partition: `3` (high volume, rolled past partitions 0-2)
- No file extension
**4. Export result:**
```
s3://my-bucket/acme_corp/export/20250331/2025-03-31T040816Z~acme_corp~export~b745910f-f529-43c5-87e1-25629d46ef40.csv
```
- Stage: `export`
- No partition number (exports never have one)
- File type: `csv`
**5. Client IDs with special characters:**
```
s3://my-bucket/A_A/import/20250331/0/2025-03-31T040816Z~A_A~import~b745910f-f529-43c5-87e1-25629d46ef40
s3://my-bucket/A-A/import/20250331/0/2025-03-31T040816Z~A-A~import~b745910f-f529-43c5-87e1-25629d46ef40
s3://my-bucket/A#A/import/20250331/0/2025-03-31T040816Z~A#A~import~b745910f-f529-43c5-87e1-25629d46ef40
```
### Parsing a Key Back Into Its Components
The `ParseBucketKey()` function in `bucketkey.go` reverses the process. It uses regex to extract the prefix components and then parses the tilde-delimited filename.
**Step 1: Split the prefix from the filename.**
Parse regex for the prefix:
```
^([a-zA-Z0-9_#-]+)/([a-z]+/?[a-z]+?)/([0-9]{8})/(.+)$
|___ ClientID ___| |__ Location __| |_ Date _| |_ rest (part/filename or just filename) _|
```
The "rest" portion is then checked for a leading integer partition number. If present, it splits into `{Part}/{Filename}`. If not, the entire rest is the filename.
**Step 2: Parse the filename.**
The filename is a `~`-delimited string with either 4 or 5 segments, plus an optional `.{extension}` suffix on the last segment. The parsing algorithm is:
1. Split on `.` first. If there are exactly 2 parts, the second part is the FileType. If there is 1 part, there is no FileType. More than 2 `.`-separated parts is invalid.
2. Split the base (part before `.`) on `~`. This yields either 4 or 5 segments:
- **4 segments:** `[Timestamp, ClientID, Location, EntityID]` -- single upload, no batch
- **5 segments:** `[Timestamp, ClientID, Location, EntityID, BatchID]` -- batch upload
3. The segment count is the only way to determine whether a BatchID is present. Count the `~` delimiters: 3 tildes = 4 segments (no batch), 4 tildes = 5 segments (has batch).
**Decision table for segment count:**
| Tilde count | Segment count | Has BatchID? | Upload source |
|-------------|---------------|--------------|---------------|
| 3 | 4 | No | Single document upload API |
| 4 | 5 | Yes | Batch (ZIP) upload API |
| Other | N/A | N/A | Invalid -- reject |
### Guide: Watching an S3 Bucket for Client Document Imports
This section shows how to monitor an S3 bucket for newly imported PDF documents belonging to a specific client.
#### Prerequisites
- AWS CLI configured with credentials that have `s3:ListBucket` and `s3:GetObject` permissions on the target bucket.
- The bucket name (from the `BUCKET` environment variable).
- The client ID you want to monitor.
#### Step 1: List All Imports for a Client
To see all imported documents for client `acme_corp`:
```bash
aws s3 ls s3://my-bucket/acme_corp/import/ --recursive
```
This lists every object under the client's `import` location across all dates and partitions.
#### Step 2: Filter by Date
To narrow down to a specific date (e.g., March 31, 2025):
```bash
aws s3 ls s3://my-bucket/acme_corp/import/20250331/
```
This shows the partition directories for that date:
```
PRE 0/
PRE 1/
```
Then list a specific partition:
```bash
aws s3 ls s3://my-bucket/acme_corp/import/20250331/0/
```
#### Step 3: Filter for PDFs Only
```bash
aws s3 ls s3://my-bucket/acme_corp/import/ --recursive | grep '\.pdf$'
```
#### Step 4: Watch for New Imports in Real Time
Use a polling loop to detect new documents as they arrive:
```bash
BUCKET="my-bucket"
CLIENT="acme_corp"
INTERVAL=30 # seconds between checks
LAST_CHECK=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
while true; do
echo "Checking for new imports since $LAST_CHECK ..."
# List objects and filter by those newer than last check
aws s3api list-objects-v2 \
--bucket "$BUCKET" \
--prefix "$CLIENT/import/" \
--query "Contents[?LastModified>=\`$LAST_CHECK\`].[Key,Size,LastModified]" \
--output table
LAST_CHECK=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
sleep "$INTERVAL"
done
```
#### Step 5: Extract Document Identity from a Key
Given a key, you can extract the client ID, document/upload ID, batch ID, and timestamp without a database query. The filename portion has either 4 or 5 `~`-delimited segments (see [Filename Segment Reference](#filename-segment-reference)).
**Example 1 -- single upload (4 segments, no batch, no extension):**
```
acme_corp/import/20250331/0/2025-03-31T040816Z~acme_corp~import~b745910f-f529-43c5-87e1-25629d46ef40
```
**Example 2 -- batch upload (5 segments, has batch ID, no extension):**
```
acme_corp/import/20250331/0/2025-03-31T040816Z~acme_corp~import~b745910f-f529-43c5-87e1-25629d46ef40~12345678-1234-5678-9012-123456789012
```
Using bash to parse either format:
```bash
KEY="$1" # pass the S3 key as argument
# Extract the filename (everything after the last /)
FILENAME="${KEY##*/}"
# Strip file extension if present (e.g. ".pdf")
# Note: import documents typically have NO extension
BASE="${FILENAME%%.*}"
if [[ "$FILENAME" == *"."* ]]; then
FILE_EXT="${FILENAME##*.}"
else
FILE_EXT=""
fi
# Split on ~ into an array
IFS='~' read -ra SEGMENTS <<< "$BASE"
SEGMENT_COUNT="${#SEGMENTS[@]}"
# First 4 segments are always present
TIMESTAMP="${SEGMENTS[0]}"
CLIENT_ID="${SEGMENTS[1]}"
LOCATION="${SEGMENTS[2]}"
ENTITY_ID="${SEGMENTS[3]}"
# 5th segment is the BatchID (only present for batch uploads)
if [[ "$SEGMENT_COUNT" -eq 5 ]]; then
BATCH_ID="${SEGMENTS[4]}"
else
BATCH_ID=""
fi
echo "Segments: $SEGMENT_COUNT"
echo "Client ID: $CLIENT_ID"
echo "Location: $LOCATION"
echo "Timestamp: $TIMESTAMP"
echo "Entity ID: $ENTITY_ID"
echo "Batch ID: ${BATCH_ID:-none}"
echo "File Type: ${FILE_EXT:-none}"
```
Output for example 1 (single upload):
```
Segments: 4
Client ID: acme_corp
Location: import
Timestamp: 2025-03-31T040816Z
Entity ID: b745910f-f529-43c5-87e1-25629d46ef40
Batch ID: none
File Type: none
```
Output for example 2 (batch upload):
```
Segments: 5
Client ID: acme_corp
Location: import
Timestamp: 2025-03-31T040816Z
Entity ID: b745910f-f529-43c5-87e1-25629d46ef40
Batch ID: 12345678-1234-5678-9012-123456789012
File Type: none
```
#### Step 6: Download a Document by Key
```bash
aws s3 cp \
"s3://my-bucket/acme_corp/import/20250331/0/2025-03-31T040816Z~acme_corp~import~b745910f-f529-43c5-87e1-25629d46ef40.pdf" \
./recovered-document.pdf
```
#### Step 7: Retrieve the Original Filename
The original filename from the user's upload is stored in S3 object metadata, not in the key:
```bash
aws s3api head-object \
--bucket my-bucket \
--key "acme_corp/import/20250331/0/2025-03-31T040816Z~acme_corp~import~b745910f-f529-43c5-87e1-25629d46ef40.pdf" \
--query "Metadata"
```
Output:
```json
{
"original-path": "contracts/2025/Q1/agreement.pdf",
"original-filename": "agreement.pdf"
}
```
#### Step 8: Bulk Recovery Script
To recover all PDFs for a client on a given date and rename them using their original filenames:
```bash
BUCKET="my-bucket"
CLIENT="acme_corp"
DATE="20250331"
OUTPUT_DIR="./recovered"
mkdir -p "$OUTPUT_DIR"
aws s3api list-objects-v2 \
--bucket "$BUCKET" \
--prefix "$CLIENT/import/$DATE/" \
--query "Contents[].Key" \
--output text | tr '\t' '\n' | grep '\.pdf$' | while read -r KEY; do
# Get original filename from metadata
ORIGINAL=$(aws s3api head-object \
--bucket "$BUCKET" \
--key "$KEY" \
--query "Metadata.\"original-filename\"" \
--output text)
# Extract entity ID for uniqueness (4th segment, works for both 4 and 5 segment filenames)
FILENAME="${KEY##*/}"
BASE="${FILENAME%%.*}"
IFS='~' read -ra SEGS <<< "$BASE"
ENTITY_ID="${SEGS[3]}"
# Use original filename if available, otherwise use entity ID
if [ "$ORIGINAL" != "None" ] && [ -n "$ORIGINAL" ]; then
DEST="$OUTPUT_DIR/${ENTITY_ID}_${ORIGINAL}"
else
DEST="$OUTPUT_DIR/${ENTITY_ID}.pdf"
fi
echo "Downloading: $KEY -> $DEST"
aws s3 cp "s3://$BUCKET/$KEY" "$DEST"
done
```
---
*Source code reference: `internal/serviceconfig/objectstore/bucketkey.go`*
Reference in New Issue Copy Permalink