jacob-mathison/query-orchestration
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

Merged in feature/demo-support (pull request #207)

Browse Source 
Feature/demo support

* index logs now debug

* demo data loader
This commit is contained in:
Jay Brown
2026-02-02 19:08:08 +00:00
parent 63c12a2f44
commit 963ccc6553
15 changed files with 2240 additions and 314 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/demo.data.loader/README.md
+193
View File
@@ -0,0 +1,193 @@
# Demo Data Loader
A Python script that imports demo data from a CSV file into the Query Orchestration API. This tool is used to populate the system with sample documents and field extraction records for demos and testing.
## Overview
The script performs the following operations:
1. **Create Demo Client** - Creates a new client in the system to hold the demo data
2. **Upload Documents** - For each unique file in the CSV, creates and uploads a unique PDF document
3. **Create Field Extractions** - Imports the field extraction data from the CSV for each document
4. **Verify Records** - Retrieves and validates the created records match the input data
## Requirements
- Python 3.x (standard library only - no third-party packages required)
- Access to a running Query Orchestration API instance
## Usage
```bash
python3 demo_data_loader.py <csv_path> <base_url> [jwt_token]
python3 demo_data_loader.py ./SaaS_Demo_Data_Sam.csv http://localhost:8080
# example for uat system with token
python3 demo_data_loader.py ./SaaS_Demo_Data_Sam.csv http://queryo-query-wheg72fhas6h-1773391169.us-east-2.elb.amazonaws.com tokenfile=token.txt
```
### Arguments
| Argument | Required | Description |
| ----------- | -------- | --------------------------------------------------------- |
| `csv_path` | Yes | Path to the CSV file containing demo data |
| `base_url` | Yes | Base URL of the API (e.g., `http://localhost:8080`) |
| `jwt_token` | No | JWT token for authentication (placeholder for future use) |
### Examples
```bash
# Run against local development server
python3 demo_data_loader.py SaaS_Demo_Data_Sam.csv http://localhost:8080
# Run against a deployed environment with authentication
python3 demo_data_loader.py SaaS_Demo_Data_Sam.csv https://api.example.com "eyJhbGciOiJS..."
```
## CSV File Format
The CSV file must have:
- **Header row** - Column names in the first row
- **Data rows** - Multiple rows where rows with the same `FILE_NAME` belong to the same document
- **Single-value fields** (columns 1-18) - Should be identical for all rows with the same `FILE_NAME`
- **Array fields** (columns 19+) - Can vary per row, forming multiple array items per document
### Column Mapping
#### Single-Value Fields (Columns 1-18)
| Column | API Field |
| ----------------------------- | -------------------------- |
| FILE_NAME | fileName |
| CONTRACT_TITLE | contractTitle |
| AARETE_DERIVED_AMENDMENT_NUM | aareteDerivedAmendmentNum |
| CLIENT_NAME | clientName |
| PAYER_NAME | payerName |
| PAYER_STATE | payerState |
| PROVIDER_STATE | providerState |
| FILENAME_TIN | filenameTin |
| PROV_GROUP_TIN | provGroupTin |
| PROV_GROUP_NPI | provGroupNpi |
| PROV_GROUP_NAME_FULL | provGroupNameFull |
| PROV_OTHER_TIN | provOtherTin |
| PROV_OTHER_NPI | provOtherNpi |
| PROV_OTHER_NAME_FULL | provOtherNameFull |
| AARETE_DERIVED_EFFECTIVE_DT | aareteDerivedEffectiveDt |
| AARETE_DERIVED_TERMINATION_DT | aareteDerivedTerminationDt |
| AUTO_RENEWAL_IND | autoRenewalInd |
| AUTO_RENEWAL_TERM | autoRenewalTerm |
#### Array Fields (Columns 19+)
See the full mapping in the script source code. Array fields include:
- Exhibit information (title, page)
- Provider information (TIN, NPI, name)
- Claim type, product, LOB, program, network
- Taxonomy and specialty codes
- Reimbursement terms and rates
- Grouper information
- Outlier and stop-loss provisions
- Facility adjustments (DSH, IME, NTAP, etc.)
- Rate escalator information
## Data Type Conversions
The script automatically converts CSV values to appropriate types:
| Type | Fields | Conversion |
| ------- | ------------------------------------ | -------------------------------------- |
| Boolean | `*Ind` fields | Y/N/TRUE/FALSE -> true/false |
| Integer | `aareteDerivedAmendmentNum` | String -> integer |
| Numeric | `*Rate`, `*Amt`, `*Threshold` fields | String -> float (handles %, $, commas) |
| Date | `*Dt` fields | M/D/YYYY -> YYYY-MM-DD |
## Output
The script provides detailed progress output:
```
============================================================
Demo Data Loader
============================================================
CSV File: SaaS_Demo_Data_Sam.csv
Base URL: http://localhost:8080
JWT Token: not provided
============================================================
Step 1: Reading CSV File
============================================================
[OK] Read 220 data rows with 130 columns
============================================================
Step 2: Analyzing Data
============================================================
[OK] Found 23 unique documents to import
- 952436878_Providence MC_Amd01_01.2025: 1 array item(s)
- 952436878_Providence MC_Amd02_01.2025: 18 array item(s)
...
============================================================
Summary
============================================================
Client ID: demo-data-loader-1769734603
Documents uploaded: 23
Field extractions created: 23
Verifications passed: 23
Verifications failed: 0
[OK] All operations completed successfully!
```
## Exit Codes
| Code | Meaning |
| ---- | ---------------------------------------------------------------------- |
| 0 | All operations completed successfully |
| 1 | Error occurred (missing arguments, file read error, API failure, etc.) |
## Rate Limiting
The script includes:
- Automatic retry with exponential backoff for 429 (rate limit) responses
- Delays between API calls to avoid triggering rate limits
## API Endpoints Used
| Method | Endpoint | Purpose |
| ------ | ------------------------------------ | ----------------------- |
| POST | `/client` | Create demo client |
| POST | `/client/{id}/document` | Upload document |
| GET | `/client/{id}/document` | List documents |
| GET | `/document/{id}` | Get document details |
| POST | `/field-extractions` | Create field extraction |
| GET | `/field-extractions?documentId={id}` | Verify field extraction |
## Troubleshooting
### "No documents available after upload"
The document processing queue may be slow. Try:
1. Check that the `storeEventRunner` service is running
2. Increase the `max_attempts` parameter in `wait_for_documents()`
### Rate Limit Errors (429)
The script handles rate limits automatically with retries. If you still see errors:
1. Increase delays between operations in the script
2. Reduce the number of documents being imported at once
### "Failed to create extraction"
Check that:
1. The document exists in the system
2. The field values are valid (proper date formats, numeric values, etc.)
## Source Files
- `demo_data_loader.py` - Main script
- `SaaS_Demo_Data_Sam.csv` - Sample CSV data file
- `README.md` - This documentation
test/demo.data.loader/demo_data_loader.py
+1055
View File
File diff suppressed because it is too large Load Diff