query-orchestration/test/demo.data.loader

Demo Data Loader

A Python script that imports demo data from a CSV file into the Query Orchestration API. This tool is used to populate the system with sample documents and field extraction records for demos and testing.

Overview

The script performs the following operations:

1. Create Demo Client - Creates a new client in the system to hold the demo data
2. Upload Documents - For each unique file in the CSV, creates and uploads a unique PDF document
3. Create Field Extractions - Imports the field extraction data from the CSV for each document
4. Verify Records - Retrieves and validates the created records match the input data

Requirements

• Python 3.x (standard library only - no third-party packages required)
• Access to a running Query Orchestration API instance

Usage

python3 demo_data_loader.py <csv_path> <base_url> [jwt_token]
python3 demo_data_loader.py ./SaaS_Demo_Data_Sam.csv  http://localhost:8080
# example for uat system with token
python3 demo_data_loader.py ./SaaS_Demo_Data_Sam.csv http://queryo-query-wheg72fhas6h-1773391169.us-east-2.elb.amazonaws.com tokenfile=token.txt

Arguments

Argument	Required	Description
csv_path	Yes	Path to the CSV file containing demo data
base_url	Yes	Base URL of the API (e.g., http://localhost:8080)
jwt_token	No	JWT token for authentication (placeholder for future use)

Examples

# Run against local development server
python3 demo_data_loader.py SaaS_Demo_Data_Sam.csv http://localhost:8080

# Run against a deployed environment with authentication
python3 demo_data_loader.py SaaS_Demo_Data_Sam.csv https://api.example.com "eyJhbGciOiJS..."

CSV File Format

The CSV file must have:

• Header row - Column names in the first row
• Data rows - Multiple rows where rows with the same FILE_NAME belong to the same document
• Single-value fields (columns 1-18) - Should be identical for all rows with the same FILE_NAME
• Array fields (columns 19+) - Can vary per row, forming multiple array items per document

Column Mapping

Single-Value Fields (Columns 1-18)

Column	API Field
FILE_NAME	fileName
CONTRACT_TITLE	contractTitle
AARETE_DERIVED_AMENDMENT_NUM	aareteDerivedAmendmentNum
CLIENT_NAME	clientName
PAYER_NAME	payerName
PAYER_STATE	payerState
PROVIDER_STATE	providerState
FILENAME_TIN	filenameTin
PROV_GROUP_TIN	provGroupTin
PROV_GROUP_NPI	provGroupNpi
PROV_GROUP_NAME_FULL	provGroupNameFull
PROV_OTHER_TIN	provOtherTin
PROV_OTHER_NPI	provOtherNpi
PROV_OTHER_NAME_FULL	provOtherNameFull
AARETE_DERIVED_EFFECTIVE_DT	aareteDerivedEffectiveDt
AARETE_DERIVED_TERMINATION_DT	aareteDerivedTerminationDt
AUTO_RENEWAL_IND	autoRenewalInd
AUTO_RENEWAL_TERM	autoRenewalTerm

Array Fields (Columns 19+)

See the full mapping in the script source code. Array fields include:

• Exhibit information (title, page)
• Provider information (TIN, NPI, name)
• Claim type, product, LOB, program, network
• Taxonomy and specialty codes
• Reimbursement terms and rates
• Grouper information
• Outlier and stop-loss provisions
• Facility adjustments (DSH, IME, NTAP, etc.)
• Rate escalator information

Data Type Conversions

The script automatically converts CSV values to appropriate types:

Type	Fields	Conversion
Boolean	*Ind fields	Y/N/TRUE/FALSE -> true/false
Integer	aareteDerivedAmendmentNum	String -> integer
Numeric	*Rate, *Amt, *Threshold fields	String -> float (handles %, $, commas)
Date	*Dt fields	M/D/YYYY -> YYYY-MM-DD

Output

The script provides detailed progress output:

============================================================
  Demo Data Loader
============================================================
  CSV File: SaaS_Demo_Data_Sam.csv
  Base URL: http://localhost:8080
  JWT Token: not provided

============================================================
  Step 1: Reading CSV File
============================================================
  [OK] Read 220 data rows with 130 columns

============================================================
  Step 2: Analyzing Data
============================================================
  [OK] Found 23 unique documents to import
    - 952436878_Providence MC_Amd01_01.2025: 1 array item(s)
    - 952436878_Providence MC_Amd02_01.2025: 18 array item(s)
    ...

============================================================
  Summary
============================================================
  Client ID: demo-data-loader-1769734603
  Documents uploaded: 23
  Field extractions created: 23
  Verifications passed: 23
  Verifications failed: 0
  [OK] All operations completed successfully!

Exit Codes

Code	Meaning
0	All operations completed successfully
1	Error occurred (missing arguments, file read error, API failure, etc.)

Rate Limiting

The script includes:

• Automatic retry with exponential backoff for 429 (rate limit) responses
• Delays between API calls to avoid triggering rate limits

API Endpoints Used

Method	Endpoint	Purpose
POST	/client	Create demo client
POST	/client/{id}/document	Upload document
GET	/client/{id}/document	List documents
GET	/document/{id}	Get document details
POST	/field-extractions	Create field extraction
GET	/field-extractions?documentId={id}	Verify field extraction

Troubleshooting

"No documents available after upload"

The document processing queue may be slow. Try:

1. Check that the storeEventRunner service is running
2. Increase the max_attempts parameter in wait_for_documents()

Rate Limit Errors (429)

The script handles rate limits automatically with retries. If you still see errors:

1. Increase delays between operations in the script
2. Reduce the number of documents being imported at once

"Failed to create extraction"

Check that:

1. The document exists in the system
2. The field values are valid (proper date formats, numeric values, etc.)

Source Files

• demo_data_loader.py - Main script
• SaaS_Demo_Data_Sam.csv - Sample CSV data file
• README.md - This documentation