query-orchestration/test/demo.data.loader/README.md

# Demo Data Loader

A Python script that imports demo data from a CSV file into the Query Orchestration API. This tool is used to populate the system with sample documents and field extraction records for demos and testing.

## Overview

The script performs the following operations:

1. **Create Demo Client** - Creates a new client in the system to hold the demo data
2. **Upload Documents** - For each unique file in the CSV, creates and uploads a unique PDF document
3. **Create Field Extractions** - Imports the field extraction data from the CSV for each document
4. **Verify Records** - Retrieves and validates the created records match the input data

## Requirements

- Python 3.x (standard library only - no third-party packages required)
- Access to a running Query Orchestration API instance

## Usage

```bash
python3 demo_data_loader.py <csv_path> <base_url> [jwt_token]
python3 demo_data_loader.py ./SaaS_Demo_Data_Sam.csv  http://localhost:8080
# example for uat system with token
python3 demo_data_loader.py ./SaaS_Demo_Data_Sam.csv http://queryo-query-wheg72fhas6h-1773391169.us-east-2.elb.amazonaws.com tokenfile=token.txt
```

### Arguments

| Argument    | Required | Description                                               |
| ----------- | -------- | --------------------------------------------------------- |
| `csv_path`  | Yes      | Path to the CSV file containing demo data                 |
| `base_url`  | Yes      | Base URL of the API (e.g., `http://localhost:8080`)       |
| `jwt_token` | No       | JWT token for authentication (placeholder for future use) |

### Examples

```bash
# Run against local development server
python3 demo_data_loader.py SaaS_Demo_Data_Sam.csv http://localhost:8080

# Run against a deployed environment with authentication
python3 demo_data_loader.py SaaS_Demo_Data_Sam.csv https://api.example.com "eyJhbGciOiJS..."
```

## CSV File Format

The CSV file must have:

- **Header row** - Column names in the first row
- **Data rows** - Multiple rows where rows with the same `FILE_NAME` belong to the same document
- **Single-value fields** (columns 1-18) - Should be identical for all rows with the same `FILE_NAME`
- **Array fields** (columns 19+) - Can vary per row, forming multiple array items per document

### Column Mapping

#### Single-Value Fields (Columns 1-18)

| Column                        | API Field                  |
| ----------------------------- | -------------------------- |
| FILE_NAME                     | fileName                   |
| CONTRACT_TITLE                | contractTitle              |
| AARETE_DERIVED_AMENDMENT_NUM  | aareteDerivedAmendmentNum  |
| CLIENT_NAME                   | clientName                 |
| PAYER_NAME                    | payerName                  |
| PAYER_STATE                   | payerState                 |
| PROVIDER_STATE                | providerState              |
| FILENAME_TIN                  | filenameTin                |
| PROV_GROUP_TIN                | provGroupTin               |
| PROV_GROUP_NPI                | provGroupNpi               |
| PROV_GROUP_NAME_FULL          | provGroupNameFull          |
| PROV_OTHER_TIN                | provOtherTin               |
| PROV_OTHER_NPI                | provOtherNpi               |
| PROV_OTHER_NAME_FULL          | provOtherNameFull          |
| AARETE_DERIVED_EFFECTIVE_DT   | aareteDerivedEffectiveDt   |
| AARETE_DERIVED_TERMINATION_DT | aareteDerivedTerminationDt |
| AUTO_RENEWAL_IND              | autoRenewalInd             |
| AUTO_RENEWAL_TERM             | autoRenewalTerm            |

#### Array Fields (Columns 19+)

See the full mapping in the script source code. Array fields include:

- Exhibit information (title, page)
- Provider information (TIN, NPI, name)
- Claim type, product, LOB, program, network
- Taxonomy and specialty codes
- Reimbursement terms and rates
- Grouper information
- Outlier and stop-loss provisions
- Facility adjustments (DSH, IME, NTAP, etc.)
- Rate escalator information

## Data Type Conversions

The script automatically converts CSV values to appropriate types:

| Type    | Fields                               | Conversion                             |
| ------- | ------------------------------------ | -------------------------------------- |
| Boolean | `*Ind` fields                        | Y/N/TRUE/FALSE -> true/false           |
| Integer | `aareteDerivedAmendmentNum`          | String -> integer                      |
| Numeric | `*Rate`, `*Amt`, `*Threshold` fields | String -> float (handles %, $, commas) |
| Date    | `*Dt` fields                         | M/D/YYYY -> YYYY-MM-DD                 |

## Output

The script provides detailed progress output:

```
============================================================
  Demo Data Loader
============================================================
  CSV File: SaaS_Demo_Data_Sam.csv
  Base URL: http://localhost:8080
  JWT Token: not provided

============================================================
  Step 1: Reading CSV File
============================================================
  [OK] Read 220 data rows with 130 columns

============================================================
  Step 2: Analyzing Data
============================================================
  [OK] Found 23 unique documents to import
    - 952436878_Providence MC_Amd01_01.2025: 1 array item(s)
    - 952436878_Providence MC_Amd02_01.2025: 18 array item(s)
    ...

============================================================
  Summary
============================================================
  Client ID: demo-data-loader-1769734603
  Documents uploaded: 23
  Field extractions created: 23
  Verifications passed: 23
  Verifications failed: 0
  [OK] All operations completed successfully!
```

## Exit Codes

| Code | Meaning                                                                |
| ---- | ---------------------------------------------------------------------- |
| 0    | All operations completed successfully                                  |
| 1    | Error occurred (missing arguments, file read error, API failure, etc.) |

## Rate Limiting

The script includes:

- Automatic retry with exponential backoff for 429 (rate limit) responses
- Delays between API calls to avoid triggering rate limits

## API Endpoints Used

| Method | Endpoint                             | Purpose                 |
| ------ | ------------------------------------ | ----------------------- |
| POST   | `/client`                            | Create demo client      |
| POST   | `/client/{id}/document`              | Upload document         |
| GET    | `/client/{id}/document`              | List documents          |
| GET    | `/document/{id}`                     | Get document details    |
| POST   | `/field-extractions`                 | Create field extraction |
| GET    | `/field-extractions?documentId={id}` | Verify field extraction |

## Troubleshooting

### "No documents available after upload"

The document processing queue may be slow. Try:

1. Check that the `storeEventRunner` service is running
2. Increase the `max_attempts` parameter in `wait_for_documents()`

### Rate Limit Errors (429)

The script handles rate limits automatically with retries. If you still see errors:

1. Increase delays between operations in the script
2. Reduce the number of documents being imported at once

### "Failed to create extraction"

Check that:

1. The document exists in the system
2. The field values are valid (proper date formats, numeric values, etc.)

## Source Files

- `demo_data_loader.py` - Main script
- `SaaS_Demo_Data_Sam.csv` - Sample CSV data file
- `README.md` - This documentation