Katon Minhas
afb6d5185d
Feature/lesser table caching refactor hybrid * chore: Remove unused duplicate main.py from shared pipeline * fix: Correct crosswalk paths in aarete_derived.py * chore: Remove unused documentation files from fieldExtraction * docs: Add documentation files to documentation folder * docs: Update README with uv setup, expanded project structure, and branching conventions * docs: Add uv installation steps with Ubuntu/WSL emphasis * Enable prompt caching for all remaining LLM calls - Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE, EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH, SPECIAL_CASE_ASSIGNMENT - Update all invoke_claude() calls in saas and clover pipelines to use cache=True with corresponding _INSTRUCTION() functions - Add new instructions to get_cacheable_instructions() for cache warming - Update tests for new instruction functions Functions now using caching: - prompt_exhibit_level - prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF) - prompt_fee_schedule_breakout - prompt_grouper_breakout - prompt_special_case_assignment - prompt_exhibit_linkage - prompt_exhibit_header - prompt_smart_chunked (ONE_TO_ONE templates) - prompt_date_fix - prompt_derived_term_date - prompt_exhibit_title_match - provider_name_match_check 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Reorder * feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction - Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction - Extract full paragraph text of offset/recoupment provisions from contracts - Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence - Fix reorder_columns to preserve extra columns not in COLUMN_ORDER - Update QC/QA output path to outputs/qc_qa/ * fix: Update dev deps and test assertions for QC/QA output path - Add pytest/pytest-mock to dev dependencies for mypy type checking - Update test assertions to expect outputs/qc_qa instead of qa_qc_output * style: Apply black formatting to prompt_templates.py * Merge main, move scripts * Archive some scripts * update py version * remove .py version file * Remove ASCII characters * Restore testbed code * restore tracking * Update testbed metrics * Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT - Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification - Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination - Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification - Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT - Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK - Add new instructions to get_cacheable_instructions() for cache warming - Add unit tests for new instruction functions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix postprocessing_funcs to remove invalid columns * Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid * Revert prompt caching changes from aed1b73c * update formatting * Update imports Approved-by: Sha Brown Approved-by: Praneel Panchigar
651 lines
50 KiB
Markdown
651 lines
50 KiB
Markdown
# Field Extraction Visual Flow Guide
|
|
|
|
## Complete System Flow Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ START: main.py │
|
|
│ │
|
|
│ 1. Load configuration (batch_id, fields, max_workers) │
|
|
│ 2. Initialize batch tracker │
|
|
│ 3. Read input contracts (from local folder or S3) │
|
|
│ 4. Filter contracts with "reimbursement" for B processing │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ Split Processing Paths │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓ ↓
|
|
╔═══════════════════════╗ ╔═══════════════════════╗
|
|
║ AC PROCESSING ║ ║ B PROCESSING ║
|
|
║ (Contract Fields) ║ ║ (Reimbursement) ║
|
|
╚═══════════════════════╝ ╚═══════════════════════╝
|
|
↓ ↓
|
|
[See AC Flow Diagram] [See B Flow Diagram]
|
|
↓ ↓
|
|
╔═══════════════════════╗ ╔═══════════════════════╗
|
|
║ AC Output CSV ║ ║ B Output CSV ║
|
|
╚═══════════════════════╝ ╚═══════════════════════╝
|
|
↓ ↓
|
|
└───────────┬───────────────┘
|
|
↓
|
|
┌───────────────────────────────┐
|
|
│ Consolidation & QA/QC │
|
|
│ - Merge AC + B data │
|
|
│ - Quality checks │
|
|
│ - Generate reports │
|
|
└───────────────────────────────┘
|
|
↓
|
|
┌───────────────────────────────┐
|
|
│ Final Outputs │
|
|
│ - ABC combined CSV │
|
|
│ - QA/QC Excel report │
|
|
└───────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## AC Processing Detailed Flow
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ AC PROCESSING START │
|
|
│ (file_processing.py: run_ac_prompts) │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 1: PREPROCESSING (preprocess.py) │
|
|
│ │
|
|
│ Input: Raw contract text │
|
|
│ │
|
|
│ Actions: │
|
|
│ ├─ Clean text (remove symbols, fix newlines) │
|
|
│ ├─ Split into pages │
|
|
│ └─ Create smart chunks based on keywords │
|
|
│ │
|
|
│ Output: text_dict (pages), ac_chunks (focused text per field) │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 2: REGEX EXTRACTION │
|
|
│ │
|
|
│ Extract using patterns: │
|
|
│ ├─ TIN (Tax ID): XX-XXXXXXX format │
|
|
│ ├─ NPI (Provider ID): 10-digit number │
|
|
│ └─ Other IRS names from specific pages │
|
|
│ │
|
|
│ Output: Partial ac_answers_dict │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 3: SMART CHUNKED PROMPTS (smart_chunking_funcs.py) │
|
|
│ │
|
|
│ For each field group with keyword matches: │
|
|
│ │
|
|
│ ┌─────────────────────────────────────────────────┐ │
|
|
│ │ Example: CONTRACT_EFFECTIVE_DT │ │
|
|
│ ├─────────────────────────────────────────────────┤ │
|
|
│ │ 1. Keywords found: "effective date", "execution"│ │
|
|
│ │ 2. Chunk contains: Pages 1-3 │ │
|
|
│ │ 3. Build prompt with focused question │ │
|
|
│ │ 4. Send to Claude AI │ │
|
|
│ │ 5. Parse response: "2024-01-15" │ │
|
|
│ │ 6. Store in ac_answers_dict │ │
|
|
│ └─────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Repeat for ~40 field groups │
|
|
│ │
|
|
│ Output: ac_answers_dict with most fields filled │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 4: FULL CONTEXT PROMPTS │
|
|
│ │
|
|
│ For fields with no keyword matches: │
|
|
│ ├─ Use entire contract as context │
|
|
│ ├─ More expensive but comprehensive │
|
|
│ └─ Ensures rare fields aren't missed │
|
|
│ │
|
|
│ Output: Remaining fields filled │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 5: CONDITIONAL PROMPTS │
|
|
│ │
|
|
│ Context-based derivations: │
|
|
│ ├─ Use existing answers to inform new questions │
|
|
│ └─ Example: If effective date is N/A, check signature date │
|
|
│ │
|
|
│ Output: Final ac_answers_dict │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 6: POSTPROCESSING (postprocess.py: ac_postprocess) │
|
|
│ │
|
|
│ Cleanup and standardization: │
|
|
│ ├─ Convert dates to YYYY-MM-DD format │
|
|
│ ├─ Clean TIN/NPI formatting │
|
|
│ ├─ Derive indicator fields (Y/N from text) │
|
|
│ ├─ Add metadata (filename, page count, parent code) │
|
|
│ ├─ Remove null/invalid values │
|
|
│ ├─ Rename columns to user-friendly names │
|
|
│ └─ Reorder columns │
|
|
│ │
|
|
│ Output: Clean DataFrame │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 7: WRITE OUTPUT │
|
|
│ │
|
|
│ Save to: {batch_id}-AC.csv │
|
|
│ Format: One row per contract, ~45 columns │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## B Processing Detailed Flow
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ B PROCESSING START │
|
|
│ (file_processing.py: run_b_prompts) │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 1: PREPROCESSING (preprocess.py) │
|
|
│ │
|
|
│ Input: Raw contract text │
|
|
│ │
|
|
│ Actions: │
|
|
│ ├─ Clean text (symbols, newlines) │
|
|
│ ├─ Split into pages │
|
|
│ ├─ Identify exhibit pages (headers like "Exhibit", "Attachment") │
|
|
│ ├─ Chunk consecutive pages together │
|
|
│ └─ Align and format tables │
|
|
│ │
|
|
│ Output: text_dict with chunked pages and aligned tables │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 2: BOTTOM-UP PRIMARY (one_to_n_funcs.py) │
|
|
│ │
|
|
│ Process each page/chunk with reimbursement info: │
|
|
│ │
|
|
│ ┌─────────────────────────────────────────────────────────────┐ │
|
|
│ │ For Page/Chunk: │ │
|
|
│ │ │ │
|
|
│ │ Prompt (prompts.py: BOTTOM_UP_PRIMARY): │ │
|
|
│ │ "Extract EVERY reimbursement term on this page" │ │
|
|
│ │ │ │
|
|
│ │ Input Page Example: │ │
|
|
│ │ ┌────────────────────────────────────────┐ │ │
|
|
│ │ │ PHYSICIAN SERVICES │ │ │
|
|
│ │ │ Office Visits: 110% Medicare │ │ │
|
|
│ │ │ Surgery: 120% Medicare │ │ │
|
|
│ │ │ Lab Tests: $25.00 per test │ │ │
|
|
│ │ └────────────────────────────────────────┘ │ │
|
|
│ │ │ │
|
|
│ │ AI Extraction: │ │
|
|
│ │ [ │ │
|
|
│ │ { │ │
|
|
│ │ "FULL_SERVICE": "Office Visits", │ │
|
|
│ │ "FULL_METHODOLOGY": "110% Medicare", │ │
|
|
│ │ "PROV_TYPE": "Professional" │ │
|
|
│ │ }, │ │
|
|
│ │ { │ │
|
|
│ │ "FULL_SERVICE": "Surgery", │ │
|
|
│ │ "FULL_METHODOLOGY": "120% Medicare", │ │
|
|
│ │ "PROV_TYPE": "Professional" │ │
|
|
│ │ }, │ │
|
|
│ │ { │ │
|
|
│ │ "FULL_SERVICE": "Lab Tests", │ │
|
|
│ │ "FULL_METHODOLOGY": "$25.00 per test", │ │
|
|
│ │ "PROV_TYPE": "Ancillary" │ │
|
|
│ │ } │ │
|
|
│ │ ] │ │
|
|
│ └─────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Output: List of dictionaries (one per service found) │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 3: BOTTOM-UP SECONDARY (one_to_n_funcs.py) │
|
|
│ │
|
|
│ For each extracted service, break down methodology: │
|
|
│ │
|
|
│ ┌─────────────────────────────────────────────────────────────┐ │
|
|
│ │ Methodology Breakout │ │
|
|
│ │ │ │
|
|
│ │ Prompt (prompts.py: BOTTOM_UP_METHODOLOGY_BREAKOUT): │ │
|
|
│ │ "Parse this methodology into structured fields" │ │
|
|
│ │ │ │
|
|
│ │ Input: "Lesser of billed charges or 110% Medicare" │ │
|
|
│ │ │ │
|
|
│ │ AI Breakdown: │ │
|
|
│ │ { │ │
|
|
│ │ "LESSER": "Y", │ │
|
|
│ │ "RATE_STANDARD": "110% Medicare", │ │
|
|
│ │ "RATE_SHORT": "1.1", │ │
|
|
│ │ "FLAT_FEE_STANDARD": "N/A", │ │
|
|
│ │ "LESSER_RATE": "100% of BC", │ │
|
|
│ │ "NOT_TO_EXCEED": "N/A" │ │
|
|
│ │ } │ │
|
|
│ └─────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ ┌─────────────────────────────────────────────────────────────┐ │
|
|
│ │ Escalator Detection │ │
|
|
│ │ │ │
|
|
│ │ Prompt (prompts.py: BOTTOM_UP_ESCALATOR): │ │
|
|
│ │ "Does this rate increase over time?" │ │
|
|
│ │ │ │
|
|
│ │ Output: │ │
|
|
│ │ { │ │
|
|
│ │ "RATE_ESCALATOR_IND": "Y", │ │
|
|
│ │ "RATE_ESCALATOR_DT": "2025-01-01" │ │
|
|
│ │ } │ │
|
|
│ └─────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Output: Enhanced list with structured rate information │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 4: TOP-DOWN EXTRACTION (one_to_n_funcs.py) │
|
|
│ │
|
|
│ Extract page-level context that applies to all services: │
|
|
│ │
|
|
│ Prompt (prompts.py: TOP_DOWN_PRIMARY): │
|
|
│ ├─ EXHIBIT: Full exhibit/attachment name │
|
|
│ ├─ CONTRACT_LOB: Line of Business (Medicare, Medicaid, etc.) │
|
|
│ ├─ DEFAULT_TERM: What happens if no rate specified │
|
|
│ ├─ CDM_IND: Chargemaster neutralization? │
|
|
│ └─ ADD_ON_REIMBURSEMENT_LANGUAGE: Extra payment clauses │
|
|
│ │
|
|
│ Example Output: │
|
|
│ { │
|
|
│ "EXHIBIT": "Attachment C: Medicare Advantage", │
|
|
│ "CONTRACT_LOB": "Medicare Advantage", │
|
|
│ "DEFAULT_TERM": "100% of Medicare Fee Schedule", │
|
|
│ "CDM_IND": "N" │
|
|
│ } │
|
|
│ │
|
|
│ This context is merged with ALL services on that page │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 5: CONDITIONAL PROMPTS (conditional_funcs.py) │
|
|
│ │
|
|
│ Fill in additional service-level details: │
|
|
│ │
|
|
│ ├─ IP/OP: Inpatient or Outpatient? │
|
|
│ │ Prompt: "Is this service IP or OP?" │
|
|
│ │ Uses keywords like "inpatient", "outpatient" │
|
|
│ │ │
|
|
│ ├─ PROV_TYPE_2: More specific provider category │
|
|
│ │ Prompt: "What specific provider type?" │
|
|
│ │ Options: PCP, Specialist, ER, Clinic, etc. │
|
|
│ │ │
|
|
│ └─ LOB_CHECK: Assign single LOB to each service │
|
|
│ Prompt: "Which LOB does this service fall under?" │
|
|
│ Narrows from page-level to service-level │
|
|
│ │
|
|
│ Output: Fully enriched service list │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 6: POSTPROCESSING (postprocess.py: b_postprocess) │
|
|
│ │
|
|
│ Cleanup and standardization: │
|
|
│ ├─ Add metadata (filename, parent code, page count) │
|
|
│ ├─ Flag single code, multiple rates (SCMR) │
|
|
│ ├─ Filter out add-on services │
|
|
│ ├─ Clean methodology text │
|
|
│ ├─ Standardize provider types │
|
|
│ ├─ Clean lesser of rates │
|
|
│ ├─ Standardize Line of Business values │
|
|
│ ├─ Fix Y/N fields │
|
|
│ ├─ Rename columns │
|
|
│ └─ Reorder columns │
|
|
│ │
|
|
│ Output: Clean DataFrame │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ STEP 7: WRITE OUTPUT │
|
|
│ │
|
|
│ Save to: {batch_id}-B.csv │
|
|
│ Format: Multiple rows per contract (one per service), ~35 columns │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Prompt Flow for Key Extractions
|
|
|
|
### Contract Effective Date Extraction
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────┐
|
|
│ CONTRACT_EFFECTIVE_DT EXTRACTION │
|
|
└────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────┐
|
|
│ Smart Chunking │
|
|
│ Keywords: "effective date", "execution date" │
|
|
│ Result: Pages 1-3 contain keywords │
|
|
└────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────┐
|
|
│ Build Prompt (prompts.py: get_effective_date_prompt) │
|
|
│ │
|
|
│ Context: [Page 1-3 text] │
|
|
│ │
|
|
│ Instruction: │
|
|
│ "Extract ONLY the effective date of the contract │
|
|
│ Follow these rules: │
|
|
│ 1. Look for 'Effective Date:' label │
|
|
│ 2. Don't use signature date │
|
|
│ 3. Return in YYYY-MM-DD format or N/A │
|
|
│ 4. Enclose answer in |pipes|" │
|
|
│ │
|
|
│ Example Text: │
|
|
│ "This Agreement is effective as of January 15, │
|
|
│ 2024 ('Effective Date')..." │
|
|
└────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────┐
|
|
│ Send to Claude AI │
|
|
│ Model: Claude 3.5 Sonnet │
|
|
└────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────┐
|
|
│ AI Response │
|
|
│ "...reviewing the contract text... │
|
|
│ Answer: |2024-01-15|" │
|
|
└────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────┐
|
|
│ Parse Response │
|
|
│ - Extract text between |pipes| │
|
|
│ - Result: "2024-01-15" │
|
|
└────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────┐
|
|
│ Validate & Convert │
|
|
│ - Check format is YYYY-MM-DD │
|
|
│ - If invalid, set to N/A │
|
|
└────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────┐
|
|
│ Store in Output │
|
|
│ CONTRACT_EFFECTIVE_DT = "2024-01-15" │
|
|
└────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Reimbursement Methodology Extraction
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ REIMBURSEMENT METHODOLOGY EXTRACTION │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ Page Text (Example) │
|
|
│ ──────────────────────────────────────────────────────────── │
|
|
│ COMPENSATION SCHEDULE - PROFESSIONAL SERVICES │
|
|
│ │
|
|
│ CCA shall reimburse Provider for Covered Services rendered │
|
|
│ to Covered Persons the lesser of: │
|
|
│ (i) Allowable Charges; or │
|
|
│ (ii) The rates specified below: │
|
|
│ │
|
|
│ Service Category | Contracted Rate │
|
|
│ ──────────────────────────────────────────── │
|
|
│ Office Visits | 110% of Medicare Fee Schedule │
|
|
│ Preventive Care | 100% of Medicare Fee Schedule │
|
|
│ Specialist Consults | 125% of Medicare Fee Schedule │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ STAGE 1: Bottom-Up Primary │
|
|
│ Prompt: BOTTOM_UP_PRIMARY │
|
|
│ │
|
|
│ Instruction: "Extract EVERY reimbursement term" │
|
|
│ │
|
|
│ Key Rules: │
|
|
│ - Include complete "lesser of" statements │
|
|
│ - Capture table headers for context │
|
|
│ - One entry per service │
|
|
│ - Include provider type │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ AI Response (JSON) │
|
|
│ [ │
|
|
│ { │
|
|
│ "FULL_SERVICE": "Office Visits", │
|
|
│ "SUBHEADER": "N/A", │
|
|
│ "FULL_METHODOLOGY": "Lesser of: (i) Allowable Charges; │
|
|
│ or (ii) The rates specified below: Service Category | │
|
|
│ Contracted Rate\nOffice Visits | 110% of Medicare │
|
|
│ Fee Schedule", │
|
|
│ "PROV_TYPE": "Professional" │
|
|
│ }, │
|
|
│ { │
|
|
│ "FULL_SERVICE": "Preventive Care", │
|
|
│ "SUBHEADER": "N/A", │
|
|
│ "FULL_METHODOLOGY": "Lesser of: (i) Allowable Charges; │
|
|
│ or (ii) The rates specified below: Service Category | │
|
|
│ Contracted Rate\nPreventive Care | 100% of Medicare │
|
|
│ Fee Schedule", │
|
|
│ "PROV_TYPE": "Professional" │
|
|
│ }, │
|
|
│ { │
|
|
│ "FULL_SERVICE": "Specialist Consults", │
|
|
│ "SUBHEADER": "N/A", │
|
|
│ "FULL_METHODOLOGY": "Lesser of: (i) Allowable Charges; │
|
|
│ or (ii) The rates specified below: Service Category | │
|
|
│ Contracted Rate\nSpecialist Consults | 125% of │
|
|
│ Medicare Fee Schedule", │
|
|
│ "PROV_TYPE": "Professional" │
|
|
│ } │
|
|
│ ] │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ STAGE 2: Bottom-Up Secondary (Methodology Breakout) │
|
|
│ │
|
|
│ For each service, parse methodology: │
|
|
│ │
|
|
│ Input (Office Visits): │
|
|
│ "Lesser of: (i) Allowable Charges; or (ii) 110% of Medicare │
|
|
│ Fee Schedule" │
|
|
│ │
|
|
│ Prompt: BOTTOM_UP_METHODOLOGY_BREAKOUT │
|
|
│ "Break this methodology into structured fields" │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ AI Response (Office Visits) │
|
|
│ { │
|
|
│ "LESSER": "Y", │
|
|
│ "RATE_STANDARD": "110% Medicare", │
|
|
│ "RATE_SHORT": "1.1", │
|
|
│ "FLAT_FEE_STANDARD": "N/A", │
|
|
│ "LESSER_RATE": "100% of AC", // AC = Allowable Charges │
|
|
│ "NOT_TO_EXCEED": "N/A" │
|
|
│ } │
|
|
│ │
|
|
│ Repeat for Preventive Care → RATE_STANDARD: "100% Medicare" │
|
|
│ Repeat for Specialist → RATE_STANDARD: "125% Medicare" │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ STAGE 3: Top-Down Context │
|
|
│ │
|
|
│ Extract page-level info: │
|
|
│ { │
|
|
│ "EXHIBIT": "COMPENSATION SCHEDULE - PROFESSIONAL SERVICES", │
|
|
│ "CONTRACT_LOB": "Commercial", │
|
|
│ "DEFAULT_TERM": "N/A", │
|
|
│ "CDM_IND": "N" │
|
|
│ } │
|
|
│ │
|
|
│ Merge with each service │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────────────────────────────────┐
|
|
│ FINAL OUTPUT (Office Visits row) │
|
|
│ ──────────────────────────────────────────────────────────── │
|
|
│ Contract Name: "contract_123.pdf" │
|
|
│ Exhibit: "COMPENSATION SCHEDULE - PROFESSIONAL SERVICES" │
|
|
│ Line of Business: "Commercial" │
|
|
│ Full Service: "Office Visits" │
|
|
│ Provider Type: "Professional" │
|
|
│ Reimbursement Methodology: "Lesser of: (i) Allowable │
|
|
│ Charges; or (ii) 110% of Medicare Fee Schedule" │
|
|
│ Rate Standard: "110% Medicare" │
|
|
│ Rate Short: "1.1" │
|
|
│ Flat Fee: "N/A" │
|
|
│ Lesser Of: "Y" │
|
|
│ Lesser Of Rate: "100% of AC" │
|
|
│ Not to Exceed: "N/A" │
|
|
│ ... (additional columns) │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Error Handling Flow
|
|
|
|
```
|
|
┌────────────────────────────────────────────┐
|
|
│ File Processing Attempt │
|
|
└────────────────────────────────────────────┘
|
|
↓
|
|
┌──────────────────────┐
|
|
│ Success? │
|
|
└──────────────────────┘
|
|
↓ Yes ↓ No
|
|
┌─────────┐ ┌─────────────────────┐
|
|
│ Log │ │ Capture Error │
|
|
│ Success │ │ - Error message │
|
|
│ │ │ - Stack trace │
|
|
│ Update │ │ - Timestamp │
|
|
│ Tracking│ └─────────────────────┘
|
|
│ │ ↓
|
|
│ Continue│ ┌─────────────────────┐
|
|
│ to Next │ │ Log to Tracking │
|
|
└─────────┘ │ Status: Failed │
|
|
│ Error: [details] │
|
|
└─────────────────────┘
|
|
↓
|
|
┌─────────────────────┐
|
|
│ Mark for Self-Repair│
|
|
└─────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────┐
|
|
│ After All Files Processed │
|
|
└────────────────────────────────────┘
|
|
↓
|
|
┌────────────────────────────────────┐
|
|
│ Self-Repair Mode │
|
|
│ - Identify failed files │
|
|
│ - Retry processing │
|
|
│ - Consolidate outputs │
|
|
└────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Data Flow Summary
|
|
|
|
```
|
|
Raw Contract Text
|
|
↓
|
|
┌─────────────┐
|
|
│ Preprocessing│
|
|
│ - Clean │
|
|
│ - Split │
|
|
│ - Chunk │
|
|
└─────────────┘
|
|
↓
|
|
┌─────────────┐ ┌──────────────┐
|
|
│ AC Path │ │ B Path │
|
|
│ │ │ │
|
|
│ Smart Chunk │ │ Bottom-Up │
|
|
│ Prompts │ │ Primary │
|
|
│ ↓ │ │ ↓ │
|
|
│ Full Context│ │ Bottom-Up │
|
|
│ Prompts │ │ Secondary │
|
|
│ ↓ │ │ ↓ │
|
|
│ Conditional │ │ Top-Down │
|
|
│ Prompts │ │ ↓ │
|
|
│ ↓ │ │ Conditional │
|
|
│ Postprocess │ │ ↓ │
|
|
│ │ │ Postprocess │
|
|
└─────────────┘ └──────────────┘
|
|
↓ ↓
|
|
┌─────────────┐ ┌──────────────┐
|
|
│ AC CSV │ │ B CSV │
|
|
│ (1 row) │ │ (many rows) │
|
|
└─────────────┘ └──────────────┘
|
|
↓ ↓
|
|
└──────────┬──────────┘
|
|
↓
|
|
┌───────────────┐
|
|
│ Consolidation │
|
|
│ - Merge AC+B │
|
|
│ - QA/QC │
|
|
└───────────────┘
|
|
↓
|
|
┌───────────────┐
|
|
│ Final Outputs │
|
|
│ - ABC CSV │
|
|
│ - QA Report │
|
|
└───────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Parallel Processing Visualization
|
|
|
|
```
|
|
Main Process
|
|
│
|
|
┌────────────────┼────────────────┐
|
|
│ │ │
|
|
Worker 1 Worker 2 Worker 3
|
|
│ │ │
|
|
Contract A Contract D Contract G
|
|
↓ ↓ ↓
|
|
[AC + B] [AC + B] [AC + B]
|
|
Processing Processing Processing
|
|
↓ ↓ ↓
|
|
Success Success Failed
|
|
│ │ │
|
|
└────────────────┼────────────────┘
|
|
│
|
|
Track Results
|
|
│
|
|
┌────────────────┼────────────────┐
|
|
│ │ │
|
|
Worker 1 Worker 2 Worker 3
|
|
│ │ │
|
|
Contract B Contract E Contract H
|
|
↓ ↓ ↓
|
|
Processing Processing Processing
|
|
│ │ │
|
|
└────────────────┼────────────────┘
|
|
│
|
|
↓
|
|
Upload Progress (every 10 files)
|
|
│
|
|
↓
|
|
Continue...
|
|
```
|
|
|
|
---
|
|
|
|
This visual guide complements the detailed text guide by showing the exact flow of data and processing through the system in an easy-to-understand diagram format.
|