# Field Extraction Visual Flow Guide ## Complete System Flow Diagram ``` ┌─────────────────────────────────────────────────────────────────────┐ │ START: main.py │ │ │ │ 1. Load configuration (batch_id, fields, max_workers) │ │ 2. Initialize batch tracker │ │ 3. Read input contracts (from local folder or S3) │ │ 4. Filter contracts with "reimbursement" for B processing │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ Split Processing Paths │ └─────────────────────────────────────────────────────────────────────┘ ↓ ↓ ╔═══════════════════════╗ ╔═══════════════════════╗ ║ AC PROCESSING ║ ║ B PROCESSING ║ ║ (Contract Fields) ║ ║ (Reimbursement) ║ ╚═══════════════════════╝ ╚═══════════════════════╝ ↓ ↓ [See AC Flow Diagram] [See B Flow Diagram] ↓ ↓ ╔═══════════════════════╗ ╔═══════════════════════╗ ║ AC Output CSV ║ ║ B Output CSV ║ ╚═══════════════════════╝ ╚═══════════════════════╝ ↓ ↓ └───────────┬───────────────┘ ↓ ┌───────────────────────────────┐ │ Consolidation & QA/QC │ │ - Merge AC + B data │ │ - Quality checks │ │ - Generate reports │ └───────────────────────────────┘ ↓ ┌───────────────────────────────┐ │ Final Outputs │ │ - ABC combined CSV │ │ - QA/QC Excel report │ └───────────────────────────────┘ ``` --- ## AC Processing Detailed Flow ``` ┌─────────────────────────────────────────────────────────────────────┐ │ AC PROCESSING START │ │ (file_processing.py: run_ac_prompts) │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 1: PREPROCESSING (preprocess.py) │ │ │ │ Input: Raw contract text │ │ │ │ Actions: │ │ ├─ Clean text (remove symbols, fix newlines) │ │ ├─ Split into pages │ │ └─ Create smart chunks based on keywords │ │ │ │ Output: text_dict (pages), ac_chunks (focused text per field) │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 2: REGEX EXTRACTION │ │ │ │ Extract using patterns: │ │ ├─ TIN (Tax ID): XX-XXXXXXX format │ │ ├─ NPI (Provider ID): 10-digit number │ │ └─ Other IRS names from specific pages │ │ │ │ Output: Partial ac_answers_dict │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 3: SMART CHUNKED PROMPTS (smart_chunking_funcs.py) │ │ │ │ For each field group with keyword matches: │ │ │ │ ┌─────────────────────────────────────────────────┐ │ │ │ Example: CONTRACT_EFFECTIVE_DT │ │ │ ├─────────────────────────────────────────────────┤ │ │ │ 1. Keywords found: "effective date", "execution"│ │ │ │ 2. Chunk contains: Pages 1-3 │ │ │ │ 3. Build prompt with focused question │ │ │ │ 4. Send to Claude AI │ │ │ │ 5. Parse response: "2024-01-15" │ │ │ │ 6. Store in ac_answers_dict │ │ │ └─────────────────────────────────────────────────┘ │ │ │ │ Repeat for ~40 field groups │ │ │ │ Output: ac_answers_dict with most fields filled │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 4: FULL CONTEXT PROMPTS │ │ │ │ For fields with no keyword matches: │ │ ├─ Use entire contract as context │ │ ├─ More expensive but comprehensive │ │ └─ Ensures rare fields aren't missed │ │ │ │ Output: Remaining fields filled │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 5: CONDITIONAL PROMPTS │ │ │ │ Context-based derivations: │ │ ├─ Use existing answers to inform new questions │ │ └─ Example: If effective date is N/A, check signature date │ │ │ │ Output: Final ac_answers_dict │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 6: POSTPROCESSING (postprocess.py: ac_postprocess) │ │ │ │ Cleanup and standardization: │ │ ├─ Convert dates to YYYY-MM-DD format │ │ ├─ Clean TIN/NPI formatting │ │ ├─ Derive indicator fields (Y/N from text) │ │ ├─ Add metadata (filename, page count, parent code) │ │ ├─ Remove null/invalid values │ │ ├─ Rename columns to user-friendly names │ │ └─ Reorder columns │ │ │ │ Output: Clean DataFrame │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 7: WRITE OUTPUT │ │ │ │ Save to: {batch_id}-AC.csv │ │ Format: One row per contract, ~45 columns │ └─────────────────────────────────────────────────────────────────────┘ ``` --- ## B Processing Detailed Flow ``` ┌─────────────────────────────────────────────────────────────────────┐ │ B PROCESSING START │ │ (file_processing.py: run_b_prompts) │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 1: PREPROCESSING (preprocess.py) │ │ │ │ Input: Raw contract text │ │ │ │ Actions: │ │ ├─ Clean text (symbols, newlines) │ │ ├─ Split into pages │ │ ├─ Identify exhibit pages (headers like "Exhibit", "Attachment") │ │ ├─ Chunk consecutive pages together │ │ └─ Align and format tables │ │ │ │ Output: text_dict with chunked pages and aligned tables │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 2: BOTTOM-UP PRIMARY (one_to_n_funcs.py) │ │ │ │ Process each page/chunk with reimbursement info: │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ For Page/Chunk: │ │ │ │ │ │ │ │ Prompt (prompts.py: BOTTOM_UP_PRIMARY): │ │ │ │ "Extract EVERY reimbursement term on this page" │ │ │ │ │ │ │ │ Input Page Example: │ │ │ │ ┌────────────────────────────────────────┐ │ │ │ │ │ PHYSICIAN SERVICES │ │ │ │ │ │ Office Visits: 110% Medicare │ │ │ │ │ │ Surgery: 120% Medicare │ │ │ │ │ │ Lab Tests: $25.00 per test │ │ │ │ │ └────────────────────────────────────────┘ │ │ │ │ │ │ │ │ AI Extraction: │ │ │ │ [ │ │ │ │ { │ │ │ │ "FULL_SERVICE": "Office Visits", │ │ │ │ "FULL_METHODOLOGY": "110% Medicare", │ │ │ │ "PROV_TYPE": "Professional" │ │ │ │ }, │ │ │ │ { │ │ │ │ "FULL_SERVICE": "Surgery", │ │ │ │ "FULL_METHODOLOGY": "120% Medicare", │ │ │ │ "PROV_TYPE": "Professional" │ │ │ │ }, │ │ │ │ { │ │ │ │ "FULL_SERVICE": "Lab Tests", │ │ │ │ "FULL_METHODOLOGY": "$25.00 per test", │ │ │ │ "PROV_TYPE": "Ancillary" │ │ │ │ } │ │ │ │ ] │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Output: List of dictionaries (one per service found) │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 3: BOTTOM-UP SECONDARY (one_to_n_funcs.py) │ │ │ │ For each extracted service, break down methodology: │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Methodology Breakout │ │ │ │ │ │ │ │ Prompt (prompts.py: BOTTOM_UP_METHODOLOGY_BREAKOUT): │ │ │ │ "Parse this methodology into structured fields" │ │ │ │ │ │ │ │ Input: "Lesser of billed charges or 110% Medicare" │ │ │ │ │ │ │ │ AI Breakdown: │ │ │ │ { │ │ │ │ "LESSER": "Y", │ │ │ │ "RATE_STANDARD": "110% Medicare", │ │ │ │ "RATE_SHORT": "1.1", │ │ │ │ "FLAT_FEE_STANDARD": "N/A", │ │ │ │ "LESSER_RATE": "100% of BC", │ │ │ │ "NOT_TO_EXCEED": "N/A" │ │ │ │ } │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Escalator Detection │ │ │ │ │ │ │ │ Prompt (prompts.py: BOTTOM_UP_ESCALATOR): │ │ │ │ "Does this rate increase over time?" │ │ │ │ │ │ │ │ Output: │ │ │ │ { │ │ │ │ "RATE_ESCALATOR_IND": "Y", │ │ │ │ "RATE_ESCALATOR_DT": "2025-01-01" │ │ │ │ } │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Output: Enhanced list with structured rate information │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 4: TOP-DOWN EXTRACTION (one_to_n_funcs.py) │ │ │ │ Extract page-level context that applies to all services: │ │ │ │ Prompt (prompts.py: TOP_DOWN_PRIMARY): │ │ ├─ EXHIBIT: Full exhibit/attachment name │ │ ├─ CONTRACT_LOB: Line of Business (Medicare, Medicaid, etc.) │ │ ├─ DEFAULT_TERM: What happens if no rate specified │ │ ├─ CDM_IND: Chargemaster neutralization? │ │ └─ ADD_ON_REIMBURSEMENT_LANGUAGE: Extra payment clauses │ │ │ │ Example Output: │ │ { │ │ "EXHIBIT": "Attachment C: Medicare Advantage", │ │ "CONTRACT_LOB": "Medicare Advantage", │ │ "DEFAULT_TERM": "100% of Medicare Fee Schedule", │ │ "CDM_IND": "N" │ │ } │ │ │ │ This context is merged with ALL services on that page │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 5: CONDITIONAL PROMPTS (conditional_funcs.py) │ │ │ │ Fill in additional service-level details: │ │ │ │ ├─ IP/OP: Inpatient or Outpatient? │ │ │ Prompt: "Is this service IP or OP?" │ │ │ Uses keywords like "inpatient", "outpatient" │ │ │ │ │ ├─ PROV_TYPE_2: More specific provider category │ │ │ Prompt: "What specific provider type?" │ │ │ Options: PCP, Specialist, ER, Clinic, etc. │ │ │ │ │ └─ LOB_CHECK: Assign single LOB to each service │ │ Prompt: "Which LOB does this service fall under?" │ │ Narrows from page-level to service-level │ │ │ │ Output: Fully enriched service list │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 6: POSTPROCESSING (postprocess.py: b_postprocess) │ │ │ │ Cleanup and standardization: │ │ ├─ Add metadata (filename, parent code, page count) │ │ ├─ Flag single code, multiple rates (SCMR) │ │ ├─ Filter out add-on services │ │ ├─ Clean methodology text │ │ ├─ Standardize provider types │ │ ├─ Clean lesser of rates │ │ ├─ Standardize Line of Business values │ │ ├─ Fix Y/N fields │ │ ├─ Rename columns │ │ └─ Reorder columns │ │ │ │ Output: Clean DataFrame │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────────┐ │ STEP 7: WRITE OUTPUT │ │ │ │ Save to: {batch_id}-B.csv │ │ Format: Multiple rows per contract (one per service), ~35 columns │ └─────────────────────────────────────────────────────────────────────┘ ``` --- ## Prompt Flow for Key Extractions ### Contract Effective Date Extraction ``` ┌────────────────────────────────────────────────────┐ │ CONTRACT_EFFECTIVE_DT EXTRACTION │ └────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────┐ │ Smart Chunking │ │ Keywords: "effective date", "execution date" │ │ Result: Pages 1-3 contain keywords │ └────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────┐ │ Build Prompt (prompts.py: get_effective_date_prompt) │ │ │ │ Context: [Page 1-3 text] │ │ │ │ Instruction: │ │ "Extract ONLY the effective date of the contract │ │ Follow these rules: │ │ 1. Look for 'Effective Date:' label │ │ 2. Don't use signature date │ │ 3. Return in YYYY-MM-DD format or N/A │ │ 4. Enclose answer in |pipes|" │ │ │ │ Example Text: │ │ "This Agreement is effective as of January 15, │ │ 2024 ('Effective Date')..." │ └────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────┐ │ Send to Claude AI │ │ Model: Claude 3.5 Sonnet │ └────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────┐ │ AI Response │ │ "...reviewing the contract text... │ │ Answer: |2024-01-15|" │ └────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────┐ │ Parse Response │ │ - Extract text between |pipes| │ │ - Result: "2024-01-15" │ └────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────┐ │ Validate & Convert │ │ - Check format is YYYY-MM-DD │ │ - If invalid, set to N/A │ └────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────┐ │ Store in Output │ │ CONTRACT_EFFECTIVE_DT = "2024-01-15" │ └────────────────────────────────────────────────────┘ ``` ### Reimbursement Methodology Extraction ``` ┌────────────────────────────────────────────────────────────────┐ │ REIMBURSEMENT METHODOLOGY EXTRACTION │ └────────────────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────────────────┐ │ Page Text (Example) │ │ ──────────────────────────────────────────────────────────── │ │ COMPENSATION SCHEDULE - PROFESSIONAL SERVICES │ │ │ │ CCA shall reimburse Provider for Covered Services rendered │ │ to Covered Persons the lesser of: │ │ (i) Allowable Charges; or │ │ (ii) The rates specified below: │ │ │ │ Service Category | Contracted Rate │ │ ──────────────────────────────────────────── │ │ Office Visits | 110% of Medicare Fee Schedule │ │ Preventive Care | 100% of Medicare Fee Schedule │ │ Specialist Consults | 125% of Medicare Fee Schedule │ └────────────────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────────────────┐ │ STAGE 1: Bottom-Up Primary │ │ Prompt: BOTTOM_UP_PRIMARY │ │ │ │ Instruction: "Extract EVERY reimbursement term" │ │ │ │ Key Rules: │ │ - Include complete "lesser of" statements │ │ - Capture table headers for context │ │ - One entry per service │ │ - Include provider type │ └────────────────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────────────────┐ │ AI Response (JSON) │ │ [ │ │ { │ │ "FULL_SERVICE": "Office Visits", │ │ "SUBHEADER": "N/A", │ │ "FULL_METHODOLOGY": "Lesser of: (i) Allowable Charges; │ │ or (ii) The rates specified below: Service Category | │ │ Contracted Rate\nOffice Visits | 110% of Medicare │ │ Fee Schedule", │ │ "PROV_TYPE": "Professional" │ │ }, │ │ { │ │ "FULL_SERVICE": "Preventive Care", │ │ "SUBHEADER": "N/A", │ │ "FULL_METHODOLOGY": "Lesser of: (i) Allowable Charges; │ │ or (ii) The rates specified below: Service Category | │ │ Contracted Rate\nPreventive Care | 100% of Medicare │ │ Fee Schedule", │ │ "PROV_TYPE": "Professional" │ │ }, │ │ { │ │ "FULL_SERVICE": "Specialist Consults", │ │ "SUBHEADER": "N/A", │ │ "FULL_METHODOLOGY": "Lesser of: (i) Allowable Charges; │ │ or (ii) The rates specified below: Service Category | │ │ Contracted Rate\nSpecialist Consults | 125% of │ │ Medicare Fee Schedule", │ │ "PROV_TYPE": "Professional" │ │ } │ │ ] │ └────────────────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────────────────┐ │ STAGE 2: Bottom-Up Secondary (Methodology Breakout) │ │ │ │ For each service, parse methodology: │ │ │ │ Input (Office Visits): │ │ "Lesser of: (i) Allowable Charges; or (ii) 110% of Medicare │ │ Fee Schedule" │ │ │ │ Prompt: BOTTOM_UP_METHODOLOGY_BREAKOUT │ │ "Break this methodology into structured fields" │ └────────────────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────────────────┐ │ AI Response (Office Visits) │ │ { │ │ "LESSER": "Y", │ │ "RATE_STANDARD": "110% Medicare", │ │ "RATE_SHORT": "1.1", │ │ "FLAT_FEE_STANDARD": "N/A", │ │ "LESSER_RATE": "100% of AC", // AC = Allowable Charges │ │ "NOT_TO_EXCEED": "N/A" │ │ } │ │ │ │ Repeat for Preventive Care → RATE_STANDARD: "100% Medicare" │ │ Repeat for Specialist → RATE_STANDARD: "125% Medicare" │ └────────────────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────────────────┐ │ STAGE 3: Top-Down Context │ │ │ │ Extract page-level info: │ │ { │ │ "EXHIBIT": "COMPENSATION SCHEDULE - PROFESSIONAL SERVICES", │ │ "CONTRACT_LOB": "Commercial", │ │ "DEFAULT_TERM": "N/A", │ │ "CDM_IND": "N" │ │ } │ │ │ │ Merge with each service │ └────────────────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────────────────┐ │ FINAL OUTPUT (Office Visits row) │ │ ──────────────────────────────────────────────────────────── │ │ Contract Name: "contract_123.pdf" │ │ Exhibit: "COMPENSATION SCHEDULE - PROFESSIONAL SERVICES" │ │ Line of Business: "Commercial" │ │ Full Service: "Office Visits" │ │ Provider Type: "Professional" │ │ Reimbursement Methodology: "Lesser of: (i) Allowable │ │ Charges; or (ii) 110% of Medicare Fee Schedule" │ │ Rate Standard: "110% Medicare" │ │ Rate Short: "1.1" │ │ Flat Fee: "N/A" │ │ Lesser Of: "Y" │ │ Lesser Of Rate: "100% of AC" │ │ Not to Exceed: "N/A" │ │ ... (additional columns) │ └────────────────────────────────────────────────────────────────┘ ``` --- ## Error Handling Flow ``` ┌────────────────────────────────────────────┐ │ File Processing Attempt │ └────────────────────────────────────────────┘ ↓ ┌──────────────────────┐ │ Success? │ └──────────────────────┘ ↓ Yes ↓ No ┌─────────┐ ┌─────────────────────┐ │ Log │ │ Capture Error │ │ Success │ │ - Error message │ │ │ │ - Stack trace │ │ Update │ │ - Timestamp │ │ Tracking│ └─────────────────────┘ │ │ ↓ │ Continue│ ┌─────────────────────┐ │ to Next │ │ Log to Tracking │ └─────────┘ │ Status: Failed │ │ Error: [details] │ └─────────────────────┘ ↓ ┌─────────────────────┐ │ Mark for Self-Repair│ └─────────────────────┘ ↓ ┌────────────────────────────────────┐ │ After All Files Processed │ └────────────────────────────────────┘ ↓ ┌────────────────────────────────────┐ │ Self-Repair Mode │ │ - Identify failed files │ │ - Retry processing │ │ - Consolidate outputs │ └────────────────────────────────────┘ ``` --- ## Data Flow Summary ``` Raw Contract Text ↓ ┌─────────────┐ │ Preprocessing│ │ - Clean │ │ - Split │ │ - Chunk │ └─────────────┘ ↓ ┌─────────────┐ ┌──────────────┐ │ AC Path │ │ B Path │ │ │ │ │ │ Smart Chunk │ │ Bottom-Up │ │ Prompts │ │ Primary │ │ ↓ │ │ ↓ │ │ Full Context│ │ Bottom-Up │ │ Prompts │ │ Secondary │ │ ↓ │ │ ↓ │ │ Conditional │ │ Top-Down │ │ Prompts │ │ ↓ │ │ ↓ │ │ Conditional │ │ Postprocess │ │ ↓ │ │ │ │ Postprocess │ └─────────────┘ └──────────────┘ ↓ ↓ ┌─────────────┐ ┌──────────────┐ │ AC CSV │ │ B CSV │ │ (1 row) │ │ (many rows) │ └─────────────┘ └──────────────┘ ↓ ↓ └──────────┬──────────┘ ↓ ┌───────────────┐ │ Consolidation │ │ - Merge AC+B │ │ - QA/QC │ └───────────────┘ ↓ ┌───────────────┐ │ Final Outputs │ │ - ABC CSV │ │ - QA Report │ └───────────────┘ ``` --- ## Parallel Processing Visualization ``` Main Process │ ┌────────────────┼────────────────┐ │ │ │ Worker 1 Worker 2 Worker 3 │ │ │ Contract A Contract D Contract G ↓ ↓ ↓ [AC + B] [AC + B] [AC + B] Processing Processing Processing ↓ ↓ ↓ Success Success Failed │ │ │ └────────────────┼────────────────┘ │ Track Results │ ┌────────────────┼────────────────┐ │ │ │ Worker 1 Worker 2 Worker 3 │ │ │ Contract B Contract E Contract H ↓ ↓ ↓ Processing Processing Processing │ │ │ └────────────────┼────────────────┘ │ ↓ Upload Progress (every 10 files) │ ↓ Continue... ``` --- This visual guide complements the detailed text guide by showing the exact flow of data and processing through the system in an easy-to-understand diagram format.