Katon Minhas
afb6d5185d
Feature/lesser table caching refactor hybrid * chore: Remove unused duplicate main.py from shared pipeline * fix: Correct crosswalk paths in aarete_derived.py * chore: Remove unused documentation files from fieldExtraction * docs: Add documentation files to documentation folder * docs: Update README with uv setup, expanded project structure, and branching conventions * docs: Add uv installation steps with Ubuntu/WSL emphasis * Enable prompt caching for all remaining LLM calls - Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE, EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH, SPECIAL_CASE_ASSIGNMENT - Update all invoke_claude() calls in saas and clover pipelines to use cache=True with corresponding _INSTRUCTION() functions - Add new instructions to get_cacheable_instructions() for cache warming - Update tests for new instruction functions Functions now using caching: - prompt_exhibit_level - prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF) - prompt_fee_schedule_breakout - prompt_grouper_breakout - prompt_special_case_assignment - prompt_exhibit_linkage - prompt_exhibit_header - prompt_smart_chunked (ONE_TO_ONE templates) - prompt_date_fix - prompt_derived_term_date - prompt_exhibit_title_match - provider_name_match_check 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Reorder * feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction - Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction - Extract full paragraph text of offset/recoupment provisions from contracts - Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence - Fix reorder_columns to preserve extra columns not in COLUMN_ORDER - Update QC/QA output path to outputs/qc_qa/ * fix: Update dev deps and test assertions for QC/QA output path - Add pytest/pytest-mock to dev dependencies for mypy type checking - Update test assertions to expect outputs/qc_qa instead of qa_qc_output * style: Apply black formatting to prompt_templates.py * Merge main, move scripts * Archive some scripts * update py version * remove .py version file * Remove ASCII characters * Restore testbed code * restore tracking * Update testbed metrics * Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT - Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification - Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination - Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification - Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT - Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK - Add new instructions to get_cacheable_instructions() for cache warming - Add unit tests for new instruction functions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix postprocessing_funcs to remove invalid columns * Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid * Revert prompt caching changes from aed1b73c * update formatting * Update imports Approved-by: Sha Brown Approved-by: Praneel Panchigar
50 KiB
50 KiB
Field Extraction Visual Flow Guide
Complete System Flow Diagram
┌─────────────────────────────────────────────────────────────────────┐
│ START: main.py │
│ │
│ 1. Load configuration (batch_id, fields, max_workers) │
│ 2. Initialize batch tracker │
│ 3. Read input contracts (from local folder or S3) │
│ 4. Filter contracts with "reimbursement" for B processing │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ Split Processing Paths │
└─────────────────────────────────────────────────────────────────────┘
↓ ↓
╔═══════════════════════╗ ╔═══════════════════════╗
║ AC PROCESSING ║ ║ B PROCESSING ║
║ (Contract Fields) ║ ║ (Reimbursement) ║
╚═══════════════════════╝ ╚═══════════════════════╝
↓ ↓
[See AC Flow Diagram] [See B Flow Diagram]
↓ ↓
╔═══════════════════════╗ ╔═══════════════════════╗
║ AC Output CSV ║ ║ B Output CSV ║
╚═══════════════════════╝ ╚═══════════════════════╝
↓ ↓
└───────────┬───────────────┘
↓
┌───────────────────────────────┐
│ Consolidation & QA/QC │
│ - Merge AC + B data │
│ - Quality checks │
│ - Generate reports │
└───────────────────────────────┘
↓
┌───────────────────────────────┐
│ Final Outputs │
│ - ABC combined CSV │
│ - QA/QC Excel report │
└───────────────────────────────┘
AC Processing Detailed Flow
┌─────────────────────────────────────────────────────────────────────┐
│ AC PROCESSING START │
│ (file_processing.py: run_ac_prompts) │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 1: PREPROCESSING (preprocess.py) │
│ │
│ Input: Raw contract text │
│ │
│ Actions: │
│ ├─ Clean text (remove symbols, fix newlines) │
│ ├─ Split into pages │
│ └─ Create smart chunks based on keywords │
│ │
│ Output: text_dict (pages), ac_chunks (focused text per field) │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 2: REGEX EXTRACTION │
│ │
│ Extract using patterns: │
│ ├─ TIN (Tax ID): XX-XXXXXXX format │
│ ├─ NPI (Provider ID): 10-digit number │
│ └─ Other IRS names from specific pages │
│ │
│ Output: Partial ac_answers_dict │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 3: SMART CHUNKED PROMPTS (smart_chunking_funcs.py) │
│ │
│ For each field group with keyword matches: │
│ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Example: CONTRACT_EFFECTIVE_DT │ │
│ ├─────────────────────────────────────────────────┤ │
│ │ 1. Keywords found: "effective date", "execution"│ │
│ │ 2. Chunk contains: Pages 1-3 │ │
│ │ 3. Build prompt with focused question │ │
│ │ 4. Send to Claude AI │ │
│ │ 5. Parse response: "2024-01-15" │ │
│ │ 6. Store in ac_answers_dict │ │
│ └─────────────────────────────────────────────────┘ │
│ │
│ Repeat for ~40 field groups │
│ │
│ Output: ac_answers_dict with most fields filled │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 4: FULL CONTEXT PROMPTS │
│ │
│ For fields with no keyword matches: │
│ ├─ Use entire contract as context │
│ ├─ More expensive but comprehensive │
│ └─ Ensures rare fields aren't missed │
│ │
│ Output: Remaining fields filled │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 5: CONDITIONAL PROMPTS │
│ │
│ Context-based derivations: │
│ ├─ Use existing answers to inform new questions │
│ └─ Example: If effective date is N/A, check signature date │
│ │
│ Output: Final ac_answers_dict │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 6: POSTPROCESSING (postprocess.py: ac_postprocess) │
│ │
│ Cleanup and standardization: │
│ ├─ Convert dates to YYYY-MM-DD format │
│ ├─ Clean TIN/NPI formatting │
│ ├─ Derive indicator fields (Y/N from text) │
│ ├─ Add metadata (filename, page count, parent code) │
│ ├─ Remove null/invalid values │
│ ├─ Rename columns to user-friendly names │
│ └─ Reorder columns │
│ │
│ Output: Clean DataFrame │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 7: WRITE OUTPUT │
│ │
│ Save to: {batch_id}-AC.csv │
│ Format: One row per contract, ~45 columns │
└─────────────────────────────────────────────────────────────────────┘
B Processing Detailed Flow
┌─────────────────────────────────────────────────────────────────────┐
│ B PROCESSING START │
│ (file_processing.py: run_b_prompts) │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 1: PREPROCESSING (preprocess.py) │
│ │
│ Input: Raw contract text │
│ │
│ Actions: │
│ ├─ Clean text (symbols, newlines) │
│ ├─ Split into pages │
│ ├─ Identify exhibit pages (headers like "Exhibit", "Attachment") │
│ ├─ Chunk consecutive pages together │
│ └─ Align and format tables │
│ │
│ Output: text_dict with chunked pages and aligned tables │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 2: BOTTOM-UP PRIMARY (one_to_n_funcs.py) │
│ │
│ Process each page/chunk with reimbursement info: │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ For Page/Chunk: │ │
│ │ │ │
│ │ Prompt (prompts.py: BOTTOM_UP_PRIMARY): │ │
│ │ "Extract EVERY reimbursement term on this page" │ │
│ │ │ │
│ │ Input Page Example: │ │
│ │ ┌────────────────────────────────────────┐ │ │
│ │ │ PHYSICIAN SERVICES │ │ │
│ │ │ Office Visits: 110% Medicare │ │ │
│ │ │ Surgery: 120% Medicare │ │ │
│ │ │ Lab Tests: $25.00 per test │ │ │
│ │ └────────────────────────────────────────┘ │ │
│ │ │ │
│ │ AI Extraction: │ │
│ │ [ │ │
│ │ { │ │
│ │ "FULL_SERVICE": "Office Visits", │ │
│ │ "FULL_METHODOLOGY": "110% Medicare", │ │
│ │ "PROV_TYPE": "Professional" │ │
│ │ }, │ │
│ │ { │ │
│ │ "FULL_SERVICE": "Surgery", │ │
│ │ "FULL_METHODOLOGY": "120% Medicare", │ │
│ │ "PROV_TYPE": "Professional" │ │
│ │ }, │ │
│ │ { │ │
│ │ "FULL_SERVICE": "Lab Tests", │ │
│ │ "FULL_METHODOLOGY": "$25.00 per test", │ │
│ │ "PROV_TYPE": "Ancillary" │ │
│ │ } │ │
│ │ ] │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ Output: List of dictionaries (one per service found) │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 3: BOTTOM-UP SECONDARY (one_to_n_funcs.py) │
│ │
│ For each extracted service, break down methodology: │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Methodology Breakout │ │
│ │ │ │
│ │ Prompt (prompts.py: BOTTOM_UP_METHODOLOGY_BREAKOUT): │ │
│ │ "Parse this methodology into structured fields" │ │
│ │ │ │
│ │ Input: "Lesser of billed charges or 110% Medicare" │ │
│ │ │ │
│ │ AI Breakdown: │ │
│ │ { │ │
│ │ "LESSER": "Y", │ │
│ │ "RATE_STANDARD": "110% Medicare", │ │
│ │ "RATE_SHORT": "1.1", │ │
│ │ "FLAT_FEE_STANDARD": "N/A", │ │
│ │ "LESSER_RATE": "100% of BC", │ │
│ │ "NOT_TO_EXCEED": "N/A" │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Escalator Detection │ │
│ │ │ │
│ │ Prompt (prompts.py: BOTTOM_UP_ESCALATOR): │ │
│ │ "Does this rate increase over time?" │ │
│ │ │ │
│ │ Output: │ │
│ │ { │ │
│ │ "RATE_ESCALATOR_IND": "Y", │ │
│ │ "RATE_ESCALATOR_DT": "2025-01-01" │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ Output: Enhanced list with structured rate information │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 4: TOP-DOWN EXTRACTION (one_to_n_funcs.py) │
│ │
│ Extract page-level context that applies to all services: │
│ │
│ Prompt (prompts.py: TOP_DOWN_PRIMARY): │
│ ├─ EXHIBIT: Full exhibit/attachment name │
│ ├─ CONTRACT_LOB: Line of Business (Medicare, Medicaid, etc.) │
│ ├─ DEFAULT_TERM: What happens if no rate specified │
│ ├─ CDM_IND: Chargemaster neutralization? │
│ └─ ADD_ON_REIMBURSEMENT_LANGUAGE: Extra payment clauses │
│ │
│ Example Output: │
│ { │
│ "EXHIBIT": "Attachment C: Medicare Advantage", │
│ "CONTRACT_LOB": "Medicare Advantage", │
│ "DEFAULT_TERM": "100% of Medicare Fee Schedule", │
│ "CDM_IND": "N" │
│ } │
│ │
│ This context is merged with ALL services on that page │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 5: CONDITIONAL PROMPTS (conditional_funcs.py) │
│ │
│ Fill in additional service-level details: │
│ │
│ ├─ IP/OP: Inpatient or Outpatient? │
│ │ Prompt: "Is this service IP or OP?" │
│ │ Uses keywords like "inpatient", "outpatient" │
│ │ │
│ ├─ PROV_TYPE_2: More specific provider category │
│ │ Prompt: "What specific provider type?" │
│ │ Options: PCP, Specialist, ER, Clinic, etc. │
│ │ │
│ └─ LOB_CHECK: Assign single LOB to each service │
│ Prompt: "Which LOB does this service fall under?" │
│ Narrows from page-level to service-level │
│ │
│ Output: Fully enriched service list │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 6: POSTPROCESSING (postprocess.py: b_postprocess) │
│ │
│ Cleanup and standardization: │
│ ├─ Add metadata (filename, parent code, page count) │
│ ├─ Flag single code, multiple rates (SCMR) │
│ ├─ Filter out add-on services │
│ ├─ Clean methodology text │
│ ├─ Standardize provider types │
│ ├─ Clean lesser of rates │
│ ├─ Standardize Line of Business values │
│ ├─ Fix Y/N fields │
│ ├─ Rename columns │
│ └─ Reorder columns │
│ │
│ Output: Clean DataFrame │
└─────────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────────┐
│ STEP 7: WRITE OUTPUT │
│ │
│ Save to: {batch_id}-B.csv │
│ Format: Multiple rows per contract (one per service), ~35 columns │
└─────────────────────────────────────────────────────────────────────┘
Prompt Flow for Key Extractions
Contract Effective Date Extraction
┌────────────────────────────────────────────────────┐
│ CONTRACT_EFFECTIVE_DT EXTRACTION │
└────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────┐
│ Smart Chunking │
│ Keywords: "effective date", "execution date" │
│ Result: Pages 1-3 contain keywords │
└────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────┐
│ Build Prompt (prompts.py: get_effective_date_prompt) │
│ │
│ Context: [Page 1-3 text] │
│ │
│ Instruction: │
│ "Extract ONLY the effective date of the contract │
│ Follow these rules: │
│ 1. Look for 'Effective Date:' label │
│ 2. Don't use signature date │
│ 3. Return in YYYY-MM-DD format or N/A │
│ 4. Enclose answer in |pipes|" │
│ │
│ Example Text: │
│ "This Agreement is effective as of January 15, │
│ 2024 ('Effective Date')..." │
└────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────┐
│ Send to Claude AI │
│ Model: Claude 3.5 Sonnet │
└────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────┐
│ AI Response │
│ "...reviewing the contract text... │
│ Answer: |2024-01-15|" │
└────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────┐
│ Parse Response │
│ - Extract text between |pipes| │
│ - Result: "2024-01-15" │
└────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────┐
│ Validate & Convert │
│ - Check format is YYYY-MM-DD │
│ - If invalid, set to N/A │
└────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────┐
│ Store in Output │
│ CONTRACT_EFFECTIVE_DT = "2024-01-15" │
└────────────────────────────────────────────────────┘
Reimbursement Methodology Extraction
┌────────────────────────────────────────────────────────────────┐
│ REIMBURSEMENT METHODOLOGY EXTRACTION │
└────────────────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────────────────┐
│ Page Text (Example) │
│ ──────────────────────────────────────────────────────────── │
│ COMPENSATION SCHEDULE - PROFESSIONAL SERVICES │
│ │
│ CCA shall reimburse Provider for Covered Services rendered │
│ to Covered Persons the lesser of: │
│ (i) Allowable Charges; or │
│ (ii) The rates specified below: │
│ │
│ Service Category | Contracted Rate │
│ ──────────────────────────────────────────── │
│ Office Visits | 110% of Medicare Fee Schedule │
│ Preventive Care | 100% of Medicare Fee Schedule │
│ Specialist Consults | 125% of Medicare Fee Schedule │
└────────────────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────────────────┐
│ STAGE 1: Bottom-Up Primary │
│ Prompt: BOTTOM_UP_PRIMARY │
│ │
│ Instruction: "Extract EVERY reimbursement term" │
│ │
│ Key Rules: │
│ - Include complete "lesser of" statements │
│ - Capture table headers for context │
│ - One entry per service │
│ - Include provider type │
└────────────────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────────────────┐
│ AI Response (JSON) │
│ [ │
│ { │
│ "FULL_SERVICE": "Office Visits", │
│ "SUBHEADER": "N/A", │
│ "FULL_METHODOLOGY": "Lesser of: (i) Allowable Charges; │
│ or (ii) The rates specified below: Service Category | │
│ Contracted Rate\nOffice Visits | 110% of Medicare │
│ Fee Schedule", │
│ "PROV_TYPE": "Professional" │
│ }, │
│ { │
│ "FULL_SERVICE": "Preventive Care", │
│ "SUBHEADER": "N/A", │
│ "FULL_METHODOLOGY": "Lesser of: (i) Allowable Charges; │
│ or (ii) The rates specified below: Service Category | │
│ Contracted Rate\nPreventive Care | 100% of Medicare │
│ Fee Schedule", │
│ "PROV_TYPE": "Professional" │
│ }, │
│ { │
│ "FULL_SERVICE": "Specialist Consults", │
│ "SUBHEADER": "N/A", │
│ "FULL_METHODOLOGY": "Lesser of: (i) Allowable Charges; │
│ or (ii) The rates specified below: Service Category | │
│ Contracted Rate\nSpecialist Consults | 125% of │
│ Medicare Fee Schedule", │
│ "PROV_TYPE": "Professional" │
│ } │
│ ] │
└────────────────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────────────────┐
│ STAGE 2: Bottom-Up Secondary (Methodology Breakout) │
│ │
│ For each service, parse methodology: │
│ │
│ Input (Office Visits): │
│ "Lesser of: (i) Allowable Charges; or (ii) 110% of Medicare │
│ Fee Schedule" │
│ │
│ Prompt: BOTTOM_UP_METHODOLOGY_BREAKOUT │
│ "Break this methodology into structured fields" │
└────────────────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────────────────┐
│ AI Response (Office Visits) │
│ { │
│ "LESSER": "Y", │
│ "RATE_STANDARD": "110% Medicare", │
│ "RATE_SHORT": "1.1", │
│ "FLAT_FEE_STANDARD": "N/A", │
│ "LESSER_RATE": "100% of AC", // AC = Allowable Charges │
│ "NOT_TO_EXCEED": "N/A" │
│ } │
│ │
│ Repeat for Preventive Care → RATE_STANDARD: "100% Medicare" │
│ Repeat for Specialist → RATE_STANDARD: "125% Medicare" │
└────────────────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────────────────┐
│ STAGE 3: Top-Down Context │
│ │
│ Extract page-level info: │
│ { │
│ "EXHIBIT": "COMPENSATION SCHEDULE - PROFESSIONAL SERVICES", │
│ "CONTRACT_LOB": "Commercial", │
│ "DEFAULT_TERM": "N/A", │
│ "CDM_IND": "N" │
│ } │
│ │
│ Merge with each service │
└────────────────────────────────────────────────────────────────┘
↓
┌────────────────────────────────────────────────────────────────┐
│ FINAL OUTPUT (Office Visits row) │
│ ──────────────────────────────────────────────────────────── │
│ Contract Name: "contract_123.pdf" │
│ Exhibit: "COMPENSATION SCHEDULE - PROFESSIONAL SERVICES" │
│ Line of Business: "Commercial" │
│ Full Service: "Office Visits" │
│ Provider Type: "Professional" │
│ Reimbursement Methodology: "Lesser of: (i) Allowable │
│ Charges; or (ii) 110% of Medicare Fee Schedule" │
│ Rate Standard: "110% Medicare" │
│ Rate Short: "1.1" │
│ Flat Fee: "N/A" │
│ Lesser Of: "Y" │
│ Lesser Of Rate: "100% of AC" │
│ Not to Exceed: "N/A" │
│ ... (additional columns) │
└────────────────────────────────────────────────────────────────┘
Error Handling Flow
┌────────────────────────────────────────────┐
│ File Processing Attempt │
└────────────────────────────────────────────┘
↓
┌──────────────────────┐
│ Success? │
└──────────────────────┘
↓ Yes ↓ No
┌─────────┐ ┌─────────────────────┐
│ Log │ │ Capture Error │
│ Success │ │ - Error message │
│ │ │ - Stack trace │
│ Update │ │ - Timestamp │
│ Tracking│ └─────────────────────┘
│ │ ↓
│ Continue│ ┌─────────────────────┐
│ to Next │ │ Log to Tracking │
└─────────┘ │ Status: Failed │
│ Error: [details] │
└─────────────────────┘
↓
┌─────────────────────┐
│ Mark for Self-Repair│
└─────────────────────┘
↓
┌────────────────────────────────────┐
│ After All Files Processed │
└────────────────────────────────────┘
↓
┌────────────────────────────────────┐
│ Self-Repair Mode │
│ - Identify failed files │
│ - Retry processing │
│ - Consolidate outputs │
└────────────────────────────────────┘
Data Flow Summary
Raw Contract Text
↓
┌─────────────┐
│ Preprocessing│
│ - Clean │
│ - Split │
│ - Chunk │
└─────────────┘
↓
┌─────────────┐ ┌──────────────┐
│ AC Path │ │ B Path │
│ │ │ │
│ Smart Chunk │ │ Bottom-Up │
│ Prompts │ │ Primary │
│ ↓ │ │ ↓ │
│ Full Context│ │ Bottom-Up │
│ Prompts │ │ Secondary │
│ ↓ │ │ ↓ │
│ Conditional │ │ Top-Down │
│ Prompts │ │ ↓ │
│ ↓ │ │ Conditional │
│ Postprocess │ │ ↓ │
│ │ │ Postprocess │
└─────────────┘ └──────────────┘
↓ ↓
┌─────────────┐ ┌──────────────┐
│ AC CSV │ │ B CSV │
│ (1 row) │ │ (many rows) │
└─────────────┘ └──────────────┘
↓ ↓
└──────────┬──────────┘
↓
┌───────────────┐
│ Consolidation │
│ - Merge AC+B │
│ - QA/QC │
└───────────────┘
↓
┌───────────────┐
│ Final Outputs │
│ - ABC CSV │
│ - QA Report │
└───────────────┘
Parallel Processing Visualization
Main Process
│
┌────────────────┼────────────────┐
│ │ │
Worker 1 Worker 2 Worker 3
│ │ │
Contract A Contract D Contract G
↓ ↓ ↓
[AC + B] [AC + B] [AC + B]
Processing Processing Processing
↓ ↓ ↓
Success Success Failed
│ │ │
└────────────────┼────────────────┘
│
Track Results
│
┌────────────────┼────────────────┐
│ │ │
Worker 1 Worker 2 Worker 3
│ │ │
Contract B Contract E Contract H
↓ ↓ ↓
Processing Processing Processing
│ │ │
└────────────────┼────────────────┘
│
↓
Upload Progress (every 10 files)
│
↓
Continue...
This visual guide complements the detailed text guide by showing the exact flow of data and processing through the system in an easy-to-understand diagram format.