doczyai-pipelines/documentation/FIELD_EXTRACTION_GUIDE.md

Field Extraction System - Complete Guide

What This System Does

This system reads healthcare provider contracts (like agreements between insurance companies and doctors or hospitals) and automatically extracts important information from them. Think of it as a smart assistant that reads through hundreds of pages of legal contracts and pulls out key details like payment rates, contract dates, provider names, and reimbursement terms.

---

Table of Contents

1. Overview
2. How to Run the System
3. Complete Processing Flow
4. Field Types Explained
5. The Extraction Process Step-by-Step
6. Understanding the Prompts
7. Output Files

---

Overview

What Gets Extracted?

The system extracts two main categories of information:

1. AC Fields (Administrative/Contract Fields): Basic contract information like:

   • Contract title and effective date
   • Payer name (insurance company)
   • Provider names and identification numbers
   • Termination clauses
   • Legal terms and conditions

2. B Fields (Benefit/Reimbursement Fields): Payment-related information like:

   • What services are covered
   • How much providers get paid
   • Payment methodologies (percentages of Medicare/Medicaid rates, flat fees)
   • Which line of business the rates apply to (Medicare, Medicaid, Commercial)

---

How to Run the System

Starting the Process

The main entry point is main.py. You can run it with different configurations:

python main.py batch_id=my_batch fields=abc max_workers=5

Key Parameters:

• batch_id: A unique name for this extraction run (e.g., "batch_january_2025")
• fields: What to extract - "ac" for contract fields, "b" for reimbursement, "abc" for both
• max_workers: How many contracts to process at the same time (parallel processing)
• read_mode: Where to read files from ("local" or "s3")
• write_to_s3: Whether to save results to cloud storage

---

Complete Processing Flow

The Big Picture

Input Contracts (PDF → Text)
         ↓
    main.py starts
         ↓
    ┌────────────────────────────┐
    │   Split into two paths     │
    └────────────────────────────┘
         ↓              ↓
    AC Processing   B Processing
    (Contract Info) (Reimbursement)
         ↓              ↓
    Preprocessing   Preprocessing
         ↓              ↓
    AI Extraction   AI Extraction
         ↓              ↓
    Postprocessing  Postprocessing
         ↓              ↓
    AC Output CSV   B Output CSV
         ↓              ↓
    ┌────────────────────────────┐
    │   Combine & Quality Check  │
    └────────────────────────────┘
         ↓
    Final Reports

Step-by-Step Flow

1. Initial Setup (main.py:220-244)

When you run the system:

• Creates a batch tracker to monitor progress
• Reads all contract text files from the input folder (or S3 bucket)
• Checks which files contain "reimbursement" information
• Filters out already-processed files if requested

2. Parallel Processing (main.py:296-334)

The system processes multiple contracts at once:

• Creates separate processing tasks for AC and B fields
• Runs them in parallel using multiple "workers" (like having multiple assistants)
• Tracks progress and handles errors for each contract
• Updates progress statistics in real-time

3. Contract Processing Paths

For Each Contract:

• If processing AC fields → calls process_ac()
• If processing B fields → calls process_b()
• Each path follows its own specialized extraction process

---

Field Types Explained

AC Fields (Contract Information)

These fields capture the administrative details of the contract. Examples:

• CONTRACT_TITLE: The official name of the agreement
• PAYER_NAME: The insurance company (e.g., "Centene", "Superior Health Plan")
• PROV_GROUP_NAME: The doctor's office or hospital name
• CONTRACT_EFFECTIVE_DT: When the contract starts
• TERMINATION_UPON_NOTICE: How many days notice to cancel
• TERM_CLAUSE: The full legal text about contract duration

Why these matter: These fields help understand who is involved in the contract, when it's valid, and basic legal terms.

B Fields (Reimbursement Information)

These fields capture payment details. Examples:

• FULL_SERVICE: What medical service is being paid for (e.g., "Office Visits", "Surgery", "Lab Tests")
• FULL_METHODOLOGY: The complete description of how payment is calculated
• RATE_STANDARD: The payment rate (e.g., "110% Medicare" or "85% Medicaid")
• FLAT_FEE_STANDARD: Fixed dollar amounts (e.g., "$50.00 Per Visit")
• LESSER_RATE: If there's a "lesser of" clause (e.g., "100% of Billed Charges")
• PROV_TYPE: Whether it's for doctors (Professional), hospitals (Facility), or other services (Ancillary)

Why these matter: These tell us exactly how much a provider gets paid for each type of service.

---

The Extraction Process Step-by-Step

AC Processing Flow

Entry Point: file_processing.py:15-100

Step 1: Preprocessing (preprocess.py:53-56)

What happens:

1. Clean up the text (remove weird symbols, fix line breaks)
2. Split the contract into individual pages
3. Create "smart chunks" - group pages by topic using keywords

Smart Chunking (smart_chunking_funcs.py):

• Looks for specific keywords related to each field
• Example: For "CONTRACT_EFFECTIVE_DT", it searches for pages containing "effective date", "execution date", etc.
• Only sends relevant pages to AI, saving time and cost

Step 2: Extract Basic Information

Regex-based extraction (file_processing.py:36-40):

• Some fields are found using patterns (like phone numbers or tax IDs)
• TIN (Tax ID): Looks for 9-digit numbers with hyphen (XX-XXXXXXX)
• NPI (Provider ID): Looks for 10-digit numbers

Step 3: Smart Chunked Prompts (smart_chunking_funcs.py:33-150)

For each field or group of fields:

1. Take the relevant chunk of text (found by keywords)
2. Build a specific question or set of questions
3. Send to AI (Claude) with the question and text
4. Parse the AI's response
5. Store the answer

Example for Contract Effective Date:

Context: [Pages 1-3 of the contract]
Question: "What is the contract effective date?"
AI Response: "2024-01-15"
System converts to: "2024-01-15" (standardized format)

Step 4: Full Context Prompts

For fields not found in chunks:

• Uses the entire contract as context
• More expensive but ensures nothing is missed
• Used for rare fields or when keyword search fails

Step 5: Postprocessing (postprocess.py:55-114)

Clean up and validate:

• Convert dates to standard format (YYYY-MM-DD)
• Clean up Tax IDs and NPIs
• Derive indicator fields (Yes/No flags)
• Add metadata (filename, page count)
• Remove invalid values
• Organize columns in standard order

B Processing Flow

Entry Point: file_processing.py:103-153

Step 1: Preprocessing (preprocess.py:23-50)

What happens:

1. Clean the text
2. Find pages with "Exhibit" or "Attachment" headers (where reimbursement info lives)
3. Chunk consecutive pages together (keeps related compensation schedules together)
4. Detect and format tables (many rates are in table format)

Table Alignment:

• Contracts often have messy tables
• System realigns columns and rows
• Preserves the structure for accurate extraction

Step 2: Bottom-Up Extraction (one_to_n_funcs.py)

Primary Bottom-Up (prompts.py:9-196):

This is the core reimbursement extraction. For each page:

Prompt asks AI to find:

• Every service mentioned (e.g., "Office Visits", "Emergency Room")
• Complete payment methodology
• Provider type (Professional/Facility/Ancillary)
• Any subheaders or categories

Example extraction:

Page text: "Physician Services: 110% of Medicare Fee Schedule"

AI extracts:
{
  "FULL_SERVICE": "Physician Services",
  "FULL_METHODOLOGY": "110% of Medicare Fee Schedule",
  "PROV_TYPE": "Professional"
}

Critical Rules in the Prompt:

• Extract EVERY rate on the page (don't skip anything)
• Include "lesser of" statements (e.g., "lesser of billed charges or 100% Medicare")
• Capture complete table context (headers + values)
• Split services with multiple rates into separate entries

Secondary Bottom-Up (prompts.py:198-458):

Breaks down each methodology into structured fields:

Methodology Breakout Prompt:

Input: "Lesser of billed charges or 110% Medicare Fee Schedule"

AI extracts:
{
  "LESSER": "Y",
  "RATE_STANDARD": "110% Medicare",
  "RATE_SHORT": "1.1",
  "LESSER_RATE": "100% of BC" (Billed Charges)
}

Escalator Detection:

• Checks if rates increase over time
• Finds effective dates for rate changes

Step 3: Top-Down Extraction (prompts.py:482-554)

Extracts page-level information:

• Exhibit names (e.g., "Attachment C: Medicare Advantage")
• Line of Business (Medicare, Medicaid, Commercial, Dual)
• Default terms (what happens if no rate is specified)
• Chargemaster protections
• Add-on reimbursement language

Why both Bottom-Up and Top-Down?

• Bottom-up finds specific rates and services
• Top-down finds context that applies to all services on a page
• Combined, they give complete information

Step 4: Conditional Prompts (conditional_funcs.py)

Fills in additional details:

• Inpatient vs Outpatient (IP/OP)
• More specific provider types
• Line of Business assignment for each service

Step 5: Postprocessing (postprocess.py:5-52)

Final cleanup:

• Filter out invalid services
• Clean methodology text
• Standardize Line of Business values
• Add metadata fields
• Rename columns to user-friendly names
• Reorder columns

---

Understanding the Prompts

What Are Prompts?

Prompts are detailed instructions given to the AI (Claude). They're like giving very specific directions to a research assistant.

Anatomy of a Prompt

Every prompt has:

1. Context: The contract text to analyze
2. Task: What to extract
3. Instructions: How to extract it
4. Examples: Show the AI what good answers look like
5. Constraints: What NOT to do
6. Output format: How to return the answer (JSON, text, etc.)

Example: Bottom-Up Primary Prompt

Location: prompts.py:9-196

Structure:
├── Give page text
├── Explain the task ("extract EVERY reimbursement term")
├── Define each field to extract
│   ├── FULL_SERVICE: What service is being reimbursed?
│   ├── FULL_METHODOLOGY: Complete payment description
│   └── PROV_TYPE: Professional, Facility, or Ancillary
├── Provide detailed rules
│   ├── How to handle tables
│   ├── How to capture "lesser of" statements
│   └── When to split into multiple entries
├── Give 3-5 examples of correct extractions
└── Request JSON output

Key Instruction Examples:

• "Extract EVERY percentage rate, dollar amount, or lesser of statement"
• "For tables, include: [Lesser of statements] + [Table header] + [Service row]"
• "Split entries when encountering multiple percentage rates for different services"

Example: Methodology Breakout Prompt

Location: prompts.py:198-324

Purpose: Parse a methodology string into structured fields

Input: "Lesser of billed charges or 110% of Medicare Fee Schedule"

Prompt asks:
1. Is there a "lesser of"? → LESSER: Y or N
2. What's the percentage rate? → RATE_STANDARD: "110% Medicare"
3. Convert to decimal → RATE_SHORT: "1.1"
4. What's the lesser of rate? → LESSER_RATE: "100% of BC"
5. Any flat fees? → FLAT_FEE_STANDARD: N/A
6. Any "not to exceed"? → NOT_TO_EXCEED: N/A

Output:
{
  "LESSER": "Y",
  "RATE_STANDARD": "110% Medicare",
  "RATE_SHORT": "1.1",
  "FLAT_FEE_STANDARD": "N/A",
  "LESSER_RATE": "100% of BC",
  "NOT_TO_EXCEED": "N/A"
}

Why this structure?

• Makes rates searchable and comparable
• Standardizes different ways of expressing the same thing
• Enables analysis (e.g., "find all services paid above 100% Medicare")

---

Preprocessing Details

Text Cleaning

Functions in preprocessing_funcs.py:

1. clean_newlines: Fixes line break issues
2. clean_law_symbols: Removes § symbols and other legal markers
3. split_text: Divides contract into page-by-page dictionary

Smart Chunking for AC Fields

How it works (preprocessing_funcs.py):

1. Define keyword groups (keywords.py):

   "CONTRACT_EFFECTIVE_DT": {
       "keywords": ["effective date", "execution date", "date of execution"],
       "case_sensitive": False
   }
   

2. Search each page for keywords

3. Extract matching pages (and nearby pages for context)

4. Build focused chunk with only relevant text

5. Result: Instead of sending 50 pages, send only 3 relevant pages

Benefits:

• Faster processing
• Lower cost (less text to analyze)
• More accurate (AI focuses on relevant content)

Chunking for B Fields

Different approach (preprocess.py:24-50):

1. Find exhibit pages: Pages starting with "Exhibit", "Attachment", "Amendment"
2. Chunk consecutive pages: Keep pages 10-15 together if they're part of the same exhibit
3. Align tables: Detect tables and reformat them for consistency

Why chunk this way?

• Compensation schedules span multiple pages
• Need full context of each exhibit
• Tables need to stay together for accurate extraction

---

Output Files

AC Output

File: {batch_id}-AC.csv

Contains one row per contract with columns like:

• Contract Name
• Contract Title
• Payer Name
• Provider Group Name
• Contract Effective Date
• Termination Terms
• Legal Clauses
• (40+ fields total)

B Output

File: {batch_id}-B.csv

Contains multiple rows per contract (one per service/rate) with columns like:

• Contract Name
• Exhibit Name
• Line of Business
• Full Service Description
• Provider Type
• Reimbursement Methodology
• Rate Standard
• Rate Short (decimal)
• Flat Fee
• Lesser Of Indicator
• (30+ fields total)

Combined ABC Output

File: {batch_id}-ABC.csv

Merges AC and B data:

• Each B row includes AC contract information
• Enables analysis like "find all services in Medicare contracts with rates above 110%"

QA/QC Report

File: Excel workbook with multiple sheets:

• Blanks Analysis: Fields with missing values
• Date Formatting Check: Invalid dates
• State Validation: Health plan state issues
• TIN/NPI Validation: Provider ID issues
• Y/N Field Check: Incorrect Yes/No values

---

Error Handling and Recovery

Built-in Safeguards

1. File-level tracking (tracking.py):

   • Records which files are processed
   • Skips already-completed files on re-run
   • Logs errors with details

2. Self-repair mode (main.py:170-218):

   • Automatically retries failed files
   • Consolidates output after repairs

3. Progress monitoring:

   • Real-time progress percentages
   • Completion tracking for AC and B separately
   • Upload every 10 files for incremental results

What Happens When Things Go Wrong

If a file fails:

1. Error is logged with details
2. Processing continues with other files
3. Failed files marked in tracking
4. Self-repair attempts to reprocess

If AI returns bad JSON:

1. System tries to fix common JSON errors
2. Falls back to asking AI to fix it (prompts.py:727-735)
3. If still broken, logs error and continues

---

Key Design Decisions

Why Two Separate Paths (AC and B)?

• Different information types: Contract metadata vs reimbursement details
• Different extraction patterns: One-to-one (AC) vs one-to-many (B)
• Different prompting strategies: Single-pass vs multi-stage

Why Multiple Prompt Stages?

For B fields:

1. Bottom-up primary: Find everything on the page
2. Bottom-up secondary: Structure each finding
3. Top-down: Add page-level context
4. Conditional: Fill in gaps

Reasoning:

• Breaking it into stages improves accuracy
• Each stage has a focused task
• Easier to debug and improve each stage independently

Why Use AI Instead of Rules?

Contracts are highly variable:

• Different formats (tables, paragraphs, bullet points)
• Different terminology (same concept, different words)
• Different structures (some have exhibits, some don't)
• Legal language that changes meaning with small variations

AI advantages:

• Understands context and meaning
• Adapts to variations
• Can follow complex instructions
• Generalizes to new contract types

Where rules ARE used:

• Date formatting (after extraction)
• TIN/NPI patterns (regex)
• Data cleanup (standardization)

---

Performance and Scalability

Processing Speed

Typical performance:

• AC fields: ~30-60 seconds per contract
• B fields: ~2-5 minutes per contract (depends on reimbursement complexity)
• With 5 workers: Can process 100 contracts in ~1-2 hours

Parallel Processing

How it works (main.py:296-334):

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Submit all contracts for processing
    futures = [executor.submit(process_ac, contract) for contract in contracts]

    # Collect results as they complete
    for future in concurrent.futures.as_completed(futures):
        result = future.result()

Benefits:

• Process multiple contracts simultaneously
• Better utilization of resources
• Significant time savings on large batches

Cost Management

Strategies:

• Smart chunking reduces tokens sent to AI
• Caching avoids re-processing
• Progress tracking prevents duplicate work
• Cost logging tracks spend per file (costlog_funcs.py)

---

Troubleshooting Common Issues

"No reimbursement information found"

Cause: Contract doesn't contain payment terms (might be administrative-only) Solution: Normal behavior - file is skipped for B processing, AC still processed

"AC output file not created"

Cause: Error during AC processing or postprocessing Solution: Check error logs, review tracking file for details

Missing fields in output

Cause: Information not found in contract OR extraction failed Solution: Check raw contract for presence of information, review prompt effectiveness

Rate extraction errors

Cause: Complex table structures, unusual formatting Solution: Review table alignment step, check if manual cleanup needed

---

Summary

This field extraction system automates the tedious work of reading healthcare contracts and pulling out critical information. By combining smart text processing, focused AI prompting, and thorough quality checks, it can process hundreds of contracts efficiently and accurately.

The key to its success:

1. Intelligent preprocessing: Only send relevant text to AI
2. Structured prompting: Clear instructions with examples
3. Multi-stage extraction: Break complex tasks into manageable steps
4. Robust postprocessing: Clean and validate results
5. Error handling: Gracefully handle failures and retry

Whether you're looking to understand how the system works, modify it for new fields, or troubleshoot issues, this guide provides the foundation you need.