Praneel Panchigar
2306334b40
Feature/document index
* Add Document Index preprocessing — Layers 1, 2, and 3 wiring
Parse the Textract-emitted Document Index block at the top of each contract
with a single cached LLM call (prompt_document_index) instead of one per-page
call per page. Layer 2 verifies parsed entries via literal string match and
structural regex sweep, escalating suspect pages back to the existing per-page
path. Layer 1+2 failure triggers a full fallback to today's per-page flow.
New symbols:
- preprocessing_funcs.extract_document_index_block — regex slice of index prefix
- preprocessing_funcs.verify_index_against_pages — structural verifier (plain dict return)
- prompt_templates.DOCUMENT_INDEX_INSTRUCTION / DOCUMENT_INDEX — cached prompt pair
- prompt_calls.prompt_document_index — LLM wrapper (usage_label DOCUMENT_INDEX_PARSE)
- config: DOCUMENT_INDEX_PARSE_ENABLED and three threshold flags
- instrumentation: DOCUMENT_INDEX_PARSE mapped to preprocessing segment
one_to_n_exhibit_chunking now accepts optional contract_text; file_processing.py
pas…
* Add exhibit-level relevance filtering and deprecate pipe-delimited output
Step 5: extend the Document Index LLM call to emit reimbursement_relevant per
exhibit (no extra cost), then filter exhibit_header_dict before downstream
one-to-n extraction. Safety net A (keyword override) and safety net B (never
drop all exhibits) prevent false negatives. Controlled by
DOCUMENT_INDEX_RELEVANCE_FILTER_ENABLED flag.
Remove all pipe-delimited output from prompt_templates.py:
EXHIBIT_HEADER_INSTRUCTION, EXHIBIT_HEADER, EXHIBIT_HEADER_DEDUP_INSTRUCTION,
and EXHIBIT_HEADER_DEDUP now use JSON_DICT_FORMAT_INSTRUCTIONS and the
standard "return your final answer as a valid JSON dictionary" closing pattern.
DOCUMENT_INDEX_INSTRUCTION already used the JSON standard; consistent across
all preprocessing prompts.
* Add Tier 1 and Tier 2 unit tests for Document Index feature
Tier 1 — src/tests/test_document_index.py (31 tests):
- TestExtractDocumentIndexBlock: regex extraction (6 tests)
- TestDocumentIndexPromptTemplates: instruction shape, parser, cache
registration (11 tests)
- TestPromptDocumentIndex: mocked LLM wrapper — usage_label, cache=True,
fallback on malformed/non-list/empty responses (6 tests)
- TestVerifyIndexAgainstPages: literal match pass/fail, coverage sweep
escalation, page drift, high-failure-ratio fallback, behavioural pin
that Layer 2 never adds headers from regex (8 tests)
Tier 2 — src/tests/test_preprocessing_funcs.py (6 new tests):
- TestOneToNExhibitChunkingHybrid: Layer 1 success path, escalation routing,
global fallback, feature-flag-off, no contract_text, no index block
* Fix relevance overwrite bug and remove unused config flag
Bug: relevance_lookup was built from all parsed entries, so a landmark or
noise entry on the same page as an exhibit_start could overwrite the
exhibit_start's reimbursement_relevant=True flag. Fixed by filtering the
lookup to exhibit_start entries only.
Remove DOCUMENT_INDEX_COVERAGE_THRESHOLD from config.py — the flag was
defined but never referenced in the verifier or orchestrator. Removing
it avoids misleading operators expecting coverage-threshold fallback.
Docstring correction: extract_document_index_block docstring claimed
BOM-tolerance; removed that claim since Textract-generated .txt files
don't carry BOMs and the implementation doesn't strip .
Add TestRelevanceFilterLogic (3 tests): landmark-overwrite regression,
irrelevant exhibit exclusion, and safety-net-B preservation.
* Add Tier 3 offline study script for Document Index
Runs Layer 1+2 (extract, LLM parse, structural verify) against every .txt
in an input dir without the full pipeline. Emits a per-file CSV with 17
metrics columns (coverage_score, classification distribution, verification
failures, escalation count, fallback flag, runtime) and a summary markdown.
Marked DELETABLE post-rollout — intended for threshold tuning before Tier 4
E2E runs, not for production.
Usage:
uv run python -m src.testbed.document_index_offline
uv run python -m src.testbed.document_index_offline --input-dir data/pacific-source
uv run python -m src.testbed.document_index_offline --max-files 10
* Fix relevance overwrite bug and remove unused config flag
Remove coverage-sweep escalation — high-trust architecture
The Document Index path is now authoritative when it passes quality
thresholds. The per-page escalation of individual uncovered pages is
removed; the only fallback is the whole-document global fallback (empty
parse, page drift, low exhibit density, high failure ratio). This
eliminates the 8–50 false-positive escalations per file observed in the
Tier 3 offline study (continuation pages containing sub-headings were
being sent back to the per-page LLM unnecessarily).
verify_index_against_pages now returns 3 keys instead of 4:
verified_dict, metrics, global_fallback (escalation_pages removed)
one_to_n_exhibit_chunking success path no longer runs get_exhibit_pages_new
or prompt_header_deduplication — it returns verified_dict directly after
the relevance filter.
Add TestVerifyIndexAgainstPagesV2 (40 tests) written contract-first by the
testing agent: return-shape invariants, all 4 fallback tr…
* Trim TestVerifyIndexAgainstPagesV2 to 10 targeted boundary tests
* Remove exhibit-level relevance filter (Step 5)
The filter classified exhibits as reimbursement-relevant or not based
purely on header text from the Document Index table of contents — never
reading the exhibit content. This is unreliable: "AMENDMENT NO. 3" or
"GENERAL TERMS" could contain critical rate changes; the header alone
cannot distinguish them from truly irrelevant exhibits.
Gate 4 (has_reimbursements check after Step 1) already handles this
correctly by reading actual content. The header-based pre-filter added
correctness risk without proportional benefit.
Removed: DOCUMENT_INDEX_RELEVANCE_FILTER_ENABLED config flag, the Step 5
filter block in one_to_n_exhibit_chunking, reimbursement_relevant from
DOCUMENT_INDEX_INSTRUCTION output schema (back to 3 keys: page, header,
classification), TestRelevanceFilterLogic test class. Plan document
updated with reasoning.
Also fix document_index_offline.py to use split_text + lightweight page
wrapper instead of split_text_with_pages, avoiding expensive table
sp…
* Enrich comparison script with page spans and DI parent mapping
* Merged dev into feature/document-index
* Restructure Document Index quality gates: fuzzy match + failure-ratio + tail-coverage
Verifier semantics changed to lead with the index and only fall back on real
quality signals:
- Fuzzy header match: literal-with-whitespace match first, then rapidfuzz
partial_ratio (threshold 85) on the top of the page. Handles dash variants
and minor formatting differences.
- Failure-ratio gate: >10% of exhibit_start entries failing the fuzzy match
triggers fallback (parse is untrustworthy).
- Tail-coverage gate: replaces the old 'first exhibit must start at page 1'
rule, which fired falsely on contracts where page 1 is preamble. New rule
measures the trailing span: tail_pages = total_pages - max_exhibit_start.
For docs >15 pages, fallback when tail > max(15, 30%) of the document — the
signal that the LLM stopped parsing partway through the index.
- Thresholds are hardcoded module constants in preprocessing_funcs.py
(_DI_FUZZY_MATCH_THRESHOLD, _DI_MAX_FAILURE_RATIO, _DI_TAIL_COVERAGE_*) —
not config fla…
* Move Document Index testbed scripts to local_scripts (untracked)
The DI offline study and comparison tools are throwaway local utilities, not
production code. Moving them to local_scripts/ (already gitignored) keeps
src/testbed/ clean and removes them from the repo.
* Document Index: add SECTION as exhibit_start keyword, disambiguate sub-clauses
Empirical signal from a 100-file generic-corpus comparison run showed 37
legacy-only exhibit boundaries on three SECTION-style contracts (Gulf Coast
Division, UTMB Amendment 10, Rehab Designs) that DI was missing because
SECTION wasn't listed alongside EXHIBIT/ATTACHMENT/ARTICLE/etc. Also added
a counter-example to the landmark bullet so numeric sub-clauses like
'2.3 Confidentiality' or '10.4 Billing Procedures' stay out of exhibit_start.
Two unit tests pin the new behavior: SECTION appears in the exhibit_start
paragraph, and the numeric-sub-clause disambiguation appears in landmark.
* Trim Document Index tests: drop low-value prompt-content pinning, consolidate
41 tests for a ~200-line feature was over-investment. Cut to 25 by:
- Dropping prompt-string-content tests that pinned copy rather than behavior
(will-break-on-tuning, value-add zero) — non-empty-string, required-section
headers, noise-example pinning, SECTION/landmark prompt-content checks
- Removing trivial happy-path tests for DOCUMENT_INDEX factory (returns tuple,
embeds block, parser parses valid JSON) that duplicate parser-level coverage
in test_json_parsers.py
- Trimming TestPromptDocumentIndex to the three meaningful contracts:
usage_label, cache=True, malformed-JSON tolerance
- Folding TestVerifyIndexAgainstPagesBoundaries into the main verify class
and consolidating four 'doesn't crash' tests into a single sweep test
Net: 41 → 25 tests. Test density now matches comparable small features in
the codebase (test_clean_header_footer 13 tests, test_chc_extraction 6).
* Document Index: rewrite verified headers with page-text via single LLM call
The Document Index returns thin TOC entries ("EXHIBIT A - FEE SCHEDULE");
downstream code (chunk-boundary matching, dynamic-primary LLM input,
EXHIBIT_TITLE column, active-rates exhibit-title standardization) is
calibrated for the rich page-text version that the legacy per-page path
produced (multi-line headers with provider names, effective dates, etc.).
After DI parse + verification, bundle the full text of each verified page
under per-page boundary markers and run a single LLM call to extract the
page-formatted headers. Two-headers-on-one-page is supported by allowing
list values per boundary — that case is the original reason DI is the
preferred path. On any LLM/parse failure the original verified_dict is
preserved so the pipeline never breaks.
Cost stays bounded by count of header-bearing pages (typically 5-15 per
contract), independent of exhibit length. A 60-page contract with 8
exhibits costs 2 LLM calls (DI parse + bundle …
* Merge remote-tracking branch 'origin/dev' into feature/document-index
* Document Index: recover orphan-anchor pages dropped by TOC
When OCR strips the "ATTACHMENT X" / "Exhibit N" label from a Document Index
line, leaving only the secondary title (e.g. "MEDICARE ADVANTAGE COMPENSATION"
where "ATTACHMENT B" was expected), the DI LLM classifies the orphan secondary
as `landmark`, the page never enters verified_dict, and the exhibit is silently
dropped - even though the actual page begins with the anchor label.
Add a regex-based recovery pass that runs between verify_index_against_pages
and replace_index_headers_with_page_text. Scans every unverified page for a
strict whole-line header anchor, then applies five guards before promoting:
1. Strict whole-line regex (optional " - Secondary Title" suffix)
2. Length cap (anchors are short; <=80 chars)
3. Running-header filter (line on >=2 pages AND >40% of pages)
4. List detection (>=3 distinct anchors on one page -> body-text list, skip)
5. Continuation guard (same anchor as previous page -> multi-page exhibit, skip)
No new …
* Document Index: strip running-header bleed from page-text rewriter
The verified-pages header LLM was concatenating document running headers
(e.g. 'PROVIDER SERVICES AGREEMENT' repeated on every page) into the
rewritten exhibit title, producing entries like
'PROVIDER SERVICES AGREEMENTExhibit 1Services & Compensation'.
Lift the running-header set computation to the DI orchestrator so a single
set is shared between orphan-anchor recovery (Fix A) and the page-text
rewriter (Fix B). The rewriter now strips lines matching the set from each
page before bundling, so the LLM never sees the bleed and can't include it.
Verified on the 6 affected Clover contracts: DAVITA p7, Amedisys p34, HMH
p3, Inspira p4/p5, PHOEBE p17, CedarBridge p6 — all now produce clean
anchor + newline + secondary-title titles.
* Merge remote-tracking branch 'origin/dev' into feature/document-index
Approved-by: Karan Desai
Approved-by: Faizan Mohiuddin
Field Extraction Pipeline
Contract field extraction using LLMs.
Setup
Install uv (if not already installed)
# Ubuntu/WSL (recommended)
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc # or restart terminal
# macOS
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell - native, not WSL)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
Verify installation: uv --version
Install dependencies
uv sync
Running the Code
Use uv run with Python's module flag from the project root:
# Run the default SaaS pipeline
uv run python -m src.pipelines.saas.main
# Run with unified runner (supports client selection)
uv run python -m src.pipelines.runner --client saas --input-dir /path/to/input
# Run with specific client
uv run python -m src.pipelines.runner --client clover --input-dir /path/to/input
Note: Always use uv run to ensure correct virtual environment. Always use -m flag for proper imports.
Development
Before pushing, always run these checks to avoid breaking the CI pipeline:
# Format code
uv run black src/
# Type checking
uv run mypy src/
# Run tests
uv run pytest
All checks must pass before merging to main.
Project Structure
├── src/
│ ├── pipelines/ # Pipeline implementations
│ │ ├── runner.py # Unified CLI entry point with client routing
│ │ ├── saas/ # Default SaaS pipeline
│ │ │ └── main.py # Main entry point for SaaS
│ │ ├── shared/ # Shared pipeline components
│ │ │ ├── preprocessing/ # Document preprocessing
│ │ │ ├── extraction/ # Field extraction logic
│ │ │ └── postprocessing/ # Result postprocessing
│ │ └── clients/ # Client-specific overrides
│ │ └── clover/ # Clover client customizations
│ ├── core/ # Core utilities (registry, fieldset)
│ ├── constants/ # Constants, mappings, and field definitions
│ │ ├── mappings/ # Crosswalk JSON files
│ │ └── lists/ # Lookup lists
│ ├── prompts/ # LLM prompt templates
│ ├── utils/ # Shared utilities (IO, string, logging, etc.)
│ ├── codes/ # Medical code extraction utilities
│ ├── crosswalk/ # Crosswalk mapping logic
│ ├── embeddings/ # Pre-computed embeddings for code matching
│ ├── qc_qa/ # QC/QA validation pipeline
│ ├── parent_child/ # Parent-child relationship mapping
│ ├── document_classification/ # Document type classification (DTC)
│ └── tests/ # Unit tests
├── documentation/ # Project documentation
├── outputs/ # Pipeline output files (gitignored)
├── logs/ # Log files (gitignored)
└── pyproject.toml # Project dependencies (uv/pip)
Branching
Naming Conventions
| Type | Pattern | Use Case | Example |
|---|---|---|---|
| Feature | feature/<ticket>-<description> |
New functionality | feature/PROJ-123-add-export-csv |
| Bugfix | bugfix/<ticket>-<description> |
Bug fixes | bugfix/PROJ-456-fix-null-handling |
| Hotfix | hotfix/<ticket>-<description> |
Urgent production fixes | hotfix/PROJ-789-critical-parse-error |
| Test | test/<description> |
Testing/experimentation | test/lesser-table-caching-refactor |
| Release | release/<version> |
Release preparation | release/v1.2.0 |
Branch Guidelines
- Use lowercase with hyphens (kebab-case) for descriptions
- Include ticket number when applicable (e.g., JIRA, GitHub issue)
- Keep branch names concise but descriptive
- Delete branches after merging
Workflow
- Create branch from
main - Make changes and commit with clear messages
- Run checks before pushing:
uv run black src/ && uv run mypy src/ && uv run pytest - Create PR to
main - Ensure all CI checks pass
- Get code review approval
- Squash and merge