Praneel Panchigar
37e1e9c228
Bugfix/implicit cpt update
* DAIP2-2310: enrich service term before implicit code extraction
- Add SERVICE_ENRICHMENT prompt + enrich_service_for_codes() that runs
after explicit extraction and before the implicit pipeline, stripping
provider/facility/contractual noise while preserving billing codes.
- Tighten CODE_IMPLICIT_ARBITRATION instructions to require strict
synonymy and prefer no_match over best-effort guesses.
- Remove single-candidate auto-accept bypass so all implicit candidates
go through LLM arbitration.
- Update unit tests for new arbitration path and enrichment mock.
* DAIP2-2310: refine service enrichment and track implicit mapping source
- Service enrichment prompt: bare 'Inpatient'/'Outpatient' place-of-service
rows now collapse to 'Covered Services' to prevent spurious Level 1 matches.
When paired with a real procedure (e.g. 'Outpatient Surgery'), the place
modifier is preserved.
- code_implicit_rag: record per-match mapping origin (CPT_LEVEL1,
HCPCS_LEVEL1, CPT_LEVEL2, HCPCS_LEVEL2) on the code answer dict for
debugging which mapping produced each implicit match.
- constants: add TODO next to HCPCS_LEVEL1_MAPPING load flagging the
upcoming DME code range change.
* Merged dev into DAIP2-2310-implicit-cpt-update
* DAIP2-2310: fix 3-digit revenue code extraction and enrichment exclusion rule
Two related fixes surfaced during short-circuit analysis of the NICU rows:
1. 3-digit revenue codes were being dropped by validate_explicit_codes,
causing the explicit path to return empty and the row to fall through
to implicit RAG (which would then semantic-match against CPT/HCPCS
and produce 'Specific - No Match'). Root causes:
- _revenue_code_format_valid required exactly 4 digits, rejecting the
common contract form like 'Revenue Code 173'.
- REV_MAPPING keys are loaded as ints by pandas (leading zero stripped
from all-numeric CSV columns), so even after accepting 3-digit form
the mapping lookup missed.
Fixes in src/codes/code_funcs.py:
- _revenue_code_format_valid now accepts 3 or 4 digits.
- New _normalize_revenue_code helper returns canonical 4-digit form.
- validate_explicit_codes stores the normalized form in REVENUE_CD.
- _has_unmatched_codes normalizes before membership chec…
* Merged dev into bugfix/implicit-cpt-update
* DAIP2-2310: accept revenue code wildcards and ranges; add unit tests
Extends the earlier 3-digit revenue code fix to cover two more forms the
CODE_EXPLICIT prompt instructs the LLM to return:
1. X-suffix wildcards (e.g. '17X', '173X') - standard UB-04 shorthand
meaning any digit across a revenue family. The prompt explicitly says
'may end in X' but the validator was rejecting them, causing the same
silent drop + implicit-RAG fall-through as the 3-digit bug.
2. Ranges (e.g. '173-174', '0170-0179') - the CODE_EXPLICIT prompt
instructs ranges in 'LOW-HIGH' form. Procedure codes support ranges
but revenue codes did not. The Gulf Coast 'Rev Codes 173-174' rows
were landing on Specific - No Match because of this.
New helpers in src/codes/code_funcs.py:
- _revenue_wildcard_format_valid / _revenue_range_format_valid /
_revenue_code_any_format_valid: targeted format checks.
- _normalize_revenue_any: canonical form for any valid kind
(single code -> '0173', wildcard -> '017X', range -> '0173-0…
* DAIP2-2310: prevent enrichment from hallucinating or dropping descriptors
Two related prompt changes to SERVICE_ENRICHMENT_INSTRUCTION based on
review feedback:
1. Add a 'no information not in input' rule. The enrichment is a
faithful rewrite that strips noise — not a knowledge-augmented
expansion. The LLM must NOT look up what a code means and inject
the description, infer category names from codes, or translate
medical shorthand into clinical terms not present in the input.
2. Add a 'preserve descriptive labels' rule. Things like 'Level 1',
'Level 2-5', 'Tier A', 'Class III', 'low complexity' etc. are
meaningful service descriptors written by the contract author and
must remain in the output even when codes are also present. The
prior ER few-shot example showed the LLM stripping 'Level 1'..
'Level 5' labels — that's information loss.
Updated few-shot examples to match the new rules:
- Emergency Room example output now preserves 'Level 1 - 99281,
Level 2 - 99282, ...' verbati…
Approved-by: Katon Minhas
Field Extraction Pipeline
Contract field extraction using LLMs.
Setup
Install uv (if not already installed)
# Ubuntu/WSL (recommended)
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc # or restart terminal
# macOS
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell - native, not WSL)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
Verify installation: uv --version
Install dependencies
uv sync
Running the Code
Use uv run with Python's module flag from the project root:
# Run the default SaaS pipeline
uv run python -m src.pipelines.saas.main
# Run with unified runner (supports client selection)
uv run python -m src.pipelines.runner --client saas --input-dir /path/to/input
# Run with specific client
uv run python -m src.pipelines.runner --client clover --input-dir /path/to/input
Note: Always use uv run to ensure correct virtual environment. Always use -m flag for proper imports.
Development
Before pushing, always run these checks to avoid breaking the CI pipeline:
# Format code
uv run black src/
# Type checking
uv run mypy src/
# Run tests
uv run pytest
All checks must pass before merging to main.
Project Structure
├── src/
│ ├── pipelines/ # Pipeline implementations
│ │ ├── runner.py # Unified CLI entry point with client routing
│ │ ├── saas/ # Default SaaS pipeline
│ │ │ └── main.py # Main entry point for SaaS
│ │ ├── shared/ # Shared pipeline components
│ │ │ ├── preprocessing/ # Document preprocessing
│ │ │ ├── extraction/ # Field extraction logic
│ │ │ └── postprocessing/ # Result postprocessing
│ │ └── clients/ # Client-specific overrides
│ │ └── clover/ # Clover client customizations
│ ├── core/ # Core utilities (registry, fieldset)
│ ├── constants/ # Constants, mappings, and field definitions
│ │ ├── mappings/ # Crosswalk JSON files
│ │ └── lists/ # Lookup lists
│ ├── prompts/ # LLM prompt templates
│ ├── utils/ # Shared utilities (IO, string, logging, etc.)
│ ├── codes/ # Medical code extraction utilities
│ ├── crosswalk/ # Crosswalk mapping logic
│ ├── embeddings/ # Pre-computed embeddings for code matching
│ ├── qc_qa/ # QC/QA validation pipeline
│ ├── parent_child/ # Parent-child relationship mapping
│ ├── document_classification/ # Document type classification (DTC)
│ └── tests/ # Unit tests
├── documentation/ # Project documentation
├── outputs/ # Pipeline output files (gitignored)
├── logs/ # Log files (gitignored)
└── pyproject.toml # Project dependencies (uv/pip)
Branching
Naming Conventions
| Type | Pattern | Use Case | Example |
|---|---|---|---|
| Feature | feature/<ticket>-<description> |
New functionality | feature/PROJ-123-add-export-csv |
| Bugfix | bugfix/<ticket>-<description> |
Bug fixes | bugfix/PROJ-456-fix-null-handling |
| Hotfix | hotfix/<ticket>-<description> |
Urgent production fixes | hotfix/PROJ-789-critical-parse-error |
| Test | test/<description> |
Testing/experimentation | test/lesser-table-caching-refactor |
| Release | release/<version> |
Release preparation | release/v1.2.0 |
Branch Guidelines
- Use lowercase with hyphens (kebab-case) for descriptions
- Include ticket number when applicable (e.g., JIRA, GitHub issue)
- Keep branch names concise but descriptive
- Delete branches after merging
Workflow
- Create branch from
main - Make changes and commit with clear messages
- Run checks before pushing:
uv run black src/ && uv run mypy src/ && uv run pytest - Create PR to
main - Ensure all CI checks pass
- Get code review approval
- Squash and merge