Venkatakrishna Reddy Avula
2da639e59b
Feature/DAIP2-2314 DAIP2 1687 hybrid * remove -files from s3 prefix requirements * Resolve input paths * fix: VendorProcessor.process_file returns (df, None) tuple runner.safe_process_file unpacks the result as (cc_df, dashboard_df), so returning a single DataFrame caused every vendor/generic file to fail with "too many values to unpack (expected 2)" — Python iterates DataFrame columns during unpacking. Vendor pipelines have no dashboard variant; second slot is None and the existing `dashboard_result is not None` guard in runner.py already handles it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * DAIP2-2314 + DAIP2-1687: pad DYNAMIC_PRIMARY + DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION over 1024-token cache floor - Pad DYNAMIC_PRIMARY_INSTRUCTION with three new sections: [SCOPE BOUNDARIES], [SOURCE TEXT INTERPRETATION], [REASONING DISCIPLINE], plus a [WORKED EXAMPLES] block. Estimated tokens: 447 -> 1117 (Sonnet 4.5 1024-min, +93 margin). All additions reinforce existing rules (sibling-field separation, alias mapping, pricing-vs-LOB distinction, contrastive-clause exclusion, exhibit-header binding) — no new directives that could bias extraction. - Pad DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION_INSTRUCTION with a [FINAL CHECKLIST BEFORE OUTPUT] block. Estimated tokens: 956 -> 1101 (Sonnet 4.5 1024-min, +77 margin). Reinforces the existing 4-step anti-duplication protocol and JSON shape requirements. - Register both new entries in cache_registry: DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION as INSTRUCTION_PLUS_CONTEXT (caches at warm-up), DYNAMIC_PRIMARY_ENTITIES as CONTEXT (instruction is intentionally short; CONTEXT c… * black format fix * Merged dev into feature/DAIP2-2314-DAIP2-1687-hybrid * fixed raw lob values in base lob field mapping and composite entities fix * black format fix * fixed LOB Program output issues * issue fixes * remove debugging code * Updated prompts * updated additional instructions * Update Program-->LOB * LLM-based AD-Program/Product mapping to LOB even when there is a crosswalk * black format fix * Merged dev into feature/DAIP2-2314-DAIP2-1687-hybrid * added logging in prompt call tracking * added updated logging in prompt call tracking * aaded min cache token per usage label * added cache registry for dynamic primary mapping prompt calls * reolved mapping prompts ambiguities * black format fix * Phase 2 modifications added * reverted phase 2 modifications Approved-by: Katon Minhas
Field Extraction Pipeline
Contract field extraction using LLMs.
Setup
Install uv (if not already installed)
# Ubuntu/WSL (recommended)
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc # or restart terminal
# macOS
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell - native, not WSL)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
Verify installation: uv --version
Install dependencies
uv sync
Running the Code
Use uv run with Python's module flag from the project root:
# Run the default SaaS pipeline
uv run python -m src.pipelines.saas.main
# Run with unified runner (supports client selection)
uv run python -m src.pipelines.runner --client saas --input-dir /path/to/input
# Run with specific client
uv run python -m src.pipelines.runner --client clover --input-dir /path/to/input
Note: Always use uv run to ensure correct virtual environment. Always use -m flag for proper imports.
Development
Before pushing, always run these checks to avoid breaking the CI pipeline:
# Format code
uv run black src/
# Type checking
uv run mypy src/
# Run tests
uv run pytest
All checks must pass before merging to main.
Project Structure
├── src/
│ ├── pipelines/ # Pipeline implementations
│ │ ├── runner.py # Unified CLI entry point with client routing
│ │ ├── saas/ # Default SaaS pipeline
│ │ │ └── main.py # Main entry point for SaaS
│ │ ├── shared/ # Shared pipeline components
│ │ │ ├── preprocessing/ # Document preprocessing
│ │ │ ├── extraction/ # Field extraction logic
│ │ │ └── postprocessing/ # Result postprocessing
│ │ └── clients/ # Client-specific overrides
│ │ └── clover/ # Clover client customizations
│ ├── core/ # Core utilities (registry, fieldset)
│ ├── constants/ # Constants, mappings, and field definitions
│ │ ├── mappings/ # Crosswalk JSON files
│ │ └── lists/ # Lookup lists
│ ├── prompts/ # LLM prompt templates
│ ├── utils/ # Shared utilities (IO, string, logging, etc.)
│ ├── codes/ # Medical code extraction utilities
│ ├── crosswalk/ # Crosswalk mapping logic
│ ├── embeddings/ # Pre-computed embeddings for code matching
│ ├── qc_qa/ # QC/QA validation pipeline
│ ├── parent_child/ # Parent-child relationship mapping
│ ├── document_classification/ # Document type classification (DTC)
│ └── tests/ # Unit tests
├── documentation/ # Project documentation
├── outputs/ # Pipeline output files (gitignored)
├── logs/ # Log files (gitignored)
└── pyproject.toml # Project dependencies (uv/pip)
Branching
Naming Conventions
| Type | Pattern | Use Case | Example |
|---|---|---|---|
| Feature | feature/<ticket>-<description> |
New functionality | feature/PROJ-123-add-export-csv |
| Bugfix | bugfix/<ticket>-<description> |
Bug fixes | bugfix/PROJ-456-fix-null-handling |
| Hotfix | hotfix/<ticket>-<description> |
Urgent production fixes | hotfix/PROJ-789-critical-parse-error |
| Test | test/<description> |
Testing/experimentation | test/lesser-table-caching-refactor |
| Release | release/<version> |
Release preparation | release/v1.2.0 |
Branch Guidelines
- Use lowercase with hyphens (kebab-case) for descriptions
- Include ticket number when applicable (e.g., JIRA, GitHub issue)
- Keep branch names concise but descriptive
- Delete branches after merging
Workflow
- Create branch from
main - Make changes and commit with clear messages
- Run checks before pushing:
uv run black src/ && uv run mypy src/ && uv run pytest - Create PR to
main - Ensure all CI checks pass
- Get code review approval
- Squash and merge