Katon Minhas
afb6d5185d
Feature/lesser table caching refactor hybrid * chore: Remove unused duplicate main.py from shared pipeline * fix: Correct crosswalk paths in aarete_derived.py * chore: Remove unused documentation files from fieldExtraction * docs: Add documentation files to documentation folder * docs: Update README with uv setup, expanded project structure, and branching conventions * docs: Add uv installation steps with Ubuntu/WSL emphasis * Enable prompt caching for all remaining LLM calls - Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE, EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH, SPECIAL_CASE_ASSIGNMENT - Update all invoke_claude() calls in saas and clover pipelines to use cache=True with corresponding _INSTRUCTION() functions - Add new instructions to get_cacheable_instructions() for cache warming - Update tests for new instruction functions Functions now using caching: - prompt_exhibit_level - prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF) - prompt_fee_schedule_breakout - prompt_grouper_breakout - prompt_special_case_assignment - prompt_exhibit_linkage - prompt_exhibit_header - prompt_smart_chunked (ONE_TO_ONE templates) - prompt_date_fix - prompt_derived_term_date - prompt_exhibit_title_match - provider_name_match_check 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Reorder * feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction - Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction - Extract full paragraph text of offset/recoupment provisions from contracts - Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence - Fix reorder_columns to preserve extra columns not in COLUMN_ORDER - Update QC/QA output path to outputs/qc_qa/ * fix: Update dev deps and test assertions for QC/QA output path - Add pytest/pytest-mock to dev dependencies for mypy type checking - Update test assertions to expect outputs/qc_qa instead of qa_qc_output * style: Apply black formatting to prompt_templates.py * Merge main, move scripts * Archive some scripts * update py version * remove .py version file * Remove ASCII characters * Restore testbed code * restore tracking * Update testbed metrics * Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT - Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification - Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination - Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification - Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT - Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK - Add new instructions to get_cacheable_instructions() for cache warming - Add unit tests for new instruction functions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix postprocessing_funcs to remove invalid columns * Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid * Revert prompt caching changes from aed1b73c * update formatting * Update imports Approved-by: Sha Brown Approved-by: Praneel Panchigar
124 lines
4.3 KiB
Markdown
124 lines
4.3 KiB
Markdown
# Field Extraction Pipeline
|
|
|
|
Contract field extraction using LLMs.
|
|
|
|
## Setup
|
|
|
|
### Install uv (if not already installed)
|
|
|
|
```bash
|
|
# Ubuntu/WSL (recommended)
|
|
curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
source ~/.bashrc # or restart terminal
|
|
|
|
# macOS
|
|
curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
|
|
# Windows (PowerShell - native, not WSL)
|
|
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
|
|
```
|
|
|
|
Verify installation: `uv --version`
|
|
|
|
### Install dependencies
|
|
|
|
```bash
|
|
uv sync
|
|
```
|
|
|
|
## Running the Code
|
|
|
|
Use `uv run` with Python's module flag from the project root:
|
|
|
|
```bash
|
|
# Run the default SaaS pipeline
|
|
uv run python -m src.pipelines.saas.main
|
|
|
|
# Run with unified runner (supports client selection)
|
|
uv run python -m src.pipelines.runner --client saas --input-dir /path/to/input
|
|
|
|
# Run with specific client
|
|
uv run python -m src.pipelines.runner --client clover --input-dir /path/to/input
|
|
```
|
|
|
|
**Note:** Always use `uv run` to ensure correct virtual environment. Always use `-m` flag for proper imports.
|
|
|
|
## Development
|
|
|
|
**Before pushing, always run these checks to avoid breaking the CI pipeline:**
|
|
|
|
```bash
|
|
# Format code
|
|
uv run black src/
|
|
|
|
# Type checking
|
|
uv run mypy src/
|
|
|
|
# Run tests
|
|
uv run pytest
|
|
```
|
|
|
|
All checks must pass before merging to main.
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
├── src/
|
|
│ ├── pipelines/ # Pipeline implementations
|
|
│ │ ├── runner.py # Unified CLI entry point with client routing
|
|
│ │ ├── saas/ # Default SaaS pipeline
|
|
│ │ │ └── main.py # Main entry point for SaaS
|
|
│ │ ├── shared/ # Shared pipeline components
|
|
│ │ │ ├── preprocessing/ # Document preprocessing
|
|
│ │ │ ├── extraction/ # Field extraction logic
|
|
│ │ │ └── postprocessing/ # Result postprocessing
|
|
│ │ └── clients/ # Client-specific overrides
|
|
│ │ └── clover/ # Clover client customizations
|
|
│ ├── core/ # Core utilities (registry, fieldset)
|
|
│ ├── constants/ # Constants, mappings, and field definitions
|
|
│ │ ├── mappings/ # Crosswalk JSON files
|
|
│ │ └── lists/ # Lookup lists
|
|
│ ├── prompts/ # LLM prompt templates
|
|
│ ├── utils/ # Shared utilities (IO, string, logging, etc.)
|
|
│ ├── codes/ # Medical code extraction utilities
|
|
│ ├── crosswalk/ # Crosswalk mapping logic
|
|
│ ├── embeddings/ # Pre-computed embeddings for code matching
|
|
│ ├── qc_qa/ # QC/QA validation pipeline
|
|
│ ├── parent_child/ # Parent-child relationship mapping
|
|
│ ├── document_classification/ # Document type classification (DTC)
|
|
│ └── tests/ # Unit tests
|
|
├── documentation/ # Project documentation
|
|
├── outputs/ # Pipeline output files (gitignored)
|
|
├── logs/ # Log files (gitignored)
|
|
└── pyproject.toml # Project dependencies (uv/pip)
|
|
```
|
|
|
|
## Branching
|
|
|
|
### Naming Conventions
|
|
|
|
| Type | Pattern | Use Case | Example |
|
|
|------|---------|----------|---------|
|
|
| Feature | `feature/<ticket>-<description>` | New functionality | `feature/PROJ-123-add-export-csv` |
|
|
| Bugfix | `bugfix/<ticket>-<description>` | Bug fixes | `bugfix/PROJ-456-fix-null-handling` |
|
|
| Hotfix | `hotfix/<ticket>-<description>` | Urgent production fixes | `hotfix/PROJ-789-critical-parse-error` |
|
|
| Test | `test/<description>` | Testing/experimentation | `test/lesser-table-caching-refactor` |
|
|
| Release | `release/<version>` | Release preparation | `release/v1.2.0` |
|
|
|
|
### Branch Guidelines
|
|
|
|
- Use lowercase with hyphens (kebab-case) for descriptions
|
|
- Include ticket number when applicable (e.g., JIRA, GitHub issue)
|
|
- Keep branch names concise but descriptive
|
|
- Delete branches after merging
|
|
|
|
### Workflow
|
|
|
|
1. Create branch from `main`
|
|
2. Make changes and commit with clear messages
|
|
3. Run checks before pushing: `uv run black src/ && uv run mypy src/ && uv run pytest`
|
|
4. Create PR to `main`
|
|
5. Ensure all CI checks pass
|
|
6. Get code review approval
|
|
7. Squash and merge
|