Bugfix/generic dynamic jan26
* Refactor: Extract dynamic primary metrics logic to separate module
- Created src/testbed/dynamic_primary_metrics.py with all dynamic primary field analysis logic
- Moved tally-based analysis functions to new module for better modularity
- Updated testbed_utils.py to import and use functions from dynamic_primary_metrics
- Simplified testbed_metrics_dynamic_only.py to use new module
- Removed unused analyze_dynamic_primary_fields function (400+ lines)
- Added field filtering in match_rows to handle missing columns gracefully
- Minimal changes to testbed_utils.py (~100 lines vs 845 before)
* feat: Add debug print blocks and prompt changes for dynamic primary fields in 1:1 processing
This commit adds prompt changes for when we pass dynamic primary to 1:1 and comprehensive debug logging for dynamic primary fields when they are escalated to 1:1 processing via Hybrid Smart Chunking (HSC) or Full Context methods.
Changes:
- Added debug print blocks in hybrid_smart_chunking_funcs.py to log:
* Retrieval question, context chunks, and final prompt for HSC processing
* Raw LLM output (reasoning + answer) for dynamic primary fields
- Added debug print blocks in prompt_calls.py to log:
* Contract context, final prompt for Full Context processing
* Raw LLM output (reasoning + answer) for dynamic primary fields
- Added FULL_CONTEXT_DYNAMIC_PRIMARY_INSTRUCTION() to prompt_templates.py:
* Provides specific guidance for dynamic primary fields in full context
* Emphasizes extracting values only if they refer to entire contract
* Includes pipe-delimited formatting instructions
- Updated dynamic_fun…
* Ran black, for formatting
* Refine dynamic primary field prompts and add Duals auto-detection
- Enhanced DYNAMIC_ASSIGNMENT_INSTRUCTION with structural boundary rules
- Added exhibit header and subsection context hierarchy for LOB assignment
- Implemented Duals auto-detection in dynamic_primary discovery (Medicare+Medicaid -> Duals)
- Added Duals value formatting instruction to prevent oversimplification
- Commented out DUAL_LOB_CHECK for performance testing
- Removed verbose debug print blocks from HSC and Full Context processing
- Added focused debug print for dynamic assignment raw LLM output
* Remove debug print blocks from pipeline files
- Removed all debug print blocks from file_processing.py (8 blocks)
- Removed all debug print blocks from dynamic_funcs.py (8 blocks)
- Removed debug print block from prompt_calls.py (dynamic assignment)
- DUAL_LOB_CHECK remains commented as requested
- Total: 188 lines of debug code removed
* Merge remote-tracking branch 'origin/main' into bugfix/generic_dynamic_jan26
* Ran black for CI
* Remove update_lob_for_duals
* Merge branch 'DEV' into bugfix/generic_dynamic_jan26
Approved-by: Katon Minhas
Feature/lesser table caching refactor hybrid
* chore: Remove unused duplicate main.py from shared pipeline
* fix: Correct crosswalk paths in aarete_derived.py
* chore: Remove unused documentation files from fieldExtraction
* docs: Add documentation files to documentation folder
* docs: Update README with uv setup, expanded project structure, and branching conventions
* docs: Add uv installation steps with Ubuntu/WSL emphasis
* Enable prompt caching for all remaining LLM calls
- Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE,
EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH,
SPECIAL_CASE_ASSIGNMENT
- Update all invoke_claude() calls in saas and clover pipelines to use
cache=True with corresponding _INSTRUCTION() functions
- Add new instructions to get_cacheable_instructions() for cache warming
- Update tests for new instruction functions
Functions now using caching:
- prompt_exhibit_level
- prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF)
- prompt_fee_schedule_breakout
- prompt_grouper_breakout
- prompt_special_case_assignment
- prompt_exhibit_linkage
- prompt_exhibit_header
- prompt_smart_chunked (ONE_TO_ONE templates)
- prompt_date_fix
- prompt_derived_term_date
- prompt_exhibit_title_match
- provider_name_match_check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Reorder
* feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction
- Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction
- Extract full paragraph text of offset/recoupment provisions from contracts
- Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence
- Fix reorder_columns to preserve extra columns not in COLUMN_ORDER
- Update QC/QA output path to outputs/qc_qa/
* fix: Update dev deps and test assertions for QC/QA output path
- Add pytest/pytest-mock to dev dependencies for mypy type checking
- Update test assertions to expect outputs/qc_qa instead of qa_qc_output
* style: Apply black formatting to prompt_templates.py
* Merge main, move scripts
* Archive some scripts
* update py version
* remove .py version file
* Remove ASCII characters
* Restore testbed code
* restore tracking
* Update testbed metrics
* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT
- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix postprocessing_funcs to remove invalid columns
* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid
* Revert prompt caching changes from aed1b73c
* update formatting
* Update imports
Approved-by: Sha Brown
Approved-by: Praneel Panchigar
Katon Minhas