Dev
* Merged in bugfix/DAIP2-2823-generic-issue-fixes-one-to-one (pull request #1011)
Bugfix/DAIP2-2823 generic issue fixes one to one
* updated auto renewal ind prompt
* updated CONTRACT_AMENDMENT_NUM prompt
* Merged dev into bugfix/DAIP2-2823-generic-issue-fixes-one-to-one
Approved-by: Praneel Panchigar
Approved-by: Siddhant Medar
* Merged in feature/DAIP2-2698-phase-3-program-product-lob-mapping (pull request #1000)
Feature/DAIP2-2698 phase 3 program product lob mapping
* added missing phase 2 modifications
* added phase 3 modifications
* Fixed acronym issues
* black format fix
* Merged dev into feature/DAIP2-2698-phase-3-program-product-lob-mapping
* standardization fixes
* Merged dev into feature/DAIP2-2698-phase-3-program-product-lob-mapping
* added hyphenated suffix names fix
* black format fix
* standardization fix
* dynamic primary fix
* Merge Dev into feature/DAIP2-2698-phase-3-program-product-lob-mapping
* updated Program Product Standardization
* updated prompt instrcutions
* added documenttaion markdown file
* add Rule 5 counter-example for Plus-less sibling in PRODUCT canonical prompt
Clarify that Rule 5's abbreviation absorption does not extend to the
bare-form sibling when the anchor's expansion itself contains 'Plus'.
Addresses a row in the may18 canonical run where 'Molina Medicare
Options' (no Plus) was…
Approved-by: Siddhant Medar
Feature/DAIP2-2314 DAIP2 1687 hybrid
* remove -files from s3 prefix requirements
* Resolve input paths
* fix: VendorProcessor.process_file returns (df, None) tuple
runner.safe_process_file unpacks the result as (cc_df, dashboard_df), so
returning a single DataFrame caused every vendor/generic file to fail with
"too many values to unpack (expected 2)" — Python iterates DataFrame columns
during unpacking. Vendor pipelines have no dashboard variant; second slot is
None and the existing `dashboard_result is not None` guard in runner.py
already handles it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* DAIP2-2314 + DAIP2-1687: pad DYNAMIC_PRIMARY + DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION over 1024-token cache floor
- Pad DYNAMIC_PRIMARY_INSTRUCTION with three new sections: [SCOPE BOUNDARIES], [SOURCE TEXT INTERPRETATION], [REASONING DISCIPLINE], plus a [WORKED EXAMPLES] block. Estimated tokens: 447 -> 1117 (Sonnet 4.5 1024-min, +93 margin). All additions reinforce existing rules (sibling-field separation, alias mapping, pricing-vs-LOB distinction, contrastive-clause exclusion, exhibit-header binding) — no new directives that could bias extraction.
- Pad DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION_INSTRUCTION with a [FINAL CHECKLIST BEFORE OUTPUT] block. Estimated tokens: 956 -> 1101 (Sonnet 4.5 1024-min, +77 margin). Reinforces the existing 4-step anti-duplication protocol and JSON shape requirements.
- Register both new entries in cache_registry: DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION as INSTRUCTION_PLUS_CONTEXT (caches at warm-up), DYNAMIC_PRIMARY_ENTITIES as CONTEXT (instruction is intentionally short; CONTEXT c…
* black format fix
* Merged dev into feature/DAIP2-2314-DAIP2-1687-hybrid
* fixed raw lob values in base lob field mapping and composite entities fix
* black format fix
* fixed LOB Program output issues
* issue fixes
* remove debugging code
* Updated prompts
* updated additional instructions
* Update Program-->LOB
* LLM-based AD-Program/Product mapping to LOB even when there is a crosswalk
* black format fix
* Merged dev into feature/DAIP2-2314-DAIP2-1687-hybrid
* added logging in prompt call tracking
* added updated logging in prompt call tracking
* aaded min cache token per usage label
* added cache registry for dynamic primary mapping prompt calls
* reolved mapping prompts ambiguities
* black format fix
* Phase 2 modifications added
* reverted phase 2 modifications
Approved-by: Katon Minhas
Bugfix/DAIP2-1870 dynamic issues
* updated list format appending
* fill bill type from claim type cd
* Fixed remaining format fixes
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* changed crosswalk mapping from PRODUCT to AARETE_DERIVED_PRODUCT
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* updating LOB field values as well from PRODUCT AND PROGRAM
* updated apply crosswalk for reverse mapping
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* added QHP and BHP
* removed lob reverse mapping
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* Remove Bill Type/Claim Type fill
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* Black format
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
Approved-by: Katon Minhas
Bugfix/parser downstream improvements
* Refactor: Implement field-aware JSON parsers with centralized normalization
This refactor introduces a robust system for normalizing LLM output based on
field format mappings, ensuring consistent data types throughout the pipeline.
Key Changes:
- Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields
- Create format_normalization.py utility for type-aware normalization
- Update json_utils.py parsers to accept field_names/field_name parameters
- Refactor prompt_templates.py to use parser factories (_create_json_dict_parser,
_create_json_list_parser) that bind field metadata for automatic normalization
- Update prompt_calls.py to pass field names to parsers, eliminating redundant
normalization logic
- Remove parse_json_dict_or_list (unused, ambiguous function)
- Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions
- Add comprehensive integration tests verifying normalization works end-to-end
Benefits:
- Single source of truth for field formats (FIELD_FORMAT_…
* refactor: normalize helper prompt outputs at prompt_calls level
- Update CARVEOUT_CHECK to use field-aware parser for CARVEOUT_CD normalization
- Update LOB_RELATIONSHIP to normalize to string format in prompt_calls.py
- Update SPLIT_REIMB_DATES to normalize date values in prompt_calls.py
- Remove defensive normalization from one_to_n_funcs.py for LOB relationships
- Remove manual normalization from split_reimb_dates() - values now normalized upstream
- All helper prompts that populate fields now normalize at prompt_calls.py level
- Downstream functions receive correctly formatted values without additional processing
* refactor: remove band-aid normalization functions and migrate HSC to field-aware parsers
- Update ONE_TO_ONE_SINGLE_FIELD_TEMPLATE to use field-aware parser with field_name parameter
- Remove list wrapping logic in hybrid_smart_chunking_funcs (field-aware parser handles normalization)
- Remove normalize_one_to_one_field_value and normalize_one_to_one_answers_dict from string_utils.py
- Remove all debug print statements from HSC processing
- Remove commented-out normalization calls from client-specific files (clover, bcbs_promise)
- All normalization now handled exclusively through FIELD_FORMAT_MAPPING via field-aware parsers
* Ran Black for formatting
* Print Statements removed, more cleaning
* Merge branch 'DEV' into bugfix/parser-downstream-improvements
* refactor: combine FIELD_FORMAT_MAPPING into investment_columns.py
- Merged field_format_mapping.py into investment_columns.py to create single source of truth
- FIELD_FORMAT_MAPPING now ordered by COLUMN_ORDER (161 fields)
- Added 5 missing fields from COLUMN_ORDER with default format types
- Updated all imports across codebase to use investment_columns
- Python dict preserves insertion order (3.7+), maintaining COLUMN_ORDER sequence
- All tests passing (38 field-aware tests verified)
* Deprecate COLUMN_ORDER, rely on Mapping only
* Merged DEV into bugfix/parser-downstream-improvements
Approved-by: Katon Minhas
- Updated all prompt_calls.py files (saas, bcbs_promise, clover) to use new parser pattern
- Updated prompt_templates.py with JSON format instructions
- Updated crosswalk_utils.py to handle JSON lists
- Updated json_utils.py with improved parsing
- Updated aarete_derived.py for JSON compatibility
- Updated tin_npi_funcs.py and qa_qc_utils.py for JSON parsing
- All changes align with Phase 4 completion of JSON standardization
- Fixed prompt_lesser_of_distribution to extract string from list when parser returns list
- Added debug print statements to trace REIMB_TERM format through processing pipeline
- Removed defensive normalization checks that are no longer needed upstream
- Added debug output in reimbursement_level and methodology_breakout_single_row to track data flow
Feature/lesser table caching refactor hybrid
* chore: Remove unused duplicate main.py from shared pipeline
* fix: Correct crosswalk paths in aarete_derived.py
* chore: Remove unused documentation files from fieldExtraction
* docs: Add documentation files to documentation folder
* docs: Update README with uv setup, expanded project structure, and branching conventions
* docs: Add uv installation steps with Ubuntu/WSL emphasis
* Enable prompt caching for all remaining LLM calls
- Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE,
EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH,
SPECIAL_CASE_ASSIGNMENT
- Update all invoke_claude() calls in saas and clover pipelines to use
cache=True with corresponding _INSTRUCTION() functions
- Add new instructions to get_cacheable_instructions() for cache warming
- Update tests for new instruction functions
Functions now using caching:
- prompt_exhibit_level
- prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF)
- prompt_fee_schedule_breakout
- prompt_grouper_breakout
- prompt_special_case_assignment
- prompt_exhibit_linkage
- prompt_exhibit_header
- prompt_smart_chunked (ONE_TO_ONE templates)
- prompt_date_fix
- prompt_derived_term_date
- prompt_exhibit_title_match
- provider_name_match_check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Reorder
* feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction
- Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction
- Extract full paragraph text of offset/recoupment provisions from contracts
- Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence
- Fix reorder_columns to preserve extra columns not in COLUMN_ORDER
- Update QC/QA output path to outputs/qc_qa/
* fix: Update dev deps and test assertions for QC/QA output path
- Add pytest/pytest-mock to dev dependencies for mypy type checking
- Update test assertions to expect outputs/qc_qa instead of qa_qc_output
* style: Apply black formatting to prompt_templates.py
* Merge main, move scripts
* Archive some scripts
* update py version
* remove .py version file
* Remove ASCII characters
* Restore testbed code
* restore tracking
* Update testbed metrics
* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT
- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix postprocessing_funcs to remove invalid columns
* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid
* Revert prompt caching changes from aed1b73c
* update formatting
* Update imports
Approved-by: Sha Brown
Approved-by: Praneel Panchigar
Praneel Panchigar