DEV
* file_processing
* Merge branch 'bugfix/code_implicit' into DEV
* Update split reimb dates condition
* Pipe line general fixes, fixed tests, ran black, and fixed mypy type check issues, still have a list of lists issue with prov other name full
* Merged in feature/update-claim-type (pull request #869)
Feature/update claim type
* Update CLAIM_TYPE_CD extraction to prioritize title/header
- Modified retrieval_question to search for title, header, and agreement name
- Updated prompt to check title/header first before analyzing body text
- Enhanced FULL_CONTEXT_CLAIM_TYPES_ADDITIONAL_INSTRUCTION to emphasize title/header as primary source
* Fix CLAIM_TYPE_CD extraction with 3-step fallback
1. Exhibit Level: Updated retrieval_question and prompt to prioritize title/header
2. Contract Title fallback: Added infer_claim_type_from_title() function that extracts
claim type from CONTRACT_TITLE when exhibit-level extraction returns empty
3. Postprocessing: fill_claim_type() now uses CONTRACT_TITLE inference when all
AARETE_DERIVED_CLAIM_TYPE_CD values are empty
Keywords mapped:
- Professional/Ancillary -> M (Physician, Professional, Ancillary, Home Health, DME, etc.)
- Institutional -> H (Hospital, Facility, Surgery Center, SNF, etc.)
* Add more prof…
* Merged in bugfix/code-funcs (pull request #872)
Bugfix/code funcs
* Try-except code extraction
* Additional try-except for protection
* Black format
Approved-by: Praneel Panchigar
* Merged in feature/update-testbed-metrics (pull request #871)
Feature/update testbed metrics
* Combine fee schedule with methodology breakout; remove Reimbursement primary; do not run trigger cap or additions
* Deprecate reimb primary functions
* combine multiple sheets to one
* Clean 1:N metrics
* Improve dynamic primary evaluation
* Resolve dynamic primary
* Update row counts
* black format
* Merged DEV into feature/update-testbed-metrics
Approved-by: Praneel Panchigar
* Merged in feature/adding_contract_admenment_num_pc (pull request #870)
Feature/adding contract admenment num pc
* Updated contract amendment
* Format code with Black
* updated to have letters
* format fixes
* Adjusted unit testing
* Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc
* apply formatting
* format fixes
* Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc
* test conflicts solved
* Format code with Black
* format fixes
* reverted the changes for prompt templates
* test_case reverted
* Sorted effective date for ranking
* format fix for lint
* Merged DEV into feature/adding_contract_admenment_num_pc
* format
* Merged DEV into feature/adding_contract_admenment_num_pc
* Merged DEV into feature/adding_contract_admenment_num_pc
Approved-by: Siddhant Medar
* Consolidate docs and documentation
* Remove PRD
* Add Essential Plan as a NY Program
* Resolve Comments
* Resolve comments
* Black format
* Resolve remaining comments
* Black format
* Merged in bugfix/fill_claim_type (pull request #874)
Bugfix/fill claim type
* Handle if AARETE_DERIVED_CLAIM_TYPE_CD comes in as list
* add unit tests
* Black format
Approved-by: Siddhant Medar
* Remove print
* Merged in bugfix/parser-downstream-improvements (pull request #875)
Bugfix/parser downstream improvements
* Refactor: Implement field-aware JSON parsers with centralized normalization
This refactor introduces a robust system for normalizing LLM output based on
field format mappings, ensuring consistent data types throughout the pipeline.
Key Changes:
- Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields
- Create format_normalization.py utility for type-aware normalization
- Update json_utils.py parsers to accept field_names/field_name parameters
- Refactor prompt_templates.py to use parser factories (_create_json_dict_parser,
_create_json_list_parser) that bind field metadata for automatic normalization
- Update prompt_calls.py to pass field names to parsers, eliminating redundant
normalization logic
- Remove parse_json_dict_or_list (unused, ambiguous function)
- Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions
- Add comprehensive integration tests…
* Update other lists
* Black format
* Remove prints
* Update qa_qc to account for lists in is_empty check
* Resolve issues
* Black format
Approved-by: Siddhant Medar
Bugfix/parser downstream improvements
* Refactor: Implement field-aware JSON parsers with centralized normalization
This refactor introduces a robust system for normalizing LLM output based on
field format mappings, ensuring consistent data types throughout the pipeline.
Key Changes:
- Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields
- Create format_normalization.py utility for type-aware normalization
- Update json_utils.py parsers to accept field_names/field_name parameters
- Refactor prompt_templates.py to use parser factories (_create_json_dict_parser,
_create_json_list_parser) that bind field metadata for automatic normalization
- Update prompt_calls.py to pass field names to parsers, eliminating redundant
normalization logic
- Remove parse_json_dict_or_list (unused, ambiguous function)
- Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions
- Add comprehensive integration tests verifying normalization works end-to-end
Benefits:
- Single source of truth for field formats (FIELD_FORMAT_…
* refactor: normalize helper prompt outputs at prompt_calls level
- Update CARVEOUT_CHECK to use field-aware parser for CARVEOUT_CD normalization
- Update LOB_RELATIONSHIP to normalize to string format in prompt_calls.py
- Update SPLIT_REIMB_DATES to normalize date values in prompt_calls.py
- Remove defensive normalization from one_to_n_funcs.py for LOB relationships
- Remove manual normalization from split_reimb_dates() - values now normalized upstream
- All helper prompts that populate fields now normalize at prompt_calls.py level
- Downstream functions receive correctly formatted values without additional processing
* refactor: remove band-aid normalization functions and migrate HSC to field-aware parsers
- Update ONE_TO_ONE_SINGLE_FIELD_TEMPLATE to use field-aware parser with field_name parameter
- Remove list wrapping logic in hybrid_smart_chunking_funcs (field-aware parser handles normalization)
- Remove normalize_one_to_one_field_value and normalize_one_to_one_answers_dict from string_utils.py
- Remove all debug print statements from HSC processing
- Remove commented-out normalization calls from client-specific files (clover, bcbs_promise)
- All normalization now handled exclusively through FIELD_FORMAT_MAPPING via field-aware parsers
* Ran Black for formatting
* Print Statements removed, more cleaning
* Merge branch 'DEV' into bugfix/parser-downstream-improvements
* refactor: combine FIELD_FORMAT_MAPPING into investment_columns.py
- Merged field_format_mapping.py into investment_columns.py to create single source of truth
- FIELD_FORMAT_MAPPING now ordered by COLUMN_ORDER (161 fields)
- Added 5 missing fields from COLUMN_ORDER with default format types
- Updated all imports across codebase to use investment_columns
- Python dict preserves insertion order (3.7+), maintaining COLUMN_ORDER sequence
- All tests passing (38 field-aware tests verified)
* Deprecate COLUMN_ORDER, rely on Mapping only
* Merged DEV into bugfix/parser-downstream-improvements
Approved-by: Katon Minhas
Feature/adding contract admenment num pc
* Updated contract amendment
* Format code with Black
* updated to have letters
* format fixes
* Adjusted unit testing
* Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc
* apply formatting
* format fixes
* Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc
* test conflicts solved
* Format code with Black
* format fixes
* reverted the changes for prompt templates
* test_case reverted
* Sorted effective date for ranking
* format fix for lint
* Merged DEV into feature/adding_contract_admenment_num_pc
* format
* Merged DEV into feature/adding_contract_admenment_num_pc
* Merged DEV into feature/adding_contract_admenment_num_pc
Approved-by: Siddhant Medar
Feature/update testbed metrics
* Combine fee schedule with methodology breakout; remove Reimbursement primary; do not run trigger cap or additions
* Deprecate reimb primary functions
* combine multiple sheets to one
* Clean 1:N metrics
* Improve dynamic primary evaluation
* Resolve dynamic primary
* Update row counts
* black format
* Merged DEV into feature/update-testbed-metrics
Approved-by: Praneel Panchigar
Feature/update claim type
* Update CLAIM_TYPE_CD extraction to prioritize title/header
- Modified retrieval_question to search for title, header, and agreement name
- Updated prompt to check title/header first before analyzing body text
- Enhanced FULL_CONTEXT_CLAIM_TYPES_ADDITIONAL_INSTRUCTION to emphasize title/header as primary source
* Fix CLAIM_TYPE_CD extraction with 3-step fallback
1. Exhibit Level: Updated retrieval_question and prompt to prioritize title/header
2. Contract Title fallback: Added infer_claim_type_from_title() function that extracts
claim type from CONTRACT_TITLE when exhibit-level extraction returns empty
3. Postprocessing: fill_claim_type() now uses CONTRACT_TITLE inference when all
AARETE_DERIVED_CLAIM_TYPE_CD values are empty
Keywords mapped:
- Professional/Ancillary -> M (Physician, Professional, Ancillary, Home Health, DME, etc.)
- Institutional -> H (Hospital, Facility, Surgery Center, SNF, etc.)
* Add more professional keywords for claim type inference
Added PROVIDER SERVICE, PROVIDER AGREEMENT, PARTICIPATING PROVIDER to handle
titles like 'PROVIDER SERVICES AGREEMENT'
* Add postprocessing safety net for claim type inference
- Added fill_claim_type_from_title() in postprocessing_funcs.py that infers
AARETE_DERIVED_CLAIM_TYPE_CD from CONTRACT_TITLE if still empty after code_breakout
- Added call in postprocess.py before attach_sid_column
- This ensures claim type is populated even if earlier extraction steps miss it
* Improve CLAIM_TYPE_CD extraction context and postprocessing
1. Added keywords to CLAIM_TYPE_CD field (same as CONTRACT_TITLE) so it retrieves
from the same document header/title context
2. Updated fill_claim_type_from_title() to first fill from other exhibits in the
same file (mode within file) before falling back to CONTRACT_TITLE keyword matching
* Merge DEV into feature/update-claim-type
Resolved conflicts:
- code_funcs.py: Kept DEV's code_breakout (claim type now handled in postprocessing)
- postprocess.py: Added fill_claim_type_from_title call
- prompt_templates.py: Kept DEV's VALIDATE_REIMBURSEMENTS_PROMPT signature
* Merge DEV into feature/update-claim-type
* move keywords to ancillary
Approved-by: Katon Minhas
- Updated all prompt_calls.py files (saas, bcbs_promise, clover) to use new parser pattern
- Updated prompt_templates.py with JSON format instructions
- Updated crosswalk_utils.py to handle JSON lists
- Updated json_utils.py with improved parsing
- Updated aarete_derived.py for JSON compatibility
- Updated tin_npi_funcs.py and qa_qc_utils.py for JSON parsing
- All changes align with Phase 4 completion of JSON standardization
- Removed all debug print statements from prompt_calls.py and one_to_n_funcs.py
- Updated PRD to reflect Phase 4 completion and recent bug fixes
- Documented REIMB_TERM normalization fixes and cleanup work
- Updated Phase 6 status to 25% complete (debug removal done)
- Fixed prompt_lesser_of_distribution to extract string from list when parser returns list
- Added debug print statements to trace REIMB_TERM format through processing pipeline
- Removed defensive normalization checks that are no longer needed upstream
- Added debug output in reimbursement_level and methodology_breakout_single_row to track data flow
Trigger error when batch_id is not specified or when max_workers is negative
* Trigger error when batch_id is not specified or when max_workers is negative
Approved-by: Katon Minhas
Feature/adding contract admenment num pc
* Updated contract amendment
* Format code with Black
* updated to have letters
* format fixes
* Adjusted unit testing
* Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc
* apply formatting
Katon Minhas