Feature/lesser table caching refactor hybrid
* chore: Remove unused duplicate main.py from shared pipeline
* fix: Correct crosswalk paths in aarete_derived.py
* chore: Remove unused documentation files from fieldExtraction
* docs: Add documentation files to documentation folder
* docs: Update README with uv setup, expanded project structure, and branching conventions
* docs: Add uv installation steps with Ubuntu/WSL emphasis
* Enable prompt caching for all remaining LLM calls
- Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE,
EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH,
SPECIAL_CASE_ASSIGNMENT
- Update all invoke_claude() calls in saas and clover pipelines to use
cache=True with corresponding _INSTRUCTION() functions
- Add new instructions to get_cacheable_instructions() for cache warming
- Update tests for new instruction functions
Functions now using caching:
- prompt_exhibit_level
- prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF)
- prompt_fee_schedule_breakout
- prompt_grouper_breakout
- prompt_special_case_assignment
- prompt_exhibit_linkage
- prompt_exhibit_header
- prompt_smart_chunked (ONE_TO_ONE templates)
- prompt_date_fix
- prompt_derived_term_date
- prompt_exhibit_title_match
- provider_name_match_check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Reorder
* feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction
- Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction
- Extract full paragraph text of offset/recoupment provisions from contracts
- Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence
- Fix reorder_columns to preserve extra columns not in COLUMN_ORDER
- Update QC/QA output path to outputs/qc_qa/
* fix: Update dev deps and test assertions for QC/QA output path
- Add pytest/pytest-mock to dev dependencies for mypy type checking
- Update test assertions to expect outputs/qc_qa instead of qa_qc_output
* style: Apply black formatting to prompt_templates.py
* Merge main, move scripts
* Archive some scripts
* update py version
* remove .py version file
* Remove ASCII characters
* Restore testbed code
* restore tracking
* Update testbed metrics
* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT
- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix postprocessing_funcs to remove invalid columns
* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid
* Revert prompt caching changes from aed1b73c
* update formatting
* Update imports
Approved-by: Sha Brown
Approved-by: Praneel Panchigar
Restructure/facility adjustments group to main
* removed _PCT_RATE and _FEE_RATE in facility adjustment_breakout
* moved facility adjustments term to exhibit level
* added facility adjustmnet breakout
* Merged main into restructure/facility_adjustments_group
* Update breakout
Approved-by: Katon Minhas
Feature/multithreading
* fixes
* Merge main into feature/multithreading
Resolved conflicts:
- Kept timing instrumentation in file_processing.py
- Kept new 3-step Exhibit-based approach for one-to-n processing
- Maintained parallelization improvements (20 workers)
Changes include:
- Timing utils integration for performance monitoring
- Increased max_workers from 5 to 20 across all components
- Parallelized code_breakout and grouper_breakout
- Fixed tin_npi_funcs function call parameters
* Fix max_workers error for empty documents
- Add check to skip parallel processing when no pages exist
- Use min(len(all_page_tasks), 20) to prevent max_workers=0
- Handles edge case of documents with no exhibits or pages
* Fix max_workers=0 errors in one_to_n_funcs
- Add checks before all ThreadPoolExecutor creations
- Prevents errors when processing empty lists:
- carveout_and_special_case
- breakout
- special_case_breakout
- filter_services_without_reimbursements
- run_lob_relationship
- Ensures executor only created when there are items to process
* Reduce code processing parallelism to prevent API throttling
Lower max_workers from 20 to 10 for code_breakout and grouper_breakout
to prevent overwhelming Bedrock API with concurrent requests
* fixed
* Merged main into feature/multithreading
* Move documentation into folder
* Fix logging statements
* Merge branch 'main' into feature/multithreading
* Refactor for clarity
* Update previous exhibit passing logic
* properly simplify exhibits
* Merged main into feature/multithreading
* update conditional for None
* exhibit multithreading changed
* exhibit multithreading changed
* Merge remote-tracking branch 'origin/main' into feature/multithreading
* synced with main
* Parallelize dynamic assignment, refactor HSC field worker, add timing/exhibit unit tests, tidy imports/ignore helpers
* analyze_regression.py edited online with Bitbucket
* count_pages.py edited online with Bitbucket
* compare_regressed_with_baseline.py edited online with Bitbucket
* simple_testbed_compare.py edited online with Bitbucket
* run_testbed_metrics_regressed.py edited online with Bitbucket
Approved-by: Katon Minhas
Bugfix/dynamic issues
* Add debug print statements for dynamic primary field discovery and 1:1 escalation tracking
* Add reimb_term to bill type CD determination context
- Updated fill_bill_type() to accept reimb_term parameter
- Modified FILL_BILL_TYPE prompt to include reimbursement term in context
- First attempt now uses: service term + reimbursement term
- Second attempt uses: service term + reimbursement term + exhibit text
- Updated extract_codes_from_service() to pass REIMB_TERM to fill_bill_type()
* Remove debug print statements from dynamic_funcs.py and file_processing.py
- Removed all DEBUG print statements for dynamic primary discovery
- Removed debug prints for field discovery status and 1:1 escalation evaluation
- Removed debug prints for exhibit level answers and final summaries
- Removed debug comment about one_to_one_fields creation
- Cleaned up all temporary debugging code
* Merge branch 'main' into bugfix/dynamic_issues
* Remove duplicate one_to_one_fields FieldSet initialization from process_file
Approved-by: Katon Minhas
Bugfix/hmk issues
* fix for missing AARETE_DERIVED_LOB
* fix for AARETE_DERIVED_SID
* bugfix in universal_json_load
* bugfix for normalize_state_field
* reimb eff date assignmnet prompt update
* prompt update for missing reimb terms
* Merged main into bugfix/hmk-issues
* test case for universal_json_loads
* reveresed the prompt update for missing reimb
* prompt fix for reimb dates
Approved-by: Katon Minhas
bugfix/carveout to main
* removed specific service in base covered
* Merged main into bugfix/carveout
* Merged main into bugfix/carveout
Approved-by: Katon Minhas
Bugfix/bill type code fixes
* Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code
- Add format requirements to LOB_RELATIONSHIP_INSTRUCTION for proper caching
- Strengthen LOB_RELATIONSHIP prompt with explicit pipe delimiter requirements
- Preserve all DEBUG blocks and print statements in dynamic_funcs.py and file_processing.py
- Update crosswalk mappings and gitignore
- Ensure format instructions are cached at API level to prevent missing pipe delimiters
* Merge main into bugfix/dynamic_issuefixes
Resolved merge conflicts in:
- fieldExtraction/src/investment/dynamic_funcs.py (merged debug print statements)
- fieldExtraction/src/investment/file_processing.py (merged debug statements and exhibit inheritance code)
All conflicts resolved successfully.
* Update FILL_BILL_TYPE prompt with place-of-service priority and optional exhibit_text
- Add optional exhibit_text parameter to FILL_BILL_TYPE function
- Update instructions to prioritize place-of-service over service-only terminology
- Add new examples: 'home dialysis' and 'dialysis' while keeping original examples
- Add conditional exhibit context section when exhibit_text is provided
- Maintains backward compatibility (exhibit_text defaults to None)
Ref: PLAN_BILL_TYPE_CD_IMPROVEMENTS.md
* Implement two-step bill type CD determination with exhibit text fallback
- Add EXHIBIT_TEXT to answer_dict in combine_one_to_n() for contextual analysis
- Update fill_bill_type() to use two-step approach:
1. First attempt: service term only (no exhibit text)
2. Second attempt: service term + exhibit text (only if first fails)
- Update extract_codes_from_service() to pass exhibit_text to fill_bill_type()
- Add debug print when full exhibit text context is used
- More efficient: only makes second LLM call when service term alone fails
Ref: PLAN_BILL_TYPE_CD_IMPROVEMENTS.md
* Remove debug print statement and merge main into bill_type_code_fixes
- Remove debug print statement from fill_bill_type() in code_funcs.py
- Merge latest changes from main branch
- Includes updates to prompt_templates.py (payer_name parameter)
- Includes updates to file_processing.py (provider info ordering)
- Includes updates to aarete_derived.py, crosswalk mappings, and tin_npi_funcs
* Remove remaining debug print statements from dynamic_funcs.py and file_processing.py
Approved-by: Katon Minhas
Bugfix/derived lob changes
* Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code
- Add format requirements to LOB_RELATIONSHIP_INSTRUCTION for proper caching
- Strengthen LOB_RELATIONSHIP prompt with explicit pipe delimiter requirements
- Preserve all DEBUG blocks and print statements in dynamic_funcs.py and file_processing.py
- Update crosswalk mappings and gitignore
- Ensure format instructions are cached at API level to prevent missing pipe delimiters
* Merge main into bugfix/dynamic_issuefixes
Resolved merge conflicts in:
- fieldExtraction/src/investment/dynamic_funcs.py (merged debug print statements)
- fieldExtraction/src/investment/file_processing.py (merged debug statements and exhibit inheritance code)
All conflicts resolved successfully.
* Fix AARETE_DERIVED_LOB to merge values from all crosswalks (PROGRAM and PRODUCT)
- Updated fill_na_mapping to always check both AARETE_DERIVED_PROGRAM and PRODUCT crosswalks
- Merges unique values from all sources (existing, PROGRAM, PRODUCT) instead of only filling when empty
- Ensures Duals is included when PRODUCT contains Molina Medicare Options Plus
- Deduplicates and sorts values before pipe-delimiting
- Each row processed individually for correct LOB assignment per reimbursement term
* Merged main into bugfix/derived_lob_changes
Approved-by: Katon Minhas
Bugfix/prov info issues
* Initial commit
* Passed payer name to tin npi llm call to avoid assigning tin npi to payer
* Removed record that contains NO_IDENTIFIERS_FOUND from PROV_INFO_JSON
* Updated unit test for test_tin_npi_funcs
* Changed name check code back to original to be consistent
* Updated test_tin_npi_funcs.py to pass in payer_name in functions
* Add fix to unit test
* Updated output in unit tests
* Merged main into bugfix/prov-info-issues
Approved-by: Katon Minhas
Bugfix/one to one and prov info
* Add post-processing logic to update PROV_INFO_JSON if needed; Updated prompt to verify the number of digitis in TIN and NPI
* Added update_prov_info_with_group_name to postprocessing_funcs.py
* Updated FIRST_PAGE_PROVIDER_NAME_TEMPLATE to ask LLM to explain resoning before answering
* Updated logic to use LLM to check if provider name matches
* Use only Hybrid Smart Chunking to get provider group name
* Cleaned up code
* Removed commented out code
* Updated unit test
* Merged main into bugfix/one_to_one_and_prov_info
* Edited prompt to distinguish provider names and to enclose answer in pipes
* Updated test_tin_npi_funcs
Approved-by: Katon Minhas
Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code
* Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code
- Add format requirements to LOB_RELATIONSHIP_INSTRUCTION for proper caching
- Strengthen LOB_RELATIONSHIP prompt with explicit pipe delimiter requirements
- Preserve all DEBUG blocks and print statements in dynamic_funcs.py and file_processing.py
- Update crosswalk mappings and gitignore
- Ensure format instructions are cached at API level to prevent missing pipe delimiters
* Merge main into bugfix/dynamic_issuefixes-clean and remove all debug print statements
* Remove redundant one_to_one_fields initialization to match main
Approved-by: Katon Minhas
Feature/cross exhibit dynamic
* Merge branch 'feature/deprecate-haiku-3' into feature/1toN-Optimization
* fix over-filtering of lesser of
* Merged in bugfix/UT-methodology-breakout (pull request #794)
Bugfix/UT methodology breakout
* updated valid values for AARETE_DERIVED_REIMB_METHOD
* removed example reimbursements
* prompt update
* prompt update
* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes
* add service term in mb prompts
* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout
* print statement removed
* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout
* primary prompt update
* remove duplicate prompt
Approved-by: Katon Minhas
* Merge branch 'main' into feature/1toN-Optimization
* Merge branch 'main' into feature/1toN-Optimization
* Update preprocessing to make the exhibit_chunk_mapping start at first page
* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into cross-exhibit-dynamic
* Address merge request comments: refactor prompt templates and logging
- Make DYNAMIC_PRIMARY_TEXT LOB-specific instructions conditional (only show when field_name is LOB)
- Remove MEDICAID FEE SCHEDULE point from DYNAMIC_ASSIGNMENT (point #2)
- Generalize SERVICE_TERM context guidance to apply to all fields (LOB, PROGRAM, NETWORK, PRODUCT)
- Refactor proximity guidance to emphasize contextual connection over strict section boundaries
- Update point #1 to explicitly prevent inferring LOB from Programs alone
- Change exhibit inheritance logging from info to debug level
* Move function to one_to_n_funcs
* Merge row_funcs.py changes from feature/cross-exhibit-dynamic
* Merged feature/1toN-Optimization into feature/cross-exhibit-dynamic
* Merged in bugfix/UT-grouper-issues (pull request #796)
Bugfix/UT grouper issues
* updated valid values for AARETE_DERIVED_REIMB_METHOD
* removed example reimbursements
* prompt update
* prompt update
* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes
* add service term in mb prompts
* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout
* print statement removed
* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout
* primary prompt update
* prompt update
* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into bugfix/UT-grouper-issues
* removed temp changes
* removed temp changes
* Update reimb primary
Approved-by: Katon Minhas
* Merged in bugfix/validation_fixes (pull request #795)
bugfix/validation_fixes to feature/1toN-Optimization
* updated validation of clean claims reimbursement
* Merged feature/1toN-Optimization into bugfix/validation_fixes
Approved-by: Katon Minhas
* Merge branch 'feature/1toN-Optimization' into feature/cross-exhibit-dynamic
* Re-add dynamic codes and reimb-info
* Re-structure empty reimbursement prompt
* Fix lesser of check overfiltering
* Merged feature/1toN-Optimization into feature/cross-exhibit-dynamic
* Update LOB inference logic and prompt guidance
- Revert DYNAMIC_PRIMARY_TEXT to original template (remove LOB-specific conditional section)
- Add explicit guidance in DYNAMIC_ASSIGNMENT: presence of Medicaid programs does NOT imply LOB is Medicaid
- Update get_dynamic_one_to_one_fields to only check AARETE_DERIVED_LOB (not raw LOB) when skipping PROGRAM/PRODUCT/NETWORK
* Fix LOB_PROGRAM_RELATIONSHIP extraction: add pipe format instructions to LOB_RELATIONSHIP_INSTRUCTION
* Update Fidelis Essential Plan mappings in crosswalk_product_lob.json
- Add mappings for Essential Plan variants (Aliessa, EP-QHP, EP, Essential Plan)
- Essential Plan Aliessa and EP Aliessa map to Medicaid
- EP-QHP maps to Commercial
- EP and Essential Plan map to Medicaid|Commercial
* Remove debug logging statements from prompt_lob_relationship
* Merge main into feature/cross-exhibit-dynamic: resolved conflicts, removed debug statements, synced prompt_templates.py with main
* Sync non-exhibit-merge files with main before merge
* Remove test.py to match main
Approved-by: Katon Minhas
Bugfix/cpt codes
* formatting fix for revenue_cd
* format fix for procedure_codes
* bugfix for revenue_codes
* working version from code main
* code formatting from main and code main
* removed dubug prints
* removed local file paths
* standardize state fields
* updated test scenerios
* minor fix
* Merge branch 'main' into bugfix/cpt-codes
merging main into cpt code fix
* checks for all fields
* reused is_empty functionality
* moved normalize_to_json_list to utils
* Merge branch 'main' into bugfix/cpt-codes
merging upto date main
* minor touches
* pytest fix
Approved-by: Katon Minhas
Issuefix/hmk 1 to n
* Merge branch 'main' into feature/1toN-Optimization
* Merge branch 'main' into feature/1toN-Optimization
* Update preprocessing to make the exhibit_chunk_mapping start at first page
* Merge row_funcs.py changes from feature/cross-exhibit-dynamic
* Merged in bugfix/UT-grouper-issues (pull request #796)
Bugfix/UT grouper issues
* updated valid values for AARETE_DERIVED_REIMB_METHOD
* removed example reimbursements
* prompt update
* prompt update
* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes
* add service term in mb prompts
* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout
* print statement removed
* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout
* primary prompt update
* prompt update
* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into bugfix/UT-grouper-issues
* removed temp changes
* removed temp changes
* Update reimb primary
Approved-by: Katon Minhas
* Merged in bugfix/validation_fixes (pull request #795)
bugfix/validation_fixes to feature/1toN-Optimization
* updated validation of clean claims reimbursement
* Merged feature/1toN-Optimization into bugfix/validation_fixes
Approved-by: Katon Minhas
* Re-add dynamic codes and reimb-info
* Re-structure empty reimbursement prompt
* Fix lesser of check overfiltering
* Merged in bugfix/generic-methodology-breakout (pull request #797)
Bugfix/generic methodology breakout
* fee schedule fix
* carveout+special case prompt changed
* fee schedule changes
* prompt changes and column order
Approved-by: Katon Minhas
* pytest fix
* remove prints
* Update valid_carveouts.json to include level designations in PAYMENT_CARVEOUT definition and add adultBasic mapping for Highmark
* Fix REIMB_DATES extraction to capture only relevant date range per term
- Updated REIMB_DATES prompt to extract only the date range that applies to the specific reimbursement term
- Added proximity-based extraction: looks for dates in same section/table as the term
- Prevents cross-contamination from other contract sections
- Handles 'Same as Effective Date of Agreement' case
- Returns single date range instead of multiple comma-separated ranges
* Update prompt templates
* Update tracked files and add untracked_tests to gitignore
* Merge branch 'main' into issuefix/hmk-1-to-n
* complete merge
* complete merge
* Merge branch 'main' into issuefix/hmk-1-to-n
* Remove prints
* Update gitignore
* Revert BASE_COVERED_SERVICES and REIMB_DATES prompts to original definitions
- Revert BASE_COVERED_SERVICES definition to original concise version
- Revert REIMB_DATES prompt to original general description (specific assignment logic handled by dedicated template)
- Add DEBUG comment blocks around print statements in dynamic_funcs.py and file_processing.py for easy identification and toggling
* remove prints
* Merge branch 'main' into issuefix/hmk-1-to-n
Approved-by: Katon Minhas
Bugfix/UT special cases
* stoploss fix
* Merged main into bugfix/UT-special-cases
* added IPPS
* added Federal PPS
* added IRF PPS
* Merged main into bugfix/UT-special-cases
Approved-by: Katon Minhas
Bugfix/prov info
* Added logic to extract provider name from first page
* Added prompt to extract provider name from first page
* Added prompts and provider name extraction using Hybrid Smart Chunking
* Cleaned up code
* Added additional logic to check if the provider name on tin/npi page matches provider name extracted from page 1.
* Added fix for IS_GROUP issue in provider info json
* Remove commented out code
* unit test fix
* uni test fix
* Merge branch 'main' into bugfix/prov_info
* Change to string_utils.is_empty
* Minor refactor
Approved-by: Katon Minhas
Issuefix/hmk 1 to 1
* prompt fix for auto renew ind
* filename tin regex update
* amendment number prompt fix
* effective date prompt tunning
* Merge remote-tracking branch 'origin/main' into issuefix/hmk-1-to-1
Approved-by: Katon Minhas
Katon Minhas