Feature/claude4 duplicate and missing rows
* Enhance crosswalk functionality with flatten_to_strings helper and improve value handling in apply_crosswalk
* Improve apply_crosswalk function to return original value as fallback when no mapping is found
* Enhance apply_crosswalk function to handle empty values and improve defensive programming tests
* Enhance reimbursement method prompt to include charge master limitations and improve mapping rules
* Merge remote-tracking branch 'origin/main' into feature/claude4-duplicate-and-missing-rows
* Refine CHECK_REDUNDANT_LESSER prompt for clarity and detail in evaluating redundancy of exhibit constraints
* Enable logging of LLM response for redundancy check in is_exhibit_lesser_of_redundant function
* Merge remote-tracking branch 'origin/main' into feature/claude4-duplicate-and-missing-rows
* rework prompt for exhibit-wide lesser of
* Refine reimbursement methodology guidelines by clarifying invalid entries and examples for risk-sharing
* Enhance exhibit linking logic to treat pages with substantial tables as separate exhibits and update function signature to include text_dict.
Approved-by: Katon Minhas
Trigger_Cap+Special_Case_Fixes to Main
* updated trigger_cap_definition
* src/prompts/investment_prompts.py
* updated trigger base threshold prompt
* updated special case check prompt
* Merged main into bugfix/trigger_cap_fixes
Approved-by: Alex Galarce
Hotfix/defensive crosswalk handling
* Enhance crosswalk functionality with flatten_to_strings helper and improve value handling in apply_crosswalk
* Improve apply_crosswalk function to return original value as fallback when no mapping is found
* Enhance apply_crosswalk function to handle empty values and improve defensive programming tests
* move flatten_to_strings helper function to string_utils
Approved-by: Katon Minhas
feature-flag ENABLE_RUNTIME_ROTATION to test NOT doing runtime rotation
* feature-flag ENABLE_RUNTIME_ROTATION to test NOT doing runtime rotation
Approved-by: Katon Minhas
Implement deduplication of provider columns in DataFrame and add corresponding unit tests
* Implement deduplication of provider columns in DataFrame and add corresponding unit tests
Approved-by: Katon Minhas
Sonnet 4 and model aliasing
* Add EXHIBIT_HEADER_MARKERS for improved document header recognition in table handling
* Refactor EXHIBIT_HEADER_MARKERS definition for improved clarity and organization
* remove unused page span tracking code; add unit tests for new functionality
* remove debugging logs from combine_continuous_tables function
* Refactor logging setup and update test file configuration in investment field testing notebook
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
* Refactor model ID handling and add support for model aliases in llm_utils
* Update model IDs to use alias in LLM invocation functions in one_to_n_funcs
* change pre-and-post-processing to new model alias method
* Update model ID references to use 'sonnet_latest' in LLM invocation functions
* Update LLM invocation functions to use 'sonnet_latest' model ID
* Update LLM invocation functions to use 'sonnet_latest' and 'haiku_latest' model IDs
* update invoke_claude function docstring
* Fix caller frame to address mypy error
Approved-by: Katon Minhas
Table Continuation Fix
* Add EXHIBIT_HEADER_MARKERS for improved document header recognition in table handling
* Refactor EXHIBIT_HEADER_MARKERS definition for improved clarity and organization
* remove unused page span tracking code; add unit tests for new functionality
* remove debugging logs from combine_continuous_tables function
* Refactor logging setup and update test file configuration in investment field testing notebook
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
Approved-by: Katon Minhas
false positives in additions (special case) fixed
* false positives in additions (special case) fixed
* Merged main into feature/additions
Approved-by: Alex Galarce
bugfix/special_case_breakout_test_metrics to main
* updated special case breakout fields to testbed metrics
* Merged main into bugfix/special_case_breakout_test_metrics
Approved-by: Alex Galarce
Exhibit-level Lesser Of
* Enhance exception language clarity in investment prompts
* Roll back exception language
* initial try at exhibit_lesser_of
* simplify `create_exhibit_lesser_of_row`
* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints in exhibit
* Remove unused `get_service_breakout` function; it was unreferenced
* run exhibit-level answers FIRST, apply exhibit lesser-of WITHIN reimbursement_level
* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints to clarify expected output and handling of multiple constraints.
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* add exhibit_lesser_of="N/A" to tests
* Add exhibit_lesser_of parameter to reimbursement_level calls in tests
* restore dynamic answer retrieval for primary, reimbursement info, and code fields
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* "fill in the blank" prompt attempt
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* Merge remote-tracking branch 'origin/main' into feature/sonnet-4
* Merge remote-tracking branch 'origin/feature/sonnet-4' into bugfix/samaritan-missing-rows
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* Remove carveout patterns from has_explicit_lesser_of_carveout function
* Add deduplicate_methodology_breakout function to remove duplicate reimbursement methodologies
* Add deduplication of methodology breakout rows in postprocess function
* Enhance create_exhibit_lesser_of_row to avoid redundancy with substring checks
* Change log level from info to debug for exhibit constraint check in create_exhibit_lesser_of_row
* Remove deduplication of methodology breakout rows from postprocess function (we handle it upstream)
* Merged main into bugfix/samaritan-missing-rows
Approved-by: Katon Minhas
Bugfix/trigger cap to main
* updated trigger base threshold prompt
* Merged main into bugfix/trigger_cap
* updated trigger cap carveout_cd desc
* updated trigger cap carveout desc
* Merged main into bugfix/trigger_cap
* updated prompts
* added units for threshold
* Merge branch 'main' into bugfix/trigger_cap
Approved-by: Katon Minhas
Feature/Rate Escalators to main
* added rate esalator fields
* Merged main into Feature/Rate_Escalators
* updated rate escalator prompts
* updated field names and prompts
* Merged main into Feature/Rate_Escalators
* updated pct prompt
Approved-by: Katon Minhas
Genericize claude-calling functions for Claude 3 and above; unify interfaces
* First-pass implementation for reimbursement filtering
* Enhance reimbursement level processing by filtering out services without reimbursement terms and logging when no valid pairs are found.
* Update mock return values in reimbursement level tests to reflect new reimbursement terms
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refine reimbursement filtering by adding 'billed' to indicators and ensuring REIMB_TERM is a string before processing.
* Refactor is_empty function signature to support multiple input types and clarify return values
* adding LLM review for ambiguous reimbursement methodology cases
* more stringent keywords for stage 1 of reimbursement filtering
* refining rate and cost patterns for reimbursement filtering
* Enhance LLM response handling in reimbursement methodology check to improve accuracy and logging for ambiguous cases.
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Refactor Claude model invocation functions to support Claude 3 and above, enhancing error handling and adding unified interfaces for local and EC2 executions.
* Add support for Claude 3.7 model in local and EC2 invocation functions
* Merge remote-tracking branch 'origin/main' into feature/sonnet-4
Approved-by: Katon Minhas
Hotfix/claim type cd
* Fix exhibit_level prompt
* e2e test passed - successfully pulling multiple Claim Types per exhibit
* Merged main into hotfix/claim-type-cd
Approved-by: Alex Galarce
Bugfix/mcs issue fixes
* debugging and initial prompt change
* Remove redundant reimb info
* Merged main into bugfix/mcs-issue-fixes
* Prep for merge
* update unit tests
Approved-by: Alex Galarce
Add script to generate and save reimbursement IDs in testbed
* Add script to generate and save reimbursement IDs in testbed
* Merged main into feature/script-add-lesser-of-id-to-testbed
Approved-by: Katon Minhas
Feature/provider state
* Add prompt for extracting provider state information in investment prompts
* Add state normalization function and integrate into context field extraction
* black format string_utils.py; add tests for normalize_state_to_abbreviation function
* black format string_utils_test.py
* Add PROVIDER_STATE to investment column order
* Merged main into feature/provider-state
Approved-by: Katon Minhas
Bugfix/mcs issue fixes
* Update code primary prompt
* Merge branch 'main' into bugfix/mcs-issue-fixes
* No longer treat non-reimbursement tables as 'tables' for the purpose of preprocessing
* No longer extract Reimb Effective Date from footer of the page
* Merge branch 'main' into bugfix/mcs-issue-fixes
* Update unit tests
* Add postprocessing step to remove Reimb Date values when the 1:1 date values are identical
* Update unit tests
* prep for merge
* Remove excess prints
* Merged main into bugfix/mcs-issue-fixes
* Merge branch 'main' into bugfix/mcs-issue-fixes
* Prep for merge
* remove print
Approved-by: Alex Galarce
Bugfix/fix overzealous duplication
* Refactor duplicate processing logic to scope seen pairs by base page, enhancing clarity and accuracy in deduplication.
* Enhance deduplication tests to include base page identifiers in seen pairs
* Merged main into bugfix/fix-overzealous-duplication
Approved-by: Katon Minhas
Feature/trigger cap to Main
* updated trigger cap prompts
* updated trigger cap prompts
* Merged main into feature/trigger_cap
* fixes field names in field order
* Merged main into feature/trigger_cap
Approved-by: Alex Galarce
Feature/split reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Add MODEL_ID_CLAUDE4_SONNET for new model integration (doesn't work right now)
* pipe delimited-output and parsing
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Add functionality to split compound reimbursement terms into distinct methodologies
* splitting logic for reimbursement terms
* put in split compound reimbursement step
* Fix JSON output formatting in reimbursement terms example
* reimbursement splitting: move prompt to investment_prompts.py, move keywords to investment_values.py
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Change order (split -> filter over filter -> split), improve logging
* Refactor reimbursement filtering logic to use a two-stage approach with pattern-based negative filtering and LLM validation
* Replace clear reimbursement indicators with obvious non-reimbursement indicators for improved clarity in reimbursement filtering
* Simplify reimbursement filtering by removing pattern-based checks and relying solely on LLM validation
* Remove obvious non-reimbursement indicators from reimbursement filtering keywords
* Enhance reimbursement level test by adding LLM validation and updating mock return values for service terms
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Fix failing test
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* update mock return values to fix failed tests
Approved-by: Siddhant Medar
Fix data quality issues in testbed preprocessing and enhance error reporting
* Fix data quality issues in testbed preprocessing and enhance error reporting
* Merge remote-tracking branch 'origin/main' into bugfix/fix_nonunique_one_to_ones_and_strip_columns
Approved-by: Siddhant Medar
remove Greater Of from prompt, as its always N
* remove Greater Of from prompt, as its always N
* Merged main into feature/remove-greaterof-ind
Approved-by: Alex Galarce
Bugfix/aarete derived effective date to Main
* added effective date keywords
* updated key word
* updated prompt
* updated new prompt
Approved-by: Katon Minhas
Feature/filter reimbursements
* First-pass implementation for reimbursement filtering
* Enhance reimbursement level processing by filtering out services without reimbursement terms and logging when no valid pairs are found.
* Update mock return values in reimbursement level tests to reflect new reimbursement terms
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refine reimbursement filtering by adding 'billed' to indicators and ensuring REIMB_TERM is a string before processing.
* Refactor is_empty function signature to support multiple input types and clarify return values
* adding LLM review for ambiguous reimbursement methodology cases
* more stringent keywords for stage 1 of reimbursement filtering
* refining rate and cost patterns for reimbursement filtering
* Enhance LLM response handling in reimbursement methodology check to improve accuracy and logging for ambiguous cases.
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Add MODEL_ID_CLAUDE4_SONNET for new model integration (doesn't work right now)
* pipe delimited-output and parsing
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refactor reimbursement filtering by moving clear indicators to constants
Approved-by: Katon Minhas
Feature/prov group tin
* don't prompt for INTRO_PAGE
* test complete
* test 2
* Update test script
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* refactor
* cleaning
* Update save
* Fix return type
* Update for reimbursement_tin_npi
* Comment out reimbursement-tin-npi
* Pass unit tests
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* Remove length validation
* Merge branch 'main' into feature/prov-group-tin
* Remove ON_INTRO from prompt
* Restore
* Remove test
* Move reimb prov info to dynamic reimb info
* e2e test
* Fix for unit test
* remove prints
* update docstrings and typehints
Approved-by: Alex Galarce
Small bug - reversed testbed and results in page-level row counts
* Add page-level row count comparison to analyze discrepancies between testbed and results
* Fix key naming in page mismatch comparison for consistency
* Merged main into feature/row-count-by-pages
* Implement page-level grouping and comparison for row counts in testbed and results
* Fix row count difference calculation in page comparison
* Merged main into feature/row-count-by-pages
* Fix exhibit page normalization logic in group count calculations
Approved-by: Katon Minhas
Alex Galarce