Feature/claude4 duplicate and missing rows
* Enhance crosswalk functionality with flatten_to_strings helper and improve value handling in apply_crosswalk
* Improve apply_crosswalk function to return original value as fallback when no mapping is found
* Enhance apply_crosswalk function to handle empty values and improve defensive programming tests
* Enhance reimbursement method prompt to include charge master limitations and improve mapping rules
* Merge remote-tracking branch 'origin/main' into feature/claude4-duplicate-and-missing-rows
* Refine CHECK_REDUNDANT_LESSER prompt for clarity and detail in evaluating redundancy of exhibit constraints
* Enable logging of LLM response for redundancy check in is_exhibit_lesser_of_redundant function
* Merge remote-tracking branch 'origin/main' into feature/claude4-duplicate-and-missing-rows
* rework prompt for exhibit-wide lesser of
* Refine reimbursement methodology guidelines by clarifying invalid entries and examples for risk-sharing
* Enhance exhibit linking logic to treat pages with substantial tables as separate exhibits and update function signature to include text_dict.
Approved-by: Katon Minhas
Trigger_Cap+Special_Case_Fixes to Main
* updated trigger_cap_definition
* src/prompts/investment_prompts.py
* updated trigger base threshold prompt
* updated special case check prompt
* Merged main into bugfix/trigger_cap_fixes
Approved-by: Alex Galarce
Hotfix/defensive crosswalk handling
* Enhance crosswalk functionality with flatten_to_strings helper and improve value handling in apply_crosswalk
* Improve apply_crosswalk function to return original value as fallback when no mapping is found
* Enhance apply_crosswalk function to handle empty values and improve defensive programming tests
* move flatten_to_strings helper function to string_utils
Approved-by: Katon Minhas
feature-flag ENABLE_RUNTIME_ROTATION to test NOT doing runtime rotation
* feature-flag ENABLE_RUNTIME_ROTATION to test NOT doing runtime rotation
Approved-by: Katon Minhas
Implement deduplication of provider columns in DataFrame and add corresponding unit tests
* Implement deduplication of provider columns in DataFrame and add corresponding unit tests
Approved-by: Katon Minhas
Sonnet 4 and model aliasing
* Add EXHIBIT_HEADER_MARKERS for improved document header recognition in table handling
* Refactor EXHIBIT_HEADER_MARKERS definition for improved clarity and organization
* remove unused page span tracking code; add unit tests for new functionality
* remove debugging logs from combine_continuous_tables function
* Refactor logging setup and update test file configuration in investment field testing notebook
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
* Refactor model ID handling and add support for model aliases in llm_utils
* Update model IDs to use alias in LLM invocation functions in one_to_n_funcs
* change pre-and-post-processing to new model alias method
* Update model ID references to use 'sonnet_latest' in LLM invocation functions
* Update LLM invocation functions to use 'sonnet_latest' model ID
* Update LLM invocation functions to use 'sonnet_latest' and 'haiku_latest' model IDs
* update invoke_claude function docstring
* Fix caller frame to address mypy error
Approved-by: Katon Minhas
Table Continuation Fix
* Add EXHIBIT_HEADER_MARKERS for improved document header recognition in table handling
* Refactor EXHIBIT_HEADER_MARKERS definition for improved clarity and organization
* remove unused page span tracking code; add unit tests for new functionality
* remove debugging logs from combine_continuous_tables function
* Refactor logging setup and update test file configuration in investment field testing notebook
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
Approved-by: Katon Minhas
false positives in additions (special case) fixed
* false positives in additions (special case) fixed
* Merged main into feature/additions
Approved-by: Alex Galarce
bugfix/special_case_breakout_test_metrics to main
* updated special case breakout fields to testbed metrics
* Merged main into bugfix/special_case_breakout_test_metrics
Approved-by: Alex Galarce
Exhibit-level Lesser Of
* Enhance exception language clarity in investment prompts
* Roll back exception language
* initial try at exhibit_lesser_of
* simplify `create_exhibit_lesser_of_row`
* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints in exhibit
* Remove unused `get_service_breakout` function; it was unreferenced
* run exhibit-level answers FIRST, apply exhibit lesser-of WITHIN reimbursement_level
* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints to clarify expected output and handling of multiple constraints.
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* add exhibit_lesser_of="N/A" to tests
* Add exhibit_lesser_of parameter to reimbursement_level calls in tests
* restore dynamic answer retrieval for primary, reimbursement info, and code fields
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* "fill in the blank" prompt attempt
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* Merge remote-tracking branch 'origin/main' into feature/sonnet-4
* Merge remote-tracking branch 'origin/feature/sonnet-4' into bugfix/samaritan-missing-rows
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* Remove carveout patterns from has_explicit_lesser_of_carveout function
* Add deduplicate_methodology_breakout function to remove duplicate reimbursement methodologies
* Add deduplication of methodology breakout rows in postprocess function
* Enhance create_exhibit_lesser_of_row to avoid redundancy with substring checks
* Change log level from info to debug for exhibit constraint check in create_exhibit_lesser_of_row
* Remove deduplication of methodology breakout rows from postprocess function (we handle it upstream)
* Merged main into bugfix/samaritan-missing-rows
Approved-by: Katon Minhas
Bugfix/trigger cap to main
* updated trigger base threshold prompt
* Merged main into bugfix/trigger_cap
* updated trigger cap carveout_cd desc
* updated trigger cap carveout desc
* Merged main into bugfix/trigger_cap
* updated prompts
* added units for threshold
* Merge branch 'main' into bugfix/trigger_cap
Approved-by: Katon Minhas
Feature/Rate Escalators to main
* added rate esalator fields
* Merged main into Feature/Rate_Escalators
* updated rate escalator prompts
* updated field names and prompts
* Merged main into Feature/Rate_Escalators
* updated pct prompt
Approved-by: Katon Minhas
Genericize claude-calling functions for Claude 3 and above; unify interfaces
* First-pass implementation for reimbursement filtering
* Enhance reimbursement level processing by filtering out services without reimbursement terms and logging when no valid pairs are found.
* Update mock return values in reimbursement level tests to reflect new reimbursement terms
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refine reimbursement filtering by adding 'billed' to indicators and ensuring REIMB_TERM is a string before processing.
* Refactor is_empty function signature to support multiple input types and clarify return values
* adding LLM review for ambiguous reimbursement methodology cases
* more stringent keywords for stage 1 of reimbursement filtering
* refining rate and cost patterns for reimbursement filtering
* Enhance LLM response handling in reimbursement methodology check to improve accuracy and logging for ambiguous cases.
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Refactor Claude model invocation functions to support Claude 3 and above, enhancing error handling and adding unified interfaces for local and EC2 executions.
* Add support for Claude 3.7 model in local and EC2 invocation functions
* Merge remote-tracking branch 'origin/main' into feature/sonnet-4
Approved-by: Katon Minhas
Hotfix/claim type cd
* Fix exhibit_level prompt
* e2e test passed - successfully pulling multiple Claim Types per exhibit
* Merged main into hotfix/claim-type-cd
Approved-by: Alex Galarce
Bugfix/mcs issue fixes
* debugging and initial prompt change
* Remove redundant reimb info
* Merged main into bugfix/mcs-issue-fixes
* Prep for merge
* update unit tests
Approved-by: Alex Galarce
Add script to generate and save reimbursement IDs in testbed
* Add script to generate and save reimbursement IDs in testbed
* Merged main into feature/script-add-lesser-of-id-to-testbed
Approved-by: Katon Minhas
Feature/provider state
* Add prompt for extracting provider state information in investment prompts
* Add state normalization function and integrate into context field extraction
* black format string_utils.py; add tests for normalize_state_to_abbreviation function
* black format string_utils_test.py
* Add PROVIDER_STATE to investment column order
* Merged main into feature/provider-state
Approved-by: Katon Minhas
Bugfix/mcs issue fixes
* Update code primary prompt
* Merge branch 'main' into bugfix/mcs-issue-fixes
* No longer treat non-reimbursement tables as 'tables' for the purpose of preprocessing
* No longer extract Reimb Effective Date from footer of the page
* Merge branch 'main' into bugfix/mcs-issue-fixes
* Update unit tests
* Add postprocessing step to remove Reimb Date values when the 1:1 date values are identical
* Update unit tests
* prep for merge
* Remove excess prints
* Merged main into bugfix/mcs-issue-fixes
* Merge branch 'main' into bugfix/mcs-issue-fixes
* Prep for merge
* remove print
Approved-by: Alex Galarce
Bugfix/fix overzealous duplication
* Refactor duplicate processing logic to scope seen pairs by base page, enhancing clarity and accuracy in deduplication.
* Enhance deduplication tests to include base page identifiers in seen pairs
* Merged main into bugfix/fix-overzealous-duplication
Approved-by: Katon Minhas
Feature/trigger cap to Main
* updated trigger cap prompts
* updated trigger cap prompts
* Merged main into feature/trigger_cap
* fixes field names in field order
* Merged main into feature/trigger_cap
Approved-by: Alex Galarce
Feature/split reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Add MODEL_ID_CLAUDE4_SONNET for new model integration (doesn't work right now)
* pipe delimited-output and parsing
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Add functionality to split compound reimbursement terms into distinct methodologies
* splitting logic for reimbursement terms
* put in split compound reimbursement step
* Fix JSON output formatting in reimbursement terms example
* reimbursement splitting: move prompt to investment_prompts.py, move keywords to investment_values.py
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Change order (split -> filter over filter -> split), improve logging
* Refactor reimbursement filtering logic to use a two-stage approach with pattern-based negative filtering and LLM validation
* Replace clear reimbursement indicators with obvious non-reimbursement indicators for improved clarity in reimbursement filtering
* Simplify reimbursement filtering by removing pattern-based checks and relying solely on LLM validation
* Remove obvious non-reimbursement indicators from reimbursement filtering keywords
* Enhance reimbursement level test by adding LLM validation and updating mock return values for service terms
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Fix failing test
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* update mock return values to fix failed tests
Approved-by: Siddhant Medar
Fix data quality issues in testbed preprocessing and enhance error reporting
* Fix data quality issues in testbed preprocessing and enhance error reporting
* Merge remote-tracking branch 'origin/main' into bugfix/fix_nonunique_one_to_ones_and_strip_columns
Approved-by: Siddhant Medar
remove Greater Of from prompt, as its always N
* remove Greater Of from prompt, as its always N
* Merged main into feature/remove-greaterof-ind
Approved-by: Alex Galarce
Bugfix/aarete derived effective date to Main
* added effective date keywords
* updated key word
* updated prompt
* updated new prompt
Approved-by: Katon Minhas
Feature/filter reimbursements
* First-pass implementation for reimbursement filtering
* Enhance reimbursement level processing by filtering out services without reimbursement terms and logging when no valid pairs are found.
* Update mock return values in reimbursement level tests to reflect new reimbursement terms
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refine reimbursement filtering by adding 'billed' to indicators and ensuring REIMB_TERM is a string before processing.
* Refactor is_empty function signature to support multiple input types and clarify return values
* adding LLM review for ambiguous reimbursement methodology cases
* more stringent keywords for stage 1 of reimbursement filtering
* refining rate and cost patterns for reimbursement filtering
* Enhance LLM response handling in reimbursement methodology check to improve accuracy and logging for ambiguous cases.
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Add MODEL_ID_CLAUDE4_SONNET for new model integration (doesn't work right now)
* pipe delimited-output and parsing
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refactor reimbursement filtering by moving clear indicators to constants
Approved-by: Katon Minhas
Feature/prov group tin
* don't prompt for INTRO_PAGE
* test complete
* test 2
* Update test script
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* refactor
* cleaning
* Update save
* Fix return type
* Update for reimbursement_tin_npi
* Comment out reimbursement-tin-npi
* Pass unit tests
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* Remove length validation
* Merge branch 'main' into feature/prov-group-tin
* Remove ON_INTRO from prompt
* Restore
* Remove test
* Move reimb prov info to dynamic reimb info
* e2e test
* Fix for unit test
* remove prints
* update docstrings and typehints
Approved-by: Alex Galarce
Small bug - reversed testbed and results in page-level row counts
* Add page-level row count comparison to analyze discrepancies between testbed and results
* Fix key naming in page mismatch comparison for consistency
* Merged main into feature/row-count-by-pages
* Implement page-level grouping and comparison for row counts in testbed and results
* Fix row count difference calculation in page comparison
* Merged main into feature/row-count-by-pages
* Fix exhibit page normalization logic in group count calculations
Approved-by: Katon Minhas
Feature/row count by pages
* Add page-level row count comparison to analyze discrepancies between testbed and results
* Fix key naming in page mismatch comparison for consistency
* Merged main into feature/row-count-by-pages
* Implement page-level grouping and comparison for row counts in testbed and results
* Fix row count difference calculation in page comparison
Approved-by: Alex Galarce
Bugfix/fee rate conversion rate
* printing incorrect recall values
* values responsible for low recall added in spreadsheet
* changing match list to unmatch list for breakouts
* recall values added for 1-to-N
* print statements removed
* removed print statements
* Merged main into feature/test_files
* output formatting fix
* fixed cases where conversion factor was shown as fee rate
* Merged main into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
* unit of measure fixed
* Merge branch 'bugfix/fee_rate_conversion_rate' of https://bitbucket.org/aarete/doczy.ai into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
* functionality to create and compare multiple rows for AARETE_DERIVED_LOB added
* Merge branch 'bugfix/fee_rate_conversion_rate' of https://bitbucket.org/aarete/doczy.ai into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
Approved-by: Katon Minhas
bugfix/reimb pct rate fix to main
* updated pct rate post process
* postprocess pct rate
* Merged main into bugfix/reimb_pct_rate_fix
* removed comments
Approved-by: Katon Minhas
Feature/rework table handling
* test: add cases for configuration boundary
* test: add table post-processing tests (cases to ensure preservation of post-table text and metadata during table operations)
* test: enhance table handling tests for small and large tables, ensuring proper suffixing and splitting behavior
* Fix and clarify intent in comments
* fix: enhance handling of post-table text when combining tables
* refactor: remove commented-out code in clean_tables function for clarity
* test: update post-table text preservation assertions and add debug tracing
* refactor: remove unused remove_table_from_page function and its test
* refactor: enhance combine method documentation and clarify split_by_size_with_smart_tables logic
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
* refactor: reduce large table threshold from 1000 to 50 rows for better table splitting
* Add simple splitting
* refactor: uncomment dynamic answer retrieval in run_one_to_n_prompts for improved functionality
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
* refactor: remove unused table handling functions and related constants for cleaner code
* refactor: remove unused test functions in preparation for new tests
* refactor: streamline continuation table handling by utilizing existing flags
* refactor: update table combination logic to include all tables from the main page
* refactor: enhance table handling to automatically resolve column mismatches, respect row limits, and improve unit/integration test coverage
* rename clean_tables_simple to clean_tables
* Update docstrings
* refactor: rename table handling functions
* Fix indentation and add tests
* isort, black
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
Approved-by: Katon Minhas
Bugfix/exhibit linking
* Crosswalk updates
* Merge branch 'main' into bugfix/dynamic-primary-issues
* dont include section information in Exhibit Header
* Add Molina Health Benefit Exchange to Product-LOB mapping
* PPO-POS separate mapping
* Prevent inference of LOB from Program or Product
* Add SoonerSelect
* Update mapping rules to allow for direct values
* update prompts
* Prep for run
* Merged main into bugfix/dynamic-primary-issues
* Update unit tests
* Update Moda Product mappings
* Remove Care1st as Product for Centene
* testing
* Mapping update
* Mapping updates
* prep for merge
* Merged main into bugfix/dynamic-primary-issues
* update prompt template
* remove debugs
* Remove prints
* Merge branch 'bugfix/dynamic-primary-issues' into bugfix/exhibit-linking
* Merged main into bugfix/exhibit-linking
* Merge branch 'main' into bugfix/exhibit-linking
Approved-by: Alex Galarce
Alex Galarce