Feature/split reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Add MODEL_ID_CLAUDE4_SONNET for new model integration (doesn't work right now)
* pipe delimited-output and parsing
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Add functionality to split compound reimbursement terms into distinct methodologies
* splitting logic for reimbursement terms
* put in split compound reimbursement step
* Fix JSON output formatting in reimbursement terms example
* reimbursement splitting: move prompt to investment_prompts.py, move keywords to investment_values.py
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Change order (split -> filter over filter -> split), improve logging
* Refactor reimbursement filtering logic to use a two-stage approach with pattern-based negative filtering and LLM validation
* Replace clear reimbursement indicators with obvious non-reimbursement indicators for improved clarity in reimbursement filtering
* Simplify reimbursement filtering by removing pattern-based checks and relying solely on LLM validation
* Remove obvious non-reimbursement indicators from reimbursement filtering keywords
* Enhance reimbursement level test by adding LLM validation and updating mock return values for service terms
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Fix failing test
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* update mock return values to fix failed tests
Approved-by: Siddhant Medar
Fix data quality issues in testbed preprocessing and enhance error reporting
* Fix data quality issues in testbed preprocessing and enhance error reporting
* Merge remote-tracking branch 'origin/main' into bugfix/fix_nonunique_one_to_ones_and_strip_columns
Approved-by: Siddhant Medar
remove Greater Of from prompt, as its always N
* remove Greater Of from prompt, as its always N
* Merged main into feature/remove-greaterof-ind
Approved-by: Alex Galarce
Bugfix/aarete derived effective date to Main
* added effective date keywords
* updated key word
* updated prompt
* updated new prompt
Approved-by: Katon Minhas
Feature/filter reimbursements
* First-pass implementation for reimbursement filtering
* Enhance reimbursement level processing by filtering out services without reimbursement terms and logging when no valid pairs are found.
* Update mock return values in reimbursement level tests to reflect new reimbursement terms
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refine reimbursement filtering by adding 'billed' to indicators and ensuring REIMB_TERM is a string before processing.
* Refactor is_empty function signature to support multiple input types and clarify return values
* adding LLM review for ambiguous reimbursement methodology cases
* more stringent keywords for stage 1 of reimbursement filtering
* refining rate and cost patterns for reimbursement filtering
* Enhance LLM response handling in reimbursement methodology check to improve accuracy and logging for ambiguous cases.
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Add MODEL_ID_CLAUDE4_SONNET for new model integration (doesn't work right now)
* pipe delimited-output and parsing
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refactor reimbursement filtering by moving clear indicators to constants
Approved-by: Katon Minhas
Feature/prov group tin
* don't prompt for INTRO_PAGE
* test complete
* test 2
* Update test script
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* refactor
* cleaning
* Update save
* Fix return type
* Update for reimbursement_tin_npi
* Comment out reimbursement-tin-npi
* Pass unit tests
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* Remove length validation
* Merge branch 'main' into feature/prov-group-tin
* Remove ON_INTRO from prompt
* Restore
* Remove test
* Move reimb prov info to dynamic reimb info
* e2e test
* Fix for unit test
* remove prints
* update docstrings and typehints
Approved-by: Alex Galarce
Small bug - reversed testbed and results in page-level row counts
* Add page-level row count comparison to analyze discrepancies between testbed and results
* Fix key naming in page mismatch comparison for consistency
* Merged main into feature/row-count-by-pages
* Implement page-level grouping and comparison for row counts in testbed and results
* Fix row count difference calculation in page comparison
* Merged main into feature/row-count-by-pages
* Fix exhibit page normalization logic in group count calculations
Approved-by: Katon Minhas
Feature/row count by pages
* Add page-level row count comparison to analyze discrepancies between testbed and results
* Fix key naming in page mismatch comparison for consistency
* Merged main into feature/row-count-by-pages
* Implement page-level grouping and comparison for row counts in testbed and results
* Fix row count difference calculation in page comparison
Approved-by: Alex Galarce
Bugfix/fee rate conversion rate
* printing incorrect recall values
* values responsible for low recall added in spreadsheet
* changing match list to unmatch list for breakouts
* recall values added for 1-to-N
* print statements removed
* removed print statements
* Merged main into feature/test_files
* output formatting fix
* fixed cases where conversion factor was shown as fee rate
* Merged main into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
* unit of measure fixed
* Merge branch 'bugfix/fee_rate_conversion_rate' of https://bitbucket.org/aarete/doczy.ai into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
* functionality to create and compare multiple rows for AARETE_DERIVED_LOB added
* Merge branch 'bugfix/fee_rate_conversion_rate' of https://bitbucket.org/aarete/doczy.ai into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
Approved-by: Katon Minhas
bugfix/reimb pct rate fix to main
* updated pct rate post process
* postprocess pct rate
* Merged main into bugfix/reimb_pct_rate_fix
* removed comments
Approved-by: Katon Minhas
Feature/rework table handling
* test: add cases for configuration boundary
* test: add table post-processing tests (cases to ensure preservation of post-table text and metadata during table operations)
* test: enhance table handling tests for small and large tables, ensuring proper suffixing and splitting behavior
* Fix and clarify intent in comments
* fix: enhance handling of post-table text when combining tables
* refactor: remove commented-out code in clean_tables function for clarity
* test: update post-table text preservation assertions and add debug tracing
* refactor: remove unused remove_table_from_page function and its test
* refactor: enhance combine method documentation and clarify split_by_size_with_smart_tables logic
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
* refactor: reduce large table threshold from 1000 to 50 rows for better table splitting
* Add simple splitting
* refactor: uncomment dynamic answer retrieval in run_one_to_n_prompts for improved functionality
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
* refactor: remove unused table handling functions and related constants for cleaner code
* refactor: remove unused test functions in preparation for new tests
* refactor: streamline continuation table handling by utilizing existing flags
* refactor: update table combination logic to include all tables from the main page
* refactor: enhance table handling to automatically resolve column mismatches, respect row limits, and improve unit/integration test coverage
* rename clean_tables_simple to clean_tables
* Update docstrings
* refactor: rename table handling functions
* Fix indentation and add tests
* isort, black
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
Approved-by: Katon Minhas
Bugfix/exhibit linking
* Crosswalk updates
* Merge branch 'main' into bugfix/dynamic-primary-issues
* dont include section information in Exhibit Header
* Add Molina Health Benefit Exchange to Product-LOB mapping
* PPO-POS separate mapping
* Prevent inference of LOB from Program or Product
* Add SoonerSelect
* Update mapping rules to allow for direct values
* update prompts
* Prep for run
* Merged main into bugfix/dynamic-primary-issues
* Update unit tests
* Update Moda Product mappings
* Remove Care1st as Product for Centene
* testing
* Mapping update
* Mapping updates
* prep for merge
* Merged main into bugfix/dynamic-primary-issues
* update prompt template
* remove debugs
* Remove prints
* Merge branch 'bugfix/dynamic-primary-issues' into bugfix/exhibit-linking
* Merged main into bugfix/exhibit-linking
* Merge branch 'main' into bugfix/exhibit-linking
Approved-by: Alex Galarce
Bugfix/dynamic primary issues
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Merged main into bugfix/dynamic-primary-issues
* Crosswalk updates
* Merge branch 'main' into bugfix/dynamic-primary-issues
* dont include section information in Exhibit Header
* Add Molina Health Benefit Exchange to Product-LOB mapping
* PPO-POS separate mapping
* Prevent inference of LOB from Program or Product
* Add SoonerSelect
* Update mapping rules to allow for direct values
* update prompts
* Prep for run
* Merged main into bugfix/dynamic-primary-issues
* Update unit tests
* Update Moda Product mappings
* Remove Care1st as Product for Centene
* testing
* Mapping update
* Mapping updates
* prep for merge
* Merged main into bugfix/dynamic-primary-issues
* Merged main into bugfix/dynamic-primary-issues
* docstrings and formatting
* Get group provider for > 1
* Merged main into bugfix/dynamic-primary-issues
Approved-by: Alex Galarce
Feature/test files
* printing incorrect recall values
* values responsible for low recall added in spreadsheet
* changing match list to unmatch list for breakouts
* recall values added for 1-to-N
* print statements removed
* removed print statements
* Merged main into feature/test_files
* output formatting fix
* Merged main into feature/test_files
Approved-by: Katon Minhas
bugfix/lesser_of_rows_and_pct_rate -> main
* updated reimb term breakout prompt
* updated lesser_of_ind prompt
* Merged main into bugfix/lesser_of_rows_and_pct_rate
* Merged main into bugfix/lesser_of_rows_and_pct_rate
Approved-by: Katon Minhas
Bugfix/date format fixes
* Switch to 12/31 for auto renews
* Remove redundant postprocessing
* Switch to 12/31 for auto renews
* auto renew = Y
Approved-by: Alex Galarce
Feature/formatted prov info json
* Add newline and pipe-delimited formatting for provider info JSON
* Add new investment column names for nicely-formatted PROV_INFO_JSON
* remove PROV_INFO_JSON_PIPE and rename PROV_INFO_JSON_NEWLINE to PROV_INFO_JSON_FORMATTED
* Merge remote-tracking branch 'origin/main' into feature/formatted-prov-info-json
Approved-by: Katon Minhas
Bugfix/NTE num rows to Main
* updated REIMB_TERM_BREAKOUTprompt
* Merged main into bugfix/NTE_num_rows
* updated NTE prompt
* updated reimb_pct_rate prompt
* updated reimb_method prompt
* added cost prompt in reimb method
* Merged main into bugfix/NTE_num_rows
* conflict fixes
* Merged main into bugfix/NTE_num_rows
* updated prompts
Approved-by: Alex Galarce
Feature/formatted prov info json
* Add newline and pipe-delimited formatting for provider info JSON
* Add new investment column names for nicely-formatted PROV_INFO_JSON
Approved-by: Katon Minhas
Add (service, methodology) deduplication with accumulator
* Add deduplication with accumulator
* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication
* Replace print statement with logging.debug for cross-exhibit duplicate detection
* Fix and add tests for one_to_n_funcs
* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication
* Add seen_pairs parameter to reimbursement_level function in tests
Approved-by: Katon Minhas
Feature/exhibit linking
* Initialize new exhibit header prompt
* Save
* Merge branch 'main' into feature/exhibit-linking
* update prompt
* cleanup return type
* remove test
* Test fixes
* Fix test
* Add tests
* Merge branch 'main' into feature/exhibit-linking
* Merge branch 'main' into feature/exhibit-linking
* Remove test notebooks
* Delete test notebook
* Save
* Merge branch 'main' into feature/exhibit-linking
* try except for exhbit page finding
* Prep for merge
* update test
* Remove test
* Merge branch 'main' into feature/exhibit-linking
* update unit test
* Fix unit test
* replace try except
* Update page sort
* Add docstring
Approved-by: Alex Galarce
Hotfix/methodology breakout issue fixes
* additional values in unit of measure removed
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* added valid list of unit of measure
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
Approved-by: Katon Minhas
Update reimb tin-npi-name passing to reimbursement primary
* Update reimb tin-npi-name passing to reimbursement primary
* Refactor reimbursement_tin_npi to streamline provider value extraction and improve handling of empty values
Approved-by: Alex Galarce
Feature/global lesser of
* Merge remote-tracking branch 'origin/feature/reimb-primary-updates'
* First try at global "lesser of"
* Generate reimb_ids further upstream
* Implement global lesser of injection and extraction logic in file processing
* Update logic to change LESSER_OF_IND in-place for global affected rows; add docstrings
* Refactor global lesser of injection logic to create new rows for REIMB_IDs lacking lesser of logic; update original rows in place and improve tracking flags.
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add logging for global lesser of logic injection and update handling of original rows
* Refactor global lesser of logic: move functions to one_to_one_funcs.py and update inject_global_lesser_of_rows to use new implementations
* Move imports for METHODOLOGY_BREAKOUT_QUESTIONS and REIMB_TERM_BREAKOUT to the top of one_to_one_funcs.py for better organization and to remove prototype comments.
* isort
* Refactor import statement for generate_reimb_ids to use the full module path for clarity
* load field from JSON instead of hardcoded values
* Update logging levels in investment_field_testing notebook
* Merged main into feature/global-lesser-of
* Remove unnecessary comment from investment_postprocessing_funcs import statement for clarity
* Merge branch 'feature/global-lesser-of' of https://bitbucket.org/aarete/doczy.ai into feature/global-lesser-of
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Remove debug print statement from process_file function
* fix call to Field.load_from_file
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add tracking flag for global lesser of logic when none is found
* explicitly sort by FILE_NAME, EXHIBIT_PAGE, and REIMB_LESSER_OF_ID columns
* Add new field type, "full_context_separate", to make sure that we don't run it twice. Also formatting fix
* Fix iteration over separate fields in run_full_context_fields function
Approved-by: Katon Minhas
Feature/fuzzy match no numbers
* Add fuzzy matching function for string comparison and enhance term extraction result evaluation
* Enhance term extraction results evaluation by adding counts and readable strings for matched, false negatives, and false positives
* Adjust text similarity threshold to 65 and improve readability of comparison results in term extraction evaluation
* Merged main into feature/fuzzy-match-no-numbers
Approved-by: Katon Minhas
Feature/further reimb primary improvements
* Refactor identify_reimbursement_exhibits to return both EXPLICIT and IMPLICIT values
* Moving from # Debugging lines to `logger`
* Enhance "lesser of" and exception language instructions in reimbursement terms
* clarify unclear print/logging statements
* Replace debugging print statements with logging in get_reimbursement_primary function
* Add logging for page processing in run_one_to_n_prompts function
Approved-by: Katon Minhas
Don't consider table-subpages each individually for reimbursement exhibit identification
* Refine EXHIBIT_CHECK prompt
* test: rework `get_exhibit_pages` for table subpages
* Fix indentation
* Experimental table de-chunking
* Increase max_tokens for Claude model in exhibit page processing
* fix get_pages_to_process docstring
* refactor: improved docstring for get_exhibit_pages
* move `get_pages_to_process` to `preprocess.py`
* test: add unit tests for get_pages_to_process function
* Comment out debugging lines
* Changed prompt that was breading universal_json_load parsing
* fix: handle "no results" case in get_reimbursement_primary function
* fixed docstring
Approved-by: Katon Minhas
default term new prompt and de-duplication before Methodology Breakout
* Refine prompts for default indicator methodology and enhance instructions for final reimbursement methodology output
* Enhance DEFAULT_INDICATOR prompt to clarify its role as a fallback reimbursement method
* Implement service-reimb tracking to prevent duplicates
* Refactor methodology breakout processing to reduce code duplication and improve clarity
* Remove debugging print statements and improve item ID formatting in reimbursement processing
* Merge remote-tracking branch 'origin/main' into feature/default-term-new-prompt
* Refactor methodology breakout processing to remove duplicate tracking and simplify logic
* Merged main into feature/default-term-new-prompt
* Merged main into feature/default-term-new-prompt
* rename _process_answer_dict to _process_carveout_or_breakout
Approved-by: Katon Minhas
Alex Galarce