Small bug - reversed testbed and results in page-level row counts
* Add page-level row count comparison to analyze discrepancies between testbed and results
* Fix key naming in page mismatch comparison for consistency
* Merged main into feature/row-count-by-pages
* Implement page-level grouping and comparison for row counts in testbed and results
* Fix row count difference calculation in page comparison
* Merged main into feature/row-count-by-pages
* Fix exhibit page normalization logic in group count calculations
Approved-by: Katon Minhas
Feature/row count by pages
* Add page-level row count comparison to analyze discrepancies between testbed and results
* Fix key naming in page mismatch comparison for consistency
* Merged main into feature/row-count-by-pages
* Implement page-level grouping and comparison for row counts in testbed and results
* Fix row count difference calculation in page comparison
Approved-by: Alex Galarce
Bugfix/fee rate conversion rate
* printing incorrect recall values
* values responsible for low recall added in spreadsheet
* changing match list to unmatch list for breakouts
* recall values added for 1-to-N
* print statements removed
* removed print statements
* Merged main into feature/test_files
* output formatting fix
* fixed cases where conversion factor was shown as fee rate
* Merged main into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
* unit of measure fixed
* Merge branch 'bugfix/fee_rate_conversion_rate' of https://bitbucket.org/aarete/doczy.ai into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
* functionality to create and compare multiple rows for AARETE_DERIVED_LOB added
* Merge branch 'bugfix/fee_rate_conversion_rate' of https://bitbucket.org/aarete/doczy.ai into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
Approved-by: Katon Minhas
bugfix/reimb pct rate fix to main
* updated pct rate post process
* postprocess pct rate
* Merged main into bugfix/reimb_pct_rate_fix
* removed comments
Approved-by: Katon Minhas
Feature/rework table handling
* test: add cases for configuration boundary
* test: add table post-processing tests (cases to ensure preservation of post-table text and metadata during table operations)
* test: enhance table handling tests for small and large tables, ensuring proper suffixing and splitting behavior
* Fix and clarify intent in comments
* fix: enhance handling of post-table text when combining tables
* refactor: remove commented-out code in clean_tables function for clarity
* test: update post-table text preservation assertions and add debug tracing
* refactor: remove unused remove_table_from_page function and its test
* refactor: enhance combine method documentation and clarify split_by_size_with_smart_tables logic
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
* refactor: reduce large table threshold from 1000 to 50 rows for better table splitting
* Add simple splitting
* refactor: uncomment dynamic answer retrieval in run_one_to_n_prompts for improved functionality
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
* refactor: remove unused table handling functions and related constants for cleaner code
* refactor: remove unused test functions in preparation for new tests
* refactor: streamline continuation table handling by utilizing existing flags
* refactor: update table combination logic to include all tables from the main page
* refactor: enhance table handling to automatically resolve column mismatches, respect row limits, and improve unit/integration test coverage
* rename clean_tables_simple to clean_tables
* Update docstrings
* refactor: rename table handling functions
* Fix indentation and add tests
* isort, black
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
Approved-by: Katon Minhas
Bugfix/exhibit linking
* Crosswalk updates
* Merge branch 'main' into bugfix/dynamic-primary-issues
* dont include section information in Exhibit Header
* Add Molina Health Benefit Exchange to Product-LOB mapping
* PPO-POS separate mapping
* Prevent inference of LOB from Program or Product
* Add SoonerSelect
* Update mapping rules to allow for direct values
* update prompts
* Prep for run
* Merged main into bugfix/dynamic-primary-issues
* Update unit tests
* Update Moda Product mappings
* Remove Care1st as Product for Centene
* testing
* Mapping update
* Mapping updates
* prep for merge
* Merged main into bugfix/dynamic-primary-issues
* update prompt template
* remove debugs
* Remove prints
* Merge branch 'bugfix/dynamic-primary-issues' into bugfix/exhibit-linking
* Merged main into bugfix/exhibit-linking
* Merge branch 'main' into bugfix/exhibit-linking
Approved-by: Alex Galarce
Bugfix/dynamic primary issues
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Merged main into bugfix/dynamic-primary-issues
* Crosswalk updates
* Merge branch 'main' into bugfix/dynamic-primary-issues
* dont include section information in Exhibit Header
* Add Molina Health Benefit Exchange to Product-LOB mapping
* PPO-POS separate mapping
* Prevent inference of LOB from Program or Product
* Add SoonerSelect
* Update mapping rules to allow for direct values
* update prompts
* Prep for run
* Merged main into bugfix/dynamic-primary-issues
* Update unit tests
* Update Moda Product mappings
* Remove Care1st as Product for Centene
* testing
* Mapping update
* Mapping updates
* prep for merge
* Merged main into bugfix/dynamic-primary-issues
* Merged main into bugfix/dynamic-primary-issues
* docstrings and formatting
* Get group provider for > 1
* Merged main into bugfix/dynamic-primary-issues
Approved-by: Alex Galarce
Feature/test files
* printing incorrect recall values
* values responsible for low recall added in spreadsheet
* changing match list to unmatch list for breakouts
* recall values added for 1-to-N
* print statements removed
* removed print statements
* Merged main into feature/test_files
* output formatting fix
* Merged main into feature/test_files
Approved-by: Katon Minhas
bugfix/lesser_of_rows_and_pct_rate -> main
* updated reimb term breakout prompt
* updated lesser_of_ind prompt
* Merged main into bugfix/lesser_of_rows_and_pct_rate
* Merged main into bugfix/lesser_of_rows_and_pct_rate
Approved-by: Katon Minhas
Bugfix/date format fixes
* Switch to 12/31 for auto renews
* Remove redundant postprocessing
* Switch to 12/31 for auto renews
* auto renew = Y
Approved-by: Alex Galarce
Feature/formatted prov info json
* Add newline and pipe-delimited formatting for provider info JSON
* Add new investment column names for nicely-formatted PROV_INFO_JSON
* remove PROV_INFO_JSON_PIPE and rename PROV_INFO_JSON_NEWLINE to PROV_INFO_JSON_FORMATTED
* Merge remote-tracking branch 'origin/main' into feature/formatted-prov-info-json
Approved-by: Katon Minhas
Bugfix/NTE num rows to Main
* updated REIMB_TERM_BREAKOUTprompt
* Merged main into bugfix/NTE_num_rows
* updated NTE prompt
* updated reimb_pct_rate prompt
* updated reimb_method prompt
* added cost prompt in reimb method
* Merged main into bugfix/NTE_num_rows
* conflict fixes
* Merged main into bugfix/NTE_num_rows
* updated prompts
Approved-by: Alex Galarce
Feature/formatted prov info json
* Add newline and pipe-delimited formatting for provider info JSON
* Add new investment column names for nicely-formatted PROV_INFO_JSON
Approved-by: Katon Minhas
Add (service, methodology) deduplication with accumulator
* Add deduplication with accumulator
* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication
* Replace print statement with logging.debug for cross-exhibit duplicate detection
* Fix and add tests for one_to_n_funcs
* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication
* Add seen_pairs parameter to reimbursement_level function in tests
Approved-by: Katon Minhas
Feature/exhibit linking
* Initialize new exhibit header prompt
* Save
* Merge branch 'main' into feature/exhibit-linking
* update prompt
* cleanup return type
* remove test
* Test fixes
* Fix test
* Add tests
* Merge branch 'main' into feature/exhibit-linking
* Merge branch 'main' into feature/exhibit-linking
* Remove test notebooks
* Delete test notebook
* Save
* Merge branch 'main' into feature/exhibit-linking
* try except for exhbit page finding
* Prep for merge
* update test
* Remove test
* Merge branch 'main' into feature/exhibit-linking
* update unit test
* Fix unit test
* replace try except
* Update page sort
* Add docstring
Approved-by: Alex Galarce
Hotfix/methodology breakout issue fixes
* additional values in unit of measure removed
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* added valid list of unit of measure
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
Approved-by: Katon Minhas
Update reimb tin-npi-name passing to reimbursement primary
* Update reimb tin-npi-name passing to reimbursement primary
* Refactor reimbursement_tin_npi to streamline provider value extraction and improve handling of empty values
Approved-by: Alex Galarce
Feature/global lesser of
* Merge remote-tracking branch 'origin/feature/reimb-primary-updates'
* First try at global "lesser of"
* Generate reimb_ids further upstream
* Implement global lesser of injection and extraction logic in file processing
* Update logic to change LESSER_OF_IND in-place for global affected rows; add docstrings
* Refactor global lesser of injection logic to create new rows for REIMB_IDs lacking lesser of logic; update original rows in place and improve tracking flags.
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add logging for global lesser of logic injection and update handling of original rows
* Refactor global lesser of logic: move functions to one_to_one_funcs.py and update inject_global_lesser_of_rows to use new implementations
* Move imports for METHODOLOGY_BREAKOUT_QUESTIONS and REIMB_TERM_BREAKOUT to the top of one_to_one_funcs.py for better organization and to remove prototype comments.
* isort
* Refactor import statement for generate_reimb_ids to use the full module path for clarity
* load field from JSON instead of hardcoded values
* Update logging levels in investment_field_testing notebook
* Merged main into feature/global-lesser-of
* Remove unnecessary comment from investment_postprocessing_funcs import statement for clarity
* Merge branch 'feature/global-lesser-of' of https://bitbucket.org/aarete/doczy.ai into feature/global-lesser-of
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Remove debug print statement from process_file function
* fix call to Field.load_from_file
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add tracking flag for global lesser of logic when none is found
* explicitly sort by FILE_NAME, EXHIBIT_PAGE, and REIMB_LESSER_OF_ID columns
* Add new field type, "full_context_separate", to make sure that we don't run it twice. Also formatting fix
* Fix iteration over separate fields in run_full_context_fields function
Approved-by: Katon Minhas
Feature/fuzzy match no numbers
* Add fuzzy matching function for string comparison and enhance term extraction result evaluation
* Enhance term extraction results evaluation by adding counts and readable strings for matched, false negatives, and false positives
* Adjust text similarity threshold to 65 and improve readability of comparison results in term extraction evaluation
* Merged main into feature/fuzzy-match-no-numbers
Approved-by: Katon Minhas
Feature/further reimb primary improvements
* Refactor identify_reimbursement_exhibits to return both EXPLICIT and IMPLICIT values
* Moving from # Debugging lines to `logger`
* Enhance "lesser of" and exception language instructions in reimbursement terms
* clarify unclear print/logging statements
* Replace debugging print statements with logging in get_reimbursement_primary function
* Add logging for page processing in run_one_to_n_prompts function
Approved-by: Katon Minhas
Don't consider table-subpages each individually for reimbursement exhibit identification
* Refine EXHIBIT_CHECK prompt
* test: rework `get_exhibit_pages` for table subpages
* Fix indentation
* Experimental table de-chunking
* Increase max_tokens for Claude model in exhibit page processing
* fix get_pages_to_process docstring
* refactor: improved docstring for get_exhibit_pages
* move `get_pages_to_process` to `preprocess.py`
* test: add unit tests for get_pages_to_process function
* Comment out debugging lines
* Changed prompt that was breading universal_json_load parsing
* fix: handle "no results" case in get_reimbursement_primary function
* fixed docstring
Approved-by: Katon Minhas
default term new prompt and de-duplication before Methodology Breakout
* Refine prompts for default indicator methodology and enhance instructions for final reimbursement methodology output
* Enhance DEFAULT_INDICATOR prompt to clarify its role as a fallback reimbursement method
* Implement service-reimb tracking to prevent duplicates
* Refactor methodology breakout processing to reduce code duplication and improve clarity
* Remove debugging print statements and improve item ID formatting in reimbursement processing
* Merge remote-tracking branch 'origin/main' into feature/default-term-new-prompt
* Refactor methodology breakout processing to remove duplicate tracking and simplify logic
* Merged main into feature/default-term-new-prompt
* Merged main into feature/default-term-new-prompt
* rename _process_answer_dict to _process_carveout_or_breakout
Approved-by: Katon Minhas
Feature/product mapping
* Initialize product-lob mapping
* Update Products for Moda
* Add valid products to the prompt
* Update fill_na_mapping
* E2E test passed
* Update product-lob mapping
* Merged main into feature/product-mapping
* Merged main into feature/product-mapping
* Add product mappings
* update mappings
* Add Healthy Families Benefit Program --> CHIP mapping
* Merged main into feature/product-mapping
* Merged main into feature/product-mapping
* Blank products
* Ready for merge
* Merged main into feature/product-mapping
* Merged main into feature/product-mapping
Approved-by: Alex Galarce
Refactor methodology breakout prompt, small testbed cleanup
* Refactor REIMB_TERM_BREAKOUT prompt. Include service AND methodology. Change to sections formatting.
* rename some columns in `Reimbursement Primary` tab of test output
Approved-by: Katon Minhas
standardized AARETE_DERIVED_FEE_SCHEDULE, removed Unit of Measure as Encounter, fix where conversion factor is captured as fee rate
* standardized AARETE_DERIVED_FEE_SCHEDULE, removed Unit of Measuer as Encounter, fix where conversion factor is captured as fee rate
* REIMB_FEE_RATE prompt updated
* updated derived fee schedule values
* updated derived fee schedule version prompt
* Merge branch 'main' into feature/daip-219
* update REIMB_PCT_RATE prompt
* Merged main into feature/daip-219
* Merged main into feature/daip-219
* Update all-caps
* Update capitalization
Approved-by: Katon Minhas
find_overlap function added to handle inconsistent chunk size
* find_overlap function added to handle inconsistent chunk size
* set min chunk overlap needed to remove overlap to 3
* Merge branch 'main' into bugfix/smart-chunking
* pipeline error fixed
* Merge branch 'main' into bugfix/smart-chunking
* spelling fix
Approved-by: Katon Minhas
Methodology Primary - New Prompt and Test Bed Utilities
* Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information
* Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked
* Add functionality to identify reimbursement exhibits
* Add functionality to identify reimbursement exhibits in run_one_to_n_prompts
* Output YAML parsing
* Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling
* New reimbursement exhibit logic
* Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers
* Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing
* Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts
* Remove commented debugging print statements in identify_reimbursement_exhibits
* Increase read_timeout in EC2 configuration from 2000 to 4000
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Remove debugging print statements in get_reimbursement_primary function
* mypy error
* Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Refactor REIMBURSEMENT_LEVEL_PRIMARY prompt
* Merge remote-tracking branch 'origin/main' into feature/explore-methodology-primary
* Rename and document row comparison function
* Merge remote-tracking branch 'origin/main' into feature/explore-methodology-primary
* Add reimbursement primary analysis and comparison function to testbed
* Enhance reimbursement exhibit identification prompt with explicit and implicit indicators
* Rename ground truth count and pairs in term extraction results for clarity
* shorten methdology_primary_row_comparison notebook
Approved-by: Katon Minhas
Alex Galarce