Feature/test files
* printing incorrect recall values
* values responsible for low recall added in spreadsheet
* changing match list to unmatch list for breakouts
* recall values added for 1-to-N
* print statements removed
* removed print statements
* Merged main into feature/test_files
* output formatting fix
* Merged main into feature/test_files
Approved-by: Katon Minhas
bugfix/lesser_of_rows_and_pct_rate -> main
* updated reimb term breakout prompt
* updated lesser_of_ind prompt
* Merged main into bugfix/lesser_of_rows_and_pct_rate
* Merged main into bugfix/lesser_of_rows_and_pct_rate
Approved-by: Katon Minhas
Bugfix/date format fixes
* Switch to 12/31 for auto renews
* Remove redundant postprocessing
* Switch to 12/31 for auto renews
* auto renew = Y
Approved-by: Alex Galarce
Feature/formatted prov info json
* Add newline and pipe-delimited formatting for provider info JSON
* Add new investment column names for nicely-formatted PROV_INFO_JSON
* remove PROV_INFO_JSON_PIPE and rename PROV_INFO_JSON_NEWLINE to PROV_INFO_JSON_FORMATTED
* Merge remote-tracking branch 'origin/main' into feature/formatted-prov-info-json
Approved-by: Katon Minhas
Bugfix/NTE num rows to Main
* updated REIMB_TERM_BREAKOUTprompt
* Merged main into bugfix/NTE_num_rows
* updated NTE prompt
* updated reimb_pct_rate prompt
* updated reimb_method prompt
* added cost prompt in reimb method
* Merged main into bugfix/NTE_num_rows
* conflict fixes
* Merged main into bugfix/NTE_num_rows
* updated prompts
Approved-by: Alex Galarce
Feature/formatted prov info json
* Add newline and pipe-delimited formatting for provider info JSON
* Add new investment column names for nicely-formatted PROV_INFO_JSON
Approved-by: Katon Minhas
Add (service, methodology) deduplication with accumulator
* Add deduplication with accumulator
* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication
* Replace print statement with logging.debug for cross-exhibit duplicate detection
* Fix and add tests for one_to_n_funcs
* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication
* Add seen_pairs parameter to reimbursement_level function in tests
Approved-by: Katon Minhas
Feature/exhibit linking
* Initialize new exhibit header prompt
* Save
* Merge branch 'main' into feature/exhibit-linking
* update prompt
* cleanup return type
* remove test
* Test fixes
* Fix test
* Add tests
* Merge branch 'main' into feature/exhibit-linking
* Merge branch 'main' into feature/exhibit-linking
* Remove test notebooks
* Delete test notebook
* Save
* Merge branch 'main' into feature/exhibit-linking
* try except for exhbit page finding
* Prep for merge
* update test
* Remove test
* Merge branch 'main' into feature/exhibit-linking
* update unit test
* Fix unit test
* replace try except
* Update page sort
* Add docstring
Approved-by: Alex Galarce
Hotfix/methodology breakout issue fixes
* additional values in unit of measure removed
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* added valid list of unit of measure
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
Approved-by: Katon Minhas
Update reimb tin-npi-name passing to reimbursement primary
* Update reimb tin-npi-name passing to reimbursement primary
* Refactor reimbursement_tin_npi to streamline provider value extraction and improve handling of empty values
Approved-by: Alex Galarce
Feature/global lesser of
* Merge remote-tracking branch 'origin/feature/reimb-primary-updates'
* First try at global "lesser of"
* Generate reimb_ids further upstream
* Implement global lesser of injection and extraction logic in file processing
* Update logic to change LESSER_OF_IND in-place for global affected rows; add docstrings
* Refactor global lesser of injection logic to create new rows for REIMB_IDs lacking lesser of logic; update original rows in place and improve tracking flags.
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add logging for global lesser of logic injection and update handling of original rows
* Refactor global lesser of logic: move functions to one_to_one_funcs.py and update inject_global_lesser_of_rows to use new implementations
* Move imports for METHODOLOGY_BREAKOUT_QUESTIONS and REIMB_TERM_BREAKOUT to the top of one_to_one_funcs.py for better organization and to remove prototype comments.
* isort
* Refactor import statement for generate_reimb_ids to use the full module path for clarity
* load field from JSON instead of hardcoded values
* Update logging levels in investment_field_testing notebook
* Merged main into feature/global-lesser-of
* Remove unnecessary comment from investment_postprocessing_funcs import statement for clarity
* Merge branch 'feature/global-lesser-of' of https://bitbucket.org/aarete/doczy.ai into feature/global-lesser-of
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Remove debug print statement from process_file function
* fix call to Field.load_from_file
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add tracking flag for global lesser of logic when none is found
* explicitly sort by FILE_NAME, EXHIBIT_PAGE, and REIMB_LESSER_OF_ID columns
* Add new field type, "full_context_separate", to make sure that we don't run it twice. Also formatting fix
* Fix iteration over separate fields in run_full_context_fields function
Approved-by: Katon Minhas
Feature/fuzzy match no numbers
* Add fuzzy matching function for string comparison and enhance term extraction result evaluation
* Enhance term extraction results evaluation by adding counts and readable strings for matched, false negatives, and false positives
* Adjust text similarity threshold to 65 and improve readability of comparison results in term extraction evaluation
* Merged main into feature/fuzzy-match-no-numbers
Approved-by: Katon Minhas
Feature/further reimb primary improvements
* Refactor identify_reimbursement_exhibits to return both EXPLICIT and IMPLICIT values
* Moving from # Debugging lines to `logger`
* Enhance "lesser of" and exception language instructions in reimbursement terms
* clarify unclear print/logging statements
* Replace debugging print statements with logging in get_reimbursement_primary function
* Add logging for page processing in run_one_to_n_prompts function
Approved-by: Katon Minhas
Don't consider table-subpages each individually for reimbursement exhibit identification
* Refine EXHIBIT_CHECK prompt
* test: rework `get_exhibit_pages` for table subpages
* Fix indentation
* Experimental table de-chunking
* Increase max_tokens for Claude model in exhibit page processing
* fix get_pages_to_process docstring
* refactor: improved docstring for get_exhibit_pages
* move `get_pages_to_process` to `preprocess.py`
* test: add unit tests for get_pages_to_process function
* Comment out debugging lines
* Changed prompt that was breading universal_json_load parsing
* fix: handle "no results" case in get_reimbursement_primary function
* fixed docstring
Approved-by: Katon Minhas
default term new prompt and de-duplication before Methodology Breakout
* Refine prompts for default indicator methodology and enhance instructions for final reimbursement methodology output
* Enhance DEFAULT_INDICATOR prompt to clarify its role as a fallback reimbursement method
* Implement service-reimb tracking to prevent duplicates
* Refactor methodology breakout processing to reduce code duplication and improve clarity
* Remove debugging print statements and improve item ID formatting in reimbursement processing
* Merge remote-tracking branch 'origin/main' into feature/default-term-new-prompt
* Refactor methodology breakout processing to remove duplicate tracking and simplify logic
* Merged main into feature/default-term-new-prompt
* Merged main into feature/default-term-new-prompt
* rename _process_answer_dict to _process_carveout_or_breakout
Approved-by: Katon Minhas
Feature/product mapping
* Initialize product-lob mapping
* Update Products for Moda
* Add valid products to the prompt
* Update fill_na_mapping
* E2E test passed
* Update product-lob mapping
* Merged main into feature/product-mapping
* Merged main into feature/product-mapping
* Add product mappings
* update mappings
* Add Healthy Families Benefit Program --> CHIP mapping
* Merged main into feature/product-mapping
* Merged main into feature/product-mapping
* Blank products
* Ready for merge
* Merged main into feature/product-mapping
* Merged main into feature/product-mapping
Approved-by: Alex Galarce
Refactor methodology breakout prompt, small testbed cleanup
* Refactor REIMB_TERM_BREAKOUT prompt. Include service AND methodology. Change to sections formatting.
* rename some columns in `Reimbursement Primary` tab of test output
Approved-by: Katon Minhas
standardized AARETE_DERIVED_FEE_SCHEDULE, removed Unit of Measure as Encounter, fix where conversion factor is captured as fee rate
* standardized AARETE_DERIVED_FEE_SCHEDULE, removed Unit of Measuer as Encounter, fix where conversion factor is captured as fee rate
* REIMB_FEE_RATE prompt updated
* updated derived fee schedule values
* updated derived fee schedule version prompt
* Merge branch 'main' into feature/daip-219
* update REIMB_PCT_RATE prompt
* Merged main into feature/daip-219
* Merged main into feature/daip-219
* Update all-caps
* Update capitalization
Approved-by: Katon Minhas
find_overlap function added to handle inconsistent chunk size
* find_overlap function added to handle inconsistent chunk size
* set min chunk overlap needed to remove overlap to 3
* Merge branch 'main' into bugfix/smart-chunking
* pipeline error fixed
* Merge branch 'main' into bugfix/smart-chunking
* spelling fix
Approved-by: Katon Minhas
Methodology Primary - New Prompt and Test Bed Utilities
* Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information
* Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked
* Add functionality to identify reimbursement exhibits
* Add functionality to identify reimbursement exhibits in run_one_to_n_prompts
* Output YAML parsing
* Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling
* New reimbursement exhibit logic
* Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers
* Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing
* Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts
* Remove commented debugging print statements in identify_reimbursement_exhibits
* Increase read_timeout in EC2 configuration from 2000 to 4000
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Remove debugging print statements in get_reimbursement_primary function
* mypy error
* Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Refactor REIMBURSEMENT_LEVEL_PRIMARY prompt
* Merge remote-tracking branch 'origin/main' into feature/explore-methodology-primary
* Rename and document row comparison function
* Merge remote-tracking branch 'origin/main' into feature/explore-methodology-primary
* Add reimbursement primary analysis and comparison function to testbed
* Enhance reimbursement exhibit identification prompt with explicit and implicit indicators
* Rename ground truth count and pairs in term extraction results for clarity
* shorten methdology_primary_row_comparison notebook
Approved-by: Katon Minhas
Reimbursement Exhibit Prefiltering
* Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information
* Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked
* Add functionality to identify reimbursement exhibits
* Add functionality to identify reimbursement exhibits in run_one_to_n_prompts
* Output YAML parsing
* Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling
* New reimbursement exhibit logic
* Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers
* Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing
* Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts
* Remove commented debugging print statements in identify_reimbursement_exhibits
* Increase read_timeout in EC2 configuration from 2000 to 4000
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Remove debugging print statements in get_reimbursement_primary function
* mypy error
* Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
Approved-by: Katon Minhas
Update testbed data sources and enhance metric calculations for blank fields
* Update testbed data sources and enhance metric calculations for blank fields
* Add order-independent provider analysis
* add tin-npi normalization and verbose
* Enhance provider field evaluation by excluding "UNKNOWN" and "NO_IDENTIFIERS_FOUND" values
* List-based method
* Enhance provider field evaluation by incorporating frequency-based metrics and excluding "UNKNOWN" values
* Fix splits
* fix headers
* fix comma splitting
* Merged main into tin-npi-acc-testing
* Merged main into tin-npi-acc-testing
* Refine provider metrics logging to exclude empty predicted values
* Merge branch 'tin-npi-acc-testing' of https://bitbucket.org/aarete/doczy.ai into tin-npi-acc-testing
* Merged main into tin-npi-acc-testing
* last commit before changing back to sets
* Refactor evaluation metrics calculation to use set operations (while still displaying counters for info)
* Merge branch 'tin-npi-acc-testing' of https://bitbucket.org/aarete/doczy.ai into tin-npi-acc-testing
Approved-by: Katon Minhas
Feature/medicare medicaid postprocess
* Initial test
* add to postprocessing funcs
* Remove from postprocessing
* Merged main into feature/medicare-medicaid-postprocess
Approved-by: Alex Galarce
Bugfix/add name to full context if no identifiers
* Add handling for missing TIN/NPI identifiers in one-to-one results
* Update provider name handling in one-to-one results for group providers
Approved-by: Katon Minhas
Katon Minhas