Hotfix/methodology breakout issue fixes
* additional values in unit of measure removed
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* added valid list of unit of measure
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
Approved-by: Katon Minhas
Update reimb tin-npi-name passing to reimbursement primary
* Update reimb tin-npi-name passing to reimbursement primary
* Refactor reimbursement_tin_npi to streamline provider value extraction and improve handling of empty values
Approved-by: Alex Galarce
Feature/global lesser of
* Merge remote-tracking branch 'origin/feature/reimb-primary-updates'
* First try at global "lesser of"
* Generate reimb_ids further upstream
* Implement global lesser of injection and extraction logic in file processing
* Update logic to change LESSER_OF_IND in-place for global affected rows; add docstrings
* Refactor global lesser of injection logic to create new rows for REIMB_IDs lacking lesser of logic; update original rows in place and improve tracking flags.
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add logging for global lesser of logic injection and update handling of original rows
* Refactor global lesser of logic: move functions to one_to_one_funcs.py and update inject_global_lesser_of_rows to use new implementations
* Move imports for METHODOLOGY_BREAKOUT_QUESTIONS and REIMB_TERM_BREAKOUT to the top of one_to_one_funcs.py for better organization and to remove prototype comments.
* isort
* Refactor import statement for generate_reimb_ids to use the full module path for clarity
* load field from JSON instead of hardcoded values
* Update logging levels in investment_field_testing notebook
* Merged main into feature/global-lesser-of
* Remove unnecessary comment from investment_postprocessing_funcs import statement for clarity
* Merge branch 'feature/global-lesser-of' of https://bitbucket.org/aarete/doczy.ai into feature/global-lesser-of
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Remove debug print statement from process_file function
* fix call to Field.load_from_file
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add tracking flag for global lesser of logic when none is found
* explicitly sort by FILE_NAME, EXHIBIT_PAGE, and REIMB_LESSER_OF_ID columns
* Add new field type, "full_context_separate", to make sure that we don't run it twice. Also formatting fix
* Fix iteration over separate fields in run_full_context_fields function
Approved-by: Katon Minhas
Feature/fuzzy match no numbers
* Add fuzzy matching function for string comparison and enhance term extraction result evaluation
* Enhance term extraction results evaluation by adding counts and readable strings for matched, false negatives, and false positives
* Adjust text similarity threshold to 65 and improve readability of comparison results in term extraction evaluation
* Merged main into feature/fuzzy-match-no-numbers
Approved-by: Katon Minhas
Feature/further reimb primary improvements
* Refactor identify_reimbursement_exhibits to return both EXPLICIT and IMPLICIT values
* Moving from # Debugging lines to `logger`
* Enhance "lesser of" and exception language instructions in reimbursement terms
* clarify unclear print/logging statements
* Replace debugging print statements with logging in get_reimbursement_primary function
* Add logging for page processing in run_one_to_n_prompts function
Approved-by: Katon Minhas
Don't consider table-subpages each individually for reimbursement exhibit identification
* Refine EXHIBIT_CHECK prompt
* test: rework `get_exhibit_pages` for table subpages
* Fix indentation
* Experimental table de-chunking
* Increase max_tokens for Claude model in exhibit page processing
* fix get_pages_to_process docstring
* refactor: improved docstring for get_exhibit_pages
* move `get_pages_to_process` to `preprocess.py`
* test: add unit tests for get_pages_to_process function
* Comment out debugging lines
* Changed prompt that was breading universal_json_load parsing
* fix: handle "no results" case in get_reimbursement_primary function
* fixed docstring
Approved-by: Katon Minhas
default term new prompt and de-duplication before Methodology Breakout
* Refine prompts for default indicator methodology and enhance instructions for final reimbursement methodology output
* Enhance DEFAULT_INDICATOR prompt to clarify its role as a fallback reimbursement method
* Implement service-reimb tracking to prevent duplicates
* Refactor methodology breakout processing to reduce code duplication and improve clarity
* Remove debugging print statements and improve item ID formatting in reimbursement processing
* Merge remote-tracking branch 'origin/main' into feature/default-term-new-prompt
* Refactor methodology breakout processing to remove duplicate tracking and simplify logic
* Merged main into feature/default-term-new-prompt
* Merged main into feature/default-term-new-prompt
* rename _process_answer_dict to _process_carveout_or_breakout
Approved-by: Katon Minhas
Feature/product mapping
* Initialize product-lob mapping
* Update Products for Moda
* Add valid products to the prompt
* Update fill_na_mapping
* E2E test passed
* Update product-lob mapping
* Merged main into feature/product-mapping
* Merged main into feature/product-mapping
* Add product mappings
* update mappings
* Add Healthy Families Benefit Program --> CHIP mapping
* Merged main into feature/product-mapping
* Merged main into feature/product-mapping
* Blank products
* Ready for merge
* Merged main into feature/product-mapping
* Merged main into feature/product-mapping
Approved-by: Alex Galarce
Refactor methodology breakout prompt, small testbed cleanup
* Refactor REIMB_TERM_BREAKOUT prompt. Include service AND methodology. Change to sections formatting.
* rename some columns in `Reimbursement Primary` tab of test output
Approved-by: Katon Minhas
standardized AARETE_DERIVED_FEE_SCHEDULE, removed Unit of Measure as Encounter, fix where conversion factor is captured as fee rate
* standardized AARETE_DERIVED_FEE_SCHEDULE, removed Unit of Measuer as Encounter, fix where conversion factor is captured as fee rate
* REIMB_FEE_RATE prompt updated
* updated derived fee schedule values
* updated derived fee schedule version prompt
* Merge branch 'main' into feature/daip-219
* update REIMB_PCT_RATE prompt
* Merged main into feature/daip-219
* Merged main into feature/daip-219
* Update all-caps
* Update capitalization
Approved-by: Katon Minhas
find_overlap function added to handle inconsistent chunk size
* find_overlap function added to handle inconsistent chunk size
* set min chunk overlap needed to remove overlap to 3
* Merge branch 'main' into bugfix/smart-chunking
* pipeline error fixed
* Merge branch 'main' into bugfix/smart-chunking
* spelling fix
Approved-by: Katon Minhas
Methodology Primary - New Prompt and Test Bed Utilities
* Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information
* Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked
* Add functionality to identify reimbursement exhibits
* Add functionality to identify reimbursement exhibits in run_one_to_n_prompts
* Output YAML parsing
* Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling
* New reimbursement exhibit logic
* Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers
* Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing
* Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts
* Remove commented debugging print statements in identify_reimbursement_exhibits
* Increase read_timeout in EC2 configuration from 2000 to 4000
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Remove debugging print statements in get_reimbursement_primary function
* mypy error
* Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Refactor REIMBURSEMENT_LEVEL_PRIMARY prompt
* Merge remote-tracking branch 'origin/main' into feature/explore-methodology-primary
* Rename and document row comparison function
* Merge remote-tracking branch 'origin/main' into feature/explore-methodology-primary
* Add reimbursement primary analysis and comparison function to testbed
* Enhance reimbursement exhibit identification prompt with explicit and implicit indicators
* Rename ground truth count and pairs in term extraction results for clarity
* shorten methdology_primary_row_comparison notebook
Approved-by: Katon Minhas
Reimbursement Exhibit Prefiltering
* Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information
* Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked
* Add functionality to identify reimbursement exhibits
* Add functionality to identify reimbursement exhibits in run_one_to_n_prompts
* Output YAML parsing
* Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling
* New reimbursement exhibit logic
* Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers
* Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing
* Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts
* Remove commented debugging print statements in identify_reimbursement_exhibits
* Increase read_timeout in EC2 configuration from 2000 to 4000
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Remove debugging print statements in get_reimbursement_primary function
* mypy error
* Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
Approved-by: Katon Minhas
Update testbed data sources and enhance metric calculations for blank fields
* Update testbed data sources and enhance metric calculations for blank fields
* Add order-independent provider analysis
* add tin-npi normalization and verbose
* Enhance provider field evaluation by excluding "UNKNOWN" and "NO_IDENTIFIERS_FOUND" values
* List-based method
* Enhance provider field evaluation by incorporating frequency-based metrics and excluding "UNKNOWN" values
* Fix splits
* fix headers
* fix comma splitting
* Merged main into tin-npi-acc-testing
* Merged main into tin-npi-acc-testing
* Refine provider metrics logging to exclude empty predicted values
* Merge branch 'tin-npi-acc-testing' of https://bitbucket.org/aarete/doczy.ai into tin-npi-acc-testing
* Merged main into tin-npi-acc-testing
* last commit before changing back to sets
* Refactor evaluation metrics calculation to use set operations (while still displaying counters for info)
* Merge branch 'tin-npi-acc-testing' of https://bitbucket.org/aarete/doczy.ai into tin-npi-acc-testing
Approved-by: Katon Minhas
Bugfix/inconsistent table sequences
* changes for inconsistent table sequence
* Added code comment block
* Merged main into bugfix/inconsistent_table_sequences
* Merged main into bugfix/inconsistent_table_sequences
* Merged main into bugfix/inconsistent_table_sequences
* Merged main into bugfix/inconsistent_table_sequences
Approved-by: Katon Minhas
Feature/medicare medicaid postprocess
* Initial test
* add to postprocessing funcs
* Remove from postprocessing
* Merged main into feature/medicare-medicaid-postprocess
Approved-by: Alex Galarce
Bugfix/add name to full context if no identifiers
* Add handling for missing TIN/NPI identifiers in one-to-one results
* Update provider name handling in one-to-one results for group providers
Approved-by: Katon Minhas
DRAFT: Bugfix/effective date fix
* Merged in bugfix/extra-fields (pull request #500)
Do not run Exhibit level prompt if there are no exhibit level fields
* Do not run Exhibit level prompt if there are no exhibit level fields
* Update dynamic test
* Remove test from one-to-n test module
* Update test
* Fix unit test
Approved-by: Alex Galarce
* Merged in bugfix/tin-npi-prompt-fixes (pull request #499)
Bugfix/tin npi prompt fixes
* Change delimiter for other provider information from comma to pipe
* filter out invalid providers
* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE
* Merged main into bugfix/tin-npi-small-bugfixes
* Refactor deduplication logic to use string_utils for checking empty provider fields
* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes
* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes
* Fix deduplication logic to not drop subsequent names
* Fix formatting of provider names in deduplication logic
* add payer name filtering
* reorder execution order to allow to find payer_name first
* Add handling for previously unfound group providers in identify_group_provider function
* Merge remote-trackin…
* Merged in feature/update-testbed-script (pull request #501)
Feature/update testbed script
* Update preprocessing
* Update preprocessing
* remove quit
* Update date for filename
* Merged main into feature/update-testbed-script
Approved-by: Alex Galarce
* Merged in bugfix/smart-chunks (pull request #503)
missing of text in chunks fixed
* missing of text in chunks fixed
Approved-by: Katon Minhas
* Merged in feature/rework-reimb-tin-npi (pull request #502)
Feature/rework reimb tin npi
* use LLM for reimb_[tin/npi/name]
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* deduplicate tin/name/npi before adding to prompt
* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging
* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly
* Reorder REIMB_PROV_NAME field in investment_prompts.json
* Update REIMB_PROV_NAME prompt to specify valid values for provider names
* Remove valid_values constraint from prov_name to raise accuracy
* Remove debugging print statements from reimbursement_tin_npi and related functions
* Remove debugging print statements from run_provider_info_fields function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Remove debugging print statem…
* Merged in feature/preformat-single-quotes-input (pull request #506)
Replace double quotes with single quotes in context strings for parsability
* Replace double quotes with single quotes in context strings for parsability
* Merged main into feature/preformat-single-quotes-input
Approved-by: Katon Minhas
* updated sig pages fnxn
* Merge branch 'main' into bugfix/effective_date_fix
* poetry add pymupdf
* created vision funcs
* Merged main into bugfix/effective_date_fix
* fixed pipeline issue
* updated string_utils_test
* updated test values
* updated extract signature page fxn
* updated test values
* updated test values
* Merged main into bugfix/effective_date_fix
* Prompt template in all-caps
* remove debugging prints
* Merged main into bugfix/effective_date_fix
* Change vision to False by default
* Change vision to False by default
* fix tests
* fix tests
Approved-by: Katon Minhas
Bugfix/exhibit header detection
* set up testing
* Increased context length, changed prompt to investment_prompts from preprocessing_prompts
* extracting the header
* minor changes
* Merged main into bugfix/exhibit-header-detection
* cleanup prints and tests
Approved-by: Alex Galarce
Add function to group provider information into JSON array format post-hoc from the test bed
* Add function to group provider information into JSON array format
* Add example notebook
Approved-by: Katon Minhas
Update prompts to use pipes (|) for multiple TINs, NPIs, and provider names instead of commas
* Update prompts to use pipes (|) for multiple TINs, NPIs, and provider names instead of commas
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-delimiter
Approved-by: Katon Minhas
Add Commercial-Exchange mapping
* Add Commercial-Exchange mapping
* Merged main into issuefix/health-insurance-exchange-issue
Approved-by: Alex Galarce
Replace double quotes with single quotes in context strings for parsability
* Replace double quotes with single quotes in context strings for parsability
* Merged main into feature/preformat-single-quotes-input
Approved-by: Katon Minhas
Feature/rework reimb tin npi
* use LLM for reimb_[tin/npi/name]
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* deduplicate tin/name/npi before adding to prompt
* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging
* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly
* Reorder REIMB_PROV_NAME field in investment_prompts.json
* Update REIMB_PROV_NAME prompt to specify valid values for provider names
* Remove valid_values constraint from prov_name to raise accuracy
* Remove debugging print statements from reimbursement_tin_npi and related functions
* Remove debugging print statements from run_provider_info_fields function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Remove debugging print statements from get_reimbursement_primary function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Refactor reimbursement_tin_npi to always add providers (when we have any) to reimbursement_level_fields
Approved-by: Katon Minhas
Feature/update testbed script
* Update preprocessing
* Update preprocessing
* remove quit
* Update date for filename
* Merged main into feature/update-testbed-script
Approved-by: Alex Galarce
Bugfix/tin npi prompt fixes
* Change delimiter for other provider information from comma to pipe
* filter out invalid providers
* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE
* Merged main into bugfix/tin-npi-small-bugfixes
* Refactor deduplication logic to use string_utils for checking empty provider fields
* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes
* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes
* Fix deduplication logic to not drop subsequent names
* Fix formatting of provider names in deduplication logic
* add payer name filtering
* reorder execution order to allow to find payer_name first
* Add handling for previously unfound group providers in identify_group_provider function
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-prompt-fixes
* Remove debugging print statements from provider info and group provider identification functions
* Refactor provider info processing to remove unused payer_name parameter and adjust execution order for clarity
* add debug lines
* Clarify instructions in TIN_NPI_TEMPLATE and GROUP_TIN_NPI_TEMPLATE for extracting provider entities, emphasizing the distinction between providers and payers, and adding identification indicators for main contracting groups.
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-prompt-fixes
* Remove debugging print statements from run_provider_info_fields function
Approved-by: Katon Minhas
Do not run Exhibit level prompt if there are no exhibit level fields
* Do not run Exhibit level prompt if there are no exhibit level fields
* Update dynamic test
* Remove test from one-to-n test module
* Update test
* Fix unit test
Approved-by: Alex Galarce
Alex Galarce