Refactor methodology breakout prompt, small testbed cleanup
* Refactor REIMB_TERM_BREAKOUT prompt. Include service AND methodology. Change to sections formatting.
* rename some columns in `Reimbursement Primary` tab of test output
Approved-by: Katon Minhas
standardized AARETE_DERIVED_FEE_SCHEDULE, removed Unit of Measure as Encounter, fix where conversion factor is captured as fee rate
* standardized AARETE_DERIVED_FEE_SCHEDULE, removed Unit of Measuer as Encounter, fix where conversion factor is captured as fee rate
* REIMB_FEE_RATE prompt updated
* updated derived fee schedule values
* updated derived fee schedule version prompt
* Merge branch 'main' into feature/daip-219
* update REIMB_PCT_RATE prompt
* Merged main into feature/daip-219
* Merged main into feature/daip-219
* Update all-caps
* Update capitalization
Approved-by: Katon Minhas
find_overlap function added to handle inconsistent chunk size
* find_overlap function added to handle inconsistent chunk size
* set min chunk overlap needed to remove overlap to 3
* Merge branch 'main' into bugfix/smart-chunking
* pipeline error fixed
* Merge branch 'main' into bugfix/smart-chunking
* spelling fix
Approved-by: Katon Minhas
Methodology Primary - New Prompt and Test Bed Utilities
* Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information
* Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked
* Add functionality to identify reimbursement exhibits
* Add functionality to identify reimbursement exhibits in run_one_to_n_prompts
* Output YAML parsing
* Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling
* New reimbursement exhibit logic
* Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers
* Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing
* Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts
* Remove commented debugging print statements in identify_reimbursement_exhibits
* Increase read_timeout in EC2 configuration from 2000 to 4000
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Remove debugging print statements in get_reimbursement_primary function
* mypy error
* Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Refactor REIMBURSEMENT_LEVEL_PRIMARY prompt
* Merge remote-tracking branch 'origin/main' into feature/explore-methodology-primary
* Rename and document row comparison function
* Merge remote-tracking branch 'origin/main' into feature/explore-methodology-primary
* Add reimbursement primary analysis and comparison function to testbed
* Enhance reimbursement exhibit identification prompt with explicit and implicit indicators
* Rename ground truth count and pairs in term extraction results for clarity
* shorten methdology_primary_row_comparison notebook
Approved-by: Katon Minhas
Reimbursement Exhibit Prefiltering
* Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information
* Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked
* Add functionality to identify reimbursement exhibits
* Add functionality to identify reimbursement exhibits in run_one_to_n_prompts
* Output YAML parsing
* Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling
* New reimbursement exhibit logic
* Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers
* Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing
* Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts
* Remove commented debugging print statements in identify_reimbursement_exhibits
* Increase read_timeout in EC2 configuration from 2000 to 4000
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Remove debugging print statements in get_reimbursement_primary function
* mypy error
* Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
Approved-by: Katon Minhas
Update testbed data sources and enhance metric calculations for blank fields
* Update testbed data sources and enhance metric calculations for blank fields
* Add order-independent provider analysis
* add tin-npi normalization and verbose
* Enhance provider field evaluation by excluding "UNKNOWN" and "NO_IDENTIFIERS_FOUND" values
* List-based method
* Enhance provider field evaluation by incorporating frequency-based metrics and excluding "UNKNOWN" values
* Fix splits
* fix headers
* fix comma splitting
* Merged main into tin-npi-acc-testing
* Merged main into tin-npi-acc-testing
* Refine provider metrics logging to exclude empty predicted values
* Merge branch 'tin-npi-acc-testing' of https://bitbucket.org/aarete/doczy.ai into tin-npi-acc-testing
* Merged main into tin-npi-acc-testing
* last commit before changing back to sets
* Refactor evaluation metrics calculation to use set operations (while still displaying counters for info)
* Merge branch 'tin-npi-acc-testing' of https://bitbucket.org/aarete/doczy.ai into tin-npi-acc-testing
Approved-by: Katon Minhas
Feature/medicare medicaid postprocess
* Initial test
* add to postprocessing funcs
* Remove from postprocessing
* Merged main into feature/medicare-medicaid-postprocess
Approved-by: Alex Galarce
Bugfix/add name to full context if no identifiers
* Add handling for missing TIN/NPI identifiers in one-to-one results
* Update provider name handling in one-to-one results for group providers
Approved-by: Katon Minhas
DRAFT: Bugfix/effective date fix
* Merged in bugfix/extra-fields (pull request #500)
Do not run Exhibit level prompt if there are no exhibit level fields
* Do not run Exhibit level prompt if there are no exhibit level fields
* Update dynamic test
* Remove test from one-to-n test module
* Update test
* Fix unit test
Approved-by: Alex Galarce
* Merged in bugfix/tin-npi-prompt-fixes (pull request #499)
Bugfix/tin npi prompt fixes
* Change delimiter for other provider information from comma to pipe
* filter out invalid providers
* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE
* Merged main into bugfix/tin-npi-small-bugfixes
* Refactor deduplication logic to use string_utils for checking empty provider fields
* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes
* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes
* Fix deduplication logic to not drop subsequent names
* Fix formatting of provider names in deduplication logic
* add payer name filtering
* reorder execution order to allow to find payer_name first
* Add handling for previously unfound group providers in identify_group_provider function
* Merge remote-trackin…
* Merged in feature/update-testbed-script (pull request #501)
Feature/update testbed script
* Update preprocessing
* Update preprocessing
* remove quit
* Update date for filename
* Merged main into feature/update-testbed-script
Approved-by: Alex Galarce
* Merged in bugfix/smart-chunks (pull request #503)
missing of text in chunks fixed
* missing of text in chunks fixed
Approved-by: Katon Minhas
* Merged in feature/rework-reimb-tin-npi (pull request #502)
Feature/rework reimb tin npi
* use LLM for reimb_[tin/npi/name]
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* deduplicate tin/name/npi before adding to prompt
* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging
* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly
* Reorder REIMB_PROV_NAME field in investment_prompts.json
* Update REIMB_PROV_NAME prompt to specify valid values for provider names
* Remove valid_values constraint from prov_name to raise accuracy
* Remove debugging print statements from reimbursement_tin_npi and related functions
* Remove debugging print statements from run_provider_info_fields function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Remove debugging print statem…
* Merged in feature/preformat-single-quotes-input (pull request #506)
Replace double quotes with single quotes in context strings for parsability
* Replace double quotes with single quotes in context strings for parsability
* Merged main into feature/preformat-single-quotes-input
Approved-by: Katon Minhas
* updated sig pages fnxn
* Merge branch 'main' into bugfix/effective_date_fix
* poetry add pymupdf
* created vision funcs
* Merged main into bugfix/effective_date_fix
* fixed pipeline issue
* updated string_utils_test
* updated test values
* updated extract signature page fxn
* updated test values
* updated test values
* Merged main into bugfix/effective_date_fix
* Prompt template in all-caps
* remove debugging prints
* Merged main into bugfix/effective_date_fix
* Change vision to False by default
* Change vision to False by default
* fix tests
* fix tests
Approved-by: Katon Minhas
Bugfix/exhibit header detection
* set up testing
* Increased context length, changed prompt to investment_prompts from preprocessing_prompts
* extracting the header
* minor changes
* Merged main into bugfix/exhibit-header-detection
* cleanup prints and tests
Approved-by: Alex Galarce
Add function to group provider information into JSON array format post-hoc from the test bed
* Add function to group provider information into JSON array format
* Add example notebook
Approved-by: Katon Minhas
Update prompts to use pipes (|) for multiple TINs, NPIs, and provider names instead of commas
* Update prompts to use pipes (|) for multiple TINs, NPIs, and provider names instead of commas
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-delimiter
Approved-by: Katon Minhas
Add Commercial-Exchange mapping
* Add Commercial-Exchange mapping
* Merged main into issuefix/health-insurance-exchange-issue
Approved-by: Alex Galarce
Replace double quotes with single quotes in context strings for parsability
* Replace double quotes with single quotes in context strings for parsability
* Merged main into feature/preformat-single-quotes-input
Approved-by: Katon Minhas
Feature/rework reimb tin npi
* use LLM for reimb_[tin/npi/name]
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* deduplicate tin/name/npi before adding to prompt
* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging
* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly
* Reorder REIMB_PROV_NAME field in investment_prompts.json
* Update REIMB_PROV_NAME prompt to specify valid values for provider names
* Remove valid_values constraint from prov_name to raise accuracy
* Remove debugging print statements from reimbursement_tin_npi and related functions
* Remove debugging print statements from run_provider_info_fields function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Remove debugging print statements from get_reimbursement_primary function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Refactor reimbursement_tin_npi to always add providers (when we have any) to reimbursement_level_fields
Approved-by: Katon Minhas
Feature/update testbed script
* Update preprocessing
* Update preprocessing
* remove quit
* Update date for filename
* Merged main into feature/update-testbed-script
Approved-by: Alex Galarce
Bugfix/tin npi prompt fixes
* Change delimiter for other provider information from comma to pipe
* filter out invalid providers
* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE
* Merged main into bugfix/tin-npi-small-bugfixes
* Refactor deduplication logic to use string_utils for checking empty provider fields
* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes
* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes
* Fix deduplication logic to not drop subsequent names
* Fix formatting of provider names in deduplication logic
* add payer name filtering
* reorder execution order to allow to find payer_name first
* Add handling for previously unfound group providers in identify_group_provider function
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-prompt-fixes
* Remove debugging print statements from provider info and group provider identification functions
* Refactor provider info processing to remove unused payer_name parameter and adjust execution order for clarity
* add debug lines
* Clarify instructions in TIN_NPI_TEMPLATE and GROUP_TIN_NPI_TEMPLATE for extracting provider entities, emphasizing the distinction between providers and payers, and adding identification indicators for main contracting groups.
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-prompt-fixes
* Remove debugging print statements from run_provider_info_fields function
Approved-by: Katon Minhas
Do not run Exhibit level prompt if there are no exhibit level fields
* Do not run Exhibit level prompt if there are no exhibit level fields
* Update dynamic test
* Remove test from one-to-n test module
* Update test
* Fix unit test
Approved-by: Alex Galarce
Bugfix/tin npi small bugfixes
* Change delimiter for other provider information from comma to pipe
* filter out invalid providers
* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE
* Merged main into bugfix/tin-npi-small-bugfixes
* Refactor deduplication logic to use string_utils for checking empty provider fields
* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes
* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME
Approved-by: Katon Minhas
Collect TIN/NPI/Name page-by-page and run a second pass for IS_GROUP
* configure first pass: no providers marked IS_GROUP. Allows simplification of prompt
* Add group provider identification logic in second pass
* group identification: don't take 20% of pages. Take first 3 and last 3
* Refine group provider identification logic to use pipes for JSON formatting and improve error handling
* fix mypy error for `identify_group_provider`
* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting
* Add "PROV_INFO_JSON" to investment column order
* Refactor group provider identification to use signature page extraction and fix JSON format
* move signature page extraction to `string_utils.py` and add unit tests for it
* Update identify_group_provider to return provider name along with TIN and NPI in JSON format
* Update identify_group_provider to always use group-provided name when available
* move out GROUP_TIN_NPI_TEMPLATE to `investment_prompts.py`
* Remove deprecated PROV_GROUP_TIN_CHECK and PROV_GROUP_NPI_CHECK prompts
* Move out provider info fields from `one_to_one_funcs` to `tin_npi_funcs`
* isort
* remove debug print statements
* Remove debug print statements from provider info functions
* add note to regex_utils.py about where functions are used
* Add page_key_sort function and corresponding tests for sorting page keys
* Refactor identify_group_provider to sort pages and avoid duplicates in extracted sections
* Remove debug print statements from identify_group_provider function
* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting
* extract_signature_page function: prioritize Textract marker for signatures and fallback to keyword search
* fix mypy error
* Update test cases to reflect changes in signature extraction logic
Approved-by: Katon Minhas
Field/initialize patient age
* Update split_text so that it works on single-page contracts (discovered when testing)
* Add PATIENT_AGE_RANGE intermediate column
* Remove PATIENT_AGE_RANGE in postprocessing
* Dont convert columns to int type
* Add unit tests
* Remove test
* Update unit test
* Update unit test
* Update unit test
* Update unit test
Approved-by: Alex Galarce
Field/dynamic codes
* remove test
* Merge branch 'main' into field/dynamic-codes
* Merged main into field/dynamic-codes
* Update test
* Fix unit test
* Merged main into field/dynamic-codes
* Merged main into field/dynamic-codes
* Merged main into field/dynamic-codes
* Update fields with crosswalks
* Update mapping to return empty string if no mapping
* Update dynamic_funcs
* Restructure
* Update file_processing
* Update dynamic primary
* Fix dynamic primary
* Update base fields
* Update one-to-n process
* Merged main into field/dynamic-codes
* genericized get_dynamic_answers
* Update tests
* Remove test file
* dynamic_funcs cleanup
* remove prints
* add exhibit_header
* Fix imports
Approved-by: Alex Galarce
Enhance universal_json_load function with detailed error handling and traceback logging
* Enhance universal_json_load function with detailed error handling and traceback logging
* Add type hint for string_dict parameter in universal_json_load function
* Update return type annotation for universal_json_load function to reflect possible return values
Approved-by: Katon Minhas
Feature/one to n testing
* Add field by field prec and rec
* Add field-wise prec and rec
* Merged main into feature/one-to-n-testing
* Merged main into feature/one-to-n-testing
* Merged main into feature/one-to-n-testing
Approved-by: Alex Galarce
Bugfix/flatten singleton list ints problem
* Fix flatten_singleton_string_list to handle non-list int strings and improve test coverage
* Fix test_flatten_singleton_string_list to handle integer input correctly
Approved-by: Katon Minhas
format_rate_fields_with_commas: change how empty values and errors are handled
* format_rate_fields_with_commas: change how empty values and errors are handled
Approved-by: Katon Minhas
Feature/align tin npi
* Add merge_provider_info function to consolidate provider data into one-to-one results
* Merge remote-tracking branch 'origin/main' into feature/align-tin-npi
* Remove debug print statement from run_one_to_one_prompts function
* turn run_regex_fields into an orchestrator function by breaking parts of it into separate functions
* Remove unused import of get_matches function and add docstring to chunk_on_matches for clarity
* Add docstring to run_regex_fields for improved clarity and documentation
* fix provider name keys in merge_provider_info
* Remove unused functions get_matches, tin_npi_prompt, and clean_tin_npi from tin_npi_funcs.py
* Set provider name to "UNKNOWN" if cleaned name is empty in clean_provider_info
* Add unit tests for clean_provider_info function to validate data processing
* Enhance docstring for get_all_matches function
* Add tests for get_all_matches and chunk_on_matches functions
* Merge remote-tracking branch 'origin/main' into feature/align-tin-npi
* Implement merge_provider_info function to consolidate provider data into one-to-one results
* Refactor merge_provider_info to simplify list conversion for group and other provider information
* Remove unnecessary TIN and NPI postprocessing step from postprocess function
* Remove clean_tin_npi_other function and its invocation from postprocess function
* Use json prompts for TIN/NPI/Name and inject them into TIN_NPI_TEMPLATE
* Rename run_regex_fields to run_provider_info_fields
* Enhance TIN_NPI_TEMPLATE to include detailed instructions for extracting provider entities and their identifying information, including IS_GROUP logic for main contracting parties.
* moved TIN/NPI regexes to regex_patterns.py
* Remove unnecessary blank line in investment_values.py
* Remove debug print statements from run_provider_info_fields and get_provider_info functions
* Enhance clean_provider_info to handle various representations of IS_GROUP as boolean
* Enhance get_provider_info to include error handling for JSON parsing and ensure consistent return format
Approved-by: Katon Minhas
Bugfix/daip2 156
* try blocks added
* fixed output 'of charges' in reimb_method
* Merged main into bugfix/daip2-156
* removed try block
* Merge branch 'bugfix/daip2-156' of https://bitbucket.org/aarete/doczy.ai into bugfix/daip2-156
* Merged main into bugfix/daip2-156
Approved-by: Alex Galarce
Hotfix/flatten list to string
* updated
* rate fields fix
* updated the singleton_list
* updated
* added
* Checked_files
* added
* Merged main into hotfix/flatten-list-to-string
* format_rate_fields_with_commas
* Update test
* flatten_singleton_string_list - new cases
* Update
* Removed unneccessary case
* Remove case
* Updated the test cases
* Merge remote-tracking branch 'origin/main' into hotfix/flatten-list-to-string
* Refactor flatten_singleton_string_list to handle empty input and return a comma-separated string for multiple elements
Approved-by: Katon Minhas
Enhance is_empty function to handle pandas Series and improve empty value checks
* Enhance is_empty function to handle pandas Series and improve empty value checks
* Enhance is_empty function to support pandas Series and add pd_mask parameter for flexible empty checks
Approved-by: Katon Minhas
Hotfix/reimb issue fixes
* updated DYNAMIC_CHECK prompt
* updated date_fix_prompt to YYYYMMDD format
* filled N/A values for REIMB_PROV_TIN
* changed YYYYMMDD to YYYY/MM/DD
* extracting TIN, NPI,NAME as a tuple
* updated TIN, NPI,NAME as a tuple prompt
* Merge remote-tracking branch 'origin/main' into hotfix/reimb_issue_fixes
* fix: correct spelling of 'memorize' to 'memoize' in cache decorator comment
* combined postprocess date format check functions
* Merged main into hotfix/reimb_issue_fixes
* updated postprocessing
* rename and slight refactor
* remove unused imports, re-sort
* remove unused imports from postprocess.py
* Fix unit tests
* re-add imports which are needed for tests to pass (?)
* Fix column name in auto-renewal termination date handling
* Fix formatting in postprocess function
Approved-by: Katon Minhas
Hotfix/methodology breakout issue fixes
* Merge branch 'hotfix/methodology_breakout_issue_fixes' of https://bitbucket.org/aarete/doczy.ai into hotfix/methodology_breakout_issue_fixes
merged with current local changes
* opps methodology added
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
merged changes from main
* Merged main into hotfix/methodology_breakout_issue_fixes
* duplicate N/A fixed
* Merge branch 'hotfix/methodology_breakout_issue_fixes' of https://bitbucket.org/aarete/doczy.ai into hotfix/methodology_breakout_issue_fixes
merged changes from main
* updated AARETE_DERIVED_REIMB_METHOD prompt
* fee schedule prompt change
* REIMB_PCT_RATE prompt updated
* Merged main into hotfix/methodology_breakout_issue_fixes
* opps moved to grouper
* fee rate format fixed
* removed incentives
* removed funding and additional reimbursement
* separate invalid reimb
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* merged main into current branch
* updated valid reimb and fee_schedule_version
* code structure fix
* code structure fix 2
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* updated methodology breakout fields
* test notebook updated
* Merged main into hotfix/methodology_breakout_issue_fixes
Approved-by: Katon Minhas
Feature/new stop loss and outlier fields
* initial commit; handle term_key cases, add flexibility for service and methodology, set prompt to `null` for term_key filled fields
* prompt changes for all outlier fields
* Merge remote-tracking branch 'origin/main' into feature/new-stop-loss-and-outlier-fields
* reorder columns, rework CD_DESC for outlier, replace redundant column OUTLIER_FIXED_LOSS_THRESHOLD (deprecated by OUTLIER_FIXED_LOSS_CHARGE_THRESHOLD)
* add OUTLIER_MAXIMUM_FREQUENCY prompt for outlier claims cap frequency
* add OUTLIER_RANGE_NBR_DAYS prompt to specify period for calculating outlier payments
* fix term key naming to align with Maria columns
* remove debug print statements
* add Maria stop loss columns
* stop loss prompts
Approved-by: Katon Minhas
Alex Galarce