DRAFT: Bugfix/effective date fix
* Merged in bugfix/extra-fields (pull request #500)
Do not run Exhibit level prompt if there are no exhibit level fields
* Do not run Exhibit level prompt if there are no exhibit level fields
* Update dynamic test
* Remove test from one-to-n test module
* Update test
* Fix unit test
Approved-by: Alex Galarce
* Merged in bugfix/tin-npi-prompt-fixes (pull request #499)
Bugfix/tin npi prompt fixes
* Change delimiter for other provider information from comma to pipe
* filter out invalid providers
* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE
* Merged main into bugfix/tin-npi-small-bugfixes
* Refactor deduplication logic to use string_utils for checking empty provider fields
* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes
* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes
* Fix deduplication logic to not drop subsequent names
* Fix formatting of provider names in deduplication logic
* add payer name filtering
* reorder execution order to allow to find payer_name first
* Add handling for previously unfound group providers in identify_group_provider function
* Merge remote-trackin…
* Merged in feature/update-testbed-script (pull request #501)
Feature/update testbed script
* Update preprocessing
* Update preprocessing
* remove quit
* Update date for filename
* Merged main into feature/update-testbed-script
Approved-by: Alex Galarce
* Merged in bugfix/smart-chunks (pull request #503)
missing of text in chunks fixed
* missing of text in chunks fixed
Approved-by: Katon Minhas
* Merged in feature/rework-reimb-tin-npi (pull request #502)
Feature/rework reimb tin npi
* use LLM for reimb_[tin/npi/name]
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* deduplicate tin/name/npi before adding to prompt
* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging
* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly
* Reorder REIMB_PROV_NAME field in investment_prompts.json
* Update REIMB_PROV_NAME prompt to specify valid values for provider names
* Remove valid_values constraint from prov_name to raise accuracy
* Remove debugging print statements from reimbursement_tin_npi and related functions
* Remove debugging print statements from run_provider_info_fields function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Remove debugging print statem…
* Merged in feature/preformat-single-quotes-input (pull request #506)
Replace double quotes with single quotes in context strings for parsability
* Replace double quotes with single quotes in context strings for parsability
* Merged main into feature/preformat-single-quotes-input
Approved-by: Katon Minhas
* updated sig pages fnxn
* Merge branch 'main' into bugfix/effective_date_fix
* poetry add pymupdf
* created vision funcs
* Merged main into bugfix/effective_date_fix
* fixed pipeline issue
* updated string_utils_test
* updated test values
* updated extract signature page fxn
* updated test values
* updated test values
* Merged main into bugfix/effective_date_fix
* Prompt template in all-caps
* remove debugging prints
* Merged main into bugfix/effective_date_fix
* Change vision to False by default
* Change vision to False by default
* fix tests
* fix tests
Approved-by: Katon Minhas
Bugfix/exhibit header detection
* set up testing
* Increased context length, changed prompt to investment_prompts from preprocessing_prompts
* extracting the header
* minor changes
* Merged main into bugfix/exhibit-header-detection
* cleanup prints and tests
Approved-by: Alex Galarce
Add function to group provider information into JSON array format post-hoc from the test bed
* Add function to group provider information into JSON array format
* Add example notebook
Approved-by: Katon Minhas
Update prompts to use pipes (|) for multiple TINs, NPIs, and provider names instead of commas
* Update prompts to use pipes (|) for multiple TINs, NPIs, and provider names instead of commas
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-delimiter
Approved-by: Katon Minhas
Add Commercial-Exchange mapping
* Add Commercial-Exchange mapping
* Merged main into issuefix/health-insurance-exchange-issue
Approved-by: Alex Galarce
Replace double quotes with single quotes in context strings for parsability
* Replace double quotes with single quotes in context strings for parsability
* Merged main into feature/preformat-single-quotes-input
Approved-by: Katon Minhas
Feature/rework reimb tin npi
* use LLM for reimb_[tin/npi/name]
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* deduplicate tin/name/npi before adding to prompt
* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging
* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly
* Reorder REIMB_PROV_NAME field in investment_prompts.json
* Update REIMB_PROV_NAME prompt to specify valid values for provider names
* Remove valid_values constraint from prov_name to raise accuracy
* Remove debugging print statements from reimbursement_tin_npi and related functions
* Remove debugging print statements from run_provider_info_fields function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Remove debugging print statements from get_reimbursement_primary function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Refactor reimbursement_tin_npi to always add providers (when we have any) to reimbursement_level_fields
Approved-by: Katon Minhas
Feature/update testbed script
* Update preprocessing
* Update preprocessing
* remove quit
* Update date for filename
* Merged main into feature/update-testbed-script
Approved-by: Alex Galarce
Bugfix/tin npi prompt fixes
* Change delimiter for other provider information from comma to pipe
* filter out invalid providers
* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE
* Merged main into bugfix/tin-npi-small-bugfixes
* Refactor deduplication logic to use string_utils for checking empty provider fields
* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes
* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes
* Fix deduplication logic to not drop subsequent names
* Fix formatting of provider names in deduplication logic
* add payer name filtering
* reorder execution order to allow to find payer_name first
* Add handling for previously unfound group providers in identify_group_provider function
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-prompt-fixes
* Remove debugging print statements from provider info and group provider identification functions
* Refactor provider info processing to remove unused payer_name parameter and adjust execution order for clarity
* add debug lines
* Clarify instructions in TIN_NPI_TEMPLATE and GROUP_TIN_NPI_TEMPLATE for extracting provider entities, emphasizing the distinction between providers and payers, and adding identification indicators for main contracting groups.
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-prompt-fixes
* Remove debugging print statements from run_provider_info_fields function
Approved-by: Katon Minhas
Do not run Exhibit level prompt if there are no exhibit level fields
* Do not run Exhibit level prompt if there are no exhibit level fields
* Update dynamic test
* Remove test from one-to-n test module
* Update test
* Fix unit test
Approved-by: Alex Galarce
Bugfix/tin npi small bugfixes
* Change delimiter for other provider information from comma to pipe
* filter out invalid providers
* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE
* Merged main into bugfix/tin-npi-small-bugfixes
* Refactor deduplication logic to use string_utils for checking empty provider fields
* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes
* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME
Approved-by: Katon Minhas
Collect TIN/NPI/Name page-by-page and run a second pass for IS_GROUP
* configure first pass: no providers marked IS_GROUP. Allows simplification of prompt
* Add group provider identification logic in second pass
* group identification: don't take 20% of pages. Take first 3 and last 3
* Refine group provider identification logic to use pipes for JSON formatting and improve error handling
* fix mypy error for `identify_group_provider`
* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting
* Add "PROV_INFO_JSON" to investment column order
* Refactor group provider identification to use signature page extraction and fix JSON format
* move signature page extraction to `string_utils.py` and add unit tests for it
* Update identify_group_provider to return provider name along with TIN and NPI in JSON format
* Update identify_group_provider to always use group-provided name when available
* move out GROUP_TIN_NPI_TEMPLATE to `investment_prompts.py`
* Remove deprecated PROV_GROUP_TIN_CHECK and PROV_GROUP_NPI_CHECK prompts
* Move out provider info fields from `one_to_one_funcs` to `tin_npi_funcs`
* isort
* remove debug print statements
* Remove debug print statements from provider info functions
* add note to regex_utils.py about where functions are used
* Add page_key_sort function and corresponding tests for sorting page keys
* Refactor identify_group_provider to sort pages and avoid duplicates in extracted sections
* Remove debug print statements from identify_group_provider function
* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting
* extract_signature_page function: prioritize Textract marker for signatures and fallback to keyword search
* fix mypy error
* Update test cases to reflect changes in signature extraction logic
Approved-by: Katon Minhas
Field/initialize patient age
* Update split_text so that it works on single-page contracts (discovered when testing)
* Add PATIENT_AGE_RANGE intermediate column
* Remove PATIENT_AGE_RANGE in postprocessing
* Dont convert columns to int type
* Add unit tests
* Remove test
* Update unit test
* Update unit test
* Update unit test
* Update unit test
Approved-by: Alex Galarce
Field/dynamic codes
* remove test
* Merge branch 'main' into field/dynamic-codes
* Merged main into field/dynamic-codes
* Update test
* Fix unit test
* Merged main into field/dynamic-codes
* Merged main into field/dynamic-codes
* Merged main into field/dynamic-codes
* Update fields with crosswalks
* Update mapping to return empty string if no mapping
* Update dynamic_funcs
* Restructure
* Update file_processing
* Update dynamic primary
* Fix dynamic primary
* Update base fields
* Update one-to-n process
* Merged main into field/dynamic-codes
* genericized get_dynamic_answers
* Update tests
* Remove test file
* dynamic_funcs cleanup
* remove prints
* add exhibit_header
* Fix imports
Approved-by: Alex Galarce
Enhance universal_json_load function with detailed error handling and traceback logging
* Enhance universal_json_load function with detailed error handling and traceback logging
* Add type hint for string_dict parameter in universal_json_load function
* Update return type annotation for universal_json_load function to reflect possible return values
Approved-by: Katon Minhas
Feature/one to n testing
* Add field by field prec and rec
* Add field-wise prec and rec
* Merged main into feature/one-to-n-testing
* Merged main into feature/one-to-n-testing
* Merged main into feature/one-to-n-testing
Approved-by: Alex Galarce
Bugfix/flatten singleton list ints problem
* Fix flatten_singleton_string_list to handle non-list int strings and improve test coverage
* Fix test_flatten_singleton_string_list to handle integer input correctly
Approved-by: Katon Minhas
format_rate_fields_with_commas: change how empty values and errors are handled
* format_rate_fields_with_commas: change how empty values and errors are handled
Approved-by: Katon Minhas
Feature/align tin npi
* Add merge_provider_info function to consolidate provider data into one-to-one results
* Merge remote-tracking branch 'origin/main' into feature/align-tin-npi
* Remove debug print statement from run_one_to_one_prompts function
* turn run_regex_fields into an orchestrator function by breaking parts of it into separate functions
* Remove unused import of get_matches function and add docstring to chunk_on_matches for clarity
* Add docstring to run_regex_fields for improved clarity and documentation
* fix provider name keys in merge_provider_info
* Remove unused functions get_matches, tin_npi_prompt, and clean_tin_npi from tin_npi_funcs.py
* Set provider name to "UNKNOWN" if cleaned name is empty in clean_provider_info
* Add unit tests for clean_provider_info function to validate data processing
* Enhance docstring for get_all_matches function
* Add tests for get_all_matches and chunk_on_matches functions
* Merge remote-tracking branch 'origin/main' into feature/align-tin-npi
* Implement merge_provider_info function to consolidate provider data into one-to-one results
* Refactor merge_provider_info to simplify list conversion for group and other provider information
* Remove unnecessary TIN and NPI postprocessing step from postprocess function
* Remove clean_tin_npi_other function and its invocation from postprocess function
* Use json prompts for TIN/NPI/Name and inject them into TIN_NPI_TEMPLATE
* Rename run_regex_fields to run_provider_info_fields
* Enhance TIN_NPI_TEMPLATE to include detailed instructions for extracting provider entities and their identifying information, including IS_GROUP logic for main contracting parties.
* moved TIN/NPI regexes to regex_patterns.py
* Remove unnecessary blank line in investment_values.py
* Remove debug print statements from run_provider_info_fields and get_provider_info functions
* Enhance clean_provider_info to handle various representations of IS_GROUP as boolean
* Enhance get_provider_info to include error handling for JSON parsing and ensure consistent return format
Approved-by: Katon Minhas
Bugfix/daip2 156
* try blocks added
* fixed output 'of charges' in reimb_method
* Merged main into bugfix/daip2-156
* removed try block
* Merge branch 'bugfix/daip2-156' of https://bitbucket.org/aarete/doczy.ai into bugfix/daip2-156
* Merged main into bugfix/daip2-156
Approved-by: Alex Galarce
Hotfix/flatten list to string
* updated
* rate fields fix
* updated the singleton_list
* updated
* added
* Checked_files
* added
* Merged main into hotfix/flatten-list-to-string
* format_rate_fields_with_commas
* Update test
* flatten_singleton_string_list - new cases
* Update
* Removed unneccessary case
* Remove case
* Updated the test cases
* Merge remote-tracking branch 'origin/main' into hotfix/flatten-list-to-string
* Refactor flatten_singleton_string_list to handle empty input and return a comma-separated string for multiple elements
Approved-by: Katon Minhas
Enhance is_empty function to handle pandas Series and improve empty value checks
* Enhance is_empty function to handle pandas Series and improve empty value checks
* Enhance is_empty function to support pandas Series and add pd_mask parameter for flexible empty checks
Approved-by: Katon Minhas
Hotfix/reimb issue fixes
* updated DYNAMIC_CHECK prompt
* updated date_fix_prompt to YYYYMMDD format
* filled N/A values for REIMB_PROV_TIN
* changed YYYYMMDD to YYYY/MM/DD
* extracting TIN, NPI,NAME as a tuple
* updated TIN, NPI,NAME as a tuple prompt
* Merge remote-tracking branch 'origin/main' into hotfix/reimb_issue_fixes
* fix: correct spelling of 'memorize' to 'memoize' in cache decorator comment
* combined postprocess date format check functions
* Merged main into hotfix/reimb_issue_fixes
* updated postprocessing
* rename and slight refactor
* remove unused imports, re-sort
* remove unused imports from postprocess.py
* Fix unit tests
* re-add imports which are needed for tests to pass (?)
* Fix column name in auto-renewal termination date handling
* Fix formatting in postprocess function
Approved-by: Katon Minhas
Hotfix/methodology breakout issue fixes
* Merge branch 'hotfix/methodology_breakout_issue_fixes' of https://bitbucket.org/aarete/doczy.ai into hotfix/methodology_breakout_issue_fixes
merged with current local changes
* opps methodology added
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
merged changes from main
* Merged main into hotfix/methodology_breakout_issue_fixes
* duplicate N/A fixed
* Merge branch 'hotfix/methodology_breakout_issue_fixes' of https://bitbucket.org/aarete/doczy.ai into hotfix/methodology_breakout_issue_fixes
merged changes from main
* updated AARETE_DERIVED_REIMB_METHOD prompt
* fee schedule prompt change
* REIMB_PCT_RATE prompt updated
* Merged main into hotfix/methodology_breakout_issue_fixes
* opps moved to grouper
* fee rate format fixed
* removed incentives
* removed funding and additional reimbursement
* separate invalid reimb
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* merged main into current branch
* updated valid reimb and fee_schedule_version
* code structure fix
* code structure fix 2
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* updated methodology breakout fields
* test notebook updated
* Merged main into hotfix/methodology_breakout_issue_fixes
Approved-by: Katon Minhas
Feature/new stop loss and outlier fields
* initial commit; handle term_key cases, add flexibility for service and methodology, set prompt to `null` for term_key filled fields
* prompt changes for all outlier fields
* Merge remote-tracking branch 'origin/main' into feature/new-stop-loss-and-outlier-fields
* reorder columns, rework CD_DESC for outlier, replace redundant column OUTLIER_FIXED_LOSS_THRESHOLD (deprecated by OUTLIER_FIXED_LOSS_CHARGE_THRESHOLD)
* add OUTLIER_MAXIMUM_FREQUENCY prompt for outlier claims cap frequency
* add OUTLIER_RANGE_NBR_DAYS prompt to specify period for calculating outlier payments
* fix term key naming to align with Maria columns
* remove debug print statements
* add Maria stop loss columns
* stop loss prompts
Approved-by: Katon Minhas
Feature/dynamic one to one
* Pass N/A to one-to-one
* E2E test passed
* Merged main into feature/dynamic-one-to-one
* Pass ANY non N/A answer to reimbursement-level
* Update test
* Remove test
* Merge branch 'main' into feature/dynamic-one-to-one
* Wrap clean_dates
* Revert postprocess
* Merged main into feature/dynamic-one-to-one
* Merged main into feature/dynamic-one-to-one
* Remove prints
Approved-by: Alex Galarce
Feature/na blank handling
* Update prompts to clarify responses for unknown vs N/A values
* Update code description mapping to differentiate between empty and unknown grouper codes
* Update prompts and code descriptions to standardize 'unknown' to 'UNKNOWN'
* Update PIPE_FORMAT_INSTRUCTIONS to replace 'concept' with 'requested information'
* Merge remote-tracking branch 'origin/main' into feature/na_blank_handling
* Merged main into feature/na_blank_handling
* Update NPI prompt for clarity and detailed instructions
* Enhance TIN prompt for clarity and detailed identification instructions
* Merged main into feature/na_blank_handling
* Merged main into feature/na_blank_handling
Approved-by: Katon Minhas
add new configurations for carveouts
* add new configurations for carveouts
* formatting, isort, and remove a couple unused imports
Approved-by: Katon Minhas
Rework reimbursement ID to associate parts of "lesser of" (or other such coupled statement) with one another
* refactor reimbursement ID generation with hashed service terms
* reduce hash length for unique reimbursement ID generation
* update reimbursement ID format to include 'exh_pg' for clarity
* clean up testing.ipynb
* one more formatting fix to testing.ipynb
* Move reimb_id generation to postprocessing
* move reimb_id function to postprocessing_funcs.py
* refactor generate_reimb_ids to group by FILE_NAME and EXHIBIT_PAGE for ID generation
* clean testing.ipynb further
Approved-by: Katon Minhas
Feature/multiple procedures
* Add stop loss and multiple procedure reduction questions to investment prompts
* Refactor field_type values in investment prompts for consistency
* Refactor methodology breakout function to accept prompt templates as parameters
* Update investment prompts to enhance clarity and add new (provisional) fields for NEVER events and experimental treatments
* Merge remote-tracking branch 'origin/main' into feature/multiple-procedures
* start consolidation of special_cases and carveouts into one omnifunction
* Enhance carveout handling by adding support for additional question types and refactoring question retrieval
* move term_key to individual cases
* Fix casing for carveout term keys in get_carveout_fields function
* Refactor carveout processing by introducing process_carveout_or_special_case function to streamline handling of various carveout types
* Enhance process_carveout_or_special_case function documentation and clarify parameter descriptions
* Add term_key parameter to get_carveout_fields function call
* Enhance type annotations for methodology and fee schedule prompt templates in process_carveout_or_special_case function
* Add SPECIAL_CASE_CONFIGS to streamline carveout processing logic
* Remove deprecated print statements and special case handling from get_special_cases function
* name carveout_config for clarity against config.py
* Remove special case functions (deprecated)
* Remove unused imports and deprecated print statements from get_special_cases function
* remove print statements for debugging, check output of universal_string_load if it's a single dict, put it to a list
* Add special case configurations for various question types in its own module
* Refactor special case handling by consolidating configurations and removing deprecated constants and comments
* Update descriptions for OUTLIER and STOP_LOSS in VALID_SPECIAL for clarity and detail
* black formatting, changed one docstring description
* isort
* Merged main into feature/multiple-procedures
Approved-by: Katon Minhas
improve contract_amendment_num prompt
* improve contract_amendment_num prompt
* Merged main into feature/numeric_amd_num
* change CONTRACT_AMENDMENT_NUM prompt to include alphanumeric, Roman numeral, and decimal formats
* add period
Approved-by: Katon Minhas
call methodology breakout on carveouts
* Refactor carveout functionality: move get_carveout_fields to one_to_n_funcs. Move out QUESTIONS as a constant at the top
* Merge remote-tracking branch 'origin/main' into feature/methodology_breakout_for_carveouts
* Enhance methodology breakout handling: flatten nested lists and improve processing logic
* Merged main into feature/methodology_breakout_for_carveouts
* Improve error handling in file processing: add traceback logging for better debugging
* integrate methodology breakout for carveouts
* ensure default indicator is set after processing
* update default indicator assignment and extend special case answers list to avoid list-list-dict structures
* add defensive programming comment for nested lists in methodology breakout function
Approved-by: Katon Minhas
Refactor/field name updates
* New names/orders
* Add MAXIMUM_FREQUENCY Placeholder
* Merged main into refactor/field-name-updates
Approved-by: Alex Galarce
VenkataKrishna Reddy Avula