Collect TIN/NPI/Name page-by-page and run a second pass for IS_GROUP
* configure first pass: no providers marked IS_GROUP. Allows simplification of prompt
* Add group provider identification logic in second pass
* group identification: don't take 20% of pages. Take first 3 and last 3
* Refine group provider identification logic to use pipes for JSON formatting and improve error handling
* fix mypy error for `identify_group_provider`
* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting
* Add "PROV_INFO_JSON" to investment column order
* Refactor group provider identification to use signature page extraction and fix JSON format
* move signature page extraction to `string_utils.py` and add unit tests for it
* Update identify_group_provider to return provider name along with TIN and NPI in JSON format
* Update identify_group_provider to always use group-provided name when available
* move out GROUP_TIN_NPI_TEMPLATE to `investment_prompts.py`
* Remove deprecated PROV_GROUP_TIN_CHECK and PROV_GROUP_NPI_CHECK prompts
* Move out provider info fields from `one_to_one_funcs` to `tin_npi_funcs`
* isort
* remove debug print statements
* Remove debug print statements from provider info functions
* add note to regex_utils.py about where functions are used
* Add page_key_sort function and corresponding tests for sorting page keys
* Refactor identify_group_provider to sort pages and avoid duplicates in extracted sections
* Remove debug print statements from identify_group_provider function
* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting
* extract_signature_page function: prioritize Textract marker for signatures and fallback to keyword search
* fix mypy error
* Update test cases to reflect changes in signature extraction logic
Approved-by: Katon Minhas
Field/initialize patient age
* Update split_text so that it works on single-page contracts (discovered when testing)
* Add PATIENT_AGE_RANGE intermediate column
* Remove PATIENT_AGE_RANGE in postprocessing
* Dont convert columns to int type
* Add unit tests
* Remove test
* Update unit test
* Update unit test
* Update unit test
* Update unit test
Approved-by: Alex Galarce
Field/dynamic codes
* remove test
* Merge branch 'main' into field/dynamic-codes
* Merged main into field/dynamic-codes
* Update test
* Fix unit test
* Merged main into field/dynamic-codes
* Merged main into field/dynamic-codes
* Merged main into field/dynamic-codes
* Update fields with crosswalks
* Update mapping to return empty string if no mapping
* Update dynamic_funcs
* Restructure
* Update file_processing
* Update dynamic primary
* Fix dynamic primary
* Update base fields
* Update one-to-n process
* Merged main into field/dynamic-codes
* genericized get_dynamic_answers
* Update tests
* Remove test file
* dynamic_funcs cleanup
* remove prints
* add exhibit_header
* Fix imports
Approved-by: Alex Galarce
Enhance universal_json_load function with detailed error handling and traceback logging
* Enhance universal_json_load function with detailed error handling and traceback logging
* Add type hint for string_dict parameter in universal_json_load function
* Update return type annotation for universal_json_load function to reflect possible return values
Approved-by: Katon Minhas
Feature/one to n testing
* Add field by field prec and rec
* Add field-wise prec and rec
* Merged main into feature/one-to-n-testing
* Merged main into feature/one-to-n-testing
* Merged main into feature/one-to-n-testing
Approved-by: Alex Galarce
Bugfix/flatten singleton list ints problem
* Fix flatten_singleton_string_list to handle non-list int strings and improve test coverage
* Fix test_flatten_singleton_string_list to handle integer input correctly
Approved-by: Katon Minhas
format_rate_fields_with_commas: change how empty values and errors are handled
* format_rate_fields_with_commas: change how empty values and errors are handled
Approved-by: Katon Minhas
Feature/align tin npi
* Add merge_provider_info function to consolidate provider data into one-to-one results
* Merge remote-tracking branch 'origin/main' into feature/align-tin-npi
* Remove debug print statement from run_one_to_one_prompts function
* turn run_regex_fields into an orchestrator function by breaking parts of it into separate functions
* Remove unused import of get_matches function and add docstring to chunk_on_matches for clarity
* Add docstring to run_regex_fields for improved clarity and documentation
* fix provider name keys in merge_provider_info
* Remove unused functions get_matches, tin_npi_prompt, and clean_tin_npi from tin_npi_funcs.py
* Set provider name to "UNKNOWN" if cleaned name is empty in clean_provider_info
* Add unit tests for clean_provider_info function to validate data processing
* Enhance docstring for get_all_matches function
* Add tests for get_all_matches and chunk_on_matches functions
* Merge remote-tracking branch 'origin/main' into feature/align-tin-npi
* Implement merge_provider_info function to consolidate provider data into one-to-one results
* Refactor merge_provider_info to simplify list conversion for group and other provider information
* Remove unnecessary TIN and NPI postprocessing step from postprocess function
* Remove clean_tin_npi_other function and its invocation from postprocess function
* Use json prompts for TIN/NPI/Name and inject them into TIN_NPI_TEMPLATE
* Rename run_regex_fields to run_provider_info_fields
* Enhance TIN_NPI_TEMPLATE to include detailed instructions for extracting provider entities and their identifying information, including IS_GROUP logic for main contracting parties.
* moved TIN/NPI regexes to regex_patterns.py
* Remove unnecessary blank line in investment_values.py
* Remove debug print statements from run_provider_info_fields and get_provider_info functions
* Enhance clean_provider_info to handle various representations of IS_GROUP as boolean
* Enhance get_provider_info to include error handling for JSON parsing and ensure consistent return format
Approved-by: Katon Minhas
Bugfix/daip2 156
* try blocks added
* fixed output 'of charges' in reimb_method
* Merged main into bugfix/daip2-156
* removed try block
* Merge branch 'bugfix/daip2-156' of https://bitbucket.org/aarete/doczy.ai into bugfix/daip2-156
* Merged main into bugfix/daip2-156
Approved-by: Alex Galarce
Hotfix/flatten list to string
* updated
* rate fields fix
* updated the singleton_list
* updated
* added
* Checked_files
* added
* Merged main into hotfix/flatten-list-to-string
* format_rate_fields_with_commas
* Update test
* flatten_singleton_string_list - new cases
* Update
* Removed unneccessary case
* Remove case
* Updated the test cases
* Merge remote-tracking branch 'origin/main' into hotfix/flatten-list-to-string
* Refactor flatten_singleton_string_list to handle empty input and return a comma-separated string for multiple elements
Approved-by: Katon Minhas
Enhance is_empty function to handle pandas Series and improve empty value checks
* Enhance is_empty function to handle pandas Series and improve empty value checks
* Enhance is_empty function to support pandas Series and add pd_mask parameter for flexible empty checks
Approved-by: Katon Minhas
Hotfix/reimb issue fixes
* updated DYNAMIC_CHECK prompt
* updated date_fix_prompt to YYYYMMDD format
* filled N/A values for REIMB_PROV_TIN
* changed YYYYMMDD to YYYY/MM/DD
* extracting TIN, NPI,NAME as a tuple
* updated TIN, NPI,NAME as a tuple prompt
* Merge remote-tracking branch 'origin/main' into hotfix/reimb_issue_fixes
* fix: correct spelling of 'memorize' to 'memoize' in cache decorator comment
* combined postprocess date format check functions
* Merged main into hotfix/reimb_issue_fixes
* updated postprocessing
* rename and slight refactor
* remove unused imports, re-sort
* remove unused imports from postprocess.py
* Fix unit tests
* re-add imports which are needed for tests to pass (?)
* Fix column name in auto-renewal termination date handling
* Fix formatting in postprocess function
Approved-by: Katon Minhas
Hotfix/methodology breakout issue fixes
* Merge branch 'hotfix/methodology_breakout_issue_fixes' of https://bitbucket.org/aarete/doczy.ai into hotfix/methodology_breakout_issue_fixes
merged with current local changes
* opps methodology added
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
merged changes from main
* Merged main into hotfix/methodology_breakout_issue_fixes
* duplicate N/A fixed
* Merge branch 'hotfix/methodology_breakout_issue_fixes' of https://bitbucket.org/aarete/doczy.ai into hotfix/methodology_breakout_issue_fixes
merged changes from main
* updated AARETE_DERIVED_REIMB_METHOD prompt
* fee schedule prompt change
* REIMB_PCT_RATE prompt updated
* Merged main into hotfix/methodology_breakout_issue_fixes
* opps moved to grouper
* fee rate format fixed
* removed incentives
* removed funding and additional reimbursement
* separate invalid reimb
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* merged main into current branch
* updated valid reimb and fee_schedule_version
* code structure fix
* code structure fix 2
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* updated methodology breakout fields
* test notebook updated
* Merged main into hotfix/methodology_breakout_issue_fixes
Approved-by: Katon Minhas
Feature/new stop loss and outlier fields
* initial commit; handle term_key cases, add flexibility for service and methodology, set prompt to `null` for term_key filled fields
* prompt changes for all outlier fields
* Merge remote-tracking branch 'origin/main' into feature/new-stop-loss-and-outlier-fields
* reorder columns, rework CD_DESC for outlier, replace redundant column OUTLIER_FIXED_LOSS_THRESHOLD (deprecated by OUTLIER_FIXED_LOSS_CHARGE_THRESHOLD)
* add OUTLIER_MAXIMUM_FREQUENCY prompt for outlier claims cap frequency
* add OUTLIER_RANGE_NBR_DAYS prompt to specify period for calculating outlier payments
* fix term key naming to align with Maria columns
* remove debug print statements
* add Maria stop loss columns
* stop loss prompts
Approved-by: Katon Minhas
Feature/dynamic one to one
* Pass N/A to one-to-one
* E2E test passed
* Merged main into feature/dynamic-one-to-one
* Pass ANY non N/A answer to reimbursement-level
* Update test
* Remove test
* Merge branch 'main' into feature/dynamic-one-to-one
* Wrap clean_dates
* Revert postprocess
* Merged main into feature/dynamic-one-to-one
* Merged main into feature/dynamic-one-to-one
* Remove prints
Approved-by: Alex Galarce
Feature/na blank handling
* Update prompts to clarify responses for unknown vs N/A values
* Update code description mapping to differentiate between empty and unknown grouper codes
* Update prompts and code descriptions to standardize 'unknown' to 'UNKNOWN'
* Update PIPE_FORMAT_INSTRUCTIONS to replace 'concept' with 'requested information'
* Merge remote-tracking branch 'origin/main' into feature/na_blank_handling
* Merged main into feature/na_blank_handling
* Update NPI prompt for clarity and detailed instructions
* Enhance TIN prompt for clarity and detailed identification instructions
* Merged main into feature/na_blank_handling
* Merged main into feature/na_blank_handling
Approved-by: Katon Minhas
add new configurations for carveouts
* add new configurations for carveouts
* formatting, isort, and remove a couple unused imports
Approved-by: Katon Minhas
Rework reimbursement ID to associate parts of "lesser of" (or other such coupled statement) with one another
* refactor reimbursement ID generation with hashed service terms
* reduce hash length for unique reimbursement ID generation
* update reimbursement ID format to include 'exh_pg' for clarity
* clean up testing.ipynb
* one more formatting fix to testing.ipynb
* Move reimb_id generation to postprocessing
* move reimb_id function to postprocessing_funcs.py
* refactor generate_reimb_ids to group by FILE_NAME and EXHIBIT_PAGE for ID generation
* clean testing.ipynb further
Approved-by: Katon Minhas
Feature/multiple procedures
* Add stop loss and multiple procedure reduction questions to investment prompts
* Refactor field_type values in investment prompts for consistency
* Refactor methodology breakout function to accept prompt templates as parameters
* Update investment prompts to enhance clarity and add new (provisional) fields for NEVER events and experimental treatments
* Merge remote-tracking branch 'origin/main' into feature/multiple-procedures
* start consolidation of special_cases and carveouts into one omnifunction
* Enhance carveout handling by adding support for additional question types and refactoring question retrieval
* move term_key to individual cases
* Fix casing for carveout term keys in get_carveout_fields function
* Refactor carveout processing by introducing process_carveout_or_special_case function to streamline handling of various carveout types
* Enhance process_carveout_or_special_case function documentation and clarify parameter descriptions
* Add term_key parameter to get_carveout_fields function call
* Enhance type annotations for methodology and fee schedule prompt templates in process_carveout_or_special_case function
* Add SPECIAL_CASE_CONFIGS to streamline carveout processing logic
* Remove deprecated print statements and special case handling from get_special_cases function
* name carveout_config for clarity against config.py
* Remove special case functions (deprecated)
* Remove unused imports and deprecated print statements from get_special_cases function
* remove print statements for debugging, check output of universal_string_load if it's a single dict, put it to a list
* Add special case configurations for various question types in its own module
* Refactor special case handling by consolidating configurations and removing deprecated constants and comments
* Update descriptions for OUTLIER and STOP_LOSS in VALID_SPECIAL for clarity and detail
* black formatting, changed one docstring description
* isort
* Merged main into feature/multiple-procedures
Approved-by: Katon Minhas
improve contract_amendment_num prompt
* improve contract_amendment_num prompt
* Merged main into feature/numeric_amd_num
* change CONTRACT_AMENDMENT_NUM prompt to include alphanumeric, Roman numeral, and decimal formats
* add period
Approved-by: Katon Minhas
call methodology breakout on carveouts
* Refactor carveout functionality: move get_carveout_fields to one_to_n_funcs. Move out QUESTIONS as a constant at the top
* Merge remote-tracking branch 'origin/main' into feature/methodology_breakout_for_carveouts
* Enhance methodology breakout handling: flatten nested lists and improve processing logic
* Merged main into feature/methodology_breakout_for_carveouts
* Improve error handling in file processing: add traceback logging for better debugging
* integrate methodology breakout for carveouts
* ensure default indicator is set after processing
* update default indicator assignment and extend special case answers list to avoid list-list-dict structures
* add defensive programming comment for nested lists in methodology breakout function
Approved-by: Katon Minhas
Refactor/field name updates
* New names/orders
* Add MAXIMUM_FREQUENCY Placeholder
* Merged main into refactor/field-name-updates
Approved-by: Alex Galarce
fix carveout_ind for special cases
* Refactor get_special_cases to improve non-special cases
* Merged main into bugfix/lesser-of
* Refactor get_special_cases to handle N/As second after default_term
* Add methodology breakout function for single row processing
* Merge remote-tracking branch 'origin/main' into bugfix/lesser-of
* Merge branch 'bugfix/lesser-of' of https://bitbucket.org/aarete/doczy.ai into bugfix/lesser-of
* Refactor methodology_breakout_single_row to return a list of results and handle empty responses
* Merge remote-tracking branch 'origin/main' into bugfix/lesser-of
* Rename ADRM and update CARVEOUT_IND logic
* Remove unused imports from carveout_funcs.py
* update docstrings
* Reorder conditions in get_special_cases for clarity and maintainability
Approved-by: Katon Minhas
DRAFT: Feature/split out error log
* Refactor result handling to separate successful and error results in processing
* Add forced exception for testing in safe_process_file function
* Merged main into feature/split-out-error-log
* main.py edited online with Bitbucket
* Add support for error output type in write_s3 function
Approved-by: Katon Minhas
Hotfix/file errors
* Set up test file
* Fix regex pattern in json.loads
* bug fix No valid JSON object found in the input string
* bug fixes
* moved try statement to parse fn
* invalid literal for int() with base 10 error fixed
* remove test
* remove excess print statements
* Raise error
* Remove todo
* Remove print statement
Approved-by: Katon Minhas
Bugfix/table failed files
* improved align and format table prompt
* Successful E2E test on San Joaquin - without running align and format table prompt
* San Joaquin e2e test passed
* Merged main into bugfix/table-error-fix
* Merge branch 'main' into bugfix/table-error-fix
* Add same number of columns condition
* Merged main into bugfix/table-error-fix
* Clean investment_values
* Remove import
* docstrings
* Remove test doc
* Add back original ALIGN_AND_FORMAT_TABLES prompt for client work
* Remove import
* Add config reference
* expected string or bytes-like object, got 'int'
* Merged main into bugfix/table-error-fix
* Merge branch 'bugfix/table-error-fix' into bugfix/table-failed-files
* Update universal json load
* update test
* Merge branch 'main' into bugfix/table-failed-files
* Position 0 error fixed
* update table_utils
* remove test.py
* revert table_utils test
* remove table_utils_test
Approved-by: Alex Galarce
Bugfix/table error fix
* handle multipage table data correctly
* same metadata for all tables on the page
* Multiple tables on page, non-continuous, split
* Add back align_and_format_tables
* update unit tests
* Merged main into bugfix/table-error-fix
* improved align and format table prompt
* Successful E2E test on San Joaquin - without running align and format table prompt
* San Joaquin e2e test passed
* Merged main into bugfix/table-error-fix
* Merge branch 'main' into bugfix/table-error-fix
* Add same number of columns condition
* Merged main into bugfix/table-error-fix
* Clean investment_values
* Remove import
* docstrings
* Remove test doc
* Add back original ALIGN_AND_FORMAT_TABLES prompt for client work
* Remove import
* Add config reference
* Merged main into bugfix/table-error-fix
* Move table_utils to investment.table_funcs
* restore table_utils.py
* Update tests
* update preprocess
Approved-by: Alex Galarce
Generate unique IDs for reimbursements
* Add reimbursement ID generation and update function signatures for unique identification
* Merge remote-tracking branch 'origin/main' into feature/unique-identifier-for-reimbursements
Approved-by: Katon Minhas
New Reimbursement Primary Prompt
* Enhance reimbursement primary extraction prompts with detailed formatting and instruction guidelines
* Clarify 'lesser of' statement application in reimbursement methods with detailed examples and guidelines
* Add examples to 'lesser of' statement application in investment prompts
* up row_limit to 20
* hybrid prompt for global table info for reimb_primary
* Merged main into feature/new-reimb-primary-prompt
* most explicit prompt
* update default row_limit in clean_tables function to 6
* Merge remote-tracking branch 'origin/main' into feature/new-reimb-primary-prompt
* Add detailed guidelines for extracting complex payment structures in anesthesia services
* Clarify handling of specific plan names in reimbursement prompts
* removed a couple examples to save tokens
Approved-by: Katon Minhas
Refactor fuzzy matching logic to use partial token sort ratio and update matched pairs structure
* Refactor fuzzy matching logic to use partial token sort ratio and update matched pairs structure
* Merged main into feature/change-1-to-n-comparison-metric
Approved-by: Katon Minhas
Small fix to comparisons (lowercase comparison string fields)
* Add accuracy calculation to fuzzy matching evaluation
* reimbursement primary accuracy test
* add match_threshold control
* clarify metrics
* Merged main into reimb-primary-accuracy-analysis
* Merge branch 'reimb-primary-accuracy-analysis' of https://bitbucket.org/aarete/doczy.ai into reimb-primary-accuracy-analysis
* add 'billed charges' to reimbursement keywords and update TODO for page checking
* add comparisons dictionary
* normalize field comparison by converting strings to lowercase
* deduplicate data sources in notebook
* Merge remote-tracking branch 'origin/main' into feature/add-billed-charges-to-reimb-keywords
Approved-by: Katon Minhas
Mayank Aamseek