Hotfix/methodology breakout issue fixes
* Merge branch 'hotfix/methodology_breakout_issue_fixes' of https://bitbucket.org/aarete/doczy.ai into hotfix/methodology_breakout_issue_fixes
merged with current local changes
* opps methodology added
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
merged changes from main
* Merged main into hotfix/methodology_breakout_issue_fixes
* duplicate N/A fixed
* Merge branch 'hotfix/methodology_breakout_issue_fixes' of https://bitbucket.org/aarete/doczy.ai into hotfix/methodology_breakout_issue_fixes
merged changes from main
* updated AARETE_DERIVED_REIMB_METHOD prompt
* fee schedule prompt change
* REIMB_PCT_RATE prompt updated
* Merged main into hotfix/methodology_breakout_issue_fixes
* opps moved to grouper
* fee rate format fixed
* removed incentives
* removed funding and additional reimbursement
* separate invalid reimb
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* merged main into current branch
* updated valid reimb and fee_schedule_version
* code structure fix
* code structure fix 2
* Merge branch 'main' into hotfix/methodology_breakout_issue_fixes
* updated methodology breakout fields
* test notebook updated
* Merged main into hotfix/methodology_breakout_issue_fixes
Approved-by: Katon Minhas
Feature/new stop loss and outlier fields
* initial commit; handle term_key cases, add flexibility for service and methodology, set prompt to `null` for term_key filled fields
* prompt changes for all outlier fields
* Merge remote-tracking branch 'origin/main' into feature/new-stop-loss-and-outlier-fields
* reorder columns, rework CD_DESC for outlier, replace redundant column OUTLIER_FIXED_LOSS_THRESHOLD (deprecated by OUTLIER_FIXED_LOSS_CHARGE_THRESHOLD)
* add OUTLIER_MAXIMUM_FREQUENCY prompt for outlier claims cap frequency
* add OUTLIER_RANGE_NBR_DAYS prompt to specify period for calculating outlier payments
* fix term key naming to align with Maria columns
* remove debug print statements
* add Maria stop loss columns
* stop loss prompts
Approved-by: Katon Minhas
Feature/dynamic one to one
* Pass N/A to one-to-one
* E2E test passed
* Merged main into feature/dynamic-one-to-one
* Pass ANY non N/A answer to reimbursement-level
* Update test
* Remove test
* Merge branch 'main' into feature/dynamic-one-to-one
* Wrap clean_dates
* Revert postprocess
* Merged main into feature/dynamic-one-to-one
* Merged main into feature/dynamic-one-to-one
* Remove prints
Approved-by: Alex Galarce
Feature/na blank handling
* Update prompts to clarify responses for unknown vs N/A values
* Update code description mapping to differentiate between empty and unknown grouper codes
* Update prompts and code descriptions to standardize 'unknown' to 'UNKNOWN'
* Update PIPE_FORMAT_INSTRUCTIONS to replace 'concept' with 'requested information'
* Merge remote-tracking branch 'origin/main' into feature/na_blank_handling
* Merged main into feature/na_blank_handling
* Update NPI prompt for clarity and detailed instructions
* Enhance TIN prompt for clarity and detailed identification instructions
* Merged main into feature/na_blank_handling
* Merged main into feature/na_blank_handling
Approved-by: Katon Minhas
add new configurations for carveouts
* add new configurations for carveouts
* formatting, isort, and remove a couple unused imports
Approved-by: Katon Minhas
Rework reimbursement ID to associate parts of "lesser of" (or other such coupled statement) with one another
* refactor reimbursement ID generation with hashed service terms
* reduce hash length for unique reimbursement ID generation
* update reimbursement ID format to include 'exh_pg' for clarity
* clean up testing.ipynb
* one more formatting fix to testing.ipynb
* Move reimb_id generation to postprocessing
* move reimb_id function to postprocessing_funcs.py
* refactor generate_reimb_ids to group by FILE_NAME and EXHIBIT_PAGE for ID generation
* clean testing.ipynb further
Approved-by: Katon Minhas
Feature/multiple procedures
* Add stop loss and multiple procedure reduction questions to investment prompts
* Refactor field_type values in investment prompts for consistency
* Refactor methodology breakout function to accept prompt templates as parameters
* Update investment prompts to enhance clarity and add new (provisional) fields for NEVER events and experimental treatments
* Merge remote-tracking branch 'origin/main' into feature/multiple-procedures
* start consolidation of special_cases and carveouts into one omnifunction
* Enhance carveout handling by adding support for additional question types and refactoring question retrieval
* move term_key to individual cases
* Fix casing for carveout term keys in get_carveout_fields function
* Refactor carveout processing by introducing process_carveout_or_special_case function to streamline handling of various carveout types
* Enhance process_carveout_or_special_case function documentation and clarify parameter descriptions
* Add term_key parameter to get_carveout_fields function call
* Enhance type annotations for methodology and fee schedule prompt templates in process_carveout_or_special_case function
* Add SPECIAL_CASE_CONFIGS to streamline carveout processing logic
* Remove deprecated print statements and special case handling from get_special_cases function
* name carveout_config for clarity against config.py
* Remove special case functions (deprecated)
* Remove unused imports and deprecated print statements from get_special_cases function
* remove print statements for debugging, check output of universal_string_load if it's a single dict, put it to a list
* Add special case configurations for various question types in its own module
* Refactor special case handling by consolidating configurations and removing deprecated constants and comments
* Update descriptions for OUTLIER and STOP_LOSS in VALID_SPECIAL for clarity and detail
* black formatting, changed one docstring description
* isort
* Merged main into feature/multiple-procedures
Approved-by: Katon Minhas
improve contract_amendment_num prompt
* improve contract_amendment_num prompt
* Merged main into feature/numeric_amd_num
* change CONTRACT_AMENDMENT_NUM prompt to include alphanumeric, Roman numeral, and decimal formats
* add period
Approved-by: Katon Minhas
call methodology breakout on carveouts
* Refactor carveout functionality: move get_carveout_fields to one_to_n_funcs. Move out QUESTIONS as a constant at the top
* Merge remote-tracking branch 'origin/main' into feature/methodology_breakout_for_carveouts
* Enhance methodology breakout handling: flatten nested lists and improve processing logic
* Merged main into feature/methodology_breakout_for_carveouts
* Improve error handling in file processing: add traceback logging for better debugging
* integrate methodology breakout for carveouts
* ensure default indicator is set after processing
* update default indicator assignment and extend special case answers list to avoid list-list-dict structures
* add defensive programming comment for nested lists in methodology breakout function
Approved-by: Katon Minhas
Refactor/field name updates
* New names/orders
* Add MAXIMUM_FREQUENCY Placeholder
* Merged main into refactor/field-name-updates
Approved-by: Alex Galarce
fix carveout_ind for special cases
* Refactor get_special_cases to improve non-special cases
* Merged main into bugfix/lesser-of
* Refactor get_special_cases to handle N/As second after default_term
* Add methodology breakout function for single row processing
* Merge remote-tracking branch 'origin/main' into bugfix/lesser-of
* Merge branch 'bugfix/lesser-of' of https://bitbucket.org/aarete/doczy.ai into bugfix/lesser-of
* Refactor methodology_breakout_single_row to return a list of results and handle empty responses
* Merge remote-tracking branch 'origin/main' into bugfix/lesser-of
* Rename ADRM and update CARVEOUT_IND logic
* Remove unused imports from carveout_funcs.py
* update docstrings
* Reorder conditions in get_special_cases for clarity and maintainability
Approved-by: Katon Minhas
DRAFT: Feature/split out error log
* Refactor result handling to separate successful and error results in processing
* Add forced exception for testing in safe_process_file function
* Merged main into feature/split-out-error-log
* main.py edited online with Bitbucket
* Add support for error output type in write_s3 function
Approved-by: Katon Minhas
Hotfix/file errors
* Set up test file
* Fix regex pattern in json.loads
* bug fix No valid JSON object found in the input string
* bug fixes
* moved try statement to parse fn
* invalid literal for int() with base 10 error fixed
* remove test
* remove excess print statements
* Raise error
* Remove todo
* Remove print statement
Approved-by: Katon Minhas
Bugfix/table failed files
* improved align and format table prompt
* Successful E2E test on San Joaquin - without running align and format table prompt
* San Joaquin e2e test passed
* Merged main into bugfix/table-error-fix
* Merge branch 'main' into bugfix/table-error-fix
* Add same number of columns condition
* Merged main into bugfix/table-error-fix
* Clean investment_values
* Remove import
* docstrings
* Remove test doc
* Add back original ALIGN_AND_FORMAT_TABLES prompt for client work
* Remove import
* Add config reference
* expected string or bytes-like object, got 'int'
* Merged main into bugfix/table-error-fix
* Merge branch 'bugfix/table-error-fix' into bugfix/table-failed-files
* Update universal json load
* update test
* Merge branch 'main' into bugfix/table-failed-files
* Position 0 error fixed
* update table_utils
* remove test.py
* revert table_utils test
* remove table_utils_test
Approved-by: Alex Galarce
Bugfix/table error fix
* handle multipage table data correctly
* same metadata for all tables on the page
* Multiple tables on page, non-continuous, split
* Add back align_and_format_tables
* update unit tests
* Merged main into bugfix/table-error-fix
* improved align and format table prompt
* Successful E2E test on San Joaquin - without running align and format table prompt
* San Joaquin e2e test passed
* Merged main into bugfix/table-error-fix
* Merge branch 'main' into bugfix/table-error-fix
* Add same number of columns condition
* Merged main into bugfix/table-error-fix
* Clean investment_values
* Remove import
* docstrings
* Remove test doc
* Add back original ALIGN_AND_FORMAT_TABLES prompt for client work
* Remove import
* Add config reference
* Merged main into bugfix/table-error-fix
* Move table_utils to investment.table_funcs
* restore table_utils.py
* Update tests
* update preprocess
Approved-by: Alex Galarce
Generate unique IDs for reimbursements
* Add reimbursement ID generation and update function signatures for unique identification
* Merge remote-tracking branch 'origin/main' into feature/unique-identifier-for-reimbursements
Approved-by: Katon Minhas
New Reimbursement Primary Prompt
* Enhance reimbursement primary extraction prompts with detailed formatting and instruction guidelines
* Clarify 'lesser of' statement application in reimbursement methods with detailed examples and guidelines
* Add examples to 'lesser of' statement application in investment prompts
* up row_limit to 20
* hybrid prompt for global table info for reimb_primary
* Merged main into feature/new-reimb-primary-prompt
* most explicit prompt
* update default row_limit in clean_tables function to 6
* Merge remote-tracking branch 'origin/main' into feature/new-reimb-primary-prompt
* Add detailed guidelines for extracting complex payment structures in anesthesia services
* Clarify handling of specific plan names in reimbursement prompts
* removed a couple examples to save tokens
Approved-by: Katon Minhas
Refactor fuzzy matching logic to use partial token sort ratio and update matched pairs structure
* Refactor fuzzy matching logic to use partial token sort ratio and update matched pairs structure
* Merged main into feature/change-1-to-n-comparison-metric
Approved-by: Katon Minhas
Small fix to comparisons (lowercase comparison string fields)
* Add accuracy calculation to fuzzy matching evaluation
* reimbursement primary accuracy test
* add match_threshold control
* clarify metrics
* Merged main into reimb-primary-accuracy-analysis
* Merge branch 'reimb-primary-accuracy-analysis' of https://bitbucket.org/aarete/doczy.ai into reimb-primary-accuracy-analysis
* add 'billed charges' to reimbursement keywords and update TODO for page checking
* add comparisons dictionary
* normalize field comparison by converting strings to lowercase
* deduplicate data sources in notebook
* Merge remote-tracking branch 'origin/main' into feature/add-billed-charges-to-reimb-keywords
Approved-by: Katon Minhas
Feature/daip2 125
* grouper prompt changed
* grouper code, type and desc updated
* removed test file
* Merged main into feature/daip2-125
* Merge remote-tracking branch 'origin/main' into feature/daip2-125
Approved-by: Alex Galarce
very small error logging fix
* Handle ValueError in code range parsing and provide fallback
* Implement safe file processing with error handling and concurrent execution
* Merge remote-tracking branch 'origin/main' into bugfix/improve-stability
* more explicit error printing
* Merge remote-tracking branch 'origin/main' into bugfix/improve-stability
Approved-by: Katon Minhas
updated prompt to get the group provider
* updated prompt to get the group provider
* Merged main into bugfix/individual-prov-group-name
Approved-by: Alex Galarce
Bugfix/improve stability
* Handle ValueError in code range parsing and provide fallback
* Implement safe file processing with error handling and concurrent execution
* Merge remote-tracking branch 'origin/main' into bugfix/improve-stability
Approved-by: Katon Minhas
Feature/field renaming
* Update column names
* Update field names
* Update field names
* Merge branch 'main' into feature/field-renaming
* _DATE to _DT in field names
* _CD_DESC to _DESC
* NOT_TO_EXCEED_IND added as valid field
* Remove print statements
Approved-by: Alex Galarce
Fix load_embeddings()
* move datetime_str function to string_utils module
* enhance load_embeddings to create directory structure and handle download errors
* Merged main into feature/reimbursement-primary-optimizization
Approved-by: Katon Minhas
Feature/test 1 to n fields
* Add function to evaluate one-to-n field comparisons and calculate metrics
* Add fuzzy matching for one-to-n field comparisons and enhance evaluation metrics
* Refactor similarity calculation to use token sort ratio for improved field comparison
* Add matched labels to evaluation metrics in one-to-n fuzzy matching
Approved-by: Katon Minhas
Feature/override aa derived eff date
* Implement special handling for AARETE_DERIVED_TERMINATION_DATE based on CONTRACT_AUTO_RENEWAL_IND
* Refine prompts for CONTRACT_AUTO_RENEWAL_IND and CONTRACT_AUTO_RENEWAL_TERM for clarity
* move special handling for derived effective date to postprocess step
* Remove unnecessary blank line in get_crosswalk_and_derived function
Approved-by: Katon Minhas
Feature/standardize column order
* add column reordering
* Standardize column order
* Merged main into feature/standardize-column-order
Approved-by: Alex Galarce
Change from AARETE_DERIVED_ to CONTRACT_- FEE_SCHEDULE_VERSION
* Change from AARETE_DERIVED_ to CONTRACT_- FEE_SCHEDULE_VERSION
Approved-by: Alex Galarce
Mayank Aamseek