Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code
* Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code
- Add format requirements to LOB_RELATIONSHIP_INSTRUCTION for proper caching
- Strengthen LOB_RELATIONSHIP prompt with explicit pipe delimiter requirements
- Preserve all DEBUG blocks and print statements in dynamic_funcs.py and file_processing.py
- Update crosswalk mappings and gitignore
- Ensure format instructions are cached at API level to prevent missing pipe delimiters
* Merge main into bugfix/dynamic_issuefixes-clean and remove all debug print statements
* Remove redundant one_to_one_fields initialization to match main
Approved-by: Katon Minhas
Issuefix/hmk 1 to n
* Merge branch 'main' into feature/1toN-Optimization
* Merge branch 'main' into feature/1toN-Optimization
* Update preprocessing to make the exhibit_chunk_mapping start at first page
* Merge row_funcs.py changes from feature/cross-exhibit-dynamic
* Merged in bugfix/UT-grouper-issues (pull request #796)
Bugfix/UT grouper issues
* updated valid values for AARETE_DERIVED_REIMB_METHOD
* removed example reimbursements
* prompt update
* prompt update
* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes
* add service term in mb prompts
* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout
* print statement removed
* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout
* primary prompt update
* prompt update
* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into bugfix/UT-grouper-issues
* removed temp changes
* removed temp changes
* Update reimb primary
Approved-by: Katon Minhas
* Merged in bugfix/validation_fixes (pull request #795)
bugfix/validation_fixes to feature/1toN-Optimization
* updated validation of clean claims reimbursement
* Merged feature/1toN-Optimization into bugfix/validation_fixes
Approved-by: Katon Minhas
* Re-add dynamic codes and reimb-info
* Re-structure empty reimbursement prompt
* Fix lesser of check overfiltering
* Merged in bugfix/generic-methodology-breakout (pull request #797)
Bugfix/generic methodology breakout
* fee schedule fix
* carveout+special case prompt changed
* fee schedule changes
* prompt changes and column order
Approved-by: Katon Minhas
* pytest fix
* remove prints
* Update valid_carveouts.json to include level designations in PAYMENT_CARVEOUT definition and add adultBasic mapping for Highmark
* Fix REIMB_DATES extraction to capture only relevant date range per term
- Updated REIMB_DATES prompt to extract only the date range that applies to the specific reimbursement term
- Added proximity-based extraction: looks for dates in same section/table as the term
- Prevents cross-contamination from other contract sections
- Handles 'Same as Effective Date of Agreement' case
- Returns single date range instead of multiple comma-separated ranges
* Update prompt templates
* Update tracked files and add untracked_tests to gitignore
* Merge branch 'main' into issuefix/hmk-1-to-n
* complete merge
* complete merge
* Merge branch 'main' into issuefix/hmk-1-to-n
* Remove prints
* Update gitignore
* Revert BASE_COVERED_SERVICES and REIMB_DATES prompts to original definitions
- Revert BASE_COVERED_SERVICES definition to original concise version
- Revert REIMB_DATES prompt to original general description (specific assignment logic handled by dedicated template)
- Add DEBUG comment blocks around print statements in dynamic_funcs.py and file_processing.py for easy identification and toggling
* remove prints
* Merge branch 'main' into issuefix/hmk-1-to-n
Approved-by: Katon Minhas
Issuefix/hmk 1 to 1
* prompt fix for auto renew ind
* filename tin regex update
* amendment number prompt fix
* effective date prompt tunning
* Merge remote-tracking branch 'origin/main' into issuefix/hmk-1-to-1
Approved-by: Katon Minhas
Feature/pc
* file read update
* removed unused code
* replace pring with logging.info
* input args for docczy output
* input args for docczy output
* added json for loading mappings
* exception for bool keys
* exception for bool keys
* added init.py
* write to s3/local functionality
* write to s3/local functionality
* logging errors and main functionality fix
* pc functionality from main
* pc functionality from main
* function signature fix
* fixed path returns
* Updated names for varaibles; Modified termination date cleaning function
* Removed commented out function
* Renamed client column names to SAAS column names
* Merge remote-tracking branch 'origin/main' into feature/pc
* removed pc utils
* black formatting
* rename function
* rename func
* additional file removed
Approved-by: Katon Minhas
Bugfix/reimb excess language issues to main
* updated trigger cap def
* updated trigger cap carveout def
* Merged main into bugfix/reimb_excess_language_issues
* updated split reimb prompt
* remove print statements
* Merged main into bugfix/reimb_excess_language_issues
* updated split reimb prompt with complex example
Approved-by: Katon Minhas
Feature/effective dt
* move prompt_smart_chunked to prompt_calls.py
* refactor pass_smart_chunked_to_full_context
* Mypy fix
* Update field to EFFECTIVE_DT
* move to prompt_calls
* Update vision funcs
* Restructure prompt-based date derivation
* E2E passed
* Remove test
* Merge branch 'main' into feature/effective-dt
* remove deprecated postprocessing_func
* Docstrings
* refactor - send global lesser of to prompt_calls.py
* remove deprecated function
* more docstrings
* Merged main into feature/effective-dt
* Merged main into feature/effective-dt
* added base field for aarete derived effective date
Approved-by: VenkataKrishna Reddy Avula
bugfix/prov_tin_npi_name_issues to main
* updated provider group full name
* removed print statements
* Merged main into bugfix/prov_tin_npi_name_issues
Approved-by: Katon Minhas
Bugfix/process only after title
* Update print
* Update exhibit dict creation
* Detect Participating Provider Agreement
* Remove test file
* Merged main into bugfix/process-only-after-title
* Merged main into bugfix/process-only-after-title
Approved-by: Faizan Mohiuddin
Bugfix/ut issues
* Update exhibit linking
* prompt_calls refactor
* remove test
* Add Healthy Kids Program
* Move Healthy Kids to UT specifically
* Remove header check from Dynamic process
* Merged main into bugfix/ut-issues
* prep for merge
* remove deprecated tests
* Merged main into bugfix/ut-issues
* Merged main into bugfix/ut-issues
Approved-by: Faizan Mohiuddin
Bugfix/special cases to Main
* prompt update
* Merge branch 'main' into bugfix/json_and_unit_of_measure
* minor bugfix
* minor bugfix
* prompt update for outlier rate
* added outlier testing jupyter notebook
* Remove special case from any empty
* updated outlier breakout prompt
* Merged main into bugfix/special-cases
* reimb_method minor fix
* Merged main into bugfix/special-cases
* bugfix field to add dynamic funcs
* merged with main
* latest outlier changes
* Merged main into bugfix/special-cases
* added table guidelines for special case primary
* updated special case primary prompt
* Merged main into bugfix/special-cases
* updated special case primary table handling
* merged main into bugfix special cases
* Merged main into bugfix/special-cases
* Remove test notebooks from tracking
* trying out pipeline fix
* reverted chnages
* pipeline fix
Approved-by: Katon Minhas
Bugfix/mi dynamic
* Refactor - split out one_to_n_cleaning
* Rework postprocess
* Update postprocessing - fill dynamic values if there is only 1 non-empty value
* Update to use string_utils
* Group by AARETE_DERIVED_LOB as well
* Update crosswalk mapping to handle | lists
* Update constants - run with MI flag
Approved-by: Siddhant Medar
Feature/black and isort
* black and isort constants
* black and isort codes
* black and isort crosswalks
* black and isort investment
* black and isort prompts
* isort testbed
* isort tracking
* black and isort utils
* poetry and black tests
Approved-by: Katon Minhas
Bugfix/provider info to refactor/one-to-n
* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test
* Update Carveout IND and Default IND logic
* Split out exhibit-level lesser
* Bugfix
* update lesser of exhibit prompt
* Bugfix
* exhibit lesser of varible name bugfix
* Update exhibit lesser of passing
* Update Exhibit-level prompt template
* Shorten full_context prompt length if the questions are long
* Special case assignment index
* Resolve unit tests
* updated UC IND prompt
* Remove print
* update one-to-n unit test
* added dot based date example
* updated date fix prompt
* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)
Bugfix/json error and unit of measure
* json parser fixes and prompt changes
* universal json load fixed
* json error fixed
* json error fixed 2
* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure
Approved-by: Katon Minhas
* added new tin format
* updated signature page prompt
* updated tin search and tin duplication
* Merge remote-tracking branch 'origin/main' into bugfix/provider_info
* extended duplicate logic to npi and names
* Merged main into bugfix/provider_info
Approved-by: Katon Minhas
Refactor/one to n
* remove print
* Update logging for fsbreakout and grouper breakout
* Additional type hints
* updated special case primary
* Remove deprecated special case check
* OUTLIER_TERMS-->OUTLIER_TERM standardization
* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test
* Update Carveout IND and Default IND logic
* Split out exhibit-level lesser
* Bugfix
* update lesser of exhibit prompt
* Bugfix
* exhibit lesser of varible name bugfix
* Update exhibit lesser of passing
* Update Exhibit-level prompt template
* Shorten full_context prompt length if the questions are long
* Special case assignment index
* Resolve unit tests
* updated UC IND prompt
* Remove print
* update one-to-n unit test
* added dot based date example
* updated date fix prompt
* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)
Bugfix/json error and unit of measure
* json parser fixes and prompt changes
* universal json load fixed
* json error fixed
* json error fixed 2
* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure
Approved-by: Katon Minhas
Approved-by: Alex Galarce
Add time-based rate handling rules and update section labels in prompt templates
* Add time-based rate handling rules and update section labels in prompt templates
* Add mapping for MyCare Ohio in crosswalk_program.json and update state mapping for Ohio in crosswalk_program_lob.json
* Update test parameters and state configuration in testing notebook
* Update exhibit-wide payment constraint prompt to exclude administrative processing language
* Merge remote-tracking branch 'origin/main' into bugfix/reimb-term-issues
* Refine exhibit-wide payment constraint prompt to clarify extraction criteria and examples
* Enhance exhibit-wide payment constraint prompt with guidance for abbreviated terms and clarify acceptable constraints
* Update SERVICE_TERM prompt to specify healthcare services and clarify extraction criteria
Approved-by: Katon Minhas
Bugfix/trigger cap fixes
* stoploss prompt added again
* stoploss prompt added again
* stoploss prompt updated
* reverted back to one stage for trigger cap cases
* Merge branch 'main' into bugfix/trigger_cap_fixes
* pipeline error fixed
* incorrect discount special case fixed
Approved-by: Alex Galarce
Feature/tin enhancements
* Add OCR-correctable TIN patterns and substitution mapping
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Refactor OCR-correctable TIN pattern and enhance documentation for clarity
* Enhance get_all_matches function with type hints and improve documentation; add get_all_matches_with_ocr for OCR-correctable matches
* Add attempt_ocr_correction function for TIN OCR error handling and validation
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Integrate OCR correction for TIN extraction in run_provider_info_fields; log corrections for monitoring
* Refactor OCR-correctable patterns in regex_patterns.py; add NPI_OCR_PATTERN and enhance documentation for clarity
* Enhance OCR correction in attempt_ocr_correction to support both TIN and NPI; update validation logic for identifier lengths
* Integrate NPI OCR correction in run_provider_info_fields; log corrections for monitoring
* Enhance logging in run_provider_info_fields to summarize TIN and NPI extraction quality; include counts of exact and OCR-corrected identifiers
* Add unit and integration tests for OCR correction
* isort
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Merged main into feature/tin-enhancements
Approved-by: Katon Minhas
Feature/payer state
* fix indentation for PROVIDER_STATE prompt
* Add PAYER_STATE prompt and normalize state abbreviation in run_full_context_fields
* Add PAYER_STATE to COLUMN_ORDER in investment_columns.py
* Merged main into feature/payer-state
* Only run PAYER_STATE prompt when more than one state in the batch
Feature/grouper cd desc
* added grouper cd desc
* bugfix
* Merge branch 'main' into feature/grouper_cd_desc
* Merged main into feature/grouper_cd_desc
Approved-by: Alex Galarce
Feature/grouper fields
* prompt changes
* print statements removed
* merged with main
* merge issue fixed
* updated grouper prompts
* updated grouper fields in fields order
* updated breakout prompt
* Merged main into feature/grouper-fields
* reverted grouper cd prompt
* updated goruper severity prompt
* Merge remote-tracking branch 'origin/main' into feature/grouper-fields
* updated grouper breakout function
* error fixes
* Merged main into feature/grouper-fields
* Merged main into feature/grouper-fields
* updated grouper_pct_rate and grouper_base_rate
* removed empty array post processing fix
* Merged main into feature/grouper-fields
* post-processing for grouper rate modified
* updated grouper type prompt
* Merged main into feature/grouper-fields
* grouper version prompt updated
* Merged main into feature/grouper-fields
* OPPS as APC
* Merged main into feature/grouper-fields
Approved-by: Katon Minhas
bugfix/outlier_fixes to main
* simplified outlier prompts
* Merged main into bugfix/outlier_fixes
* Merged main into bugfix/outlier_fixes
Approved-by: Alex Galarce
Approved-by: Katon Minhas
Refine descriptions for BASE_COVERED_SERVICES and PAYMENT_CARVEOUT in valid_carveouts.json
* Refine descriptions for BASE_COVERED_SERVICES and PAYMENT_CARVEOUT in valid_carveouts.json
* Merge remote-tracking branch 'origin/main' into bugfix/carveout-cd-issues
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* format and restructure
* checkpoint
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Update constants attributes to base class
* Revert "Update constants attributes to base class"
This reverts commit 936b1e41f2e2d445e69e840b69af08eac6661120.
* Fix mapping
* E2E test
* remove prints
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Black
* Update any dynamic
* Prompt update - more leniency
* Remove prints
* Merged main into bugfix/dynamic-primary-issues
* update crosswalk
* Merged main into bugfix/dynamic-primary-issues
Approved-by: Alex Galarce
Bugfix/lesser of
* Refine SPLIT_REIMBURSEMENTS_PROMPT to enhance clarity and preserve payment limiters in output
* Add debug logging for LLM invocation and response in split_compound_reimbursement_llm
* Add debug logging for reimbursement validation prompts and responses
* Refine SPLIT_REIMBURSEMENTS_PROMPT to clearly indicate payment limiters in split components
* Refine SPLIT_REIMBURSEMENTS_PROMPT to clarify payment limiters and their application in split methodologies
* Refine SPLIT_REIMBURSEMENTS_PROMPT to clarify handling of conditional splits and preserve original service context
* fix brackets
* Merge remote-tracking branch 'origin/main' into bugfix/lesser-of
* Remove unused SERVICE_METHODOLOGY_BREAKOUT function
* Update compound indicators to refine phrasing and ensure consistency
Approved-by: Katon Minhas
Bugfix/uom
* Add "Per Procedure" to valid unit of measure list
* Enhance unit of measure prompt for clarity and specificity in reimbursement methodologies
* Refine unit of measure prompt to ensure defaulting to 'Per Unit' regardless of procedure codes for flat rate methodologies
* Merge remote-tracking branch 'origin/main' into bugfix/uom
* Merged main into bugfix/uom
* Enhance logic for setting UNIT_OF_MEASURE to 'Per Unit' for Flat Rate reimbursement, handling all edge cases including NaN and empty strings.
* Enhance is_empty function to handle additional empty value cases for pd.Series and strings
* Enhance test cases for is_empty function to cover additional whitespace and null-like values
* Merge branch 'bugfix/uom' of https://bitbucket.org/aarete/doczy.ai into bugfix/uom
* Enhance type checking for is_empty function with overloads for str, list, and pd.Series
* Overload fix
* Merged main into bugfix/uom
Approved-by: Katon Minhas
Bugfix/carveout cd
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Set carveout indicator for two-stage process; prepped set_carveout_indicators and run_methodology_breakout_with_carveout_indicators to replace process_carveout_or_special_case
* Fix unit tests
* Remove test file
* Refactor carveout handling: streamline indicator setting and enhance methodology breakout integration
* Refactor get_carveout_and_special_fields: ensure methodology breakout runs regardless of carveout/special case configs; streamline handling of indicators and questions
* Remove deprecated process_carveout_or_special_case function
* Merge remote-tracking branch 'origin/bugfix/dynamic-primary-issues' into bugfix/carveout-cd
* Add tests for carveout indicator setting and special cases processing
* Add tests for unrecognized carveout handling and legacy stop-loss processing
* Refactor special case configurations: streamline STOP_LOSS handling and update test cases for methodology prompt templates
* Refactor special case handling: improve argument formatting in get_special_cases and run_methodology_breakout_with_carveout_indicators functions
* Add test for run_methodology_breakout_with_carveout_indicators function
* Merge remote-tracking branch 'origin/main' into bugfix/carveout-cd
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Refactor TRIGGER_CAP special case configuration to utilize only trigger cap questions for two-stage processing
* Refactor get_special_cases to improve handling of N/A responses and merge methodology results with special case results
* Fix field type in get_special_cases for fee schedule breakout configuration
* Create fieldset.py - refactor
* Fix placeholders
* string concatenate new questions
* Update trigger cap questions output format
* Update TRIGGER_CAP_THRESHOLD_AMT prompt for clarity and specificity in reimbursement methodology
* Update constants class
* Update Field class
* Resolve constants passing
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Fix unit tests
* Remove test file
Approved-by: Alex Galarce
VenkataKrishna Reddy Avula