Bugfix/multi tin npi fix
* Add debug logging for OCR candidates and provider info extraction
* Update TIN and NPI extraction instructions to handle multiple values in a single record
* Add validation layer to cross-check TINs and NPIs with regex findings in provider info extraction
* Enhance exact match extraction by normalizing identifiers and validating lengths for TIN and NPI
Approved-by: Katon Minhas
Feature/tin enhancements
* Add OCR-correctable TIN patterns and substitution mapping
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Refactor OCR-correctable TIN pattern and enhance documentation for clarity
* Enhance get_all_matches function with type hints and improve documentation; add get_all_matches_with_ocr for OCR-correctable matches
* Add attempt_ocr_correction function for TIN OCR error handling and validation
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Integrate OCR correction for TIN extraction in run_provider_info_fields; log corrections for monitoring
* Refactor OCR-correctable patterns in regex_patterns.py; add NPI_OCR_PATTERN and enhance documentation for clarity
* Enhance OCR correction in attempt_ocr_correction to support both TIN and NPI; update validation logic for identifier lengths
* Integrate NPI OCR correction in run_provider_info_fields; log corrections for monitoring
* Enhance logging in run_provider_info_fields to summarize TIN and NPI extraction quality; include counts of exact and OCR-corrected identifiers
* Add unit and integration tests for OCR correction
* isort
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Merged main into feature/tin-enhancements
Approved-by: Katon Minhas
Feature/initialize phase 5
* Initialize at Exhibit-Level
* Merge branch 'main' into feature/initialize-phase-5
* Add all breakout fields
* Prep for merge
Approved-by: Alex Galarce
bugfix/multiple_outlier_fixes to main
* updated prompts
* Merged main into bugfix/multiple_outliers_fixes
* updated outlier maximum prompts
Approved-by: Katon Minhas
Bugfix/lob fixes
* Refactor update_lob_for_duals and update_grouper_base_rate_and_grouper_pct_rate functions for improved clarity and logic; add handling for 'neither' and 'n/a' cases.
* Change logging level from info to debug for various functions to reduce log verbosity.
Approved-by: Katon Minhas
Feature/payer state
* fix indentation for PROVIDER_STATE prompt
* Add PAYER_STATE prompt and normalize state abbreviation in run_full_context_fields
* Add PAYER_STATE to COLUMN_ORDER in investment_columns.py
* Merged main into feature/payer-state
* Only run PAYER_STATE prompt when more than one state in the batch
Bugfix/dynamic reimbursement primary nv issues
* Add reimb_level_prompt to Field class for dynamic reimbursement handling
* Add reimb_level_prompt for dynamic extraction of LOBs, Networks, Products, and Programs
* Merged main into bugfix/dynamic-reimbursement-primary-nv-issues
Approved-by: Katon Minhas
Feature/grouper cd desc
* added grouper cd desc
* bugfix
* Merge branch 'main' into feature/grouper_cd_desc
* Merged main into feature/grouper_cd_desc
Approved-by: Alex Galarce
Feature/per file logging
* Add custom per-file logging handler
* Add close method in PerFileHandler to properly close file handlers
* Add return type annotation to setup_per_file_logging function
* Set up per-file logging in main and process_file functions
* Add per-file logging configuration options and update setup logic
* Refactor logging setup in testing notebook to use per-file logging and improve organization
* Clean up whitespace and improve organization in testing notebook
* Merge remote-tracking branch 'origin/main' into feature/per-file-logging
* Enhance per-file logging setup by adding debug level configuration and informative print statement
* Change logging from info to debug level in table processing functions for improved verbosity control
* isort
Approved-by: Katon Minhas
pyproject requirements, prep for logging fixes
* remove chroma
* update project name to "doczy-field-extraction" in pyproject.toml
* update lock file
* replace print statements with logging for better traceability in process_file function
* replace print statements with logging for improved error tracking and information logging
* replace print statements with logging for improved error handling and traceability
* replace print statements with logging for improved error handling in llm_utils and string_utils
* remove unused table_utils module; supplanted by table_funcs and unreferenced
* replace capsys with caplog in test_invalid_input_type for improved logging of invalid input
Approved-by: Katon Minhas
Feature/faiss
* Migrate to FAISS from chroma
* Refactor safe_process_file to remove unnecessary garbage collection and update docstring for clarity
* Merged main into feature/faiss
Approved-by: Katon Minhas
Feature/grouper fields
* prompt changes
* print statements removed
* merged with main
* merge issue fixed
* updated grouper prompts
* updated grouper fields in fields order
* updated breakout prompt
* Merged main into feature/grouper-fields
* reverted grouper cd prompt
* updated goruper severity prompt
* Merge remote-tracking branch 'origin/main' into feature/grouper-fields
* updated grouper breakout function
* error fixes
* Merged main into feature/grouper-fields
* Merged main into feature/grouper-fields
* updated grouper_pct_rate and grouper_base_rate
* removed empty array post processing fix
* Merged main into feature/grouper-fields
* post-processing for grouper rate modified
* updated grouper type prompt
* Merged main into feature/grouper-fields
* grouper version prompt updated
* Merged main into feature/grouper-fields
* OPPS as APC
* Merged main into feature/grouper-fields
Approved-by: Katon Minhas
Bugfix/too many files
* Implement caching for field data loading to optimize performance and prevent "too many open files" errors
* Documentation
* isort
* fix breaking test to clear cache and ensure mock data is actually being used
* Merge remote-tracking branch 'origin/main' into bugfix/too-many-files
* Implement thread-safe vectorstore caching and cleanup in field context extraction
* Refactor imports in smart_chunking_funcs.py for clarity and organization
* Update module docstring in smart_chunking_funcs.py
* Fix mypy errors
* Fix variable naming and error handling in run_smart_chunked_fields function
* Reorganize imports and enhance module documentation for clarity
* Add logging for error handling in field_context function
Approved-by: Katon Minhas
Remove cost logging functionality and related tests
* Remove cost logging functionality and related tests
* Remove cost log entry test and related CSV writer mock
* Merged main into feature/remove-costlog
Approved-by: Katon Minhas
bugfix/outlier_fixes to main
* simplified outlier prompts
* Merged main into bugfix/outlier_fixes
* Merged main into bugfix/outlier_fixes
Approved-by: Alex Galarce
Approved-by: Katon Minhas
Bugfix/medicare advantage duals
* Add logging for debugging in crosswalk utilities and mapping functions
* Add debug logging for crosswalk processing and mapping in get_crosswalk_fields
* Add logging for dual LOB check prompts and responses in update_lob_for_duals
* Enhance dual LOB check logic to ensure PROGRAM and PRODUCT fields are empty before proceeding
* Update dual LOB check to use AARETE_DERIVED fields for validation
* Refactor logging in fill_na_mapping and get_crosswalk_fields for clarity and conciseness
* Remove debug logging from get_lob_relationship for cleaner output
* Remove debug logging from apply_crosswalk for cleaner execution
Approved-by: Katon Minhas
Bugfix/global lesser of overpulling
* Update global reimbursement constraint prompt for clarity and examples
* Refine global reimbursement constraint prompt for clarity and specificity
Approved-by: Katon Minhas
Dynamic primary loosen prompts
* Add logging for dynamic primary prompt and Claude's response
* Update prompts to allow for synonyms and equivalent terms in product and program fields
* Refactor get_crosswalk_fields to handle 'N/A' values and improve base field tracking
* Refactor get_crosswalk_fields to remove unnecessary base field processing
* put back JUST_MAPPED
* Fix N/A handling in get_crosswalk_fields to trim whitespace before comparison
* Merge remote-tracking branch 'origin/main' into feature/alex-duals-issues
* Merge remote-tracking branch 'origin/main' into feature/alex-duals-issues
* Fix N/A handling in get_crosswalk_fields to compare without stripping whitespace (in case of `None`)
Approved-by: Katon Minhas
Refine descriptions for BASE_COVERED_SERVICES and PAYMENT_CARVEOUT in valid_carveouts.json
* Refine descriptions for BASE_COVERED_SERVICES and PAYMENT_CARVEOUT in valid_carveouts.json
* Merge remote-tracking branch 'origin/main' into bugfix/carveout-cd-issues
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* format and restructure
* checkpoint
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Update constants attributes to base class
* Revert "Update constants attributes to base class"
This reverts commit 936b1e41f2e2d445e69e840b69af08eac6661120.
* Fix mapping
* E2E test
* remove prints
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Black
* Update any dynamic
* Prompt update - more leniency
* Remove prints
* Merged main into bugfix/dynamic-primary-issues
* update crosswalk
* Merged main into bugfix/dynamic-primary-issues
Approved-by: Alex Galarce
Bugfix/lesser of
* Refine SPLIT_REIMBURSEMENTS_PROMPT to enhance clarity and preserve payment limiters in output
* Add debug logging for LLM invocation and response in split_compound_reimbursement_llm
* Add debug logging for reimbursement validation prompts and responses
* Refine SPLIT_REIMBURSEMENTS_PROMPT to clearly indicate payment limiters in split components
* Refine SPLIT_REIMBURSEMENTS_PROMPT to clarify payment limiters and their application in split methodologies
* Refine SPLIT_REIMBURSEMENTS_PROMPT to clarify handling of conditional splits and preserve original service context
* fix brackets
* Merge remote-tracking branch 'origin/main' into bugfix/lesser-of
* Remove unused SERVICE_METHODOLOGY_BREAKOUT function
* Update compound indicators to refine phrasing and ensure consistency
Approved-by: Katon Minhas
Bugfix/uom
* Add "Per Procedure" to valid unit of measure list
* Enhance unit of measure prompt for clarity and specificity in reimbursement methodologies
* Refine unit of measure prompt to ensure defaulting to 'Per Unit' regardless of procedure codes for flat rate methodologies
* Merge remote-tracking branch 'origin/main' into bugfix/uom
* Merged main into bugfix/uom
* Enhance logic for setting UNIT_OF_MEASURE to 'Per Unit' for Flat Rate reimbursement, handling all edge cases including NaN and empty strings.
* Enhance is_empty function to handle additional empty value cases for pd.Series and strings
* Enhance test cases for is_empty function to cover additional whitespace and null-like values
* Merge branch 'bugfix/uom' of https://bitbucket.org/aarete/doczy.ai into bugfix/uom
* Enhance type checking for is_empty function with overloads for str, list, and pd.Series
* Overload fix
* Merged main into bugfix/uom
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Update get prompt for direct valid value setting
* prep for merge
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Remove test
* Update unit tests with real constants object
* Ready for review
* direct replace {valid_values}
Approved-by: Alex Galarce
Bugfix/carveout cd
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Set carveout indicator for two-stage process; prepped set_carveout_indicators and run_methodology_breakout_with_carveout_indicators to replace process_carveout_or_special_case
* Fix unit tests
* Remove test file
* Refactor carveout handling: streamline indicator setting and enhance methodology breakout integration
* Refactor get_carveout_and_special_fields: ensure methodology breakout runs regardless of carveout/special case configs; streamline handling of indicators and questions
* Remove deprecated process_carveout_or_special_case function
* Merge remote-tracking branch 'origin/bugfix/dynamic-primary-issues' into bugfix/carveout-cd
* Add tests for carveout indicator setting and special cases processing
* Add tests for unrecognized carveout handling and legacy stop-loss processing
* Refactor special case configurations: streamline STOP_LOSS handling and update test cases for methodology prompt templates
* Refactor special case handling: improve argument formatting in get_special_cases and run_methodology_breakout_with_carveout_indicators functions
* Add test for run_methodology_breakout_with_carveout_indicators function
* Merge remote-tracking branch 'origin/main' into bugfix/carveout-cd
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Refactor TRIGGER_CAP special case configuration to utilize only trigger cap questions for two-stage processing
* Refactor get_special_cases to improve handling of N/A responses and merge methodology results with special case results
* Fix field type in get_special_cases for fee schedule breakout configuration
* Create fieldset.py - refactor
* Fix placeholders
* string concatenate new questions
* Update trigger cap questions output format
* Update TRIGGER_CAP_THRESHOLD_AMT prompt for clarity and specificity in reimbursement methodology
* Update constants class
* Update Field class
* Resolve constants passing
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Fix unit tests
* Remove test file
Approved-by: Alex Galarce
Hotfix/fix constants
* Update TRIGGER_CAP methodology prompt and adjust special case configuration
* Refactor synonym, valid carveouts, and valid special mappings to include metadata structure
* remove TRIGGER_CAP_BREAKOUT function
* special_case_configs.py edited online with Bitbucket
Approved-by: Katon Minhas
Bugfix/reimb dates to Main
* added reimb dates split
* Merged main into bugfix/reimb_dates
* prompt changes
* added to postprocessing
* fixed key issues
* modified split function
* removed print statements
* Merge branch 'main' into bugfix/reimb_dates
* update import
* moved split date upstream
* moved split dates to one to n funcs
* Merged main into bugfix/reimb_dates
Approved-by: Katon Minhas
Feature/remove nte
* Revert prompts
* Remove qa_qc tests
* Remove valid lines
* Update crosswalk path
* remove extra argument from split compound reimbursement
* model aliasing
* Haiku alias
* remove erroneous import
* Merge branch 'refactor/update-claude-models' into bugfix/aug-5-issue-fixes
* Convert all matching codes to strings
* Exit if service is empty
* Merged main into bugfix/aug-5-issue-fixes
* global lesser of fix
* Merge branch 'main' into feature/remove-nte
* Remove NTE
Approved-by: Alex Galarce
Bugfix/aug 5 issue fixes
* Revert prompts
* Remove qa_qc tests
* Remove valid lines
* Update crosswalk path
* remove extra argument from split compound reimbursement
* model aliasing
* Haiku alias
* remove erroneous import
* Merge branch 'refactor/update-claude-models' into bugfix/aug-5-issue-fixes
* Convert all matching codes to strings
* Exit if service is empty
* Merged main into bugfix/aug-5-issue-fixes
* global lesser of fix
Approved-by: Alex Galarce
Katon Minhas