Feature/black and isort
* black and isort constants
* black and isort codes
* black and isort crosswalks
* black and isort investment
* black and isort prompts
* isort testbed
* isort tracking
* black and isort utils
* poetry and black tests
Approved-by: Katon Minhas
Feature/one to n docs
* Refactor function signatures and improve imports in one_to_n_funcs.py
* Enhance type hints and documentation for carveout_check and methodology_breakout functions
* Add docstring to methodology_breakout_secondary function for clarity on parameters and return values
* Add docstring to special_case_breakout function for clarity on parameters and return values
* Enhance docstring for deduplicate_with_accumulator function to clarify behavior and parameters
* Enhance docstring for filter_services_without_reimbursements function to clarify validation process and answer structure
* Enhance docstring for filter_services_without_reimbursements function to clarify filtering criteria and validation process
* Enhance docstring for split_compound_reimbursements function to clarify the splitting process and provide an example
* Enhance docstring for split_reimb_dates function to clarify date processing, parameters, return values, and provide an example
* Remove unused regex import from one_to_n_funcs.py
* Remove unused Delimiter import from one_to_n_funcs.py
Approved-by: Katon Minhas
io_utils documentation
* add docstring to read_local function and handle file read errors gracefully by returning None on failure
* update read_local function to support .xlsx files and improve error handling
* add docstring to read_s3 function to clarify its purpose and return value
* refactor read_s3 and read_s3_csv functions to improve error handling and add type hints
* improve error handling in read_s3_csv function for CSV file parsing
* enhance read_input function with error handling and detailed docstring
* Add TODOs for read_input_csv and filter_already_processed functions; add type hints and docstring for remove_txt_extension
* enhance write_local function documentation; make docstrings more consistent when there is a "notes" section
* enhance write_local and write_s3 functions; improve documentation, add error output type handling, and ensure consistent CSV writing options
* refactor test_io_utils.py; reorganize imports and enhance read_input_local test with additional mock for file existence
Approved-by: Katon Minhas
Bugfix/provider info to refactor/one-to-n
* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test
* Update Carveout IND and Default IND logic
* Split out exhibit-level lesser
* Bugfix
* update lesser of exhibit prompt
* Bugfix
* exhibit lesser of varible name bugfix
* Update exhibit lesser of passing
* Update Exhibit-level prompt template
* Shorten full_context prompt length if the questions are long
* Special case assignment index
* Resolve unit tests
* updated UC IND prompt
* Remove print
* update one-to-n unit test
* added dot based date example
* updated date fix prompt
* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)
Bugfix/json error and unit of measure
* json parser fixes and prompt changes
* universal json load fixed
* json error fixed
* json error fixed 2
* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure
Approved-by: Katon Minhas
* added new tin format
* updated signature page prompt
* updated tin search and tin duplication
* Merge remote-tracking branch 'origin/main' into bugfix/provider_info
* extended duplicate logic to npi and names
* Merged main into bugfix/provider_info
Approved-by: Katon Minhas
json parsing fix
* OUTLIER_TERMS-->OUTLIER_TERM standardization
* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test
* Update Carveout IND and Default IND logic
* Split out exhibit-level lesser
* Bugfix
* update lesser of exhibit prompt
* Bugfix
* exhibit lesser of varible name bugfix
* Update exhibit lesser of passing
* Update Exhibit-level prompt template
* Shorten full_context prompt length if the questions are long
* Special case assignment index
* Resolve unit tests
* updated UC IND prompt
* Remove print
* update one-to-n unit test
* added dot based date example
* updated date fix prompt
* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)
Bugfix/json error and unit of measure
* json parser fixes and prompt changes
* universal json load fixed
* json error fixed
* json error fixed 2
* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure
Approved-by: Katon Minhas
* json parsing fix
* prompt update
* Merge branch 'main' into bugfix/json_and_unit_of_measure
* minor bugfix
* minor bugfix
Approved-by: Alex Galarce
Bugfix/fee schedule
* Initialize grouper pct and base rate prmopts
* Update postprocessing
* Merged main into bugfix/fee-schedule
* Prep for merge
Approved-by: Alex Galarce
Feature/improve logging docs
* Enhance PerFileHandler documentation and functionality for improved logging context management
* Add more docuementation
* Update logging_utils.py documentation for clarity and usage examples
* Enhance logging comments for clarity on per-file logging setup and concurrent processing
* Enhance logging comments for clarity on per-file logging context and its implications
Approved-by: Katon Minhas
Refactor/one to n
* remove print
* Update logging for fsbreakout and grouper breakout
* Additional type hints
* updated special case primary
* Remove deprecated special case check
* OUTLIER_TERMS-->OUTLIER_TERM standardization
* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test
* Update Carveout IND and Default IND logic
* Split out exhibit-level lesser
* Bugfix
* update lesser of exhibit prompt
* Bugfix
* exhibit lesser of varible name bugfix
* Update exhibit lesser of passing
* Update Exhibit-level prompt template
* Shorten full_context prompt length if the questions are long
* Special case assignment index
* Resolve unit tests
* updated UC IND prompt
* Remove print
* update one-to-n unit test
* added dot based date example
* updated date fix prompt
* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)
Bugfix/json error and unit of measure
* json parser fixes and prompt changes
* universal json load fixed
* json error fixed
* json error fixed 2
* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure
Approved-by: Katon Minhas
Approved-by: Alex Galarce
Bugfix/provider info tracker issues to main
* testing TIN AND NPI
* print statements added
* updated condition for header and footer tin and npi check
* tin testing for hyphen format
* removed print statements
* updated comments
Approved-by: Katon Minhas
Add time-based rate handling rules and update section labels in prompt templates
* Add time-based rate handling rules and update section labels in prompt templates
* Add mapping for MyCare Ohio in crosswalk_program.json and update state mapping for Ohio in crosswalk_program_lob.json
* Update test parameters and state configuration in testing notebook
* Update exhibit-wide payment constraint prompt to exclude administrative processing language
* Merge remote-tracking branch 'origin/main' into bugfix/reimb-term-issues
* Refine exhibit-wide payment constraint prompt to clarify extraction criteria and examples
* Enhance exhibit-wide payment constraint prompt with guidance for abbreviated terms and clarify acceptable constraints
* Update SERVICE_TERM prompt to specify healthcare services and clarify extraction criteria
Approved-by: Katon Minhas
Bugfix/trigger cap fixes
* stoploss prompt added again
* stoploss prompt added again
* stoploss prompt updated
* reverted back to one stage for trigger cap cases
* Merge branch 'main' into bugfix/trigger_cap_fixes
* pipeline error fixed
* incorrect discount special case fixed
Approved-by: Alex Galarce
Change logging level from info to debug for OCR correction and extraction summaries
* Change logging level from info to debug for OCR correction and extraction summaries
Approved-by: Katon Minhas
Bugfix/multi tin npi fix
* Add debug logging for OCR candidates and provider info extraction
* Update TIN and NPI extraction instructions to handle multiple values in a single record
* Add validation layer to cross-check TINs and NPIs with regex findings in provider info extraction
* Enhance exact match extraction by normalizing identifiers and validating lengths for TIN and NPI
Approved-by: Katon Minhas
Feature/tin enhancements
* Add OCR-correctable TIN patterns and substitution mapping
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Refactor OCR-correctable TIN pattern and enhance documentation for clarity
* Enhance get_all_matches function with type hints and improve documentation; add get_all_matches_with_ocr for OCR-correctable matches
* Add attempt_ocr_correction function for TIN OCR error handling and validation
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Integrate OCR correction for TIN extraction in run_provider_info_fields; log corrections for monitoring
* Refactor OCR-correctable patterns in regex_patterns.py; add NPI_OCR_PATTERN and enhance documentation for clarity
* Enhance OCR correction in attempt_ocr_correction to support both TIN and NPI; update validation logic for identifier lengths
* Integrate NPI OCR correction in run_provider_info_fields; log corrections for monitoring
* Enhance logging in run_provider_info_fields to summarize TIN and NPI extraction quality; include counts of exact and OCR-corrected identifiers
* Add unit and integration tests for OCR correction
* isort
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Merged main into feature/tin-enhancements
Approved-by: Katon Minhas
Feature/initialize phase 5
* Initialize at Exhibit-Level
* Merge branch 'main' into feature/initialize-phase-5
* Add all breakout fields
* Prep for merge
Approved-by: Alex Galarce
bugfix/multiple_outlier_fixes to main
* updated prompts
* Merged main into bugfix/multiple_outliers_fixes
* updated outlier maximum prompts
Approved-by: Katon Minhas
Bugfix/lob fixes
* Refactor update_lob_for_duals and update_grouper_base_rate_and_grouper_pct_rate functions for improved clarity and logic; add handling for 'neither' and 'n/a' cases.
* Change logging level from info to debug for various functions to reduce log verbosity.
Approved-by: Katon Minhas
Feature/payer state
* fix indentation for PROVIDER_STATE prompt
* Add PAYER_STATE prompt and normalize state abbreviation in run_full_context_fields
* Add PAYER_STATE to COLUMN_ORDER in investment_columns.py
* Merged main into feature/payer-state
* Only run PAYER_STATE prompt when more than one state in the batch
Bugfix/dynamic reimbursement primary nv issues
* Add reimb_level_prompt to Field class for dynamic reimbursement handling
* Add reimb_level_prompt for dynamic extraction of LOBs, Networks, Products, and Programs
* Merged main into bugfix/dynamic-reimbursement-primary-nv-issues
Approved-by: Katon Minhas
Feature/grouper cd desc
* added grouper cd desc
* bugfix
* Merge branch 'main' into feature/grouper_cd_desc
* Merged main into feature/grouper_cd_desc
Approved-by: Alex Galarce
Feature/per file logging
* Add custom per-file logging handler
* Add close method in PerFileHandler to properly close file handlers
* Add return type annotation to setup_per_file_logging function
* Set up per-file logging in main and process_file functions
* Add per-file logging configuration options and update setup logic
* Refactor logging setup in testing notebook to use per-file logging and improve organization
* Clean up whitespace and improve organization in testing notebook
* Merge remote-tracking branch 'origin/main' into feature/per-file-logging
* Enhance per-file logging setup by adding debug level configuration and informative print statement
* Change logging from info to debug level in table processing functions for improved verbosity control
* isort
Approved-by: Katon Minhas
pyproject requirements, prep for logging fixes
* remove chroma
* update project name to "doczy-field-extraction" in pyproject.toml
* update lock file
* replace print statements with logging for better traceability in process_file function
* replace print statements with logging for improved error tracking and information logging
* replace print statements with logging for improved error handling and traceability
* replace print statements with logging for improved error handling in llm_utils and string_utils
* remove unused table_utils module; supplanted by table_funcs and unreferenced
* replace capsys with caplog in test_invalid_input_type for improved logging of invalid input
Approved-by: Katon Minhas
Feature/faiss
* Migrate to FAISS from chroma
* Refactor safe_process_file to remove unnecessary garbage collection and update docstring for clarity
* Merged main into feature/faiss
Approved-by: Katon Minhas
Feature/grouper fields
* prompt changes
* print statements removed
* merged with main
* merge issue fixed
* updated grouper prompts
* updated grouper fields in fields order
* updated breakout prompt
* Merged main into feature/grouper-fields
* reverted grouper cd prompt
* updated goruper severity prompt
* Merge remote-tracking branch 'origin/main' into feature/grouper-fields
* updated grouper breakout function
* error fixes
* Merged main into feature/grouper-fields
* Merged main into feature/grouper-fields
* updated grouper_pct_rate and grouper_base_rate
* removed empty array post processing fix
* Merged main into feature/grouper-fields
* post-processing for grouper rate modified
* updated grouper type prompt
* Merged main into feature/grouper-fields
* grouper version prompt updated
* Merged main into feature/grouper-fields
* OPPS as APC
* Merged main into feature/grouper-fields
Approved-by: Katon Minhas
Bugfix/too many files
* Implement caching for field data loading to optimize performance and prevent "too many open files" errors
* Documentation
* isort
* fix breaking test to clear cache and ensure mock data is actually being used
* Merge remote-tracking branch 'origin/main' into bugfix/too-many-files
* Implement thread-safe vectorstore caching and cleanup in field context extraction
* Refactor imports in smart_chunking_funcs.py for clarity and organization
* Update module docstring in smart_chunking_funcs.py
* Fix mypy errors
* Fix variable naming and error handling in run_smart_chunked_fields function
* Reorganize imports and enhance module documentation for clarity
* Add logging for error handling in field_context function
Approved-by: Katon Minhas
Remove cost logging functionality and related tests
* Remove cost logging functionality and related tests
* Remove cost log entry test and related CSV writer mock
* Merged main into feature/remove-costlog
Approved-by: Katon Minhas
bugfix/outlier_fixes to main
* simplified outlier prompts
* Merged main into bugfix/outlier_fixes
* Merged main into bugfix/outlier_fixes
Approved-by: Alex Galarce
Approved-by: Katon Minhas
Bugfix/medicare advantage duals
* Add logging for debugging in crosswalk utilities and mapping functions
* Add debug logging for crosswalk processing and mapping in get_crosswalk_fields
* Add logging for dual LOB check prompts and responses in update_lob_for_duals
* Enhance dual LOB check logic to ensure PROGRAM and PRODUCT fields are empty before proceeding
* Update dual LOB check to use AARETE_DERIVED fields for validation
* Refactor logging in fill_na_mapping and get_crosswalk_fields for clarity and conciseness
* Remove debug logging from get_lob_relationship for cleaner output
* Remove debug logging from apply_crosswalk for cleaner execution
Approved-by: Katon Minhas
Bugfix/global lesser of overpulling
* Update global reimbursement constraint prompt for clarity and examples
* Refine global reimbursement constraint prompt for clarity and specificity
Approved-by: Katon Minhas
Dynamic primary loosen prompts
* Add logging for dynamic primary prompt and Claude's response
* Update prompts to allow for synonyms and equivalent terms in product and program fields
* Refactor get_crosswalk_fields to handle 'N/A' values and improve base field tracking
* Refactor get_crosswalk_fields to remove unnecessary base field processing
* put back JUST_MAPPED
* Fix N/A handling in get_crosswalk_fields to trim whitespace before comparison
* Merge remote-tracking branch 'origin/main' into feature/alex-duals-issues
* Merge remote-tracking branch 'origin/main' into feature/alex-duals-issues
* Fix N/A handling in get_crosswalk_fields to compare without stripping whitespace (in case of `None`)
Approved-by: Katon Minhas
Refine descriptions for BASE_COVERED_SERVICES and PAYMENT_CARVEOUT in valid_carveouts.json
* Refine descriptions for BASE_COVERED_SERVICES and PAYMENT_CARVEOUT in valid_carveouts.json
* Merge remote-tracking branch 'origin/main' into bugfix/carveout-cd-issues
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* format and restructure
* checkpoint
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Update constants attributes to base class
* Revert "Update constants attributes to base class"
This reverts commit 936b1e41f2e2d445e69e840b69af08eac6661120.
* Fix mapping
* E2E test
* remove prints
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Black
* Update any dynamic
* Prompt update - more leniency
* Remove prints
* Merged main into bugfix/dynamic-primary-issues
* update crosswalk
* Merged main into bugfix/dynamic-primary-issues
Approved-by: Alex Galarce
Bugfix/lesser of
* Refine SPLIT_REIMBURSEMENTS_PROMPT to enhance clarity and preserve payment limiters in output
* Add debug logging for LLM invocation and response in split_compound_reimbursement_llm
* Add debug logging for reimbursement validation prompts and responses
* Refine SPLIT_REIMBURSEMENTS_PROMPT to clearly indicate payment limiters in split components
* Refine SPLIT_REIMBURSEMENTS_PROMPT to clarify payment limiters and their application in split methodologies
* Refine SPLIT_REIMBURSEMENTS_PROMPT to clarify handling of conditional splits and preserve original service context
* fix brackets
* Merge remote-tracking branch 'origin/main' into bugfix/lesser-of
* Remove unused SERVICE_METHODOLOGY_BREAKOUT function
* Update compound indicators to refine phrasing and ensure consistency
Approved-by: Katon Minhas
Alex Galarce