Feature/multithreading
* fixes
* Merge main into feature/multithreading
Resolved conflicts:
- Kept timing instrumentation in file_processing.py
- Kept new 3-step Exhibit-based approach for one-to-n processing
- Maintained parallelization improvements (20 workers)
Changes include:
- Timing utils integration for performance monitoring
- Increased max_workers from 5 to 20 across all components
- Parallelized code_breakout and grouper_breakout
- Fixed tin_npi_funcs function call parameters
* Fix max_workers error for empty documents
- Add check to skip parallel processing when no pages exist
- Use min(len(all_page_tasks), 20) to prevent max_workers=0
- Handles edge case of documents with no exhibits or pages
* Fix max_workers=0 errors in one_to_n_funcs
- Add checks before all ThreadPoolExecutor creations
- Prevents errors when processing empty lists:
- carveout_and_special_case
- breakout
- special_case_breakout
- filter_services_without_reimbursements
- run_lob_relationship
- Ensures executor only created when there are items to process
* Reduce code processing parallelism to prevent API throttling
Lower max_workers from 20 to 10 for code_breakout and grouper_breakout
to prevent overwhelming Bedrock API with concurrent requests
* fixed
* Merged main into feature/multithreading
* Move documentation into folder
* Fix logging statements
* Merge branch 'main' into feature/multithreading
* Refactor for clarity
* Update previous exhibit passing logic
* properly simplify exhibits
* Merged main into feature/multithreading
* update conditional for None
* exhibit multithreading changed
* exhibit multithreading changed
* Merge remote-tracking branch 'origin/main' into feature/multithreading
* synced with main
* Parallelize dynamic assignment, refactor HSC field worker, add timing/exhibit unit tests, tidy imports/ignore helpers
* analyze_regression.py edited online with Bitbucket
* count_pages.py edited online with Bitbucket
* compare_regressed_with_baseline.py edited online with Bitbucket
* simple_testbed_compare.py edited online with Bitbucket
* run_testbed_metrics_regressed.py edited online with Bitbucket
Approved-by: Katon Minhas
Bugfix/hmk issues
* fix for missing AARETE_DERIVED_LOB
* fix for AARETE_DERIVED_SID
* bugfix in universal_json_load
* bugfix for normalize_state_field
* reimb eff date assignmnet prompt update
* prompt update for missing reimb terms
* Merged main into bugfix/hmk-issues
* test case for universal_json_loads
* reveresed the prompt update for missing reimb
* prompt fix for reimb dates
Approved-by: Katon Minhas
Bugfix/prov info issues
* Initial commit
* Passed payer name to tin npi llm call to avoid assigning tin npi to payer
* Removed record that contains NO_IDENTIFIERS_FOUND from PROV_INFO_JSON
* Updated unit test for test_tin_npi_funcs
* Changed name check code back to original to be consistent
* Updated test_tin_npi_funcs.py to pass in payer_name in functions
* Add fix to unit test
* Updated output in unit tests
* Merged main into bugfix/prov-info-issues
Approved-by: Katon Minhas
Bugfix/one to one and prov info
* Add post-processing logic to update PROV_INFO_JSON if needed; Updated prompt to verify the number of digitis in TIN and NPI
* Added update_prov_info_with_group_name to postprocessing_funcs.py
* Updated FIRST_PAGE_PROVIDER_NAME_TEMPLATE to ask LLM to explain resoning before answering
* Updated logic to use LLM to check if provider name matches
* Use only Hybrid Smart Chunking to get provider group name
* Cleaned up code
* Removed commented out code
* Updated unit test
* Merged main into bugfix/one_to_one_and_prov_info
* Edited prompt to distinguish provider names and to enclose answer in pipes
* Updated test_tin_npi_funcs
Approved-by: Katon Minhas
Bugfix/cpt codes
* formatting fix for revenue_cd
* format fix for procedure_codes
* bugfix for revenue_codes
* working version from code main
* code formatting from main and code main
* removed dubug prints
* removed local file paths
* standardize state fields
* updated test scenerios
* minor fix
* Merge branch 'main' into bugfix/cpt-codes
merging main into cpt code fix
* checks for all fields
* reused is_empty functionality
* moved normalize_to_json_list to utils
* Merge branch 'main' into bugfix/cpt-codes
merging upto date main
* minor touches
* pytest fix
Approved-by: Katon Minhas
Bugfix/prov info
* Added logic to extract provider name from first page
* Added prompt to extract provider name from first page
* Added prompts and provider name extraction using Hybrid Smart Chunking
* Cleaned up code
* Added additional logic to check if the provider name on tin/npi page matches provider name extracted from page 1.
* Added fix for IS_GROUP issue in provider info json
* Remove commented out code
* unit test fix
* uni test fix
* Merge branch 'main' into bugfix/prov_info
* Change to string_utils.is_empty
* Minor refactor
Approved-by: Katon Minhas
Bugfix/code list format
* replace | with comma in lists
* partial revert
* remove test changes
* Merged main into bugfix/code-list-format
* fix list format
* pipeline error fixed
* pipeline error fixed
* pipeline error fixed
Approved-by: Katon Minhas
Feature/automated qc qa
* Moved automation_saas.py from alien_strings to qc_qa module
* Moved from alien_string
* Moved from alien_string
* clean up
* WIP: prepare saas automation script for integration
* format code and fix mypy issue
* minor fixes during e2e testing
* add tests
* format and fix mypy issues
* Merge remote-tracking branch 'origin/main' into feature/automated-qc-qa
* Remove print - replace with logging
Approved-by: Katon Minhas
Feature/llm utils refactor
* - Removed some temporary logging code
- Removed unused imports (uuid, datetime) from llm_utils.py
- Added cache token columns (cache_creation_tokens, cache_read_tokens) to CSV exports
- Added tracking for cache warming
- Updated usage_tracking.py to track cache tokens separately in data structure
All token tracking functionality remains intact.
* Split input tokens into fresh and cache in summary section
- Add total_fresh_input_tokens and total_cache_tokens to GLOBAL_USAGE
- Update summary CSV to include split token columns for better tractability
- Add averages for fresh_input_tokens and cache_tokens
- Maintain total_input_tokens = fresh_input_tokens + cache_tokens relationship
- All calculations verified and match per-file totals correctly
* upload files functionality
* Add comprehensive unit tests for usage_tracking module
- Implement Phase 1-4 tests covering all 10 functions in usage_tracking.py
- Add 98 test cases with parametrized tests for comprehensive coverage
- Fix extract_usage_from_response to handle None input gracefully
- Add python-dotenv dependency to pyproject.toml
- Tests cover: core functions, data access, export functions, and utilities
- All tests passing (98/98)
* Add S3 upload tests for main.py and fix f-string syntax error
- Add tests/test_investment_main_s3.py with 3 test cases:
* test_main_writes_final_and_error_to_s3: validates both final and error DataFrames uploaded when processing has errors
* test_main_writes_only_final_when_no_errors: validates only final DataFrame uploaded when all files succeed
* test_main_writes_usage_when_present: validates usage tracking files uploaded when usage data exists
- Tests use monkeypatching and module stubs to isolate S3 write logic without external dependencies
- Fix syntax error in src/codes/code_funcs.py: nested f-string quotes (level_dict['level_suffix'])
- All tests passing
* Add comprehensive test for usage_tracking.get_usage_dataframes S3 uploads
- Add test_usage_dataframes_written_to_s3 to validate per_file_usage_df and batch_summary_df
- Test verifies both dataframes from usage_tracking.get_usage_dataframes() are written to S3
- Validates 'usage' suffix writes per_file_usage_df with correct schema (file_name, tokens, cost)
- Validates 'usage_summary' suffix writes batch_summary_df with correct schema (totals, averages, region_mode)
- Uses pd.testing.assert_frame_equal to ensure exact dataframes are uploaded
- Fix module caching issue between tests by clearing sys.modules cache
- All 4 tests passing
* Add cleanup fixture to prevent test pollution
- Add autouse pytest fixture to clean up stubbed modules after each test
- Remove manual cache clearing from individual tests (now handled by fixture)
- Prevents our module stubs from affecting other test files in CI/CD pipeline
- Fixes Bitbucket pipeline failures where other tests couldn't find llm_utils attributes
* Refactor tests: add comprehensive io_utils coverage; disable usage_tracking suite
- Added 22 focused tests for write_local and write_s3 plus existing IO behaviors
- Introduced FakeS3Client to avoid boto3 network/credential dependency
- Added preserve_config fixture (no monkeypatch) to isolate config side effects
- Marked test_usage_tracking.py skipped per new consolidation approach
- Verified 26 tests (io_utils + investment_main) all pass locally without AWS exceptions
* Fix S3_CLIENT mocking: use mocker.patch instead of direct assignment
- Changed preserve_config fixture to NOT save/restore S3_CLIENT
- Updated all 5 write_s3 tests to use mocker.patch('src.config.S3_CLIENT', fake)
- This prevents pollution of config.S3_CLIENT across test modules
- Fixes NoCredentialsError in other tests that were importing config after our tests set FakeS3Client
- All 22 io_utils tests pass + 4 investment_main_s3 tests pass
- test_llm_utils::test_invoke_claude_local_mode now passes
* Add tests for write_s3 function
* Resolve merge conflict: keep mocker-based write_s3 tests
* Remove rogue skip mark from test_usage_tracking.py
The skip mark was incorrectly added by remote branch claiming tests were
migrated to io_utils. However, test_io_utils.py only tests io_utils functions
(read/write operations), not usage_tracking functions.
Restore the comprehensive usage_tracking tests from commit 26a556f0 which
includes 98 test cases covering all 10 functions in usage_tracking.py.
* Fix test pollution: add config cleanup fixture and restore RUN_MODE
- Add autouse fixture in test_io_utils.py to restore config values after each test
- Fix test_llm_utils.py to restore RUN_MODE after test_invoke_claude_local_mode
- Prevents config patches from leaking into subsequent tests causing NoCredentialsError
* Remove redundant test_investment_main_s3.py
- File was testing same write_s3 functionality already covered in test_io_utils.py
- Was causing module pollution by stubbing src.utils.llm_utils
- Caused test failures in other test files due to module cache issues
- Integration testing was minimal and mocked write_s3 anyway
* Phase 1: Remove unused code and align cost constants
- Remove count_tokens() function (unused, replaced by exact API counts)
- Remove unused cost constants from invoke_claude() and invoke_claude_with_image_array()
- Remove unused math import
- Update usage_tracking.py to match correct cost constants from llm_utils.py:
- Corrected Haiku cache_read: 0.000025 -> 0.00003
- Implemented correct cache creation cost calculation:
* Sonnet models: 25% higher than input (0.00375 vs 0.003)
* Haiku: 20% higher than input (0.0003 vs 0.00025)
- Remove test_count_tokens() test
- Update test_track_usage_single_call() to account for corrected cache creation costs
All 437 tests passing. Token counts verified against previous runs.
* Phase 2: Remove redundant update_stats() tracking
- Remove all update_stats() calls from EC2 mode (3 locations)
- Remove update_stats() function definition
- Remove estimated token counting (tokens_sent, tokens_received)
- Remove elapsed_time calculations
- Remove unused start_time assignments
- Remove test_update_stats() test case
- Clean up setUp() method in test_llm_utils.py
- Update docstrings to reflect usage_tracking.py as source of truth
All token tracking now uses exact API counts via usage_tracking.track_usage().
All 436 tests passing.
* Merge main into feature/llm-utils-refactor
Resolved merge conflicts:
- llm_utils.py: Kept refactored code (removed legacy update_stats, math import)
- usage_tracking.py: Kept corrected cost constants and cache creation pricing logic
- io_utils.py: Merged main's new 'dtc' and 'dtc_summary' output types
- test_usage_tracking.py: Kept corrected test values for cache_read and cache_creation costs
All conflicts resolved while preserving Phase 1 & 2 refactoring work.
* Fix merge conflict artifact in test_usage_tracking.py
Removed leftover merge conflict marker (<<<<<<< HEAD) that was missed
during merge resolution. The test now correctly calculates cache creation
costs for Sonnet (25% higher) and Haiku (20% higher) models.
* Phase 3: Add thread safety and enhance cache key generation
Step 3.1: Add thread safety to cache operations
- Added _cache_lock = threading.Lock() at module level
- Wrapped all cache operations (get, set, move_to_end, popitem) with lock
- All 4 cache operation locations now thread-safe:
* Cache check/retrieval in invoke_claude()
* Cache storage in invoke_claude()
* Cache check/retrieval in invoke_claude_with_image_array()
* Cache storage in invoke_claude_with_image_array()
Step 3.2: Enhance cache key generation
- Updated get_cache_key() to include instruction, max_tokens, and cache flag
- Updated get_cache_key_with_image_array() with same enhancements
- Cache keys now differentiate based on all relevant parameters
- Backward compatible (all new parameters are optional)
Step 3.3: Update call sites
- Updated invoke_claude() to pass instruction, max_tokens, cache to get_cache_key()
- Updated invoke_claude_with_image_array() to pass max_tokens
All 412 tests passing. Cache operations are now thread-saf…
* Phase 4: Add comprehensive tests for cache operations and token tracking
Step 4.1: Remove legacy test cases
- Already completed in Phase 2 (test_update_stats, test_count_tokens removed)
- setUp() cleaned up (no MODEL_STATS/GLOBAL_STATS setup needed)
Step 4.2: Add tests for thread-safe cache operations
- test_cache_thread_safety_concurrent_reads: 10 concurrent cache reads
- test_cache_thread_safety_concurrent_writes: 10 concurrent cache writes
- test_cache_thread_safety_mixed_operations: mixed read/write operations
Step 4.3: Add tests for enhanced cache key generation
- test_get_cache_key_basic: backward compatibility test
- test_get_cache_key_with_instruction: instruction parameter
- test_get_cache_key_with_max_tokens: max_tokens parameter
- test_get_cache_key_with_cache_flag: cache flag
- test_get_cache_key_all_parameters: all parameters together
- test_get_cache_key_with_image_array: image array variant
- test_invoke_claude_uses_enhanced_cache_key: integration test
Step 4.4: Verify token tracking uses e…
* Phase 5: Replace legacy CSV export with usage_tracking.py
Step 5.1: Replace write_stats_to_csv() implementation
- Updated write_stats_to_csv() to use usage_tracking.get_usage_dataframes() instead of config.MODEL_STATS/GLOBAL_STATS
- Added import for usage_tracking module
- Function now reads from usage_tracking.py as the single source of truth
Step 5.2: Add optional run_id and timestamp parameters
- Added optional run_id parameter (defaults to config.BATCH_ID)
- Added optional timestamp parameter (defaults to current timestamp)
- Updated upload_tracking_logs() to pass batch_id and run_timestamp
Step 5.3: Transform usage_tracking format to legacy CSV format
- Aggregates per-file, per-model data to per-file data
- Maps model names to Claude 2/3/3.5 request counts
- Calculates tokens_sent (fresh input tokens) and tokens_received (output tokens)
- Maintains backward compatibility with legacy CSV structure
- Handles empty data gracefully (writes headers-only CSV)
Step 5.4: Verify call sites work with updated f…
* refactor(llm_utils): extract helper functions and reduce code duplication
- Add private helper functions: _track_usage_from_response(), _build_claude_3_request_body(), _store_in_cache()
- Simplify routing logic using model sets (_CLAUDE_2_MODELS, _CLAUDE_3_AND_UP_MODELS)
- Replace 8 instances of usage tracking pattern with single function call
- Replace 3 instances of request body building with helper function
- Replace 2 instances of cache storage with helper function
- Add Claude 4.5 Sonnet to cache support list
- Reduce file size from 1,004 to 860 lines (14.3% reduction, 144 lines removed)
- All tests passing (422 tests)
- Maintains backward compatibility and functionality
* Re-support Claude 3 Haiku and add comprehensive tests for new helper functions
- Re-added Claude 3 Haiku to supported models (removed from deprecated patterns)
- Updated model validation to include Haiku in supported models list
- Added comprehensive test coverage for new helper functions:
- _validate_model_support() - 5 tests covering deprecation, support, and context
- _supports_prompt_cache() - 2 tests for cache support detection
- _build_claude_3_request_body() - 4 tests for request body building with/without cache
- Updated error messages to reflect Claude 3+ models (Haiku, Sonnet 3.5/3.7/4/4.5)
- All 23 tests passing
* Merged main into feature/llm-utils-refactor
* Merged main into feature/llm-utils-refactor
Approved-by: Siddhant Medar
Approved-by: Katon Minhas
Feature/dtc clean
* dtc v1 initial commit
* second layer of prompt
* Added MAX_PAGES_TO_CHECK and page looping to get answer from llm
* Edited contract_type prompt to get cleaner output with no additional text
* Updated document_type_classification prompt
* Added non-contract type classifcation
* Added dtc_v1_test.py for testing only
* Updated prompt
* Updated prompt
* at scale changes
* add save feature for individual file output
* update dockerfile
* clean dtc to prepare for merge
* clean up dtc script
* Merge remote-tracking branch 'origin/main' into feature/dtc-clean
* reafactor code
* Merge remote-tracking branch 'origin/main' into feature/dtc-clean
Approved-by: Katon Minhas
Feature/automated cost logging csv
* feat: Add usage and cost tracking with CSV export
- Add usage_tracking.py module for thread-safe token and cost tracking
- Extract actual tokens from Bedrock API responses (replaces word count estimation)
- Integrate usage tracking into llm_utils.py for all LLM invocations
- Add CSV export functionality (per-file/per-model and batch summary)
- Integrate CSV export into main.py for local runs
- Update .gitignore to exclude PRD
Phase 1: Data collection and local CSV export implemented
- Tracks all LLM calls including utility functions (date_fix, derive_term_date)
- Calculates costs per model using centralized cost constants
- Generates two CSV files: USAGE.csv and USAGE-SUMMARY.csv
- Exports to same directory as final results when write_to_s3=False
* - Removed some temporary logging code
- Removed unused imports (uuid, datetime) from llm_utils.py
- Added cache token columns (cache_creation_tokens, cache_read_tokens) to CSV exports
- Added tracking for cache warming
- Updated usage_tracking.py to track cache tokens separately in data structure
All token tracking functionality remains intact.
* Split input tokens into fresh and cache in summary section
- Add total_fresh_input_tokens and total_cache_tokens to GLOBAL_USAGE
- Update summary CSV to include split token columns for better tractability
- Add averages for fresh_input_tokens and cache_tokens
- Maintain total_input_tokens = fresh_input_tokens + cache_tokens relationship
- All calculations verified and match per-file totals correctly
* upload files functionality
* Add comprehensive unit tests for usage_tracking module
- Implement Phase 1-4 tests covering all 10 functions in usage_tracking.py
- Add 98 test cases with parametrized tests for comprehensive coverage
- Fix extract_usage_from_response to handle None input gracefully
- Add python-dotenv dependency to pyproject.toml
- Tests cover: core functions, data access, export functions, and utilities
- All tests passing (98/98)
* Add S3 upload tests for main.py and fix f-string syntax error
- Add tests/test_investment_main_s3.py with 3 test cases:
* test_main_writes_final_and_error_to_s3: validates both final and error DataFrames uploaded when processing has errors
* test_main_writes_only_final_when_no_errors: validates only final DataFrame uploaded when all files succeed
* test_main_writes_usage_when_present: validates usage tracking files uploaded when usage data exists
- Tests use monkeypatching and module stubs to isolate S3 write logic without external dependencies
- Fix syntax error in src/codes/code_funcs.py: nested f-string quotes (level_dict['level_suffix'])
- All tests passing
* Add comprehensive test for usage_tracking.get_usage_dataframes S3 uploads
- Add test_usage_dataframes_written_to_s3 to validate per_file_usage_df and batch_summary_df
- Test verifies both dataframes from usage_tracking.get_usage_dataframes() are written to S3
- Validates 'usage' suffix writes per_file_usage_df with correct schema (file_name, tokens, cost)
- Validates 'usage_summary' suffix writes batch_summary_df with correct schema (totals, averages, region_mode)
- Uses pd.testing.assert_frame_equal to ensure exact dataframes are uploaded
- Fix module caching issue between tests by clearing sys.modules cache
- All 4 tests passing
* Add cleanup fixture to prevent test pollution
- Add autouse pytest fixture to clean up stubbed modules after each test
- Remove manual cache clearing from individual tests (now handled by fixture)
- Prevents our module stubs from affecting other test files in CI/CD pipeline
- Fixes Bitbucket pipeline failures where other tests couldn't find llm_utils attributes
* Refactor tests: add comprehensive io_utils coverage; disable usage_tracking suite
- Added 22 focused tests for write_local and write_s3 plus existing IO behaviors
- Introduced FakeS3Client to avoid boto3 network/credential dependency
- Added preserve_config fixture (no monkeypatch) to isolate config side effects
- Marked test_usage_tracking.py skipped per new consolidation approach
- Verified 26 tests (io_utils + investment_main) all pass locally without AWS exceptions
* Fix S3_CLIENT mocking: use mocker.patch instead of direct assignment
- Changed preserve_config fixture to NOT save/restore S3_CLIENT
- Updated all 5 write_s3 tests to use mocker.patch('src.config.S3_CLIENT', fake)
- This prevents pollution of config.S3_CLIENT across test modules
- Fixes NoCredentialsError in other tests that were importing config after our tests set FakeS3Client
- All 22 io_utils tests pass + 4 investment_main_s3 tests pass
- test_llm_utils::test_invoke_claude_local_mode now passes
* Add tests for write_s3 function
* Resolve merge conflict: keep mocker-based write_s3 tests
* Remove rogue skip mark from test_usage_tracking.py
The skip mark was incorrectly added by remote branch claiming tests were
migrated to io_utils. However, test_io_utils.py only tests io_utils functions
(read/write operations), not usage_tracking functions.
Restore the comprehensive usage_tracking tests from commit 26a556f0 which
includes 98 test cases covering all 10 functions in usage_tracking.py.
* Fix test pollution: add config cleanup fixture and restore RUN_MODE
- Add autouse fixture in test_io_utils.py to restore config values after each test
- Fix test_llm_utils.py to restore RUN_MODE after test_invoke_claude_local_mode
- Prevents config patches from leaking into subsequent tests causing NoCredentialsError
* Remove redundant test_investment_main_s3.py
- File was testing same write_s3 functionality already covered in test_io_utils.py
- Was causing module pollution by stubbing src.utils.llm_utils
- Caused test failures in other test files due to module cache issues
- Integration testing was minimal and mocked write_s3 anyway
* Merged main into feature/automated-cost-logging-csv
Approved-by: Karan Desai
Approved-by: Katon Minhas
Bugfix/ut errors
* Update universal_json_load and unit tests
* Update tests
* Fix unit tests
* remove test
* remove imports
* Update function
* update pd Series is_empty
* Update reimbursement check
* added some test cases
* Merged main into bugfix/ut-errors
* Merged main into bugfix/ut-errors
* remove import
* Merged main into bugfix/ut-errors
Approved-by: Karan Desai
Feature/effective dt
* move prompt_smart_chunked to prompt_calls.py
* refactor pass_smart_chunked_to_full_context
* Mypy fix
* Update field to EFFECTIVE_DT
* move to prompt_calls
* Update vision funcs
* Restructure prompt-based date derivation
* E2E passed
* Remove test
* Merge branch 'main' into feature/effective-dt
* remove deprecated postprocessing_func
* Docstrings
* refactor - send global lesser of to prompt_calls.py
* remove deprecated function
* more docstrings
* Merged main into feature/effective-dt
* Merged main into feature/effective-dt
* added base field for aarete derived effective date
Approved-by: VenkataKrishna Reddy Avula
Bugfix/mi reimb terms
* test changes
* logging added
* Merge branch 'main' into bugfix/generic-reimb-terms
* parsing fixes
* fix missing reimbursements
* make service terms unique by adding additional context
* Merge branch 'main' into bugfix/mi-reimb-terms
* remomoving test changes
* remomoving test changes
* remomoving test changes
* test case for universal_json_load added
Approved-by: Katon Minhas
bugfix/code_breakout_rev to main
* added methodology to code explicit
* pipeline fixes
* Merged main into bugfix/code_breakout_rev
* Remove prints
Approved-by: Katon Minhas
Bugfix/ut issues
* Update exhibit linking
* prompt_calls refactor
* remove test
* Add Healthy Kids Program
* Move Healthy Kids to UT specifically
* Remove header check from Dynamic process
* Merged main into bugfix/ut-issues
* prep for merge
* remove deprecated tests
* Merged main into bugfix/ut-issues
* Merged main into bugfix/ut-issues
Approved-by: Faizan Mohiuddin
Bugfix/special cases to Main
* prompt update
* Merge branch 'main' into bugfix/json_and_unit_of_measure
* minor bugfix
* minor bugfix
* prompt update for outlier rate
* added outlier testing jupyter notebook
* Remove special case from any empty
* updated outlier breakout prompt
* Merged main into bugfix/special-cases
* reimb_method minor fix
* Merged main into bugfix/special-cases
* bugfix field to add dynamic funcs
* merged with main
* latest outlier changes
* Merged main into bugfix/special-cases
* added table guidelines for special case primary
* updated special case primary prompt
* Merged main into bugfix/special-cases
* updated special case primary table handling
* merged main into bugfix special cases
* Merged main into bugfix/special-cases
* Remove test notebooks from tracking
* trying out pipeline fix
* reverted chnages
* pipeline fix
Approved-by: Katon Minhas
Bugfix/mi dynamic
* Refactor - split out one_to_n_cleaning
* Rework postprocess
* Update postprocessing - fill dynamic values if there is only 1 non-empty value
* Update to use string_utils
* Group by AARETE_DERIVED_LOB as well
* Update crosswalk mapping to handle | lists
* Update constants - run with MI flag
Approved-by: Siddhant Medar
Feature/black and isort
* black and isort constants
* black and isort codes
* black and isort crosswalks
* black and isort investment
* black and isort prompts
* isort testbed
* isort tracking
* black and isort utils
* poetry and black tests
Approved-by: Katon Minhas
io_utils documentation
* add docstring to read_local function and handle file read errors gracefully by returning None on failure
* update read_local function to support .xlsx files and improve error handling
* add docstring to read_s3 function to clarify its purpose and return value
* refactor read_s3 and read_s3_csv functions to improve error handling and add type hints
* improve error handling in read_s3_csv function for CSV file parsing
* enhance read_input function with error handling and detailed docstring
* Add TODOs for read_input_csv and filter_already_processed functions; add type hints and docstring for remove_txt_extension
* enhance write_local function documentation; make docstrings more consistent when there is a "notes" section
* enhance write_local and write_s3 functions; improve documentation, add error output type handling, and ensure consistent CSV writing options
* refactor test_io_utils.py; reorganize imports and enhance read_input_local test with additional mock for file existence
Approved-by: Katon Minhas
Refactor/one to n
* remove print
* Update logging for fsbreakout and grouper breakout
* Additional type hints
* updated special case primary
* Remove deprecated special case check
* OUTLIER_TERMS-->OUTLIER_TERM standardization
* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test
* Update Carveout IND and Default IND logic
* Split out exhibit-level lesser
* Bugfix
* update lesser of exhibit prompt
* Bugfix
* exhibit lesser of varible name bugfix
* Update exhibit lesser of passing
* Update Exhibit-level prompt template
* Shorten full_context prompt length if the questions are long
* Special case assignment index
* Resolve unit tests
* updated UC IND prompt
* Remove print
* update one-to-n unit test
* added dot based date example
* updated date fix prompt
* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)
Bugfix/json error and unit of measure
* json parser fixes and prompt changes
* universal json load fixed
* json error fixed
* json error fixed 2
* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure
Approved-by: Katon Minhas
Approved-by: Alex Galarce
Bugfix/trigger cap fixes
* stoploss prompt added again
* stoploss prompt added again
* stoploss prompt updated
* reverted back to one stage for trigger cap cases
* Merge branch 'main' into bugfix/trigger_cap_fixes
* pipeline error fixed
* incorrect discount special case fixed
Approved-by: Alex Galarce
Feature/tin enhancements
* Add OCR-correctable TIN patterns and substitution mapping
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Refactor OCR-correctable TIN pattern and enhance documentation for clarity
* Enhance get_all_matches function with type hints and improve documentation; add get_all_matches_with_ocr for OCR-correctable matches
* Add attempt_ocr_correction function for TIN OCR error handling and validation
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Integrate OCR correction for TIN extraction in run_provider_info_fields; log corrections for monitoring
* Refactor OCR-correctable patterns in regex_patterns.py; add NPI_OCR_PATTERN and enhance documentation for clarity
* Enhance OCR correction in attempt_ocr_correction to support both TIN and NPI; update validation logic for identifier lengths
* Integrate NPI OCR correction in run_provider_info_fields; log corrections for monitoring
* Enhance logging in run_provider_info_fields to summarize TIN and NPI extraction quality; include counts of exact and OCR-corrected identifiers
* Add unit and integration tests for OCR correction
* isort
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Merged main into feature/tin-enhancements
Approved-by: Katon Minhas
pyproject requirements, prep for logging fixes
* remove chroma
* update project name to "doczy-field-extraction" in pyproject.toml
* update lock file
* replace print statements with logging for better traceability in process_file function
* replace print statements with logging for improved error tracking and information logging
* replace print statements with logging for improved error handling and traceability
* replace print statements with logging for improved error handling in llm_utils and string_utils
* remove unused table_utils module; supplanted by table_funcs and unreferenced
* replace capsys with caplog in test_invalid_input_type for improved logging of invalid input
Approved-by: Katon Minhas
Bugfix/too many files
* Implement caching for field data loading to optimize performance and prevent "too many open files" errors
* Documentation
* isort
* fix breaking test to clear cache and ensure mock data is actually being used
* Merge remote-tracking branch 'origin/main' into bugfix/too-many-files
* Implement thread-safe vectorstore caching and cleanup in field context extraction
* Refactor imports in smart_chunking_funcs.py for clarity and organization
* Update module docstring in smart_chunking_funcs.py
* Fix mypy errors
* Fix variable naming and error handling in run_smart_chunked_fields function
* Reorganize imports and enhance module documentation for clarity
* Add logging for error handling in field_context function
Approved-by: Katon Minhas
Remove cost logging functionality and related tests
* Remove cost logging functionality and related tests
* Remove cost log entry test and related CSV writer mock
* Merged main into feature/remove-costlog
Approved-by: Katon Minhas
Bugfix/uom
* Add "Per Procedure" to valid unit of measure list
* Enhance unit of measure prompt for clarity and specificity in reimbursement methodologies
* Refine unit of measure prompt to ensure defaulting to 'Per Unit' regardless of procedure codes for flat rate methodologies
* Merge remote-tracking branch 'origin/main' into bugfix/uom
* Merged main into bugfix/uom
* Enhance logic for setting UNIT_OF_MEASURE to 'Per Unit' for Flat Rate reimbursement, handling all edge cases including NaN and empty strings.
* Enhance is_empty function to handle additional empty value cases for pd.Series and strings
* Enhance test cases for is_empty function to cover additional whitespace and null-like values
* Merge branch 'bugfix/uom' of https://bitbucket.org/aarete/doczy.ai into bugfix/uom
* Enhance type checking for is_empty function with overloads for str, list, and pd.Series
* Overload fix
* Merged main into bugfix/uom
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Update get prompt for direct valid value setting
* prep for merge
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Remove test
* Update unit tests with real constants object
* Ready for review
* direct replace {valid_values}
Approved-by: Alex Galarce
Bugfix/carveout cd
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Set carveout indicator for two-stage process; prepped set_carveout_indicators and run_methodology_breakout_with_carveout_indicators to replace process_carveout_or_special_case
* Fix unit tests
* Remove test file
* Refactor carveout handling: streamline indicator setting and enhance methodology breakout integration
* Refactor get_carveout_and_special_fields: ensure methodology breakout runs regardless of carveout/special case configs; streamline handling of indicators and questions
* Remove deprecated process_carveout_or_special_case function
* Merge remote-tracking branch 'origin/bugfix/dynamic-primary-issues' into bugfix/carveout-cd
* Add tests for carveout indicator setting and special cases processing
* Add tests for unrecognized carveout handling and legacy stop-loss processing
* Refactor special case configurations: streamline STOP_LOSS handling and update test cases for methodology prompt templates
* Refactor special case handling: improve argument formatting in get_special_cases and run_methodology_breakout_with_carveout_indicators functions
* Add test for run_methodology_breakout_with_carveout_indicators function
* Merge remote-tracking branch 'origin/main' into bugfix/carveout-cd
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Refactor TRIGGER_CAP special case configuration to utilize only trigger cap questions for two-stage processing
* Refactor get_special_cases to improve handling of N/A responses and merge methodology results with special case results
* Fix field type in get_special_cases for fee schedule breakout configuration
* Create fieldset.py - refactor
* Fix placeholders
* string concatenate new questions
* Update trigger cap questions output format
* Update TRIGGER_CAP_THRESHOLD_AMT prompt for clarity and specificity in reimbursement methodology
* Update constants class
* Update Field class
* Resolve constants passing
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Fix unit tests
* Remove test file
Approved-by: Alex Galarce
Refactor/one time io
* Delete client work
* Streamline imports
* Fix list class
* Update tests
* Clear code funcs test
* Fix unit tests
* Single read code funcs
* Single-read model
* Single-load embeddings
* Successful E2E test
* update unit tests
* Update importsg
* Reload poetry.lock
* remove old exhibit header function
* Remove aarete_derived generic function
* remove align and format tables
* Remove strings to dict
* references
* Clear code
* Move preprocessing_funcs
* remove keywords
* refactor postprocessing_funcs
* Pass unit test - remove qa_qc directory
* Black and isort
* remove print
Approved-by: Alex Galarce
Implement deduplication of provider columns in DataFrame and add corresponding unit tests
* Implement deduplication of provider columns in DataFrame and add corresponding unit tests
Approved-by: Katon Minhas
Table Continuation Fix
* Add EXHIBIT_HEADER_MARKERS for improved document header recognition in table handling
* Refactor EXHIBIT_HEADER_MARKERS definition for improved clarity and organization
* remove unused page span tracking code; add unit tests for new functionality
* remove debugging logs from combine_continuous_tables function
* Refactor logging setup and update test file configuration in investment field testing notebook
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
Approved-by: Katon Minhas
Exhibit-level Lesser Of
* Enhance exception language clarity in investment prompts
* Roll back exception language
* initial try at exhibit_lesser_of
* simplify `create_exhibit_lesser_of_row`
* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints in exhibit
* Remove unused `get_service_breakout` function; it was unreferenced
* run exhibit-level answers FIRST, apply exhibit lesser-of WITHIN reimbursement_level
* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints to clarify expected output and handling of multiple constraints.
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* add exhibit_lesser_of="N/A" to tests
* Add exhibit_lesser_of parameter to reimbursement_level calls in tests
* restore dynamic answer retrieval for primary, reimbursement info, and code fields
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* "fill in the blank" prompt attempt
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* Merge remote-tracking branch 'origin/main' into feature/sonnet-4
* Merge remote-tracking branch 'origin/feature/sonnet-4' into bugfix/samaritan-missing-rows
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* Remove carveout patterns from has_explicit_lesser_of_carveout function
* Add deduplicate_methodology_breakout function to remove duplicate reimbursement methodologies
* Add deduplication of methodology breakout rows in postprocess function
* Enhance create_exhibit_lesser_of_row to avoid redundancy with substring checks
* Change log level from info to debug for exhibit constraint check in create_exhibit_lesser_of_row
* Remove deduplication of methodology breakout rows from postprocess function (we handle it upstream)
* Merged main into bugfix/samaritan-missing-rows
Approved-by: Katon Minhas
Mayank Aamseek