Bugfix/Molina UT conversion factor
* handle multiple default_terms in one reimb_term
* remove incorrect trigger cap cases
* carveout_cd fixes
* Merge branch 'main' into bugfix/Molina-UT-carveout-cd
* fix variable naming
* methodology prompt updated
* Merge branch 'main' into bugfix/Molina-UT-carveout-cd
* revert test changes
* revert test changes
* fixed incorrect conversion factor
* Merge branch 'main' into bugfix/Molina-UT-conversion-factor
* test to fix service term
* Merge branch 'main' into bugfix/Molina-UT-conversion-factor
* revert test changes
* Merged main into bugfix/Molina-UT-conversion-factor
Approved-by: Katon Minhas
Bugfix/outlier terms to main
* added split sepcial term
* additional instrcution added
* Merged main into bugfix/outlier_terms
* modified special case primary
* removed split special case primary
* remove additional instruction
* remove print statements
* removed print statements
* updated special case primary prompt
* Merged main into bugfix/outlier_terms
Approved-by: Katon Minhas
Feature/dtc clean
* dtc v1 initial commit
* second layer of prompt
* Added MAX_PAGES_TO_CHECK and page looping to get answer from llm
* Edited contract_type prompt to get cleaner output with no additional text
* Updated document_type_classification prompt
* Added non-contract type classifcation
* Added dtc_v1_test.py for testing only
* Updated prompt
* Updated prompt
* at scale changes
* add save feature for individual file output
* update dockerfile
* clean dtc to prepare for merge
* clean up dtc script
* Merge remote-tracking branch 'origin/main' into feature/dtc-clean
* reafactor code
* Merge remote-tracking branch 'origin/main' into feature/dtc-clean
Approved-by: Katon Minhas
Feature/automated cost logging csv
* feat: Add usage and cost tracking with CSV export
- Add usage_tracking.py module for thread-safe token and cost tracking
- Extract actual tokens from Bedrock API responses (replaces word count estimation)
- Integrate usage tracking into llm_utils.py for all LLM invocations
- Add CSV export functionality (per-file/per-model and batch summary)
- Integrate CSV export into main.py for local runs
- Update .gitignore to exclude PRD
Phase 1: Data collection and local CSV export implemented
- Tracks all LLM calls including utility functions (date_fix, derive_term_date)
- Calculates costs per model using centralized cost constants
- Generates two CSV files: USAGE.csv and USAGE-SUMMARY.csv
- Exports to same directory as final results when write_to_s3=False
* - Removed some temporary logging code
- Removed unused imports (uuid, datetime) from llm_utils.py
- Added cache token columns (cache_creation_tokens, cache_read_tokens) to CSV exports
- Added tracking for cache warming
- Updated usage_tracking.py to track cache tokens separately in data structure
All token tracking functionality remains intact.
* Split input tokens into fresh and cache in summary section
- Add total_fresh_input_tokens and total_cache_tokens to GLOBAL_USAGE
- Update summary CSV to include split token columns for better tractability
- Add averages for fresh_input_tokens and cache_tokens
- Maintain total_input_tokens = fresh_input_tokens + cache_tokens relationship
- All calculations verified and match per-file totals correctly
* upload files functionality
* Add comprehensive unit tests for usage_tracking module
- Implement Phase 1-4 tests covering all 10 functions in usage_tracking.py
- Add 98 test cases with parametrized tests for comprehensive coverage
- Fix extract_usage_from_response to handle None input gracefully
- Add python-dotenv dependency to pyproject.toml
- Tests cover: core functions, data access, export functions, and utilities
- All tests passing (98/98)
* Add S3 upload tests for main.py and fix f-string syntax error
- Add tests/test_investment_main_s3.py with 3 test cases:
* test_main_writes_final_and_error_to_s3: validates both final and error DataFrames uploaded when processing has errors
* test_main_writes_only_final_when_no_errors: validates only final DataFrame uploaded when all files succeed
* test_main_writes_usage_when_present: validates usage tracking files uploaded when usage data exists
- Tests use monkeypatching and module stubs to isolate S3 write logic without external dependencies
- Fix syntax error in src/codes/code_funcs.py: nested f-string quotes (level_dict['level_suffix'])
- All tests passing
* Add comprehensive test for usage_tracking.get_usage_dataframes S3 uploads
- Add test_usage_dataframes_written_to_s3 to validate per_file_usage_df and batch_summary_df
- Test verifies both dataframes from usage_tracking.get_usage_dataframes() are written to S3
- Validates 'usage' suffix writes per_file_usage_df with correct schema (file_name, tokens, cost)
- Validates 'usage_summary' suffix writes batch_summary_df with correct schema (totals, averages, region_mode)
- Uses pd.testing.assert_frame_equal to ensure exact dataframes are uploaded
- Fix module caching issue between tests by clearing sys.modules cache
- All 4 tests passing
* Add cleanup fixture to prevent test pollution
- Add autouse pytest fixture to clean up stubbed modules after each test
- Remove manual cache clearing from individual tests (now handled by fixture)
- Prevents our module stubs from affecting other test files in CI/CD pipeline
- Fixes Bitbucket pipeline failures where other tests couldn't find llm_utils attributes
* Refactor tests: add comprehensive io_utils coverage; disable usage_tracking suite
- Added 22 focused tests for write_local and write_s3 plus existing IO behaviors
- Introduced FakeS3Client to avoid boto3 network/credential dependency
- Added preserve_config fixture (no monkeypatch) to isolate config side effects
- Marked test_usage_tracking.py skipped per new consolidation approach
- Verified 26 tests (io_utils + investment_main) all pass locally without AWS exceptions
* Fix S3_CLIENT mocking: use mocker.patch instead of direct assignment
- Changed preserve_config fixture to NOT save/restore S3_CLIENT
- Updated all 5 write_s3 tests to use mocker.patch('src.config.S3_CLIENT', fake)
- This prevents pollution of config.S3_CLIENT across test modules
- Fixes NoCredentialsError in other tests that were importing config after our tests set FakeS3Client
- All 22 io_utils tests pass + 4 investment_main_s3 tests pass
- test_llm_utils::test_invoke_claude_local_mode now passes
* Add tests for write_s3 function
* Resolve merge conflict: keep mocker-based write_s3 tests
* Remove rogue skip mark from test_usage_tracking.py
The skip mark was incorrectly added by remote branch claiming tests were
migrated to io_utils. However, test_io_utils.py only tests io_utils functions
(read/write operations), not usage_tracking functions.
Restore the comprehensive usage_tracking tests from commit 26a556f0 which
includes 98 test cases covering all 10 functions in usage_tracking.py.
* Fix test pollution: add config cleanup fixture and restore RUN_MODE
- Add autouse fixture in test_io_utils.py to restore config values after each test
- Fix test_llm_utils.py to restore RUN_MODE after test_invoke_claude_local_mode
- Prevents config patches from leaking into subsequent tests causing NoCredentialsError
* Remove redundant test_investment_main_s3.py
- File was testing same write_s3 functionality already covered in test_io_utils.py
- Was causing module pollution by stubbing src.utils.llm_utils
- Caused test failures in other test files due to module cache issues
- Integration testing was minimal and mocked write_s3 anyway
* Merged main into feature/automated-cost-logging-csv
Approved-by: Karan Desai
Approved-by: Katon Minhas
bugfix/table_reimb_term to main
* table instruction added
* Merged main into bugfix/table_reimb_term
* Merged main into bugfix/table_reimb_term
* Merged main into bugfix/table_reimb_term
Approved-by: Katon Minhas
Bugfix/ut errors
* Update universal_json_load and unit tests
* Update tests
* Fix unit tests
* remove test
* remove imports
* Update function
* update pd Series is_empty
* Update reimbursement check
* added some test cases
* Merged main into bugfix/ut-errors
* Merged main into bugfix/ut-errors
* remove import
* Merged main into bugfix/ut-errors
Approved-by: Karan Desai
Feature/update testbed metrics
* Clean up filename filter function
* Strip out unused tests
* Strip out page-level comparison
* Add 1:1 fields back to analysis:
* Run through
* Merged main into feature/update-testbed-metrics
* Merged main into feature/update-testbed-metrics
* Merged main into feature/update-testbed-metrics
* Merged main into feature/update-testbed-metrics
* Merged main into feature/update-testbed-metrics
Approved-by: Faizan Mohiuddin
Bugfix/reimb excess language issues to main
* updated trigger cap def
* updated trigger cap carveout def
* Merged main into bugfix/reimb_excess_language_issues
* updated split reimb prompt
* remove print statements
* Merged main into bugfix/reimb_excess_language_issues
* updated split reimb prompt with complex example
Approved-by: Katon Minhas
Feature/effective dt
* move prompt_smart_chunked to prompt_calls.py
* refactor pass_smart_chunked_to_full_context
* Mypy fix
* Update field to EFFECTIVE_DT
* move to prompt_calls
* Update vision funcs
* Restructure prompt-based date derivation
* E2E passed
* Remove test
* Merge branch 'main' into feature/effective-dt
* remove deprecated postprocessing_func
* Docstrings
* refactor - send global lesser of to prompt_calls.py
* remove deprecated function
* more docstrings
* Merged main into feature/effective-dt
* Merged main into feature/effective-dt
* added base field for aarete derived effective date
Approved-by: VenkataKrishna Reddy Avula
Cache only for necessary prompts
* Cache only for necessary prompts
* Merged main into feature/cache-field-split
* Merged main into feature/cache-field-split
Approved-by: Faizan Mohiuddin
Bugfix/claim type cd to main
* test claim type cd
* Merged main into bugfix/claim_type_cd
* updated claim type prompt
* updated prompt
* empty one to n results initiated
* remove print statements
* src/investment/prompt_calls.py
* full context prompt update
Approved-by: Katon Minhas
new functionality to generate SID
* new functionality to generate SID
* replacing xx with '' if new col created
* Merged main into feature/attach_sid
Approved-by: Katon Minhas
bugfix/prov_tin_npi_name_issues to main
* updated provider group full name
* removed print statements
* Merged main into bugfix/prov_tin_npi_name_issues
Approved-by: Katon Minhas
Bugfix/mi reimb terms
* test changes
* logging added
* Merge branch 'main' into bugfix/generic-reimb-terms
* parsing fixes
* fix missing reimbursements
* make service terms unique by adding additional context
* Merge branch 'main' into bugfix/mi-reimb-terms
* remomoving test changes
* remomoving test changes
* remomoving test changes
* test case for universal_json_load added
Approved-by: Katon Minhas
bugfix/code_breakout_rev to main
* added methodology to code explicit
* pipeline fixes
* Merged main into bugfix/code_breakout_rev
* Remove prints
Approved-by: Katon Minhas
Bugfix/molina nv reimb method issues to main
* added asc and apc full forms in prompt
* updated grouper instructions for reimb method
* updated reimb method prompt
* Merged main into bugfix/molina_nv_reimb_method_issues
Approved-by: Katon Minhas
Bugfix/process only after title
* Update print
* Update exhibit dict creation
* Detect Participating Provider Agreement
* Remove test file
* Merged main into bugfix/process-only-after-title
* Merged main into bugfix/process-only-after-title
Approved-by: Faizan Mohiuddin
Feature/field type config
* Added field type procesing conditional logic
* Updated field type naming convention in config.py; Updated date_postprocessing to avoid column key missing issue
* Cleaned up code; Move merging functionality to one to one processing block
* Merged main into feature/field_type_config
Approved-by: Katon Minhas
Bugfix/ut issues
* Update exhibit linking
* prompt_calls refactor
* remove test
* Add Healthy Kids Program
* Move Healthy Kids to UT specifically
* Remove header check from Dynamic process
* Merged main into bugfix/ut-issues
* prep for merge
* remove deprecated tests
* Merged main into bugfix/ut-issues
* Merged main into bugfix/ut-issues
Approved-by: Faizan Mohiuddin
Bugfix/special cases to Main
* prompt update
* Merge branch 'main' into bugfix/json_and_unit_of_measure
* minor bugfix
* minor bugfix
* prompt update for outlier rate
* added outlier testing jupyter notebook
* Remove special case from any empty
* updated outlier breakout prompt
* Merged main into bugfix/special-cases
* reimb_method minor fix
* Merged main into bugfix/special-cases
* bugfix field to add dynamic funcs
* merged with main
* latest outlier changes
* Merged main into bugfix/special-cases
* added table guidelines for special case primary
* updated special case primary prompt
* Merged main into bugfix/special-cases
* updated special case primary table handling
* merged main into bugfix special cases
* Merged main into bugfix/special-cases
* Remove test notebooks from tracking
* trying out pipeline fix
* reverted chnages
* pipeline fix
Approved-by: Katon Minhas
Bugfix/mi issue fixes
* Update mapping to allow for , or | lists
* Pipe-separation for full-context
* Update VALIDATE_REIMBURSEMENTS
* Designate Provider Submission Form as cover sheet language - make cover sheet detection more efficient
* remove deprecated functions
* remove test
Approved-by: Faizan Mohiuddin
Bugfix/mi dynamic
* Refactor - split out one_to_n_cleaning
* Rework postprocess
* Update postprocessing - fill dynamic values if there is only 1 non-empty value
* Update to use string_utils
* Group by AARETE_DERIVED_LOB as well
* Update crosswalk mapping to handle | lists
* Update constants - run with MI flag
Approved-by: Siddhant Medar
bugfix/molina_tracker_issues to main
* provider name updated prompt
* updated provider name prompt
* Merged main into bugfix/molina_tracker_issues
Approved-by: Katon Minhas
VenkataKrishna Reddy Avula