Bugfix/bill type code fixes
* Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code
- Add format requirements to LOB_RELATIONSHIP_INSTRUCTION for proper caching
- Strengthen LOB_RELATIONSHIP prompt with explicit pipe delimiter requirements
- Preserve all DEBUG blocks and print statements in dynamic_funcs.py and file_processing.py
- Update crosswalk mappings and gitignore
- Ensure format instructions are cached at API level to prevent missing pipe delimiters
* Merge main into bugfix/dynamic_issuefixes
Resolved merge conflicts in:
- fieldExtraction/src/investment/dynamic_funcs.py (merged debug print statements)
- fieldExtraction/src/investment/file_processing.py (merged debug statements and exhibit inheritance code)
All conflicts resolved successfully.
* Update FILL_BILL_TYPE prompt with place-of-service priority and optional exhibit_text
- Add optional exhibit_text parameter to FILL_BILL_TYPE function
- Update instructions to prioritize place-of-service over service-only terminology
- Add new examples: 'home dialysis' and 'dialysis' while keeping original examples
- Add conditional exhibit context section when exhibit_text is provided
- Maintains backward compatibility (exhibit_text defaults to None)
Ref: PLAN_BILL_TYPE_CD_IMPROVEMENTS.md
* Implement two-step bill type CD determination with exhibit text fallback
- Add EXHIBIT_TEXT to answer_dict in combine_one_to_n() for contextual analysis
- Update fill_bill_type() to use two-step approach:
1. First attempt: service term only (no exhibit text)
2. Second attempt: service term + exhibit text (only if first fails)
- Update extract_codes_from_service() to pass exhibit_text to fill_bill_type()
- Add debug print when full exhibit text context is used
- More efficient: only makes second LLM call when service term alone fails
Ref: PLAN_BILL_TYPE_CD_IMPROVEMENTS.md
* Remove debug print statement and merge main into bill_type_code_fixes
- Remove debug print statement from fill_bill_type() in code_funcs.py
- Merge latest changes from main branch
- Includes updates to prompt_templates.py (payer_name parameter)
- Includes updates to file_processing.py (provider info ordering)
- Includes updates to aarete_derived.py, crosswalk mappings, and tin_npi_funcs
* Remove remaining debug print statements from dynamic_funcs.py and file_processing.py
Approved-by: Katon Minhas
Bugfix/cpt codes
* formatting fix for revenue_cd
* format fix for procedure_codes
* bugfix for revenue_codes
* working version from code main
* code formatting from main and code main
* removed dubug prints
* removed local file paths
* standardize state fields
* updated test scenerios
* minor fix
* Merge branch 'main' into bugfix/cpt-codes
merging main into cpt code fix
* checks for all fields
* reused is_empty functionality
* moved normalize_to_json_list to utils
* Merge branch 'main' into bugfix/cpt-codes
merging upto date main
* minor touches
* pytest fix
Approved-by: Katon Minhas
Bugfix/code list format
* replace | with comma in lists
* partial revert
* remove test changes
* Merged main into bugfix/code-list-format
* fix list format
* pipeline error fixed
* pipeline error fixed
* pipeline error fixed
Approved-by: Katon Minhas
Feature/automated cost logging csv
* feat: Add usage and cost tracking with CSV export
- Add usage_tracking.py module for thread-safe token and cost tracking
- Extract actual tokens from Bedrock API responses (replaces word count estimation)
- Integrate usage tracking into llm_utils.py for all LLM invocations
- Add CSV export functionality (per-file/per-model and batch summary)
- Integrate CSV export into main.py for local runs
- Update .gitignore to exclude PRD
Phase 1: Data collection and local CSV export implemented
- Tracks all LLM calls including utility functions (date_fix, derive_term_date)
- Calculates costs per model using centralized cost constants
- Generates two CSV files: USAGE.csv and USAGE-SUMMARY.csv
- Exports to same directory as final results when write_to_s3=False
* - Removed some temporary logging code
- Removed unused imports (uuid, datetime) from llm_utils.py
- Added cache token columns (cache_creation_tokens, cache_read_tokens) to CSV exports
- Added tracking for cache warming
- Updated usage_tracking.py to track cache tokens separately in data structure
All token tracking functionality remains intact.
* Split input tokens into fresh and cache in summary section
- Add total_fresh_input_tokens and total_cache_tokens to GLOBAL_USAGE
- Update summary CSV to include split token columns for better tractability
- Add averages for fresh_input_tokens and cache_tokens
- Maintain total_input_tokens = fresh_input_tokens + cache_tokens relationship
- All calculations verified and match per-file totals correctly
* upload files functionality
* Add comprehensive unit tests for usage_tracking module
- Implement Phase 1-4 tests covering all 10 functions in usage_tracking.py
- Add 98 test cases with parametrized tests for comprehensive coverage
- Fix extract_usage_from_response to handle None input gracefully
- Add python-dotenv dependency to pyproject.toml
- Tests cover: core functions, data access, export functions, and utilities
- All tests passing (98/98)
* Add S3 upload tests for main.py and fix f-string syntax error
- Add tests/test_investment_main_s3.py with 3 test cases:
* test_main_writes_final_and_error_to_s3: validates both final and error DataFrames uploaded when processing has errors
* test_main_writes_only_final_when_no_errors: validates only final DataFrame uploaded when all files succeed
* test_main_writes_usage_when_present: validates usage tracking files uploaded when usage data exists
- Tests use monkeypatching and module stubs to isolate S3 write logic without external dependencies
- Fix syntax error in src/codes/code_funcs.py: nested f-string quotes (level_dict['level_suffix'])
- All tests passing
* Add comprehensive test for usage_tracking.get_usage_dataframes S3 uploads
- Add test_usage_dataframes_written_to_s3 to validate per_file_usage_df and batch_summary_df
- Test verifies both dataframes from usage_tracking.get_usage_dataframes() are written to S3
- Validates 'usage' suffix writes per_file_usage_df with correct schema (file_name, tokens, cost)
- Validates 'usage_summary' suffix writes batch_summary_df with correct schema (totals, averages, region_mode)
- Uses pd.testing.assert_frame_equal to ensure exact dataframes are uploaded
- Fix module caching issue between tests by clearing sys.modules cache
- All 4 tests passing
* Add cleanup fixture to prevent test pollution
- Add autouse pytest fixture to clean up stubbed modules after each test
- Remove manual cache clearing from individual tests (now handled by fixture)
- Prevents our module stubs from affecting other test files in CI/CD pipeline
- Fixes Bitbucket pipeline failures where other tests couldn't find llm_utils attributes
* Refactor tests: add comprehensive io_utils coverage; disable usage_tracking suite
- Added 22 focused tests for write_local and write_s3 plus existing IO behaviors
- Introduced FakeS3Client to avoid boto3 network/credential dependency
- Added preserve_config fixture (no monkeypatch) to isolate config side effects
- Marked test_usage_tracking.py skipped per new consolidation approach
- Verified 26 tests (io_utils + investment_main) all pass locally without AWS exceptions
* Fix S3_CLIENT mocking: use mocker.patch instead of direct assignment
- Changed preserve_config fixture to NOT save/restore S3_CLIENT
- Updated all 5 write_s3 tests to use mocker.patch('src.config.S3_CLIENT', fake)
- This prevents pollution of config.S3_CLIENT across test modules
- Fixes NoCredentialsError in other tests that were importing config after our tests set FakeS3Client
- All 22 io_utils tests pass + 4 investment_main_s3 tests pass
- test_llm_utils::test_invoke_claude_local_mode now passes
* Add tests for write_s3 function
* Resolve merge conflict: keep mocker-based write_s3 tests
* Remove rogue skip mark from test_usage_tracking.py
The skip mark was incorrectly added by remote branch claiming tests were
migrated to io_utils. However, test_io_utils.py only tests io_utils functions
(read/write operations), not usage_tracking functions.
Restore the comprehensive usage_tracking tests from commit 26a556f0 which
includes 98 test cases covering all 10 functions in usage_tracking.py.
* Fix test pollution: add config cleanup fixture and restore RUN_MODE
- Add autouse fixture in test_io_utils.py to restore config values after each test
- Fix test_llm_utils.py to restore RUN_MODE after test_invoke_claude_local_mode
- Prevents config patches from leaking into subsequent tests causing NoCredentialsError
* Remove redundant test_investment_main_s3.py
- File was testing same write_s3 functionality already covered in test_io_utils.py
- Was causing module pollution by stubbing src.utils.llm_utils
- Caused test failures in other test files due to module cache issues
- Integration testing was minimal and mocked write_s3 anyway
* Merged main into feature/automated-cost-logging-csv
Approved-by: Karan Desai
Approved-by: Katon Minhas
bugfix/code_breakout_rev to main
* added methodology to code explicit
* pipeline fixes
* Merged main into bugfix/code_breakout_rev
* Remove prints
Approved-by: Katon Minhas
Feature/black and isort
* black and isort constants
* black and isort codes
* black and isort crosswalks
* black and isort investment
* black and isort prompts
* isort testbed
* isort tracking
* black and isort utils
* poetry and black tests
Approved-by: Katon Minhas
Refactor/one to n
* remove print
* Update logging for fsbreakout and grouper breakout
* Additional type hints
* updated special case primary
* Remove deprecated special case check
* OUTLIER_TERMS-->OUTLIER_TERM standardization
* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test
* Update Carveout IND and Default IND logic
* Split out exhibit-level lesser
* Bugfix
* update lesser of exhibit prompt
* Bugfix
* exhibit lesser of varible name bugfix
* Update exhibit lesser of passing
* Update Exhibit-level prompt template
* Shorten full_context prompt length if the questions are long
* Special case assignment index
* Resolve unit tests
* updated UC IND prompt
* Remove print
* update one-to-n unit test
* added dot based date example
* updated date fix prompt
* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)
Bugfix/json error and unit of measure
* json parser fixes and prompt changes
* universal json load fixed
* json error fixed
* json error fixed 2
* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure
Approved-by: Katon Minhas
Approved-by: Alex Galarce
Feature/grouper cd desc
* added grouper cd desc
* bugfix
* Merge branch 'main' into feature/grouper_cd_desc
* Merged main into feature/grouper_cd_desc
Approved-by: Alex Galarce
Feature/grouper fields
* prompt changes
* print statements removed
* merged with main
* merge issue fixed
* updated grouper prompts
* updated grouper fields in fields order
* updated breakout prompt
* Merged main into feature/grouper-fields
* reverted grouper cd prompt
* updated goruper severity prompt
* Merge remote-tracking branch 'origin/main' into feature/grouper-fields
* updated grouper breakout function
* error fixes
* Merged main into feature/grouper-fields
* Merged main into feature/grouper-fields
* updated grouper_pct_rate and grouper_base_rate
* removed empty array post processing fix
* Merged main into feature/grouper-fields
* post-processing for grouper rate modified
* updated grouper type prompt
* Merged main into feature/grouper-fields
* grouper version prompt updated
* Merged main into feature/grouper-fields
* OPPS as APC
* Merged main into feature/grouper-fields
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Refactor TRIGGER_CAP special case configuration to utilize only trigger cap questions for two-stage processing
* Refactor get_special_cases to improve handling of N/A responses and merge methodology results with special case results
* Fix field type in get_special_cases for fee schedule breakout configuration
* Create fieldset.py - refactor
* Fix placeholders
* string concatenate new questions
* Update trigger cap questions output format
* Update TRIGGER_CAP_THRESHOLD_AMT prompt for clarity and specificity in reimbursement methodology
* Update constants class
* Update Field class
* Resolve constants passing
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Fix unit tests
* Remove test file
Approved-by: Alex Galarce
Bugfix/aug 5 issue fixes
* Revert prompts
* Remove qa_qc tests
* Remove valid lines
* Update crosswalk path
* remove extra argument from split compound reimbursement
* model aliasing
* Haiku alias
* remove erroneous import
* Merge branch 'refactor/update-claude-models' into bugfix/aug-5-issue-fixes
* Convert all matching codes to strings
* Exit if service is empty
* Merged main into bugfix/aug-5-issue-fixes
* global lesser of fix
Approved-by: Alex Galarce
Refactor/one time io
* Delete client work
* Streamline imports
* Fix list class
* Update tests
* Clear code funcs test
* Fix unit tests
* Single read code funcs
* Single-read model
* Single-load embeddings
* Successful E2E test
* update unit tests
* Update importsg
* Reload poetry.lock
* remove old exhibit header function
* Remove aarete_derived generic function
* remove align and format tables
* Remove strings to dict
* references
* Clear code
* Move preprocessing_funcs
* remove keywords
* refactor postprocessing_funcs
* Pass unit test - remove qa_qc directory
* Black and isort
* remove print
Approved-by: Alex Galarce
Praneel Panchigar