Merged in feature/automated-cost-logging-csv (pull request #768)

Feature/automated cost logging csv

* feat: Add usage and cost tracking with CSV export

- Add usage_tracking.py module for thread-safe token and cost tracking
- Extract actual tokens from Bedrock API responses (replaces word count estimation)
- Integrate usage tracking into llm_utils.py for all LLM invocations
- Add CSV export functionality (per-file/per-model and batch summary)
- Integrate CSV export into main.py for local runs
- Update .gitignore to exclude PRD

Phase 1: Data collection and local CSV export implemented
- Tracks all LLM calls including utility functions (date_fix, derive_term_date)
- Calculates costs per model using centralized cost constants
- Generates two CSV files: USAGE.csv and USAGE-SUMMARY.csv
- Exports to same directory as final results when write_to_s3=False

* - Removed some temporary logging code
- Removed unused imports (uuid, datetime) from llm_utils.py
- Added cache token columns (cache_creation_tokens, cache_read_tokens) to CSV exports
- Added tracking for cache warming
- Updated usage_tracking.py to track cache tokens separately in data structure

All token tracking functionality remains intact.

* Split input tokens into fresh and cache in summary section

- Add total_fresh_input_tokens and total_cache_tokens to GLOBAL_USAGE
- Update summary CSV to include split token columns for better tractability
- Add averages for fresh_input_tokens and cache_tokens
- Maintain total_input_tokens = fresh_input_tokens + cache_tokens relationship
- All calculations verified and match per-file totals correctly

* upload files functionality

* Add comprehensive unit tests for usage_tracking module

- Implement Phase 1-4 tests covering all 10 functions in usage_tracking.py
- Add 98 test cases with parametrized tests for comprehensive coverage
- Fix extract_usage_from_response to handle None input gracefully
- Add python-dotenv dependency to pyproject.toml
- Tests cover: core functions, data access, export functions, and utilities
- All tests passing (98/98)

* Add S3 upload tests for main.py and fix f-string syntax error

- Add tests/test_investment_main_s3.py with 3 test cases:
  * test_main_writes_final_and_error_to_s3: validates both final and error DataFrames uploaded when processing has errors
  * test_main_writes_only_final_when_no_errors: validates only final DataFrame uploaded when all files succeed
  * test_main_writes_usage_when_present: validates usage tracking files uploaded when usage data exists
- Tests use monkeypatching and module stubs to isolate S3 write logic without external dependencies
- Fix syntax error in src/codes/code_funcs.py: nested f-string quotes (level_dict['level_suffix'])
- All tests passing

* Add comprehensive test for usage_tracking.get_usage_dataframes S3 uploads

- Add test_usage_dataframes_written_to_s3 to validate per_file_usage_df and batch_summary_df
- Test verifies both dataframes from usage_tracking.get_usage_dataframes() are written to S3
- Validates 'usage' suffix writes per_file_usage_df with correct schema (file_name, tokens, cost)
- Validates 'usage_summary' suffix writes batch_summary_df with correct schema (totals, averages, region_mode)
- Uses pd.testing.assert_frame_equal to ensure exact dataframes are uploaded
- Fix module caching issue between tests by clearing sys.modules cache
- All 4 tests passing

* Add cleanup fixture to prevent test pollution

- Add autouse pytest fixture to clean up stubbed modules after each test
- Remove manual cache clearing from individual tests (now handled by fixture)
- Prevents our module stubs from affecting other test files in CI/CD pipeline
- Fixes Bitbucket pipeline failures where other tests couldn't find llm_utils attributes

* Refactor tests: add comprehensive io_utils coverage; disable usage_tracking suite

- Added 22 focused tests for write_local and write_s3 plus existing IO behaviors
- Introduced FakeS3Client to avoid boto3 network/credential dependency
- Added preserve_config fixture (no monkeypatch) to isolate config side effects
- Marked test_usage_tracking.py skipped per new consolidation approach
- Verified 26 tests (io_utils + investment_main) all pass locally without AWS exceptions

* Fix S3_CLIENT mocking: use mocker.patch instead of direct assignment

- Changed preserve_config fixture to NOT save/restore S3_CLIENT
- Updated all 5 write_s3 tests to use mocker.patch('src.config.S3_CLIENT', fake)
- This prevents pollution of config.S3_CLIENT across test modules
- Fixes NoCredentialsError in other tests that were importing config after our tests set FakeS3Client
- All 22 io_utils tests pass + 4 investment_main_s3 tests pass
- test_llm_utils::test_invoke_claude_local_mode now passes

* Add tests for write_s3 function

* Resolve merge conflict: keep mocker-based write_s3 tests

* Remove rogue skip mark from test_usage_tracking.py

The skip mark was incorrectly added by remote branch claiming tests were
migrated to io_utils. However, test_io_utils.py only tests io_utils functions
(read/write operations), not usage_tracking functions.

Restore the comprehensive usage_tracking tests from commit 26a556f0 which
includes 98 test cases covering all 10 functions in usage_tracking.py.

* Fix test pollution: add config cleanup fixture and restore RUN_MODE

- Add autouse fixture in test_io_utils.py to restore config values after each test
- Fix test_llm_utils.py to restore RUN_MODE after test_invoke_claude_local_mode
- Prevents config patches from leaking into subsequent tests causing NoCredentialsError

* Remove redundant test_investment_main_s3.py

- File was testing same write_s3 functionality already covered in test_io_utils.py
- Was causing module pollution by stubbing src.utils.llm_utils
- Caused test failures in other test files due to module cache issues
- Integration testing was minimal and mocked write_s3 anyway

* Merged main into feature/automated-cost-logging-csv


Approved-by: Karan Desai
Approved-by: Katon Minhas

fieldExtraction/src/codes/code_funcs.py

@@ -356,13 +356,13 @@ def code_implicit_rag(service, implicit_run_dict, filename, constants):
             code_answer_dict["PROCEDURE_CD"] = "|".join(proc_codes)
             code_answer_dict["PROCEDURE_CD_DESC"] = "|".join(proc_descs)
             code_answer_dict["CODE_METHODOLOGY"] = (
-                f"Implicit - Level {level_dict["level_suffix"]}"
+                f"Implicit - Level {level_dict['level_suffix']}"
             )
         if rev_codes:
             code_answer_dict["REVENUE_CD"] = "|".join(rev_codes)
             code_answer_dict["REVENUE_CD_DESC"] = "|".join(rev_descs)
             code_answer_dict["CODE_METHODOLOGY"] = (
-                f"Implicit - Level {level_dict["level_suffix"]}"
+                f"Implicit - Level {level_dict['level_suffix']}"
             )
 
         if code_answer_dict: