jacob-mathison/doczyai-pipelines
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity
1,621 Commits 1 Branch 0 Tags
929ff4abf8148f0acb5dd26ae9e1c2c4f3efeab8
Commit Graph

6 Commits

Author SHA1 Message Date
Katon Minhas afb6d5185d Merged in feature/lesser-table-caching-refactor-hybrid (pull request #847) 
Feature/lesser table caching refactor hybrid

* chore: Remove unused duplicate main.py from shared pipeline

* fix: Correct crosswalk paths in aarete_derived.py

* chore: Remove unused documentation files from fieldExtraction

* docs: Add documentation files to documentation folder

* docs: Update README with uv setup, expanded project structure, and branching conventions

* docs: Add uv installation steps with Ubuntu/WSL emphasis

* Enable prompt caching for all remaining LLM calls

- Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE,
  EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH,
  SPECIAL_CASE_ASSIGNMENT
- Update all invoke_claude() calls in saas and clover pipelines to use
  cache=True with corresponding _INSTRUCTION() functions
- Add new instructions to get_cacheable_instructions() for cache warming
- Update tests for new instruction functions

Functions now using caching:
- prompt_exhibit_level
- prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF)
- prompt_fee_schedule_breakout
- prompt_grouper_breakout
- prompt_special_case_assignment
- prompt_exhibit_linkage
- prompt_exhibit_header
- prompt_smart_chunked (ONE_TO_ONE templates)
- prompt_date_fix
- prompt_derived_term_date
- prompt_exhibit_title_match
- provider_name_match_check

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Reorder

* feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction

- Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction
- Extract full paragraph text of offset/recoupment provisions from contracts
- Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence
- Fix reorder_columns to preserve extra columns not in COLUMN_ORDER
- Update QC/QA output path to outputs/qc_qa/

* fix: Update dev deps and test assertions for QC/QA output path

- Add pytest/pytest-mock to dev dependencies for mypy type checking
- Update test assertions to expect outputs/qc_qa instead of qa_qc_output

* style: Apply black formatting to prompt_templates.py

* Merge main, move scripts

* Archive some scripts

* update py version

* remove .py version file

* Remove ASCII characters

* Restore testbed code

* restore tracking

* Update testbed metrics

* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT

- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix postprocessing_funcs to remove invalid columns

* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid

* Revert prompt caching changes from aed1b73c

* update formatting

* Update imports


Approved-by: Sha Brown
Approved-by: Praneel Panchigar
 2026-01-26 16:52:55 +00:00
Praneel Panchigar 7056c1c687 Merged in feature/automated-cost-logging-csv (pull request #768) 
Feature/automated cost logging csv

* feat: Add usage and cost tracking with CSV export

- Add usage_tracking.py module for thread-safe token and cost tracking
- Extract actual tokens from Bedrock API responses (replaces word count estimation)
- Integrate usage tracking into llm_utils.py for all LLM invocations
- Add CSV export functionality (per-file/per-model and batch summary)
- Integrate CSV export into main.py for local runs
- Update .gitignore to exclude PRD

Phase 1: Data collection and local CSV export implemented
- Tracks all LLM calls including utility functions (date_fix, derive_term_date)
- Calculates costs per model using centralized cost constants
- Generates two CSV files: USAGE.csv and USAGE-SUMMARY.csv
- Exports to same directory as final results when write_to_s3=False

* - Removed some temporary logging code
- Removed unused imports (uuid, datetime) from llm_utils.py
- Added cache token columns (cache_creation_tokens, cache_read_tokens) to CSV exports
- Added tracking for cache warming
- Updated usage_tracking.py to track cache tokens separately in data structure

All token tracking functionality remains intact.

* Split input tokens into fresh and cache in summary section

- Add total_fresh_input_tokens and total_cache_tokens to GLOBAL_USAGE
- Update summary CSV to include split token columns for better tractability
- Add averages for fresh_input_tokens and cache_tokens
- Maintain total_input_tokens = fresh_input_tokens + cache_tokens relationship
- All calculations verified and match per-file totals correctly

* upload files functionality

* Add comprehensive unit tests for usage_tracking module

- Implement Phase 1-4 tests covering all 10 functions in usage_tracking.py
- Add 98 test cases with parametrized tests for comprehensive coverage
- Fix extract_usage_from_response to handle None input gracefully
- Add python-dotenv dependency to pyproject.toml
- Tests cover: core functions, data access, export functions, and utilities
- All tests passing (98/98)

* Add S3 upload tests for main.py and fix f-string syntax error

- Add tests/test_investment_main_s3.py with 3 test cases:
  * test_main_writes_final_and_error_to_s3: validates both final and error DataFrames uploaded when processing has errors
  * test_main_writes_only_final_when_no_errors: validates only final DataFrame uploaded when all files succeed
  * test_main_writes_usage_when_present: validates usage tracking files uploaded when usage data exists
- Tests use monkeypatching and module stubs to isolate S3 write logic without external dependencies
- Fix syntax error in src/codes/code_funcs.py: nested f-string quotes (level_dict['level_suffix'])
- All tests passing

* Add comprehensive test for usage_tracking.get_usage_dataframes S3 uploads

- Add test_usage_dataframes_written_to_s3 to validate per_file_usage_df and batch_summary_df
- Test verifies both dataframes from usage_tracking.get_usage_dataframes() are written to S3
- Validates 'usage' suffix writes per_file_usage_df with correct schema (file_name, tokens, cost)
- Validates 'usage_summary' suffix writes batch_summary_df with correct schema (totals, averages, region_mode)
- Uses pd.testing.assert_frame_equal to ensure exact dataframes are uploaded
- Fix module caching issue between tests by clearing sys.modules cache
- All 4 tests passing

* Add cleanup fixture to prevent test pollution

- Add autouse pytest fixture to clean up stubbed modules after each test
- Remove manual cache clearing from individual tests (now handled by fixture)
- Prevents our module stubs from affecting other test files in CI/CD pipeline
- Fixes Bitbucket pipeline failures where other tests couldn't find llm_utils attributes

* Refactor tests: add comprehensive io_utils coverage; disable usage_tracking suite

- Added 22 focused tests for write_local and write_s3 plus existing IO behaviors
- Introduced FakeS3Client to avoid boto3 network/credential dependency
- Added preserve_config fixture (no monkeypatch) to isolate config side effects
- Marked test_usage_tracking.py skipped per new consolidation approach
- Verified 26 tests (io_utils + investment_main) all pass locally without AWS exceptions

* Fix S3_CLIENT mocking: use mocker.patch instead of direct assignment

- Changed preserve_config fixture to NOT save/restore S3_CLIENT
- Updated all 5 write_s3 tests to use mocker.patch('src.config.S3_CLIENT', fake)
- This prevents pollution of config.S3_CLIENT across test modules
- Fixes NoCredentialsError in other tests that were importing config after our tests set FakeS3Client
- All 22 io_utils tests pass + 4 investment_main_s3 tests pass
- test_llm_utils::test_invoke_claude_local_mode now passes

* Add tests for write_s3 function

* Resolve merge conflict: keep mocker-based write_s3 tests

* Remove rogue skip mark from test_usage_tracking.py

The skip mark was incorrectly added by remote branch claiming tests were
migrated to io_utils. However, test_io_utils.py only tests io_utils functions
(read/write operations), not usage_tracking functions.

Restore the comprehensive usage_tracking tests from commit 26a556f0 which
includes 98 test cases covering all 10 functions in usage_tracking.py.

* Fix test pollution: add config cleanup fixture and restore RUN_MODE

- Add autouse fixture in test_io_utils.py to restore config values after each test
- Fix test_llm_utils.py to restore RUN_MODE after test_invoke_claude_local_mode
- Prevents config patches from leaking into subsequent tests causing NoCredentialsError

* Remove redundant test_investment_main_s3.py

- File was testing same write_s3 functionality already covered in test_io_utils.py
- Was causing module pollution by stubbing src.utils.llm_utils
- Caused test failures in other test files due to module cache issues
- Integration testing was minimal and mocked write_s3 anyway

* Merged main into feature/automated-cost-logging-csv


Approved-by: Karan Desai
Approved-by: Katon Minhas
 2025-11-10 19:29:45 +00:00
VenkataKrishna Reddy Avula 0d6bdf9539 Merged in bugfix/effective_date_fix (pull request #504) 
DRAFT: Bugfix/effective date fix

* Merged in bugfix/extra-fields (pull request #500)

Do not run Exhibit level prompt if there are no exhibit level fields

* Do not run Exhibit level prompt if there are no exhibit level fields

* Update dynamic test

* Remove test from one-to-n test module

* Update test

* Fix unit test


Approved-by: Alex Galarce

* Merged in bugfix/tin-npi-prompt-fixes (pull request #499)

Bugfix/tin npi prompt fixes

* Change delimiter for other provider information from comma to pipe

* filter out invalid providers

* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE

* Merged main into bugfix/tin-npi-small-bugfixes

* Refactor deduplication logic to use string_utils for checking empty provider fields

* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes

* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME

* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes

* Fix deduplication logic to not drop subsequent names

* Fix formatting of provider names in deduplication logic

* add payer name filtering

* reorder execution order to allow to find payer_name first

* Add handling for previously unfound group providers in identify_group_provider function

* Merge remote-trackin…
* Merged in feature/update-testbed-script (pull request #501)

Feature/update testbed script

* Update preprocessing

* Update preprocessing

* remove quit

* Update date for filename

* Merged main into feature/update-testbed-script


Approved-by: Alex Galarce

* Merged in bugfix/smart-chunks (pull request #503)

missing of text in chunks fixed

* missing of text in chunks fixed


Approved-by: Katon Minhas

* Merged in feature/rework-reimb-tin-npi (pull request #502)

Feature/rework reimb tin npi

* use LLM for reimb_[tin/npi/name]

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* deduplicate tin/name/npi before adding to prompt

* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging

* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly

* Reorder REIMB_PROV_NAME field in investment_prompts.json

* Update REIMB_PROV_NAME prompt to specify valid values for provider names

* Remove valid_values constraint from prov_name to raise accuracy

* Remove debugging print statements from reimbursement_tin_npi and related functions

* Remove debugging print statements from run_provider_info_fields function

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* Remove debugging print statem…
* Merged in feature/preformat-single-quotes-input (pull request #506)

Replace double quotes with single quotes in context strings for parsability

* Replace double quotes with single quotes in context strings for parsability

* Merged main into feature/preformat-single-quotes-input


Approved-by: Katon Minhas

* updated sig pages fnxn

* Merge branch 'main' into bugfix/effective_date_fix

* poetry add pymupdf

* created vision funcs

* Merged main into bugfix/effective_date_fix

* fixed pipeline issue

* updated string_utils_test

* updated test values

* updated extract signature page fxn

* updated test values

* updated test values

* Merged main into bugfix/effective_date_fix

* Prompt template in all-caps

* remove debugging prints

* Merged main into bugfix/effective_date_fix

* Change vision to False by default

* Change vision to False by default

* fix tests

* fix tests


Approved-by: Katon Minhas
 2025-05-07 18:21:09 +00:00
Alex Galarce 56e3fdd1f8 Merged in bugfix/all_cnc_hotfixes (pull request #302) 
DRAFT: Bugfix/all cnc hotfixes

* Pipeline fix - health plan state

* Update for prompt vs non-prompt fixes

* Revert utils.find_regex_matches()

* Remove prints

* Non-prompt run

* Health Plan State, IRS, NPI, LOB, Lesser, Prov 2, Default, Rate Standard, Contract Effective Date, Term Group

* Merged in bugfix/agreement_name (pull request #294)

contract title fixed

* contract title fixed

* Merged bugfix/all_cnc_hotfixes into bugfix/agreement_name


Approved-by: Katon Minhas

* All hotfixes added

* Fixed utils

* fix for clean_contract_effective_date
* Merged in default-fix (pull request #296)

moved default funcs to the top, dropped intermediate columns

* moved default funcs to the top, dropped intermediate columns


Approved-by: Katon Minhas

* quick fix to dropping extra columns

* Merged in bugfix/effective_date_meridian (pull request #297)

fixed effective date for meridian contracts

* fixed effective date for meridian contracts


Approved-by: Katon Minhas

* Default rate and Lesser fix

* Merged in hotfix/irs_npi_updated (pull request #298)

updated irs and npi hotfix funcs

* updated irs and npi hotfix funcs

* Merged bugfix/all_cnc_hotfixes into hotfix/irs_npi_updated


Approved-by: Katon Minhas

* non-functional commit - default only

* debug commit - premerge

* Batch 1 Dup File Reconciliation

* Default rate fix

* Full CNC 1A Run Config

* Standard column refactor

* Working stitching for 1A

* Merged in Alex-Galarce/cnc_hotfix_effective_date_utilspy-edited-1732659983687 (pull request #301)

Remove "license" from effective date smart chunking

* Remove "license" from effective date smart chunking
* Merged bugfix/all_cnc_hotfixes into Alex-Galarce/cnc_hotfix_effective_date_utilspy-edited-1732659983687


Approved-by: Katon Minhas

* Merged in hotfix/irs_batch2_feedback (pull request #300)

IRS fix for feedback on batch2

* IRS fix for feedback on batch2

* Merged bugfix/all_cnc_hotfixes into hotfix/irs_batch2_feedback


Approved-by: Katon Minhas

* Merged main into bugfix/all_cnc_hotfixes


Approved-by: Katon Minhas
 2024-11-27 15:08:04 +00:00
Michael McGuinness 815fa7f0bc Merged in feature/poetry (pull request #238) 
Draft: Adding Poetry

* addpoetry

* pipes

* mypy

* streamlit

* searchforchange


Approved-by: Alex Galarce
 2024-10-15 16:20:27 +00:00
Michael McGuinness 02169d8a5e add reqs 2024-09-30 12:30:30 +01:00