0d00d581b111feb3f5446dab766a45c358e518fd
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Katon Minhas
|
deef52193c |
Merged in DEV (pull request #873)
DEV * file_processing * Merge branch 'bugfix/code_implicit' into DEV * Update split reimb dates condition * Pipe line general fixes, fixed tests, ran black, and fixed mypy type check issues, still have a list of lists issue with prov other name full * Merged in feature/update-claim-type (pull request #869) Feature/update claim type * Update CLAIM_TYPE_CD extraction to prioritize title/header - Modified retrieval_question to search for title, header, and agreement name - Updated prompt to check title/header first before analyzing body text - Enhanced FULL_CONTEXT_CLAIM_TYPES_ADDITIONAL_INSTRUCTION to emphasize title/header as primary source * Fix CLAIM_TYPE_CD extraction with 3-step fallback 1. Exhibit Level: Updated retrieval_question and prompt to prioritize title/header 2. Contract Title fallback: Added infer_claim_type_from_title() function that extracts claim type from CONTRACT_TITLE when exhibit-level extraction returns empty 3. Postprocessing: fill_claim_type() now uses CONTRACT_TITLE inference when all AARETE_DERIVED_CLAIM_TYPE_CD values are empty Keywords mapped: - Professional/Ancillary -> M (Physician, Professional, Ancillary, Home Health, DME, etc.) - Institutional -> H (Hospital, Facility, Surgery Center, SNF, etc.) * Add more prof… * Merged in bugfix/code-funcs (pull request #872) Bugfix/code funcs * Try-except code extraction * Additional try-except for protection * Black format Approved-by: Praneel Panchigar * Merged in feature/update-testbed-metrics (pull request #871) Feature/update testbed metrics * Combine fee schedule with methodology breakout; remove Reimbursement primary; do not run trigger cap or additions * Deprecate reimb primary functions * combine multiple sheets to one * Clean 1:N metrics * Improve dynamic primary evaluation * Resolve dynamic primary * Update row counts * black format * Merged DEV into feature/update-testbed-metrics Approved-by: Praneel Panchigar * Merged in feature/adding_contract_admenment_num_pc (pull request #870) Feature/adding contract admenment num pc * Updated contract amendment * Format code with Black * updated to have letters * format fixes * Adjusted unit testing * Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc * apply formatting * format fixes * Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc * test conflicts solved * Format code with Black * format fixes * reverted the changes for prompt templates * test_case reverted * Sorted effective date for ranking * format fix for lint * Merged DEV into feature/adding_contract_admenment_num_pc * format * Merged DEV into feature/adding_contract_admenment_num_pc * Merged DEV into feature/adding_contract_admenment_num_pc Approved-by: Siddhant Medar * Consolidate docs and documentation * Remove PRD * Add Essential Plan as a NY Program * Resolve Comments * Resolve comments * Black format * Resolve remaining comments * Black format * Merged in bugfix/fill_claim_type (pull request #874) Bugfix/fill claim type * Handle if AARETE_DERIVED_CLAIM_TYPE_CD comes in as list * add unit tests * Black format Approved-by: Siddhant Medar * Remove print * Merged in bugfix/parser-downstream-improvements (pull request #875) Bugfix/parser downstream improvements * Refactor: Implement field-aware JSON parsers with centralized normalization This refactor introduces a robust system for normalizing LLM output based on field format mappings, ensuring consistent data types throughout the pipeline. Key Changes: - Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields - Create format_normalization.py utility for type-aware normalization - Update json_utils.py parsers to accept field_names/field_name parameters - Refactor prompt_templates.py to use parser factories (_create_json_dict_parser, _create_json_list_parser) that bind field metadata for automatic normalization - Update prompt_calls.py to pass field names to parsers, eliminating redundant normalization logic - Remove parse_json_dict_or_list (unused, ambiguous function) - Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions - Add comprehensive integration tests… * Update other lists * Black format * Remove prints * Update qa_qc to account for lists in is_empty check * Resolve issues * Black format Approved-by: Siddhant Medar |
||
|
Siddhant Medar
|
a5c743f37f |
Merged in feature/chc-client (pull request #857)
Feature/chc client * Add CHC client pipeline with Claims Coding/Editing Determinations extraction - Add CHC client to pipelines with regex-based extraction - Create extraction/claims_coding_funcs.py for section extraction - Add chc_prompts.json with non_prompt field type - Add CHC_PROMPTS_PATH to config - Add tests for extraction function (11 test cases) * Add black[jupyter] for code formatting * Format testing notebook with black * Remove unused prompt field from chc_prompts.json * Merged main into feature/chc-client Approved-by: Katon Minhas |
||
|
Katon Minhas
|
afb6d5185d |
Merged in feature/lesser-table-caching-refactor-hybrid (pull request #847)
Feature/lesser table caching refactor hybrid * chore: Remove unused duplicate main.py from shared pipeline * fix: Correct crosswalk paths in aarete_derived.py * chore: Remove unused documentation files from fieldExtraction * docs: Add documentation files to documentation folder * docs: Update README with uv setup, expanded project structure, and branching conventions * docs: Add uv installation steps with Ubuntu/WSL emphasis * Enable prompt caching for all remaining LLM calls - Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE, EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH, SPECIAL_CASE_ASSIGNMENT - Update all invoke_claude() calls in saas and clover pipelines to use cache=True with corresponding _INSTRUCTION() functions - Add new instructions to get_cacheable_instructions() for cache warming - Update tests for new instruction functions Functions now using caching: - prompt_exhibit_level - prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF) - prompt_fee_schedule_breakout - prompt_grouper_breakout - prompt_special_case_assignment - prompt_exhibit_linkage - prompt_exhibit_header - prompt_smart_chunked (ONE_TO_ONE templates) - prompt_date_fix - prompt_derived_term_date - prompt_exhibit_title_match - provider_name_match_check 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Reorder * feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction - Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction - Extract full paragraph text of offset/recoupment provisions from contracts - Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence - Fix reorder_columns to preserve extra columns not in COLUMN_ORDER - Update QC/QA output path to outputs/qc_qa/ * fix: Update dev deps and test assertions for QC/QA output path - Add pytest/pytest-mock to dev dependencies for mypy type checking - Update test assertions to expect outputs/qc_qa instead of qa_qc_output * style: Apply black formatting to prompt_templates.py * Merge main, move scripts * Archive some scripts * update py version * remove .py version file * Remove ASCII characters * Restore testbed code * restore tracking * Update testbed metrics * Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT - Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification - Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination - Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification - Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT - Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK - Add new instructions to get_cacheable_instructions() for cache warming - Add unit tests for new instruction functions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix postprocessing_funcs to remove invalid columns * Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid * Revert prompt caching changes from aed1b73c * update formatting * Update imports Approved-by: Sha Brown Approved-by: Praneel Panchigar |
||
|
Praneel Panchigar
|
7056c1c687 |
Merged in feature/automated-cost-logging-csv (pull request #768)
Feature/automated cost logging csv
* feat: Add usage and cost tracking with CSV export
- Add usage_tracking.py module for thread-safe token and cost tracking
- Extract actual tokens from Bedrock API responses (replaces word count estimation)
- Integrate usage tracking into llm_utils.py for all LLM invocations
- Add CSV export functionality (per-file/per-model and batch summary)
- Integrate CSV export into main.py for local runs
- Update .gitignore to exclude PRD
Phase 1: Data collection and local CSV export implemented
- Tracks all LLM calls including utility functions (date_fix, derive_term_date)
- Calculates costs per model using centralized cost constants
- Generates two CSV files: USAGE.csv and USAGE-SUMMARY.csv
- Exports to same directory as final results when write_to_s3=False
* - Removed some temporary logging code
- Removed unused imports (uuid, datetime) from llm_utils.py
- Added cache token columns (cache_creation_tokens, cache_read_tokens) to CSV exports
- Added tracking for cache warming
- Updated usage_tracking.py to track cache tokens separately in data structure
All token tracking functionality remains intact.
* Split input tokens into fresh and cache in summary section
- Add total_fresh_input_tokens and total_cache_tokens to GLOBAL_USAGE
- Update summary CSV to include split token columns for better tractability
- Add averages for fresh_input_tokens and cache_tokens
- Maintain total_input_tokens = fresh_input_tokens + cache_tokens relationship
- All calculations verified and match per-file totals correctly
* upload files functionality
* Add comprehensive unit tests for usage_tracking module
- Implement Phase 1-4 tests covering all 10 functions in usage_tracking.py
- Add 98 test cases with parametrized tests for comprehensive coverage
- Fix extract_usage_from_response to handle None input gracefully
- Add python-dotenv dependency to pyproject.toml
- Tests cover: core functions, data access, export functions, and utilities
- All tests passing (98/98)
* Add S3 upload tests for main.py and fix f-string syntax error
- Add tests/test_investment_main_s3.py with 3 test cases:
* test_main_writes_final_and_error_to_s3: validates both final and error DataFrames uploaded when processing has errors
* test_main_writes_only_final_when_no_errors: validates only final DataFrame uploaded when all files succeed
* test_main_writes_usage_when_present: validates usage tracking files uploaded when usage data exists
- Tests use monkeypatching and module stubs to isolate S3 write logic without external dependencies
- Fix syntax error in src/codes/code_funcs.py: nested f-string quotes (level_dict['level_suffix'])
- All tests passing
* Add comprehensive test for usage_tracking.get_usage_dataframes S3 uploads
- Add test_usage_dataframes_written_to_s3 to validate per_file_usage_df and batch_summary_df
- Test verifies both dataframes from usage_tracking.get_usage_dataframes() are written to S3
- Validates 'usage' suffix writes per_file_usage_df with correct schema (file_name, tokens, cost)
- Validates 'usage_summary' suffix writes batch_summary_df with correct schema (totals, averages, region_mode)
- Uses pd.testing.assert_frame_equal to ensure exact dataframes are uploaded
- Fix module caching issue between tests by clearing sys.modules cache
- All 4 tests passing
* Add cleanup fixture to prevent test pollution
- Add autouse pytest fixture to clean up stubbed modules after each test
- Remove manual cache clearing from individual tests (now handled by fixture)
- Prevents our module stubs from affecting other test files in CI/CD pipeline
- Fixes Bitbucket pipeline failures where other tests couldn't find llm_utils attributes
* Refactor tests: add comprehensive io_utils coverage; disable usage_tracking suite
- Added 22 focused tests for write_local and write_s3 plus existing IO behaviors
- Introduced FakeS3Client to avoid boto3 network/credential dependency
- Added preserve_config fixture (no monkeypatch) to isolate config side effects
- Marked test_usage_tracking.py skipped per new consolidation approach
- Verified 26 tests (io_utils + investment_main) all pass locally without AWS exceptions
* Fix S3_CLIENT mocking: use mocker.patch instead of direct assignment
- Changed preserve_config fixture to NOT save/restore S3_CLIENT
- Updated all 5 write_s3 tests to use mocker.patch('src.config.S3_CLIENT', fake)
- This prevents pollution of config.S3_CLIENT across test modules
- Fixes NoCredentialsError in other tests that were importing config after our tests set FakeS3Client
- All 22 io_utils tests pass + 4 investment_main_s3 tests pass
- test_llm_utils::test_invoke_claude_local_mode now passes
* Add tests for write_s3 function
* Resolve merge conflict: keep mocker-based write_s3 tests
* Remove rogue skip mark from test_usage_tracking.py
The skip mark was incorrectly added by remote branch claiming tests were
migrated to io_utils. However, test_io_utils.py only tests io_utils functions
(read/write operations), not usage_tracking functions.
Restore the comprehensive usage_tracking tests from commit 26a556f0 which
includes 98 test cases covering all 10 functions in usage_tracking.py.
* Fix test pollution: add config cleanup fixture and restore RUN_MODE
- Add autouse fixture in test_io_utils.py to restore config values after each test
- Fix test_llm_utils.py to restore RUN_MODE after test_invoke_claude_local_mode
- Prevents config patches from leaking into subsequent tests causing NoCredentialsError
* Remove redundant test_investment_main_s3.py
- File was testing same write_s3 functionality already covered in test_io_utils.py
- Was causing module pollution by stubbing src.utils.llm_utils
- Caused test failures in other test files due to module cache issues
- Integration testing was minimal and mocked write_s3 anyway
* Merged main into feature/automated-cost-logging-csv
Approved-by: Karan Desai
Approved-by: Katon Minhas
|
||
|
VenkataKrishna Reddy Avula
|
0d6bdf9539 |
Merged in bugfix/effective_date_fix (pull request #504)
DRAFT: Bugfix/effective date fix * Merged in bugfix/extra-fields (pull request #500) Do not run Exhibit level prompt if there are no exhibit level fields * Do not run Exhibit level prompt if there are no exhibit level fields * Update dynamic test * Remove test from one-to-n test module * Update test * Fix unit test Approved-by: Alex Galarce * Merged in bugfix/tin-npi-prompt-fixes (pull request #499) Bugfix/tin npi prompt fixes * Change delimiter for other provider information from comma to pipe * filter out invalid providers * Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE * Merged main into bugfix/tin-npi-small-bugfixes * Refactor deduplication logic to use string_utils for checking empty provider fields * Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes * Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME * Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes * Fix deduplication logic to not drop subsequent names * Fix formatting of provider names in deduplication logic * add payer name filtering * reorder execution order to allow to find payer_name first * Add handling for previously unfound group providers in identify_group_provider function * Merge remote-trackin… * Merged in feature/update-testbed-script (pull request #501) Feature/update testbed script * Update preprocessing * Update preprocessing * remove quit * Update date for filename * Merged main into feature/update-testbed-script Approved-by: Alex Galarce * Merged in bugfix/smart-chunks (pull request #503) missing of text in chunks fixed * missing of text in chunks fixed Approved-by: Katon Minhas * Merged in feature/rework-reimb-tin-npi (pull request #502) Feature/rework reimb tin npi * use LLM for reimb_[tin/npi/name] * Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi * Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi * deduplicate tin/name/npi before adding to prompt * Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging * Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly * Reorder REIMB_PROV_NAME field in investment_prompts.json * Update REIMB_PROV_NAME prompt to specify valid values for provider names * Remove valid_values constraint from prov_name to raise accuracy * Remove debugging print statements from reimbursement_tin_npi and related functions * Remove debugging print statements from run_provider_info_fields function * Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi * Remove debugging print statem… * Merged in feature/preformat-single-quotes-input (pull request #506) Replace double quotes with single quotes in context strings for parsability * Replace double quotes with single quotes in context strings for parsability * Merged main into feature/preformat-single-quotes-input Approved-by: Katon Minhas * updated sig pages fnxn * Merge branch 'main' into bugfix/effective_date_fix * poetry add pymupdf * created vision funcs * Merged main into bugfix/effective_date_fix * fixed pipeline issue * updated string_utils_test * updated test values * updated extract signature page fxn * updated test values * updated test values * Merged main into bugfix/effective_date_fix * Prompt template in all-caps * remove debugging prints * Merged main into bugfix/effective_date_fix * Change vision to False by default * Change vision to False by default * fix tests * fix tests Approved-by: Katon Minhas |
||
|
Alex Galarce
|
56e3fdd1f8 |
Merged in bugfix/all_cnc_hotfixes (pull request #302)
DRAFT: Bugfix/all cnc hotfixes * Pipeline fix - health plan state * Update for prompt vs non-prompt fixes * Revert utils.find_regex_matches() * Remove prints * Non-prompt run * Health Plan State, IRS, NPI, LOB, Lesser, Prov 2, Default, Rate Standard, Contract Effective Date, Term Group * Merged in bugfix/agreement_name (pull request #294) contract title fixed * contract title fixed * Merged bugfix/all_cnc_hotfixes into bugfix/agreement_name Approved-by: Katon Minhas * All hotfixes added * Fixed utils * fix for clean_contract_effective_date * Merged in default-fix (pull request #296) moved default funcs to the top, dropped intermediate columns * moved default funcs to the top, dropped intermediate columns Approved-by: Katon Minhas * quick fix to dropping extra columns * Merged in bugfix/effective_date_meridian (pull request #297) fixed effective date for meridian contracts * fixed effective date for meridian contracts Approved-by: Katon Minhas * Default rate and Lesser fix * Merged in hotfix/irs_npi_updated (pull request #298) updated irs and npi hotfix funcs * updated irs and npi hotfix funcs * Merged bugfix/all_cnc_hotfixes into hotfix/irs_npi_updated Approved-by: Katon Minhas * non-functional commit - default only * debug commit - premerge * Batch 1 Dup File Reconciliation * Default rate fix * Full CNC 1A Run Config * Standard column refactor * Working stitching for 1A * Merged in Alex-Galarce/cnc_hotfix_effective_date_utilspy-edited-1732659983687 (pull request #301) Remove "license" from effective date smart chunking * Remove "license" from effective date smart chunking * Merged bugfix/all_cnc_hotfixes into Alex-Galarce/cnc_hotfix_effective_date_utilspy-edited-1732659983687 Approved-by: Katon Minhas * Merged in hotfix/irs_batch2_feedback (pull request #300) IRS fix for feedback on batch2 * IRS fix for feedback on batch2 * Merged bugfix/all_cnc_hotfixes into hotfix/irs_batch2_feedback Approved-by: Katon Minhas * Merged main into bugfix/all_cnc_hotfixes Approved-by: Katon Minhas |
||
|
Michael McGuinness
|
815fa7f0bc |
Merged in feature/poetry (pull request #238)
Draft: Adding Poetry * addpoetry * pipes * mypy * streamlit * searchforchange Approved-by: Alex Galarce |
||
|
Michael McGuinness
|
02169d8a5e | add reqs |