4f53b5286edec3f43cd2e45c686d0ca79120057d
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ppanchigar
|
4f53b5286e |
Revert premature merge of bugfix/retire_stale_client_file_processing
PR #959 was merged into dev without approval. This reverts commits |
||
|
ppanchigar
|
63f32c418c |
Remove debug routing print statements
Strip all [DEBUG ROUTING] print statements added for E2E verification.
Cherry-pick the previous commit (
|
||
|
ppanchigar
|
5e143c10f0 |
Retire stale client file_processing forks, unify pipeline entry point
Delete BCBS/Clover/CHC client file_processing.py files (1090 lines of 96% stale drift with 3 crash points). Recover the 4% real business logic (BCBS OFFSET_TERM, Clover 12 full_context fields) into the shared pipeline via config-driven field loading. Make runner.py the canonical pipeline entry point by merging all production features from main.py (timing, duplicate detection, aarete derived fields, TIN statistics, column reordering). Reduce main.py to a thin wrapper that delegates to runner.main(). Fix the root cause of all client overrides being dead in production: set_active_client is now called with the correct client name instead of being hardcoded to "saas". Both resolver shims (file_processing and prompt_calls) now route correctly. Includes temporary [DEBUG ROUTING] print statements at 6 routing decision points for E2E verification. Cherry-pick this commit to restore debug instrumentation for future testing. E2E verified: BCBS Promise and Clover both pass with correct routing. |
||
|
Praneel Panchigar
|
5743a4c1be |
Merged in feature/DAIP2-1615-refactor-maximize-code-reusability (pull request #930)
Feature/DAIP2-1615 refactor maximize code reusability * added reusability * client resolver added * Merged DEV into feature/DAIP2-1615-refactor-maximize-code-reusability * Refactor client prompt reuse and harden resolver test coverage. Remove duplicated client prompt functions so unresolved names fall back to SaaS, add resolver contract tests with runtime-context isolation, and clean resolver-path test imports to prevent cross-test client leakage. * Merge origin/dev into feature/DAIP2-1615-refactor-maximize-code-reusability. Bring in latest vendor pipeline framework changes from dev and resolve runner routing conflict by preserving vendor processor dispatch while keeping shared function-level client fallback behavior. * Merged dev into feature/DAIP2-1615-refactor-maximize-code-reusability * Remove tracked drift review report from documentation. Keep drift/alignment notes as local-only artifacts under ignored docs/ and stop tracking documentation/function_level_override_drift_review.md in the branch. * Merged dev into feature/DAIP2-1615-refactor-maximize-code-reusability Approved-by: Katon Minhas |
||
|
Karan Desai
|
729fef7790 |
Merged in feature/vendor-generic (pull request #907)
Feature/vendor generic * initial commit for generic vendor logic * updated generic field list * combined special fields * joint fields template * payment term and amount extraction * model update * model update * added constants * care-source fields added * care-source constants * fields update for bcbs-az * prompt update for term type * prompt update for payment type * post process additions for generic * Merge remote-tracking branch 'origin/DEV' into feature/vendor-generic * Merged DEV into feature/vendor-generic * Merged DEV into feature/vendor-generic * Black format * Merge remote-tracking branch 'origin/DEV' into feature/vendor-generic * Merge branch 'DEV' into feature/vendor-generic * PR comments * deleted redundant file * Merged dev into feature/vendor-generic Approved-by: Katon Minhas |
||
|
Faizan Mohiuddin
|
14cb727967 |
Merged in improve-logging (pull request #886)
Improve logging * Add timing blocks for comprehensive pipeline logging - Add timing_utils.timed_block() to key extraction functions in one_to_n_funcs.py: - exhibit_level sub-functions (prompt_exhibit_level, dynamic_primary, etc.) - reimbursement_level extraction and cleaning - breakout functions (methodology_breakout, special_case_breakout) - carveout_and_special_case parallel processing - one_to_n_cleaning sub-steps (crosswalk, lob_relationship, split_reimb_dates) - Add timing to dynamic_funcs.py: - dynamic_assignment parallel processing - Add timing to postprocess.py: - standard_postprocess, contract_config_postprocess, dashboard_postprocess - Add timing to preprocess.py: - exhibit_chunking sub-steps (get_exhibit_pages, link_exhibit_pages, chunk_by_exhibit) - Add info-level logging statements for completed operations with row/item counts * Standardize logging levels and remove deprecated code DAIP2-1798: Ensure logging statements are at the right level - Demote per-file processing status messages from info to debug in file_processing.py - Demote operational details (counts, timing) from info to debug in runner.py - Demote detailed extraction messages from info to debug across extraction modules - Keep only high-level milestone messages (section headers, major ops) at info level DAIP2-1796: Ensure console output matches content of logs/ - Add LOG_LEVEL config setting for console output (defaults to INFO) - Update logging_utils.py to use config.LOG_LEVEL for console handler DAIP2-1797: Remove deprecated functionality - Remove deprecated check_and_combine_exhibit_inheritance function from one_to_n_funcs.py - Remove deprecated WRITE_PC_TO_S3 config setting (now uses WRITE_TO_S3) - Remove deprecated MODEL_STATS/GLOBAL_STATS comment from config.py * Merged DEV into improve-logging * Format code for improved readability * Remove duplicate logger.propagate line * Merge remote-tracking branch 'origin/DEV' into improve-logging Approved-by: Siddhant Medar |
||
|
Praneel Panchigar
|
a0c7e7a738 |
Merged in bugfix/dynamic_issues_feb12 (pull request #882)
Bugfix/dynamic issues feb12
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_qa-qc/ for QC/QA validated results and statistics
- parent-child/ for parent-child mapping outputs
- tracking/ for usage and cost tracking data
- individual/ for per-file CC results (dashboard individual files removed)
- Update file naming conventions to match new structure
- Remove QC/QA processing for error files (error files are saved without validation)
Post-Processing Changes:
- Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.)
when they are the only value in a cell (applies before CC/dashboard split)
- Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks)
- Ensure standard cleaning runs before splitting into CC …
* Ran Black for CI
* Refactor file splitting logic and add comprehensive tests
- Refactored splitting logic in io_utils.py:
- Consolidated repeated splitting code into two focused helper functions:
- _write_local_split_files() for local file writing with splitting
- _write_s3_split_files() for S3 file writing with splitting
- Both helpers use shared split_dataframe_by_filename() function
- Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py
- Added comprehensive test coverage for splitting logic:
- Tests for split_dataframe_by_filename() with various scenarios
- Tests for local and S3 write operations with single and multiple splits
- Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types
- Fixed existing test failures (write_s3 error handling, path assertions)
- Improved code maintainability and readability
* Fix failing tests in test_postprocess.py
- Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING
(PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped
- Added FILE_NAME column to all file structure test DataFrames (required for splitting logic)
- Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests
- Fixed patch decorators for S3 tests to properly mock logging
All 41 tests now passing.
* Black for CI
* Blank [] and ['[]'] in output instead of displaying them
- Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"]
- Update format_as_json_list to return blank for empty lists instead of []
- Filter out empty-list placeholder items from list values in format_as_json_list
- Add tests for clean_na_values empty list handling and format_as_json_list
Co-authored-by: Cursor <cursoragent@cursor.com>
* Merged DEV into feature/new_output_format
* feat: dynamic primary debug improvements and LOB partial 1:1 escalation
- Add per-exhibit debug summary for dynamic primary discovery (page, values, raw LLM)
- Pass exhibit_page to dynamic_primary for debug; mark debug-only params for removal
- Fix 1:1 escalation skip: use base_field for PROGRAM/PRODUCT/NETWORK when LOB detected
- Pass LOB to 1:1 when empty in any row (partial detection from stripped headers)
- Add debug exhibit text preview; mark check_and_combine_exhibit_inheritance for deletion
- prompt_dynamic_primary returns (answer, raw); remove per-field prints
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: pass PROGRAM and PRODUCT to 1:1 when partial LOB
When LOB is empty in some rows (partial detection), pass LOB, PROGRAM, and
PRODUCT to 1:1 for contract-level extraction. Merge fills only empty cells
so 1:N values are preserved. Skip PROGRAM/PRODUCT/NETWORK only when full LOB.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Removed debug print blocks
* Merge origin/DEV into bugfix/dynamic_issues_feb12
Resolved conflicts:
- runner.py: Use RUN_DASHBOARD_POSTPROCESSING for conditional dashboard; parent-child output_dir and S3 upload
- main.py: Use RUN_DASHBOARD_POSTPROCESSING for conditional dashboard
- postprocess.py: Optional dashboard postprocessing when RUN_DASHBOARD_POSTPROCESSING is True
Co-authored-by: Cursor <cursoragent@cursor.com>
Approved-by: Katon Minhas
|
||
|
Praneel Panchigar
|
637d2dea1f |
Merged in feature/new_output_format (pull request #879)
Feature/new output format
* bugfix
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* requested postprocessing changes
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_qa-qc/ for QC/QA validated results and statistics
- parent-child/ for parent-child mapping outputs
- tracking/ for usage and cost tracking data
- individual/ for per-file CC results (dashboard individual files removed)
- Update file naming conventions to match new structure
- Remove QC/QA processing for error files (error files are saved without validation)
Post-Processing Changes:
- Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.)
when they are the only value in a cell (applies before CC/dashboard split)
- Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks)
- Ensure standard cleaning runs before splitting into CC …
* Ran Black for CI
* Refactor file splitting logic and add comprehensive tests
- Refactored splitting logic in io_utils.py:
- Consolidated repeated splitting code into two focused helper functions:
- _write_local_split_files() for local file writing with splitting
- _write_s3_split_files() for S3 file writing with splitting
- Both helpers use shared split_dataframe_by_filename() function
- Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py
- Added comprehensive test coverage for splitting logic:
- Tests for split_dataframe_by_filename() with various scenarios
- Tests for local and S3 write operations with single and multiple splits
- Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types
- Fixed existing test failures (write_s3 error handling, path assertions)
- Improved code maintainability and readability
* Fix failing tests in test_postprocess.py
- Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING
(PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped
- Added FILE_NAME column to all file structure test DataFrames (required for splitting logic)
- Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests
- Fixed patch decorators for S3 tests to properly mock logging
All 41 tests now passing.
* Black for CI
* Blank [] and ['[]'] in output instead of displaying them
- Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"]
- Update format_as_json_list to return blank for empty lists instead of []
- Filter out empty-list placeholder items from list values in format_as_json_list
- Add tests for clean_na_values empty list handling and format_as_json_list
Co-authored-by: Cursor <cursoragent@cursor.com>
* Merged DEV into feature/new_output_format
* Pipeline config, parent-child, dashboard, and runner fixes
- Parent-child: enable by default, write to run directory, S3 upload via io_utils
- Dashboard: optional (CC only by default), run_dashboard=True to enable
- Add io_utils.upload_local_file_to_s3 for centralized file uploads
- Parent-child returns (row_count, local_path); pipeline handles S3 upload
- Add TODO in config for WRITE_PC_TO_S3 removal after approval
- Fix indentation errors in runner.py
Co-authored-by: Cursor <cursoragent@cursor.com>
Approved-by: Katon Minhas
|
||
|
ppanchigar
|
5b619b2f5f | fixed code implicit special and ran black for CI | ||
|
Sha Brown
|
05b00ab6a1 |
Merged in bugfix/max_workers_and_batch_id (pull request #864)
Trigger error when batch_id is not specified or when max_workers is negative * Trigger error when batch_id is not specified or when max_workers is negative Approved-by: Katon Minhas |
||
|
Siddhant Medar
|
f412754167 |
Merged in feature/generic-parent-child (pull request #852)
Feature/generic parent child
* Refactor parent_child module with generic hierarchy lineage
- Add generic_hierarchy_lineage.py for core parent-child mapping
- Add generic_hierarchy_lineage_qc.py for QC validation engine
- Add bcbs_generic_hierarchy_lineage.py for BCBS-specific processing
- Add column_mapper.py for automatic column detection
- Rename parent_child_preprocessing.py to generic_hierarchy_lineage_preprocessing.py
- Remove old parent_child_mapping.py (replaced by generic_hierarchy_lineage.py)
- Fix import paths from src.generic_hierarchy_lineage to src.parent_child
- Replace hardcoded BCBS xwalk path with configurable parameter
- Remove global warning suppression
* Fix code review issues in parent_child module
- Fix import paths from src.generic_hierarchy_lineage to src.parent_child
- Replace hardcoded BCBS crosswalk path with configurable xwalk_path param
- Remove global warnings.filterwarnings('ignore') suppressions
- Change bare except clauses to except Exception with logging
- Add explicit client= CLI argument with backward compatible fallback
- Convert print statements to logging in bcbs_generic_hierarchy_lineage.py
* Add tests for parent_child module and fix JSON path resolution
- Add 55 unit tests covering grouping keys, parent identification,
child ranking, and QC utilities
- Use pathlib to resolve JSON config paths relative to module location,
allowing tests to run from any directory
* WIP: Refactor parent_child module structure
* Extract constants to centralized location for parent_child module
- Create src/constants/parent_child/ with generic.py and bcbs.py
- Update pipeline.py to use centralized constants (ASSIGNMENT_NO_PARENT, tier values, grouping key config)
- Update bcbsnc/mapping.py to use BCBS-specific constants
- Update imports in __main__.py and bcbsnc/__init__.py
- Pass client parameter from runner.py and saas/main.py
* Fix code formatting in parent_child module
* Merge remote-tracking branch 'origin/main' into feature/generic-parent-child
* Fix client detection and output path in parent_child module
* Fix typo in numeric_mappings.json filename and update logging
* Add type hints to parent_child module
* Merged main into feature/generic-parent-child
Approved-by: Katon Minhas
|
||
|
Katon Minhas
|
afb6d5185d |
Merged in feature/lesser-table-caching-refactor-hybrid (pull request #847)
Feature/lesser table caching refactor hybrid * chore: Remove unused duplicate main.py from shared pipeline * fix: Correct crosswalk paths in aarete_derived.py * chore: Remove unused documentation files from fieldExtraction * docs: Add documentation files to documentation folder * docs: Update README with uv setup, expanded project structure, and branching conventions * docs: Add uv installation steps with Ubuntu/WSL emphasis * Enable prompt caching for all remaining LLM calls - Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE, EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH, SPECIAL_CASE_ASSIGNMENT - Update all invoke_claude() calls in saas and clover pipelines to use cache=True with corresponding _INSTRUCTION() functions - Add new instructions to get_cacheable_instructions() for cache warming - Update tests for new instruction functions Functions now using caching: - prompt_exhibit_level - prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF) - prompt_fee_schedule_breakout - prompt_grouper_breakout - prompt_special_case_assignment - prompt_exhibit_linkage - prompt_exhibit_header - prompt_smart_chunked (ONE_TO_ONE templates) - prompt_date_fix - prompt_derived_term_date - prompt_exhibit_title_match - provider_name_match_check 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Reorder * feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction - Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction - Extract full paragraph text of offset/recoupment provisions from contracts - Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence - Fix reorder_columns to preserve extra columns not in COLUMN_ORDER - Update QC/QA output path to outputs/qc_qa/ * fix: Update dev deps and test assertions for QC/QA output path - Add pytest/pytest-mock to dev dependencies for mypy type checking - Update test assertions to expect outputs/qc_qa instead of qa_qc_output * style: Apply black formatting to prompt_templates.py * Merge main, move scripts * Archive some scripts * update py version * remove .py version file * Remove ASCII characters * Restore testbed code * restore tracking * Update testbed metrics * Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT - Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification - Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination - Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification - Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT - Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK - Add new instructions to get_cacheable_instructions() for cache warming - Add unit tests for new instruction functions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix postprocessing_funcs to remove invalid columns * Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid * Revert prompt caching changes from aed1b73c * update formatting * Update imports Approved-by: Sha Brown Approved-by: Praneel Panchigar |