ea1f4a2c8c6a560cf7aa5415542869a9fd8ddada
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Venkatakrishna Reddy Avula
|
ea1f4a2c8c |
Merged in feature/DAIP2-2023-eliminate-full-context-processing (pull request #905)
Feature/DAIP2-2023 eliminate full context processing * testing full context fields * remove full context processing * merge Dev with DAIP2-2-23 * full context removal in client codes * AARETE_DERIVED_PROVIDER_NAME field changes * Merged DEV into feature/DAIP2-2023-eliminate-full-context-processing * optimized provider name * black format fix * contract title fixes * PAYER NAME AUTO RENEWAL IND fixes * Merged DEV into feature/DAIP2-2023-eliminate-full-context-processing * Merge branch 'DEV' into feature/DAIP2-2023-eliminate-full-context-processing * Remove prints Approved-by: Katon Minhas |
||
|
Mayank Aamseek
|
3331da2a8c |
Merged in bugfix/prov_info_json_fixes (pull request #899)
Bugfix/prov info json fixes * fix: robust PROV_INFO_JSON sanitization and TIN backfill logic json_utils: - Add sanitize_prov_info_json with layered parsing (JSON, literal_eval, empty-value-after-colon fix, best-effort dict extraction). - Add _normalize_prov_entries and _prov_value_to_str for uniform str-valued output; flatten list values, strip TIN hyphens. - format_prov_info_json now delegates to sanitize_prov_info_json. postprocessing_funcs: - Add fill_prov_info_tin_from_filename_tin for TIN backfill. - Add validate_and_reformat_date (pipe-wrapped, datetime strings). - Add format_as_json_list (pipe-delimited, comma-separated, quote stripping). postprocess: - Integrate new postprocessing helpers into pipeline flow. postprocess_existing_output: - Support CSV and Excel input, configurable paths, fillna for CSV. tests: - Add test_json_parsers.py for PROV_INFO_JSON parsing coverage. - Add test_postprocess.py for date/list formatting and default_ind. * Merge branch 'DAIP2-1947-tin-and-prov-info-json-issues' into bugfix/prov_info_json_fixes * Merged DEV into bugfix/prov_info_json_fixes * Strip out unused functionality * Update filename_tin functionality * Add docstring * Update filename_tin cleaning in PROV_INFO_JSON * test prep * black format * missing function added * Merge branch 'DEV' into bugfix/prov_info_json_fixes * Black format * Strip unused functions * Strip unused code * update unit tests * Merge branch 'DEV' into bugfix/prov_info_json_fixes * Update test * Simplify process * handle list of group names * Resolve run_provider_info_field call * Merge branch 'DEV' into bugfix/prov_info_json_fixes * Correct type hints * Fix unit tests * Fix unit test * Remove redundant postprocessing_funcs Approved-by: Katon Minhas |
||
|
Praneel Panchigar
|
637d2dea1f |
Merged in feature/new_output_format (pull request #879)
Feature/new output format
* bugfix
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* requested postprocessing changes
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_qa-qc/ for QC/QA validated results and statistics
- parent-child/ for parent-child mapping outputs
- tracking/ for usage and cost tracking data
- individual/ for per-file CC results (dashboard individual files removed)
- Update file naming conventions to match new structure
- Remove QC/QA processing for error files (error files are saved without validation)
Post-Processing Changes:
- Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.)
when they are the only value in a cell (applies before CC/dashboard split)
- Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks)
- Ensure standard cleaning runs before splitting into CC …
* Ran Black for CI
* Refactor file splitting logic and add comprehensive tests
- Refactored splitting logic in io_utils.py:
- Consolidated repeated splitting code into two focused helper functions:
- _write_local_split_files() for local file writing with splitting
- _write_s3_split_files() for S3 file writing with splitting
- Both helpers use shared split_dataframe_by_filename() function
- Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py
- Added comprehensive test coverage for splitting logic:
- Tests for split_dataframe_by_filename() with various scenarios
- Tests for local and S3 write operations with single and multiple splits
- Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types
- Fixed existing test failures (write_s3 error handling, path assertions)
- Improved code maintainability and readability
* Fix failing tests in test_postprocess.py
- Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING
(PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped
- Added FILE_NAME column to all file structure test DataFrames (required for splitting logic)
- Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests
- Fixed patch decorators for S3 tests to properly mock logging
All 41 tests now passing.
* Black for CI
* Blank [] and ['[]'] in output instead of displaying them
- Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"]
- Update format_as_json_list to return blank for empty lists instead of []
- Filter out empty-list placeholder items from list values in format_as_json_list
- Add tests for clean_na_values empty list handling and format_as_json_list
Co-authored-by: Cursor <cursoragent@cursor.com>
* Merged DEV into feature/new_output_format
* Pipeline config, parent-child, dashboard, and runner fixes
- Parent-child: enable by default, write to run directory, S3 upload via io_utils
- Dashboard: optional (CC only by default), run_dashboard=True to enable
- Add io_utils.upload_local_file_to_s3 for centralized file uploads
- Parent-child returns (row_count, local_path); pipeline handles S3 upload
- Add TODO in config for WRITE_PC_TO_S3 removal after approval
- Fix indentation errors in runner.py
Co-authored-by: Cursor <cursoragent@cursor.com>
Approved-by: Katon Minhas
|
||
|
Praneel Panchigar
|
9b0a344b13 |
Merged in bugfix/parser-downstream-improvements (pull request #875)
Bugfix/parser downstream improvements * Refactor: Implement field-aware JSON parsers with centralized normalization This refactor introduces a robust system for normalizing LLM output based on field format mappings, ensuring consistent data types throughout the pipeline. Key Changes: - Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields - Create format_normalization.py utility for type-aware normalization - Update json_utils.py parsers to accept field_names/field_name parameters - Refactor prompt_templates.py to use parser factories (_create_json_dict_parser, _create_json_list_parser) that bind field metadata for automatic normalization - Update prompt_calls.py to pass field names to parsers, eliminating redundant normalization logic - Remove parse_json_dict_or_list (unused, ambiguous function) - Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions - Add comprehensive integration tests verifying normalization works end-to-end Benefits: - Single source of truth for field formats (FIELD_FORMAT_… * refactor: normalize helper prompt outputs at prompt_calls level - Update CARVEOUT_CHECK to use field-aware parser for CARVEOUT_CD normalization - Update LOB_RELATIONSHIP to normalize to string format in prompt_calls.py - Update SPLIT_REIMB_DATES to normalize date values in prompt_calls.py - Remove defensive normalization from one_to_n_funcs.py for LOB relationships - Remove manual normalization from split_reimb_dates() - values now normalized upstream - All helper prompts that populate fields now normalize at prompt_calls.py level - Downstream functions receive correctly formatted values without additional processing * refactor: remove band-aid normalization functions and migrate HSC to field-aware parsers - Update ONE_TO_ONE_SINGLE_FIELD_TEMPLATE to use field-aware parser with field_name parameter - Remove list wrapping logic in hybrid_smart_chunking_funcs (field-aware parser handles normalization) - Remove normalize_one_to_one_field_value and normalize_one_to_one_answers_dict from string_utils.py - Remove all debug print statements from HSC processing - Remove commented-out normalization calls from client-specific files (clover, bcbs_promise) - All normalization now handled exclusively through FIELD_FORMAT_MAPPING via field-aware parsers * Ran Black for formatting * Print Statements removed, more cleaning * Merge branch 'DEV' into bugfix/parser-downstream-improvements * refactor: combine FIELD_FORMAT_MAPPING into investment_columns.py - Merged field_format_mapping.py into investment_columns.py to create single source of truth - FIELD_FORMAT_MAPPING now ordered by COLUMN_ORDER (161 fields) - Added 5 missing fields from COLUMN_ORDER with default format types - Updated all imports across codebase to use investment_columns - Python dict preserves insertion order (3.7+), maintaining COLUMN_ORDER sequence - All tests passing (38 field-aware tests verified) * Deprecate COLUMN_ORDER, rely on Mapping only * Merged DEV into bugfix/parser-downstream-improvements Approved-by: Katon Minhas |
||
|
Katon Minhas
|
f8783ae968 | Update prov_info | ||
|
ppanchigar
|
cff0472717 | Latest updates | ||
|
Katon Minhas
|
afb6d5185d |
Merged in feature/lesser-table-caching-refactor-hybrid (pull request #847)
Feature/lesser table caching refactor hybrid * chore: Remove unused duplicate main.py from shared pipeline * fix: Correct crosswalk paths in aarete_derived.py * chore: Remove unused documentation files from fieldExtraction * docs: Add documentation files to documentation folder * docs: Update README with uv setup, expanded project structure, and branching conventions * docs: Add uv installation steps with Ubuntu/WSL emphasis * Enable prompt caching for all remaining LLM calls - Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE, EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH, SPECIAL_CASE_ASSIGNMENT - Update all invoke_claude() calls in saas and clover pipelines to use cache=True with corresponding _INSTRUCTION() functions - Add new instructions to get_cacheable_instructions() for cache warming - Update tests for new instruction functions Functions now using caching: - prompt_exhibit_level - prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF) - prompt_fee_schedule_breakout - prompt_grouper_breakout - prompt_special_case_assignment - prompt_exhibit_linkage - prompt_exhibit_header - prompt_smart_chunked (ONE_TO_ONE templates) - prompt_date_fix - prompt_derived_term_date - prompt_exhibit_title_match - provider_name_match_check 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Reorder * feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction - Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction - Extract full paragraph text of offset/recoupment provisions from contracts - Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence - Fix reorder_columns to preserve extra columns not in COLUMN_ORDER - Update QC/QA output path to outputs/qc_qa/ * fix: Update dev deps and test assertions for QC/QA output path - Add pytest/pytest-mock to dev dependencies for mypy type checking - Update test assertions to expect outputs/qc_qa instead of qa_qc_output * style: Apply black formatting to prompt_templates.py * Merge main, move scripts * Archive some scripts * update py version * remove .py version file * Remove ASCII characters * Restore testbed code * restore tracking * Update testbed metrics * Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT - Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification - Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination - Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification - Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT - Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK - Add new instructions to get_cacheable_instructions() for cache warming - Add unit tests for new instruction functions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix postprocessing_funcs to remove invalid columns * Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid * Revert prompt caching changes from aed1b73c * update formatting * Update imports Approved-by: Sha Brown Approved-by: Praneel Panchigar |