DEV
* Merged in bugfix/parser-downstream-improvements (pull request #875)
Bugfix/parser downstream improvements
* Refactor: Implement field-aware JSON parsers with centralized normalization
This refactor introduces a robust system for normalizing LLM output based on
field format mappings, ensuring consistent data types throughout the pipeline.
Key Changes:
- Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields
- Create format_normalization.py utility for type-aware normalization
- Update json_utils.py parsers to accept field_names/field_name parameters
- Refactor prompt_templates.py to use parser factories (_create_json_dict_parser,
_create_json_list_parser) that bind field metadata for automatic normalization
- Update prompt_calls.py to pass field names to parsers, eliminating redundant
normalization logic
- Remove parse_json_dict_or_list (unused, ambiguous function)
- Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions
- Add comprehensive integration tests…
* Update other lists
* Black format
* Remove prints
* Update qa_qc to account for lists in is_empty check
* Resolve issues
* Black format
* Merged main into DEV
* Merged in feature/claim-type-only-runner (pull request #876)
Feature/claim type only runner
* Add specific_fields config for running extraction on field groups
## What Changed
4 files modified:
1. src/config.py - Added configuration for field-specific extraction:
- SPECIFIC_FIELDS arg (default: 'all') - pass field group name or comma-separated field names
- FIELD_GROUPS dict - predefined groups: claim_type, dates, provider
- get_specific_fields_list() - resolves config to actual field list
2. src/prompts/fieldset.py - Added filter_by_names() method to FieldSet class to filter fields by a list of names
3. src/pipelines/shared/extraction/one_to_n_funcs.py - Updated exhibit_level() to accept specific_fields parameter and skip prompts for fields not in the list
4. src/pipelines/saas/file_processing.py - Passes specific_fields through the call chain to both one_to_n and one_to_one extraction
## How It Works
When specific_fields is set to something other than 'all':
1. Config resolves the field l…
* Merged in bugfix/generic_lesser_of (pull request #880)
Bugfix/generic lesser of
* Updated lesser of ditribution and lesser of check prompt
* Remove excessive logging
* Updated prompt for combining and formatting lesser of statement and reimb term
* Code clean upt
* Removed unneeded instruction on output format
* style: format prompt_calls.py with black
Approved-by: Katon Minhas
* Merged in feature/new_output_format (pull request #879)
Feature/new output format
* bugfix
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* requested postprocessing changes
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full…
* Merged in bugfix/dynamic_issues_feb12 (pull request #882)
Bugfix/dynamic issues feb12
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_q…
* Merged in feature/DAIP2-1562-add-aarete_derived_payer_name (pull request #867)
Feature/DAIP2-1562 add aarete derived payer name
* black format
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* llm_choose_derived_payer_name function added
* threshold updated
* Merge branch 'DEV' into feature/DAIP2-1562-add-aarete_derived_payer_name
* aarete_derived_payer_name column added
* prompt structure updated
* state_flag added
* pipeline error fixed
* pipeline error fixed
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* remove debug print statement for similarity matrix in clustering function
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* state logic added in clustering
* removed print statements
* pull request updates
* black format fix
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* added scalability feature and optimization
* black format
* Pull Request Changes
* updated config parameters
* updated main
* no pay…
* Merged in bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout (pull request #885)
Bugfix/DAIP2-1701 issue tracker fixes methodology breakout
* testing fee schedules issues
* logging debug added for lesser of
* Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout
* special case term fix
* updated lesser of distribution prompt
* Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout
* refactor: use f-string for special case term concatenation
Approved-by: Siddhant Medar
* Merged in bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info (pull request #881)
bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* contract title fixes
* Add instruction to strip hyphens from TIN and NPI
* prompt update
* Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* removed print statements
* pipeline error fixed
* signed_ind logic changed
* signatory_ind logic updated
* prompt update
* prompt update
* pipeline error fixed
* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* requested changes applied
* test case added
* prompt update
* contract title prompt update
* latest prompt added
Approved-by: Siddhant Medar
* Merged in bugfix/default_ind_postprocess (pull request #887)
Bugfix/default ind postprocess
* Add logic to standardize UNIT_OF_MEASURE for flat-rate reimbursement methods
- Implemented functionality in `standardize_reimb_method_and_fee_schedule` to set UNIT_OF_MEASURE to blank for rows where DEFAULT_IND is 'Y' and AARETE_DERIVED_REIMB_METHOD is 'flat rate'.
- Added unit tests to verify behavior for various scenarios, including case insensitivity and non-default conditions.
- Ensured that UNIT_OF_MEASURE remains unchanged for non-flat rate methods.
* Enhance child rank handling and ensure column consistency in parent-child mapping
- Added initialization for the `child_rank` column in both parents and children DataFrames to prevent KeyError during concatenation when no children exist.
- Updated `cols_to_keep` in `parent_child_mapping` to filter out columns not present in `pc_df`, ensuring robustness in data processing.
* Ran Black
* made a small change in code_last_check, fixed so it returns string and n…
* Apply formatting fix
* Merged in DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title (pull request #888)
DAIP2-1898 one to one issues filename tin auto renewal and contract title
* contract title fixes
* Add instruction to strip hyphens from TIN and NPI
* prompt update
* Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* removed print statements
* pipeline error fixed
* signed_ind logic changed
* signatory_ind logic updated
* prompt update
* prompt update
* pipeline error fixed
* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* requested changes applied
* test case added
* prompt update
* contract title prompt update
* prompt update
* Merge branch 'DEV' into DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title
Approved-by: Siddhant Medar
* Merged in feature/DAIP2-1803-add-aarete-derived-provider-name (pull request #884)
Feature/DAIP2-1803 add aarete derived provider name
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* state logic added in clustering
* removed print statements
* pull request updates
* black format fix
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* added scalability feature and optimization
* black format
* Pull Request Changes
* generalised funcs from aarete derived payer name
* updated remove states and updated docstring for build similarity matrix
* derived payer name functionality added
* config and main file changes
* sync dev into ADD AARETE DERIVED PROVIDER GROUP NAME FULL
* updated derived_provider name
* pipeline fixes
* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name
* config changes
* field name updated to AARETE_DERIVED_PROVIDER_NAME
* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name
* black format fix
* Fix docstr…
* Merged in improve-logging (pull request #886)
Improve logging
* Add timing blocks for comprehensive pipeline logging
- Add timing_utils.timed_block() to key extraction functions in one_to_n_funcs.py:
- exhibit_level sub-functions (prompt_exhibit_level, dynamic_primary, etc.)
- reimbursement_level extraction and cleaning
- breakout functions (methodology_breakout, special_case_breakout)
- carveout_and_special_case parallel processing
- one_to_n_cleaning sub-steps (crosswalk, lob_relationship, split_reimb_dates)
- Add timing to dynamic_funcs.py:
- dynamic_assignment parallel processing
- Add timing to postprocess.py:
- standard_postprocess, contract_config_postprocess, dashboard_postprocess
- Add timing to preprocess.py:
- exhibit_chunking sub-steps (get_exhibit_pages, link_exhibit_pages, chunk_by_exhibit)
- Add info-level logging statements for completed operations with row/item counts
* Standardize logging levels and remove deprecated code
DAIP2-1798: Ensure logging statements are at …
* Merged in bugfix/DAIP2-1913-lesser-of-indicator-issues (pull request #889)
Bugfix/DAIP2-1913 lesser of indicator issues
* contract title fixes
* Add instruction to strip hyphens from TIN and NPI
* prompt update
* Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* removed print statements
* pipeline error fixed
* signed_ind logic changed
* signatory_ind logic updated
* prompt update
* prompt update
* pipeline error fixed
* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* requested changes applied
* test case added
* prompt update
* contract title prompt update
* prompt update
* Merge branch 'DEV' into DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title
* handle nested cases of lesser_of and greater_of
* Merge branch 'DEV' into bugfix/DAIP2-1913-lesser-of-indicator-issues
* pi…
* Merged in bugfix/reimb_primary (pull request #890)
Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements
* Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements
- REIMBURSEMENT_PRIMARY: append unit of measure (PMPM, per visit, etc.) from table headers into REIMB_TERM
- METHODOLOGY_BREAKOUT: collapse mathematically equivalent time-unit rates into single entry
- VALIDATE_REIMBURSEMENTS: add umbrella terms as valid service terms, accept CPT/HCPCS percentage as concrete rate, narrow COB disqualifier to primary subject only
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Approved-by: Siddhant Medar
* Merged in feature/update-postprocessing-columns (pull request #893)
move column order to final step
* move column order to final step
* Black
Approved-by: Siddhant Medar
* Merged in bugfix/update-postprocessing-paths (pull request #896)
Update postprocessing paths
* Update postprocessing paths
Approved-by: Siddhant Medar
* Merged in feature/TIN_stats_update (pull request #898)
Feature/TIN stats update
* Tin_stats_report
* black formatting
* minor changes
* black format applied
* Addressed the katons comments
* Merged DEV into feature/TIN_stats_update
* black reformat
Approved-by: Katon Minhas
Feature/TIN stats update
* Tin_stats_report
* black formatting
* minor changes
* black format applied
* Addressed the katons comments
* Merged DEV into feature/TIN_stats_update
* black reformat
Approved-by: Katon Minhas
Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements
* Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements
- REIMBURSEMENT_PRIMARY: append unit of measure (PMPM, per visit, etc.) from table headers into REIMB_TERM
- METHODOLOGY_BREAKOUT: collapse mathematically equivalent time-unit rates into single entry
- VALIDATE_REIMBURSEMENTS: add umbrella terms as valid service terms, accept CPT/HCPCS percentage as concrete rate, narrow COB disqualifier to primary subject only
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Approved-by: Siddhant Medar
Feature/DAIP2-1803 add aarete derived provider name
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* state logic added in clustering
* removed print statements
* pull request updates
* black format fix
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* added scalability feature and optimization
* black format
* Pull Request Changes
* generalised funcs from aarete derived payer name
* updated remove states and updated docstring for build similarity matrix
* derived payer name functionality added
* config and main file changes
* sync dev into ADD AARETE DERIVED PROVIDER GROUP NAME FULL
* updated derived_provider name
* pipeline fixes
* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name
* config changes
* field name updated to AARETE_DERIVED_PROVIDER_NAME
* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name
* black format fix
* Fix docstring and add empty column when not found
* Rename test file and add missing column test case
- Renamed test_derived_payer_name.py to test_derived_names.py since
file now covers both payer and provider name functionality
- Added test_derived_provider_names_column_missing to verify empty
column is added when AARETE_DERIVED_PROVIDER_NAME is not present
* Remove trailing whitespace
* Merge remote-tracking branch 'origin/DEV' into feature/DAIP2-1803-add-aarete-derived-provider-name
Approved-by: Siddhant Medar
Bugfix/default ind postprocess
* Add logic to standardize UNIT_OF_MEASURE for flat-rate reimbursement methods
- Implemented functionality in `standardize_reimb_method_and_fee_schedule` to set UNIT_OF_MEASURE to blank for rows where DEFAULT_IND is 'Y' and AARETE_DERIVED_REIMB_METHOD is 'flat rate'.
- Added unit tests to verify behavior for various scenarios, including case insensitivity and non-default conditions.
- Ensured that UNIT_OF_MEASURE remains unchanged for non-flat rate methods.
* Enhance child rank handling and ensure column consistency in parent-child mapping
- Added initialization for the `child_rank` column in both parents and children DataFrames to prevent KeyError during concatenation when no children exist.
- Updated `cols_to_keep` in `parent_child_mapping` to filter out columns not present in `pc_df`, ensuring robustness in data processing.
* Ran Black
* made a small change in code_last_check, fixed so it returns string and not single char
* Made changes to make sure that default_ind postprocess only happens to the cc output and not dashboard
* Merged DEV into bugfix/default_ind_postprocess
* Restore deleted AARETE_DERIVED_PAYER_NAME functions
Functions were removed during previous commit. Restored from DEV to fix
AttributeError in prompt_calls.py.
* Simplify code_last_check return logic and add error logging
- Simplified return to single line with fallback
- Added error logging for failed LLM response parsing
Approved-by: Siddhant Medar
Bugfix/DAIP2-1701 issue tracker fixes methodology breakout
* testing fee schedules issues
* logging debug added for lesser of
* Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout
* special case term fix
* updated lesser of distribution prompt
* Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout
* refactor: use f-string for special case term concatenation
Approved-by: Siddhant Medar
Feature/DAIP2-1562 add aarete derived payer name
* black format
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* llm_choose_derived_payer_name function added
* threshold updated
* Merge branch 'DEV' into feature/DAIP2-1562-add-aarete_derived_payer_name
* aarete_derived_payer_name column added
* prompt structure updated
* state_flag added
* pipeline error fixed
* pipeline error fixed
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* remove debug print statement for similarity matrix in clustering function
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* state logic added in clustering
* removed print statements
* pull request updates
* black format fix
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* added scalability feature and optimization
* black format
* Pull Request Changes
* updated config parameters
* updated main
* no payer name column changes
* Business team feedback: always run derived payer name with state_flag=True
- Add empty DataFrame guard and proper docstring
- Clean up not needed flags
Approved-by: Siddhant Medar
Bugfix/dynamic issues feb12
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_qa-qc/ for QC/QA validated results and statistics
- parent-child/ for parent-child mapping outputs
- tracking/ for usage and cost tracking data
- individual/ for per-file CC results (dashboard individual files removed)
- Update file naming conventions to match new structure
- Remove QC/QA processing for error files (error files are saved without validation)
Post-Processing Changes:
- Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.)
when they are the only value in a cell (applies before CC/dashboard split)
- Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks)
- Ensure standard cleaning runs before splitting into CC …
* Ran Black for CI
* Refactor file splitting logic and add comprehensive tests
- Refactored splitting logic in io_utils.py:
- Consolidated repeated splitting code into two focused helper functions:
- _write_local_split_files() for local file writing with splitting
- _write_s3_split_files() for S3 file writing with splitting
- Both helpers use shared split_dataframe_by_filename() function
- Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py
- Added comprehensive test coverage for splitting logic:
- Tests for split_dataframe_by_filename() with various scenarios
- Tests for local and S3 write operations with single and multiple splits
- Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types
- Fixed existing test failures (write_s3 error handling, path assertions)
- Improved code maintainability and readability
* Fix failing tests in test_postprocess.py
- Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING
(PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped
- Added FILE_NAME column to all file structure test DataFrames (required for splitting logic)
- Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests
- Fixed patch decorators for S3 tests to properly mock logging
All 41 tests now passing.
* Black for CI
* Blank [] and ['[]'] in output instead of displaying them
- Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"]
- Update format_as_json_list to return blank for empty lists instead of []
- Filter out empty-list placeholder items from list values in format_as_json_list
- Add tests for clean_na_values empty list handling and format_as_json_list
Co-authored-by: Cursor <cursoragent@cursor.com>
* Merged DEV into feature/new_output_format
* feat: dynamic primary debug improvements and LOB partial 1:1 escalation
- Add per-exhibit debug summary for dynamic primary discovery (page, values, raw LLM)
- Pass exhibit_page to dynamic_primary for debug; mark debug-only params for removal
- Fix 1:1 escalation skip: use base_field for PROGRAM/PRODUCT/NETWORK when LOB detected
- Pass LOB to 1:1 when empty in any row (partial detection from stripped headers)
- Add debug exhibit text preview; mark check_and_combine_exhibit_inheritance for deletion
- prompt_dynamic_primary returns (answer, raw); remove per-field prints
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: pass PROGRAM and PRODUCT to 1:1 when partial LOB
When LOB is empty in some rows (partial detection), pass LOB, PROGRAM, and
PRODUCT to 1:1 for contract-level extraction. Merge fills only empty cells
so 1:N values are preserved. Skip PROGRAM/PRODUCT/NETWORK only when full LOB.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Removed debug print blocks
* Merge origin/DEV into bugfix/dynamic_issues_feb12
Resolved conflicts:
- runner.py: Use RUN_DASHBOARD_POSTPROCESSING for conditional dashboard; parent-child output_dir and S3 upload
- main.py: Use RUN_DASHBOARD_POSTPROCESSING for conditional dashboard
- postprocess.py: Optional dashboard postprocessing when RUN_DASHBOARD_POSTPROCESSING is True
Co-authored-by: Cursor <cursoragent@cursor.com>
Approved-by: Katon Minhas
Feature/new output format
* bugfix
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* requested postprocessing changes
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_qa-qc/ for QC/QA validated results and statistics
- parent-child/ for parent-child mapping outputs
- tracking/ for usage and cost tracking data
- individual/ for per-file CC results (dashboard individual files removed)
- Update file naming conventions to match new structure
- Remove QC/QA processing for error files (error files are saved without validation)
Post-Processing Changes:
- Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.)
when they are the only value in a cell (applies before CC/dashboard split)
- Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks)
- Ensure standard cleaning runs before splitting into CC …
* Ran Black for CI
* Refactor file splitting logic and add comprehensive tests
- Refactored splitting logic in io_utils.py:
- Consolidated repeated splitting code into two focused helper functions:
- _write_local_split_files() for local file writing with splitting
- _write_s3_split_files() for S3 file writing with splitting
- Both helpers use shared split_dataframe_by_filename() function
- Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py
- Added comprehensive test coverage for splitting logic:
- Tests for split_dataframe_by_filename() with various scenarios
- Tests for local and S3 write operations with single and multiple splits
- Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types
- Fixed existing test failures (write_s3 error handling, path assertions)
- Improved code maintainability and readability
* Fix failing tests in test_postprocess.py
- Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING
(PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped
- Added FILE_NAME column to all file structure test DataFrames (required for splitting logic)
- Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests
- Fixed patch decorators for S3 tests to properly mock logging
All 41 tests now passing.
* Black for CI
* Blank [] and ['[]'] in output instead of displaying them
- Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"]
- Update format_as_json_list to return blank for empty lists instead of []
- Filter out empty-list placeholder items from list values in format_as_json_list
- Add tests for clean_na_values empty list handling and format_as_json_list
Co-authored-by: Cursor <cursoragent@cursor.com>
* Merged DEV into feature/new_output_format
* Pipeline config, parent-child, dashboard, and runner fixes
- Parent-child: enable by default, write to run directory, S3 upload via io_utils
- Dashboard: optional (CC only by default), run_dashboard=True to enable
- Add io_utils.upload_local_file_to_s3 for centralized file uploads
- Parent-child returns (row_count, local_path); pipeline handles S3 upload
- Add TODO in config for WRITE_PC_TO_S3 removal after approval
- Fix indentation errors in runner.py
Co-authored-by: Cursor <cursoragent@cursor.com>
Approved-by: Katon Minhas
Bugfix/generic lesser of
* Updated lesser of ditribution and lesser of check prompt
* Remove excessive logging
* Updated prompt for combining and formatting lesser of statement and reimb term
* Code clean upt
* Removed unneeded instruction on output format
* style: format prompt_calls.py with black
Approved-by: Katon Minhas
Feature/claim type only runner
* Add specific_fields config for running extraction on field groups
## What Changed
4 files modified:
1. src/config.py - Added configuration for field-specific extraction:
- SPECIFIC_FIELDS arg (default: 'all') - pass field group name or comma-separated field names
- FIELD_GROUPS dict - predefined groups: claim_type, dates, provider
- get_specific_fields_list() - resolves config to actual field list
2. src/prompts/fieldset.py - Added filter_by_names() method to FieldSet class to filter fields by a list of names
3. src/pipelines/shared/extraction/one_to_n_funcs.py - Updated exhibit_level() to accept specific_fields parameter and skip prompts for fields not in the list
4. src/pipelines/saas/file_processing.py - Passes specific_fields through the call chain to both one_to_n and one_to_one extraction
## How It Works
When specific_fields is set to something other than 'all':
1. Config resolves the field list (either from FIELD_GROUPS dict or comma-separated names)
2. Before running extraction prompt…
* Add specific_fields filtering for claim_type only runs and fix UTF-8 logging
* Merge remote-tracking branch 'origin/DEV' into feature/claim-type-only-runner
* Fix black formatting in one_to_n_funcs.py
* Fix black formatting
* Fix CLAIM_TYPE_CD and AARETE_DERIVED_CLAIM_TYPE_CD to be single values
- Handle pipe-delimited strings in crosswalk by splitting them
- For CLAIM_TYPE_CD, take only first value in crosswalk since claim type should be singular
- Add single_value_fields handling in normalize_field_value for CLAIM_TYPE_CD fields
- Ensures AARETE_DERIVED_CLAIM_TYPE_CD is always a single M or H value
* Merged in feature/fix-claim-type-mapping (pull request #878)
Feature/fix claim type mapping
* Fix CLAIM_TYPE_CD and AARETE_DERIVED_CLAIM_TYPE_CD to be single values
- Handle pipe-delimited strings in crosswalk by splitting them
- For CLAIM_TYPE_CD, take only first value in crosswalk since claim type should be singular
- Add single_value_fields handling in normalize_field_value for CLAIM_TYPE_CD fields
- Ensures AARETE_DERIVED_CLAIM_TYPE_CD is always a single M or H value
* Update prompt template
* Merged feature/claim-type-only-runner into feature/fix-claim-type-mapping
Approved-by: Faizan Mohiuddin
* Address PR review comments
- Remove single_value_fields special handling from formatting_utils.py
- Remove CLAIM_TYPE_CD only section and pipe-delimited handling from aarete_derived.py
- Reference config.FIELD_GROUPS for provider_fields and date_fields in file_processing.py
* Fix CLAIM_TYPE_CD to AARETE_DERIVED_CLAIM_TYPE_CD mapping for claim_type only runs
- Apply crosswalk mapping in skip_reimbursement_processing block so derived
fields are correctly mapped even when running claim_type only extraction
- Fix _normalize_to_str to return first value when list has multiple elements
instead of returning the list itself (violated str return type)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* normalize list to str-list
Approved-by: Katon Minhas
DEV
* file_processing
* Merge branch 'bugfix/code_implicit' into DEV
* Update split reimb dates condition
* Pipe line general fixes, fixed tests, ran black, and fixed mypy type check issues, still have a list of lists issue with prov other name full
* Merged in feature/update-claim-type (pull request #869)
Feature/update claim type
* Update CLAIM_TYPE_CD extraction to prioritize title/header
- Modified retrieval_question to search for title, header, and agreement name
- Updated prompt to check title/header first before analyzing body text
- Enhanced FULL_CONTEXT_CLAIM_TYPES_ADDITIONAL_INSTRUCTION to emphasize title/header as primary source
* Fix CLAIM_TYPE_CD extraction with 3-step fallback
1. Exhibit Level: Updated retrieval_question and prompt to prioritize title/header
2. Contract Title fallback: Added infer_claim_type_from_title() function that extracts
claim type from CONTRACT_TITLE when exhibit-level extraction returns empty
3. Postprocessing: fill_claim_type() now uses CONTRACT_TITLE inference when all
AARETE_DERIVED_CLAIM_TYPE_CD values are empty
Keywords mapped:
- Professional/Ancillary -> M (Physician, Professional, Ancillary, Home Health, DME, etc.)
- Institutional -> H (Hospital, Facility, Surgery Center, SNF, etc.)
* Add more prof…
* Merged in bugfix/code-funcs (pull request #872)
Bugfix/code funcs
* Try-except code extraction
* Additional try-except for protection
* Black format
Approved-by: Praneel Panchigar
* Merged in feature/update-testbed-metrics (pull request #871)
Feature/update testbed metrics
* Combine fee schedule with methodology breakout; remove Reimbursement primary; do not run trigger cap or additions
* Deprecate reimb primary functions
* combine multiple sheets to one
* Clean 1:N metrics
* Improve dynamic primary evaluation
* Resolve dynamic primary
* Update row counts
* black format
* Merged DEV into feature/update-testbed-metrics
Approved-by: Praneel Panchigar
* Merged in feature/adding_contract_admenment_num_pc (pull request #870)
Feature/adding contract admenment num pc
* Updated contract amendment
* Format code with Black
* updated to have letters
* format fixes
* Adjusted unit testing
* Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc
* apply formatting
* format fixes
* Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc
* test conflicts solved
* Format code with Black
* format fixes
* reverted the changes for prompt templates
* test_case reverted
* Sorted effective date for ranking
* format fix for lint
* Merged DEV into feature/adding_contract_admenment_num_pc
* format
* Merged DEV into feature/adding_contract_admenment_num_pc
* Merged DEV into feature/adding_contract_admenment_num_pc
Approved-by: Siddhant Medar
* Consolidate docs and documentation
* Remove PRD
* Add Essential Plan as a NY Program
* Resolve Comments
* Resolve comments
* Black format
* Resolve remaining comments
* Black format
* Merged in bugfix/fill_claim_type (pull request #874)
Bugfix/fill claim type
* Handle if AARETE_DERIVED_CLAIM_TYPE_CD comes in as list
* add unit tests
* Black format
Approved-by: Siddhant Medar
* Remove print
* Merged in bugfix/parser-downstream-improvements (pull request #875)
Bugfix/parser downstream improvements
* Refactor: Implement field-aware JSON parsers with centralized normalization
This refactor introduces a robust system for normalizing LLM output based on
field format mappings, ensuring consistent data types throughout the pipeline.
Key Changes:
- Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields
- Create format_normalization.py utility for type-aware normalization
- Update json_utils.py parsers to accept field_names/field_name parameters
- Refactor prompt_templates.py to use parser factories (_create_json_dict_parser,
_create_json_list_parser) that bind field metadata for automatic normalization
- Update prompt_calls.py to pass field names to parsers, eliminating redundant
normalization logic
- Remove parse_json_dict_or_list (unused, ambiguous function)
- Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions
- Add comprehensive integration tests…
* Update other lists
* Black format
* Remove prints
* Update qa_qc to account for lists in is_empty check
* Resolve issues
* Black format
Approved-by: Siddhant Medar
Bugfix/parser downstream improvements
* Refactor: Implement field-aware JSON parsers with centralized normalization
This refactor introduces a robust system for normalizing LLM output based on
field format mappings, ensuring consistent data types throughout the pipeline.
Key Changes:
- Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields
- Create format_normalization.py utility for type-aware normalization
- Update json_utils.py parsers to accept field_names/field_name parameters
- Refactor prompt_templates.py to use parser factories (_create_json_dict_parser,
_create_json_list_parser) that bind field metadata for automatic normalization
- Update prompt_calls.py to pass field names to parsers, eliminating redundant
normalization logic
- Remove parse_json_dict_or_list (unused, ambiguous function)
- Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions
- Add comprehensive integration tests verifying normalization works end-to-end
Benefits:
- Single source of truth for field formats (FIELD_FORMAT_…
* refactor: normalize helper prompt outputs at prompt_calls level
- Update CARVEOUT_CHECK to use field-aware parser for CARVEOUT_CD normalization
- Update LOB_RELATIONSHIP to normalize to string format in prompt_calls.py
- Update SPLIT_REIMB_DATES to normalize date values in prompt_calls.py
- Remove defensive normalization from one_to_n_funcs.py for LOB relationships
- Remove manual normalization from split_reimb_dates() - values now normalized upstream
- All helper prompts that populate fields now normalize at prompt_calls.py level
- Downstream functions receive correctly formatted values without additional processing
* refactor: remove band-aid normalization functions and migrate HSC to field-aware parsers
- Update ONE_TO_ONE_SINGLE_FIELD_TEMPLATE to use field-aware parser with field_name parameter
- Remove list wrapping logic in hybrid_smart_chunking_funcs (field-aware parser handles normalization)
- Remove normalize_one_to_one_field_value and normalize_one_to_one_answers_dict from string_utils.py
- Remove all debug print statements from HSC processing
- Remove commented-out normalization calls from client-specific files (clover, bcbs_promise)
- All normalization now handled exclusively through FIELD_FORMAT_MAPPING via field-aware parsers
* Ran Black for formatting
* Print Statements removed, more cleaning
* Merge branch 'DEV' into bugfix/parser-downstream-improvements
* refactor: combine FIELD_FORMAT_MAPPING into investment_columns.py
- Merged field_format_mapping.py into investment_columns.py to create single source of truth
- FIELD_FORMAT_MAPPING now ordered by COLUMN_ORDER (161 fields)
- Added 5 missing fields from COLUMN_ORDER with default format types
- Updated all imports across codebase to use investment_columns
- Python dict preserves insertion order (3.7+), maintaining COLUMN_ORDER sequence
- All tests passing (38 field-aware tests verified)
* Deprecate COLUMN_ORDER, rely on Mapping only
* Merged DEV into bugfix/parser-downstream-improvements
Approved-by: Katon Minhas
Feature/adding contract admenment num pc
* Updated contract amendment
* Format code with Black
* updated to have letters
* format fixes
* Adjusted unit testing
* Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc
* apply formatting
* format fixes
* Merge remote-tracking branch 'origin/DEV' into feature/adding_contract_admenment_num_pc
* test conflicts solved
* Format code with Black
* format fixes
* reverted the changes for prompt templates
* test_case reverted
* Sorted effective date for ranking
* format fix for lint
* Merged DEV into feature/adding_contract_admenment_num_pc
* format
* Merged DEV into feature/adding_contract_admenment_num_pc
* Merged DEV into feature/adding_contract_admenment_num_pc
Approved-by: Siddhant Medar
Feature/update testbed metrics
* Combine fee schedule with methodology breakout; remove Reimbursement primary; do not run trigger cap or additions
* Deprecate reimb primary functions
* combine multiple sheets to one
* Clean 1:N metrics
* Improve dynamic primary evaluation
* Resolve dynamic primary
* Update row counts
* black format
* Merged DEV into feature/update-testbed-metrics
Approved-by: Praneel Panchigar
Feature/update claim type
* Update CLAIM_TYPE_CD extraction to prioritize title/header
- Modified retrieval_question to search for title, header, and agreement name
- Updated prompt to check title/header first before analyzing body text
- Enhanced FULL_CONTEXT_CLAIM_TYPES_ADDITIONAL_INSTRUCTION to emphasize title/header as primary source
* Fix CLAIM_TYPE_CD extraction with 3-step fallback
1. Exhibit Level: Updated retrieval_question and prompt to prioritize title/header
2. Contract Title fallback: Added infer_claim_type_from_title() function that extracts
claim type from CONTRACT_TITLE when exhibit-level extraction returns empty
3. Postprocessing: fill_claim_type() now uses CONTRACT_TITLE inference when all
AARETE_DERIVED_CLAIM_TYPE_CD values are empty
Keywords mapped:
- Professional/Ancillary -> M (Physician, Professional, Ancillary, Home Health, DME, etc.)
- Institutional -> H (Hospital, Facility, Surgery Center, SNF, etc.)
* Add more professional keywords for claim type inference
Added PROVIDER SERVICE, PROVIDER AGREEMENT, PARTICIPATING PROVIDER to handle
titles like 'PROVIDER SERVICES AGREEMENT'
* Add postprocessing safety net for claim type inference
- Added fill_claim_type_from_title() in postprocessing_funcs.py that infers
AARETE_DERIVED_CLAIM_TYPE_CD from CONTRACT_TITLE if still empty after code_breakout
- Added call in postprocess.py before attach_sid_column
- This ensures claim type is populated even if earlier extraction steps miss it
* Improve CLAIM_TYPE_CD extraction context and postprocessing
1. Added keywords to CLAIM_TYPE_CD field (same as CONTRACT_TITLE) so it retrieves
from the same document header/title context
2. Updated fill_claim_type_from_title() to first fill from other exhibits in the
same file (mode within file) before falling back to CONTRACT_TITLE keyword matching
* Merge DEV into feature/update-claim-type
Resolved conflicts:
- code_funcs.py: Kept DEV's code_breakout (claim type now handled in postprocessing)
- postprocess.py: Added fill_claim_type_from_title call
- prompt_templates.py: Kept DEV's VALIDATE_REIMBURSEMENTS_PROMPT signature
* Merge DEV into feature/update-claim-type
* move keywords to ancillary
Approved-by: Katon Minhas
Katon Minhas