Feature/dynamic codes final
* Move Claim Type to dynamic code group
* Update CLaim Type and Bill TYpe prompts
* Prompt optimization
* Prompt tuning
* streamline
* Format
* Merged dev into feature/dynamic-codes
* test claim and bill type
* Merged dev into DAIP2-2186-service-claim-type-testing
* Merged dev into feature/dynamic-codes
* prompt updated for claim type
* Merge branch 'feature/dynamic-codes' into DAIP2-2186-service-claim-type-testing
* Merged dev into DAIP2-2186-service-claim-type-testing
* Remove print
* Switch to map
* Merged dev into feature/dynamic-codes-final
* prompt inconsistency fixed
Approved-by: Praneel Panchigar
Approved-by: Siddhant Medar
Bugfix/DAIP2-2164 reimbursement primary issue fixes
* sample calculations reimbursements removed
* split service term added
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* updated prompt and unit test added
* added prompt for codes listed below
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Output format at end
* Genericize prompt
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* context missing issue fixed
* updated service term split prompt for program varints
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* updated split service term prompt
* optimized and minimized split service term prompt
* updated service term split prompt
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged dev into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Updated Prompt
* grouped instructions
* removed print statements
Approved-by: Katon Minhas
Feature/DAIP2-2023 eliminate full context processing
* testing full context fields
* remove full context processing
* merge Dev with DAIP2-2-23
* full context removal in client codes
* AARETE_DERIVED_PROVIDER_NAME field changes
* Merged DEV into feature/DAIP2-2023-eliminate-full-context-processing
* optimized provider name
* black format fix
* contract title fixes
* PAYER NAME AUTO RENEWAL IND fixes
* Merged DEV into feature/DAIP2-2023-eliminate-full-context-processing
* Merge branch 'DEV' into feature/DAIP2-2023-eliminate-full-context-processing
* Remove prints
Approved-by: Katon Minhas
DAIP2-1980 aarete derived effective dt issues
* fixed earlier of a and b
* prompt update
* prompt shortened
* test changes reverted
* Merge branch 'DEV' into DAIP2-1980-aarete-derived-effective-dt-issues
* test changes reverted
* Merged DEV into DAIP2-1980-aarete-derived-effective-dt-issues
* Fix effective date prompt
* Replace em-dashes with readable version
Approved-by: Katon Minhas
Bugfix/prov info json fixes
* fix: robust PROV_INFO_JSON sanitization and TIN backfill logic
json_utils:
- Add sanitize_prov_info_json with layered parsing (JSON, literal_eval,
empty-value-after-colon fix, best-effort dict extraction).
- Add _normalize_prov_entries and _prov_value_to_str for uniform
str-valued output; flatten list values, strip TIN hyphens.
- format_prov_info_json now delegates to sanitize_prov_info_json.
postprocessing_funcs:
- Add fill_prov_info_tin_from_filename_tin for TIN backfill.
- Add validate_and_reformat_date (pipe-wrapped, datetime strings).
- Add format_as_json_list (pipe-delimited, comma-separated, quote
stripping).
postprocess:
- Integrate new postprocessing helpers into pipeline flow.
postprocess_existing_output:
- Support CSV and Excel input, configurable paths, fillna for CSV.
tests:
- Add test_json_parsers.py for PROV_INFO_JSON parsing coverage.
- Add test_postprocess.py for date/list formatting and default_ind.
* Merge branch 'DAIP2-1947-tin-and-prov-info-json-issues' into bugfix/prov_info_json_fixes
* Merged DEV into bugfix/prov_info_json_fixes
* Strip out unused functionality
* Update filename_tin functionality
* Add docstring
* Update filename_tin cleaning in PROV_INFO_JSON
* test prep
* black format
* missing function added
* Merge branch 'DEV' into bugfix/prov_info_json_fixes
* Black format
* Strip unused functions
* Strip unused code
* update unit tests
* Merge branch 'DEV' into bugfix/prov_info_json_fixes
* Update test
* Simplify process
* handle list of group names
* Resolve run_provider_info_field call
* Merge branch 'DEV' into bugfix/prov_info_json_fixes
* Correct type hints
* Fix unit tests
* Fix unit test
* Remove redundant postprocessing_funcs
Approved-by: Katon Minhas
Feature/exhibit smart chunking
* exhibit processing per page
* header dict deduplication
* dedup prompt refinment
* refinment for header extraction proecess
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* merge updates
* minor fix
* prompt fix for reimb type
* COB defenition for clear understanding
* black formatting
* remove quit statement
* pipiline test
* pipeline test
* black formatting
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* black formating
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* typo
* PR comment fixes
* exhibit funcs refactored
* black formatting
* Refactor exhibit chunking config into dedicated class
Created ExhibitChunkingConfig class to centralize exhibit smart chunking
configuration parameters (DEFAULT_SUBCHUNK_SIZE, MIN_PARENT_CHUNK_SIZE,
CHUNK_RELEVANCE_THRESHOLD). This improves code organization by consolidating
related constants and makes configuration more maintainable.
Changes:
- Created ExhibitChunkingConfig class with ESC_CONFIG instance
- Moved CHUNK_RELEVANCE_THRESHOLD from config.py to ExhibitChunkingConfig
- Updated all constant references to use ESC_CONFIG prefix
- Added missing EXHIBIT_HEADER_MARKERS parameter documentation
* Fix logging levels and refactor imports for exhibit chunking
- Upgrade logging from WARNING to ERROR for embedding and semantic search failures
- Remove unused constant imports from exhibit_funcs.py
- Update test imports to use ESC_CONFIG pattern for configuration constants
- Add warning when no exhibit headers found during deduplication
- Expand mypy type checking by removing s3_utilities from exclude list
* Merged DEV into feature/exhibit-smart-chunking
* Merged DEV into feature/exhibit-smart-chunking
Approved-by: Siddhant Medar
Feature/new output format
* bugfix
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* requested postprocessing changes
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_qa-qc/ for QC/QA validated results and statistics
- parent-child/ for parent-child mapping outputs
- tracking/ for usage and cost tracking data
- individual/ for per-file CC results (dashboard individual files removed)
- Update file naming conventions to match new structure
- Remove QC/QA processing for error files (error files are saved without validation)
Post-Processing Changes:
- Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.)
when they are the only value in a cell (applies before CC/dashboard split)
- Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks)
- Ensure standard cleaning runs before splitting into CC …
* Ran Black for CI
* Refactor file splitting logic and add comprehensive tests
- Refactored splitting logic in io_utils.py:
- Consolidated repeated splitting code into two focused helper functions:
- _write_local_split_files() for local file writing with splitting
- _write_s3_split_files() for S3 file writing with splitting
- Both helpers use shared split_dataframe_by_filename() function
- Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py
- Added comprehensive test coverage for splitting logic:
- Tests for split_dataframe_by_filename() with various scenarios
- Tests for local and S3 write operations with single and multiple splits
- Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types
- Fixed existing test failures (write_s3 error handling, path assertions)
- Improved code maintainability and readability
* Fix failing tests in test_postprocess.py
- Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING
(PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped
- Added FILE_NAME column to all file structure test DataFrames (required for splitting logic)
- Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests
- Fixed patch decorators for S3 tests to properly mock logging
All 41 tests now passing.
* Black for CI
* Blank [] and ['[]'] in output instead of displaying them
- Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"]
- Update format_as_json_list to return blank for empty lists instead of []
- Filter out empty-list placeholder items from list values in format_as_json_list
- Add tests for clean_na_values empty list handling and format_as_json_list
Co-authored-by: Cursor <cursoragent@cursor.com>
* Merged DEV into feature/new_output_format
* Pipeline config, parent-child, dashboard, and runner fixes
- Parent-child: enable by default, write to run directory, S3 upload via io_utils
- Dashboard: optional (CC only by default), run_dashboard=True to enable
- Add io_utils.upload_local_file_to_s3 for centralized file uploads
- Parent-child returns (row_count, local_path); pipeline handles S3 upload
- Add TODO in config for WRITE_PC_TO_S3 removal after approval
- Fix indentation errors in runner.py
Co-authored-by: Cursor <cursoragent@cursor.com>
Approved-by: Katon Minhas
Feature/claim type only runner
* Add specific_fields config for running extraction on field groups
## What Changed
4 files modified:
1. src/config.py - Added configuration for field-specific extraction:
- SPECIFIC_FIELDS arg (default: 'all') - pass field group name or comma-separated field names
- FIELD_GROUPS dict - predefined groups: claim_type, dates, provider
- get_specific_fields_list() - resolves config to actual field list
2. src/prompts/fieldset.py - Added filter_by_names() method to FieldSet class to filter fields by a list of names
3. src/pipelines/shared/extraction/one_to_n_funcs.py - Updated exhibit_level() to accept specific_fields parameter and skip prompts for fields not in the list
4. src/pipelines/saas/file_processing.py - Passes specific_fields through the call chain to both one_to_n and one_to_one extraction
## How It Works
When specific_fields is set to something other than 'all':
1. Config resolves the field list (either from FIELD_GROUPS dict or comma-separated names)
2. Before running extraction prompt…
* Add specific_fields filtering for claim_type only runs and fix UTF-8 logging
* Merge remote-tracking branch 'origin/DEV' into feature/claim-type-only-runner
* Fix black formatting in one_to_n_funcs.py
* Fix black formatting
* Fix CLAIM_TYPE_CD and AARETE_DERIVED_CLAIM_TYPE_CD to be single values
- Handle pipe-delimited strings in crosswalk by splitting them
- For CLAIM_TYPE_CD, take only first value in crosswalk since claim type should be singular
- Add single_value_fields handling in normalize_field_value for CLAIM_TYPE_CD fields
- Ensures AARETE_DERIVED_CLAIM_TYPE_CD is always a single M or H value
* Merged in feature/fix-claim-type-mapping (pull request #878)
Feature/fix claim type mapping
* Fix CLAIM_TYPE_CD and AARETE_DERIVED_CLAIM_TYPE_CD to be single values
- Handle pipe-delimited strings in crosswalk by splitting them
- For CLAIM_TYPE_CD, take only first value in crosswalk since claim type should be singular
- Add single_value_fields handling in normalize_field_value for CLAIM_TYPE_CD fields
- Ensures AARETE_DERIVED_CLAIM_TYPE_CD is always a single M or H value
* Update prompt template
* Merged feature/claim-type-only-runner into feature/fix-claim-type-mapping
Approved-by: Faizan Mohiuddin
* Address PR review comments
- Remove single_value_fields special handling from formatting_utils.py
- Remove CLAIM_TYPE_CD only section and pipe-delimited handling from aarete_derived.py
- Reference config.FIELD_GROUPS for provider_fields and date_fields in file_processing.py
* Fix CLAIM_TYPE_CD to AARETE_DERIVED_CLAIM_TYPE_CD mapping for claim_type only runs
- Apply crosswalk mapping in skip_reimbursement_processing block so derived
fields are correctly mapped even when running claim_type only extraction
- Fix _normalize_to_str to return first value when list has multiple elements
instead of returning the list itself (violated str return type)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* normalize list to str-list
Approved-by: Katon Minhas
Bugfix/parser downstream improvements
* Refactor: Implement field-aware JSON parsers with centralized normalization
This refactor introduces a robust system for normalizing LLM output based on
field format mappings, ensuring consistent data types throughout the pipeline.
Key Changes:
- Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields
- Create format_normalization.py utility for type-aware normalization
- Update json_utils.py parsers to accept field_names/field_name parameters
- Refactor prompt_templates.py to use parser factories (_create_json_dict_parser,
_create_json_list_parser) that bind field metadata for automatic normalization
- Update prompt_calls.py to pass field names to parsers, eliminating redundant
normalization logic
- Remove parse_json_dict_or_list (unused, ambiguous function)
- Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions
- Add comprehensive integration tests verifying normalization works end-to-end
Benefits:
- Single source of truth for field formats (FIELD_FORMAT_…
* refactor: normalize helper prompt outputs at prompt_calls level
- Update CARVEOUT_CHECK to use field-aware parser for CARVEOUT_CD normalization
- Update LOB_RELATIONSHIP to normalize to string format in prompt_calls.py
- Update SPLIT_REIMB_DATES to normalize date values in prompt_calls.py
- Remove defensive normalization from one_to_n_funcs.py for LOB relationships
- Remove manual normalization from split_reimb_dates() - values now normalized upstream
- All helper prompts that populate fields now normalize at prompt_calls.py level
- Downstream functions receive correctly formatted values without additional processing
* refactor: remove band-aid normalization functions and migrate HSC to field-aware parsers
- Update ONE_TO_ONE_SINGLE_FIELD_TEMPLATE to use field-aware parser with field_name parameter
- Remove list wrapping logic in hybrid_smart_chunking_funcs (field-aware parser handles normalization)
- Remove normalize_one_to_one_field_value and normalize_one_to_one_answers_dict from string_utils.py
- Remove all debug print statements from HSC processing
- Remove commented-out normalization calls from client-specific files (clover, bcbs_promise)
- All normalization now handled exclusively through FIELD_FORMAT_MAPPING via field-aware parsers
* Ran Black for formatting
* Print Statements removed, more cleaning
* Merge branch 'DEV' into bugfix/parser-downstream-improvements
* refactor: combine FIELD_FORMAT_MAPPING into investment_columns.py
- Merged field_format_mapping.py into investment_columns.py to create single source of truth
- FIELD_FORMAT_MAPPING now ordered by COLUMN_ORDER (161 fields)
- Added 5 missing fields from COLUMN_ORDER with default format types
- Updated all imports across codebase to use investment_columns
- Python dict preserves insertion order (3.7+), maintaining COLUMN_ORDER sequence
- All tests passing (38 field-aware tests verified)
* Deprecate COLUMN_ORDER, rely on Mapping only
* Merged DEV into bugfix/parser-downstream-improvements
Approved-by: Katon Minhas
Feature/lesser table caching refactor hybrid
* chore: Remove unused duplicate main.py from shared pipeline
* fix: Correct crosswalk paths in aarete_derived.py
* chore: Remove unused documentation files from fieldExtraction
* docs: Add documentation files to documentation folder
* docs: Update README with uv setup, expanded project structure, and branching conventions
* docs: Add uv installation steps with Ubuntu/WSL emphasis
* Enable prompt caching for all remaining LLM calls
- Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE,
EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH,
SPECIAL_CASE_ASSIGNMENT
- Update all invoke_claude() calls in saas and clover pipelines to use
cache=True with corresponding _INSTRUCTION() functions
- Add new instructions to get_cacheable_instructions() for cache warming
- Update tests for new instruction functions
Functions now using caching:
- prompt_exhibit_level
- prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF)
- prompt_fee_schedule_breakout
- prompt_grouper_breakout
- prompt_special_case_assignment
- prompt_exhibit_linkage
- prompt_exhibit_header
- prompt_smart_chunked (ONE_TO_ONE templates)
- prompt_date_fix
- prompt_derived_term_date
- prompt_exhibit_title_match
- provider_name_match_check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Reorder
* feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction
- Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction
- Extract full paragraph text of offset/recoupment provisions from contracts
- Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence
- Fix reorder_columns to preserve extra columns not in COLUMN_ORDER
- Update QC/QA output path to outputs/qc_qa/
* fix: Update dev deps and test assertions for QC/QA output path
- Add pytest/pytest-mock to dev dependencies for mypy type checking
- Update test assertions to expect outputs/qc_qa instead of qa_qc_output
* style: Apply black formatting to prompt_templates.py
* Merge main, move scripts
* Archive some scripts
* update py version
* remove .py version file
* Remove ASCII characters
* Restore testbed code
* restore tracking
* Update testbed metrics
* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT
- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix postprocessing_funcs to remove invalid columns
* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid
* Revert prompt caching changes from aed1b73c
* update formatting
* Update imports
Approved-by: Sha Brown
Approved-by: Praneel Panchigar
Katon Minhas