Feature/new output format
* bugfix
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* requested postprocessing changes
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_qa-qc/ for QC/QA validated results and statistics
- parent-child/ for parent-child mapping outputs
- tracking/ for usage and cost tracking data
- individual/ for per-file CC results (dashboard individual files removed)
- Update file naming conventions to match new structure
- Remove QC/QA processing for error files (error files are saved without validation)
Post-Processing Changes:
- Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.)
when they are the only value in a cell (applies before CC/dashboard split)
- Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks)
- Ensure standard cleaning runs before splitting into CC …
* Ran Black for CI
* Refactor file splitting logic and add comprehensive tests
- Refactored splitting logic in io_utils.py:
- Consolidated repeated splitting code into two focused helper functions:
- _write_local_split_files() for local file writing with splitting
- _write_s3_split_files() for S3 file writing with splitting
- Both helpers use shared split_dataframe_by_filename() function
- Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py
- Added comprehensive test coverage for splitting logic:
- Tests for split_dataframe_by_filename() with various scenarios
- Tests for local and S3 write operations with single and multiple splits
- Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types
- Fixed existing test failures (write_s3 error handling, path assertions)
- Improved code maintainability and readability
* Fix failing tests in test_postprocess.py
- Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING
(PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped
- Added FILE_NAME column to all file structure test DataFrames (required for splitting logic)
- Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests
- Fixed patch decorators for S3 tests to properly mock logging
All 41 tests now passing.
* Black for CI
* Blank [] and ['[]'] in output instead of displaying them
- Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"]
- Update format_as_json_list to return blank for empty lists instead of []
- Filter out empty-list placeholder items from list values in format_as_json_list
- Add tests for clean_na_values empty list handling and format_as_json_list
Co-authored-by: Cursor <cursoragent@cursor.com>
* Merged DEV into feature/new_output_format
* Pipeline config, parent-child, dashboard, and runner fixes
- Parent-child: enable by default, write to run directory, S3 upload via io_utils
- Dashboard: optional (CC only by default), run_dashboard=True to enable
- Add io_utils.upload_local_file_to_s3 for centralized file uploads
- Parent-child returns (row_count, local_path); pipeline handles S3 upload
- Add TODO in config for WRITE_PC_TO_S3 removal after approval
- Fix indentation errors in runner.py
Co-authored-by: Cursor <cursoragent@cursor.com>
Approved-by: Katon Minhas
Bugfix/parser downstream improvements
* Refactor: Implement field-aware JSON parsers with centralized normalization
This refactor introduces a robust system for normalizing LLM output based on
field format mappings, ensuring consistent data types throughout the pipeline.
Key Changes:
- Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields
- Create format_normalization.py utility for type-aware normalization
- Update json_utils.py parsers to accept field_names/field_name parameters
- Refactor prompt_templates.py to use parser factories (_create_json_dict_parser,
_create_json_list_parser) that bind field metadata for automatic normalization
- Update prompt_calls.py to pass field names to parsers, eliminating redundant
normalization logic
- Remove parse_json_dict_or_list (unused, ambiguous function)
- Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions
- Add comprehensive integration tests verifying normalization works end-to-end
Benefits:
- Single source of truth for field formats (FIELD_FORMAT_…
* refactor: normalize helper prompt outputs at prompt_calls level
- Update CARVEOUT_CHECK to use field-aware parser for CARVEOUT_CD normalization
- Update LOB_RELATIONSHIP to normalize to string format in prompt_calls.py
- Update SPLIT_REIMB_DATES to normalize date values in prompt_calls.py
- Remove defensive normalization from one_to_n_funcs.py for LOB relationships
- Remove manual normalization from split_reimb_dates() - values now normalized upstream
- All helper prompts that populate fields now normalize at prompt_calls.py level
- Downstream functions receive correctly formatted values without additional processing
* refactor: remove band-aid normalization functions and migrate HSC to field-aware parsers
- Update ONE_TO_ONE_SINGLE_FIELD_TEMPLATE to use field-aware parser with field_name parameter
- Remove list wrapping logic in hybrid_smart_chunking_funcs (field-aware parser handles normalization)
- Remove normalize_one_to_one_field_value and normalize_one_to_one_answers_dict from string_utils.py
- Remove all debug print statements from HSC processing
- Remove commented-out normalization calls from client-specific files (clover, bcbs_promise)
- All normalization now handled exclusively through FIELD_FORMAT_MAPPING via field-aware parsers
* Ran Black for formatting
* Print Statements removed, more cleaning
* Merge branch 'DEV' into bugfix/parser-downstream-improvements
* refactor: combine FIELD_FORMAT_MAPPING into investment_columns.py
- Merged field_format_mapping.py into investment_columns.py to create single source of truth
- FIELD_FORMAT_MAPPING now ordered by COLUMN_ORDER (161 fields)
- Added 5 missing fields from COLUMN_ORDER with default format types
- Updated all imports across codebase to use investment_columns
- Python dict preserves insertion order (3.7+), maintaining COLUMN_ORDER sequence
- All tests passing (38 field-aware tests verified)
* Deprecate COLUMN_ORDER, rely on Mapping only
* Merged DEV into bugfix/parser-downstream-improvements
Approved-by: Katon Minhas
Feature/update claim type
* Update CLAIM_TYPE_CD extraction to prioritize title/header
- Modified retrieval_question to search for title, header, and agreement name
- Updated prompt to check title/header first before analyzing body text
- Enhanced FULL_CONTEXT_CLAIM_TYPES_ADDITIONAL_INSTRUCTION to emphasize title/header as primary source
* Fix CLAIM_TYPE_CD extraction with 3-step fallback
1. Exhibit Level: Updated retrieval_question and prompt to prioritize title/header
2. Contract Title fallback: Added infer_claim_type_from_title() function that extracts
claim type from CONTRACT_TITLE when exhibit-level extraction returns empty
3. Postprocessing: fill_claim_type() now uses CONTRACT_TITLE inference when all
AARETE_DERIVED_CLAIM_TYPE_CD values are empty
Keywords mapped:
- Professional/Ancillary -> M (Physician, Professional, Ancillary, Home Health, DME, etc.)
- Institutional -> H (Hospital, Facility, Surgery Center, SNF, etc.)
* Add more professional keywords for claim type inference
Added PROVIDER SERVICE, PROVIDER AGREEMENT, PARTICIPATING PROVIDER to handle
titles like 'PROVIDER SERVICES AGREEMENT'
* Add postprocessing safety net for claim type inference
- Added fill_claim_type_from_title() in postprocessing_funcs.py that infers
AARETE_DERIVED_CLAIM_TYPE_CD from CONTRACT_TITLE if still empty after code_breakout
- Added call in postprocess.py before attach_sid_column
- This ensures claim type is populated even if earlier extraction steps miss it
* Improve CLAIM_TYPE_CD extraction context and postprocessing
1. Added keywords to CLAIM_TYPE_CD field (same as CONTRACT_TITLE) so it retrieves
from the same document header/title context
2. Updated fill_claim_type_from_title() to first fill from other exhibits in the
same file (mode within file) before falling back to CONTRACT_TITLE keyword matching
* Merge DEV into feature/update-claim-type
Resolved conflicts:
- code_funcs.py: Kept DEV's code_breakout (claim type now handled in postprocessing)
- postprocess.py: Added fill_claim_type_from_title call
- prompt_templates.py: Kept DEV's VALIDATE_REIMBURSEMENTS_PROMPT signature
* Merge DEV into feature/update-claim-type
* move keywords to ancillary
Approved-by: Katon Minhas
Bugfix/generic dynamic jan26
* Refactor: Extract dynamic primary metrics logic to separate module
- Created src/testbed/dynamic_primary_metrics.py with all dynamic primary field analysis logic
- Moved tally-based analysis functions to new module for better modularity
- Updated testbed_utils.py to import and use functions from dynamic_primary_metrics
- Simplified testbed_metrics_dynamic_only.py to use new module
- Removed unused analyze_dynamic_primary_fields function (400+ lines)
- Added field filtering in match_rows to handle missing columns gracefully
- Minimal changes to testbed_utils.py (~100 lines vs 845 before)
* feat: Add debug print blocks and prompt changes for dynamic primary fields in 1:1 processing
This commit adds prompt changes for when we pass dynamic primary to 1:1 and comprehensive debug logging for dynamic primary fields when they are escalated to 1:1 processing via Hybrid Smart Chunking (HSC) or Full Context methods.
Changes:
- Added debug print blocks in hybrid_smart_chunking_funcs.py to log:
* Retrieval question, context chunks, and final prompt for HSC processing
* Raw LLM output (reasoning + answer) for dynamic primary fields
- Added debug print blocks in prompt_calls.py to log:
* Contract context, final prompt for Full Context processing
* Raw LLM output (reasoning + answer) for dynamic primary fields
- Added FULL_CONTEXT_DYNAMIC_PRIMARY_INSTRUCTION() to prompt_templates.py:
* Provides specific guidance for dynamic primary fields in full context
* Emphasizes extracting values only if they refer to entire contract
* Includes pipe-delimited formatting instructions
- Updated dynamic_fun…
* Ran black, for formatting
* Refine dynamic primary field prompts and add Duals auto-detection
- Enhanced DYNAMIC_ASSIGNMENT_INSTRUCTION with structural boundary rules
- Added exhibit header and subsection context hierarchy for LOB assignment
- Implemented Duals auto-detection in dynamic_primary discovery (Medicare+Medicaid -> Duals)
- Added Duals value formatting instruction to prevent oversimplification
- Commented out DUAL_LOB_CHECK for performance testing
- Removed verbose debug print blocks from HSC and Full Context processing
- Added focused debug print for dynamic assignment raw LLM output
* Remove debug print blocks from pipeline files
- Removed all debug print blocks from file_processing.py (8 blocks)
- Removed all debug print blocks from dynamic_funcs.py (8 blocks)
- Removed debug print block from prompt_calls.py (dynamic assignment)
- DUAL_LOB_CHECK remains commented as requested
- Total: 188 lines of debug code removed
* Merge remote-tracking branch 'origin/main' into bugfix/generic_dynamic_jan26
* Ran black for CI
* Remove update_lob_for_duals
* Merge branch 'DEV' into bugfix/generic_dynamic_jan26
Approved-by: Katon Minhas
Feature/update caching
* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT
- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix postprocessing_funcs to remove invalid columns
* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid
* Apply black formatting to 11 files
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix import paths in testbed files
* Merge main into feature/update-caching
* Add SPLIT_REIMB_DATES_INSTRUCTION for prompt caching
* Merged main into feature/update-caching
* Merge remote-tracking branch 'origin/main' into feature/update-caching
* Merge branch 'feature/update-caching' of https://bitbucket.org/aarete/doczy.ai into feature/update-caching
* Add CODE_* instruction functions for prompt caching
* Merged main into feature/update-caching
* Merged main into feature/update-caching
* Maximize prompt caching by moving static content to instructions
- Refactored 12+ prompt functions to move static rules/examples to _INSTRUCTION()
- VALIDATE_REIMBURSEMENTS: Moved 45+ lines of validation rules to instruction
- DATE_FIX: Moved conversion rules and examples to instruction
- EXHIBIT_LEVEL, EXHIBIT_TITLE_MATCH, EXHIBIT_HEADER, EXHIBIT_LINKAGE: Moved rules
- FEE_SCHEDULE_BREAKOUT, GROUPER_BREAKOUT: Moved extraction rules
- CHECK_PROVIDER_NAME_MATCH: Moved matching rules to instruction
- LOB_RELATIONSHIP: Consolidated split instructions
- DERIVED_TERM_DATE: Moved calculation logic to instruction
- DYNAMIC_PRIMARY: Created separate HEADER and TEXT instructions
- Created new REIMB_DATES_ASSIGNMENT_INSTRUCTION with all date rules
- Updated get_cacheable_instructions() with new instruction functions
- Updated all callers in saas, clover, bcbs_promise pipelines
This enables ~90% cost savings via Claude API prompt caching by ensuring
static instruction text is cached and only dynamic content varies.
* Remove Dynamic Primary Header
* Standardize prompt caching across all LLM prompts for ~90% cost savings
Move static content to _INSTRUCTION() functions for caching:
- METHODOLOGY_BREAKOUT: 8 field definitions with resolved valid_values
- FEE_SCHEDULE_BREAKOUT: 3 field definitions
- GROUPER_BREAKOUT: 11 field definitions
- OUTLIER_BREAKOUT: 8 field definitions
- CODE_EXPLICIT: 8 field definitions
- EXHIBIT_HEADER: static header markers list
- CARVEOUT_CHECK: carveout and special case definitions
- SPECIAL_CASE_ASSIGNMENT: task rules and output format
Update callers to use cache=True, instruction, and usage_label params.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Apply black formatting to prompt_templates.py
* Update unit tests for prompt caching changes
- Fix function signatures in tests (METHODOLOGY_BREAKOUT, FEE_SCHEDULE_BREAKOUT, GROUPER_BREAKOUT, CODE_EXPLICIT)
- Add tests for CARVEOUT_CHECK and EXHIBIT_HEADER prompts
- Add TestInstructionFieldContent class to verify instruction functions contain field definitions
- Add tests for CARVEOUT_CHECK_INSTRUCTION and CODE_EXPLICIT_INSTRUCTION
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Refactor instructions to load fields from investment_prompts.json and Constants
Address PR review feedback:
- Add _get_constants() helper for lazy Constants instantiation
- Add _get_fields_text() helper to load fields from investment_prompts.json
- METHODOLOGY_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- FEE_SCHEDULE_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- GROUPER_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- OUTLIER_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- EXHIBIT_HEADER_INSTRUCTION: Load markers from Constants.EXHIBIT_HEADER_MARKERS
- CARVEOUT_CHECK_INSTRUCTION: Load carveouts from Constants.VALID_CARVEOUTS,
special cases from investment_prompts.json
- CODE_EXPLICIT_INSTRUCTION: Keep hardcoded (no code_explicit field_type in JSON)
Update tests to match actual field names from investment_prompts.json.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Merge main into feature/update-caching
* Remove deprecated dynamic_primary_header
* Update CODE_EXPLICIT_INSTRUCTION to load fields from investment_prompts.json
Use field_type=code_primary_breakout to load code fields dynamically.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Merged main into feature/update-caching
* Merge main into feature/update-caching
* Merge branch 'feature/update-caching' of https://bitbucket.org/aarete/doczy.ai into feature/update-caching
Approved-by: Katon Minhas
Feature/lesser table caching refactor hybrid
* chore: Remove unused duplicate main.py from shared pipeline
* fix: Correct crosswalk paths in aarete_derived.py
* chore: Remove unused documentation files from fieldExtraction
* docs: Add documentation files to documentation folder
* docs: Update README with uv setup, expanded project structure, and branching conventions
* docs: Add uv installation steps with Ubuntu/WSL emphasis
* Enable prompt caching for all remaining LLM calls
- Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE,
EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH,
SPECIAL_CASE_ASSIGNMENT
- Update all invoke_claude() calls in saas and clover pipelines to use
cache=True with corresponding _INSTRUCTION() functions
- Add new instructions to get_cacheable_instructions() for cache warming
- Update tests for new instruction functions
Functions now using caching:
- prompt_exhibit_level
- prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF)
- prompt_fee_schedule_breakout
- prompt_grouper_breakout
- prompt_special_case_assignment
- prompt_exhibit_linkage
- prompt_exhibit_header
- prompt_smart_chunked (ONE_TO_ONE templates)
- prompt_date_fix
- prompt_derived_term_date
- prompt_exhibit_title_match
- provider_name_match_check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Reorder
* feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction
- Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction
- Extract full paragraph text of offset/recoupment provisions from contracts
- Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence
- Fix reorder_columns to preserve extra columns not in COLUMN_ORDER
- Update QC/QA output path to outputs/qc_qa/
* fix: Update dev deps and test assertions for QC/QA output path
- Add pytest/pytest-mock to dev dependencies for mypy type checking
- Update test assertions to expect outputs/qc_qa instead of qa_qc_output
* style: Apply black formatting to prompt_templates.py
* Merge main, move scripts
* Archive some scripts
* update py version
* remove .py version file
* Remove ASCII characters
* Restore testbed code
* restore tracking
* Update testbed metrics
* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT
- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix postprocessing_funcs to remove invalid columns
* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid
* Revert prompt caching changes from aed1b73c
* update formatting
* Update imports
Approved-by: Sha Brown
Approved-by: Praneel Panchigar
Praneel Panchigar