- Updated all prompt_calls.py files (saas, bcbs_promise, clover) to use new parser pattern
- Updated prompt_templates.py with JSON format instructions
- Updated crosswalk_utils.py to handle JSON lists
- Updated json_utils.py with improved parsing
- Updated aarete_derived.py for JSON compatibility
- Updated tin_npi_funcs.py and qa_qc_utils.py for JSON parsing
- All changes align with Phase 4 completion of JSON standardization
- Removed all debug print statements from prompt_calls.py and one_to_n_funcs.py
- Updated PRD to reflect Phase 4 completion and recent bug fixes
- Documented REIMB_TERM normalization fixes and cleanup work
- Updated Phase 6 status to 25% complete (debug removal done)
- Fixed prompt_lesser_of_distribution to extract string from list when parser returns list
- Added debug print statements to trace REIMB_TERM format through processing pipeline
- Removed defensive normalization checks that are no longer needed upstream
- Added debug output in reimbursement_level and methodology_breakout_single_row to track data flow
- Updated progress table: Phase 4 is now 100% complete
- Marked model_evaluation_utils.py as complete (commit 13f3cbea)
- Verified testbed/QC files process internal data (out of scope)
- Updated remaining work section
- Updated overall status to Phase 4 of 6 Complete
- Fixed DYNAMIC_PRIMARY_TEXT: now unpacks (prompt, parser) tuple and uses parser
- Fixed REIMBURSEMENT_PRIMARY: corrected function signature (only takes context)
- Replaced extract_text_from_delimiters with JSON parser
- Replaced universal_json_load with parser from tuple
- Added proper instruction caching for both prompts
Other testbed/QC files verified: they process internal data formats (out of scope per PRD)
- Updated progress table: Phase 3 is now 100% complete
- Marked code_funcs.py as complete (commit a7a32b90)
- Updated remaining work section
- Added latest commit to commit history
- Line 498: Changed from universal_json_load to use _parser from FILL_BILL_TYPE tuple
- Line 856: Updated GROUPER_BREAKOUT call to unpack tuple and use grouper_parser
- Removed unused GROUPER_QUESTIONS variable (fields now in INSTRUCTION function for caching)
- All universal_json_load calls in code_funcs.py have been replaced
- Phase 3 is now 100% complete
- Updated EFFECTIVE_DT field: Replaced all pipe format examples (|date|) with JSON dictionary format
- Changed examples from 'return |3-1 , , 03|' to 'return "3-1 , , 03"'
- Updated final answer format from 'ENCLOSED in |pipes|' to JSON dict: {"EFFECTIVE_DT": "date"}
- Updated PROVIDER_NAME field: Replaced pipe-separated format with JSON list format
- Changed from 'separated by |' to JSON list: ["Provider A", "Provider B"]
- Updated to return JSON dict: {"PROVIDER_NAME": [...]} or {"PROVIDER_NAME": "single"}
- All pipe references removed from investment_prompts.json
- Updated PRD to document this change in Phase 2 checklist
- JSON file validated and syntax confirmed correct
- Create json_utils.py with parse_json_dict() and parse_json_list() functions
- Add comprehensive unit tests (49 tests, all passing)
- Add deprecation warnings to string_utils.py for pipe-delimited parsing
- Follows PRD Phase 1: Parser Infrastructure
This is the first step in replacing pipe-delimited LLM output format with
structured JSON format as per PRD_STANDARDIZE_LIST_FORMATS.md
Feature/standardize list formats
* updated unit test for code funcs
* pipeline fixes
* Merge branch 'main' into feature/standardize-list-formats
* black format
* removed duplicates
* Merge remote-tracking branch 'origin/main' into feature/standardize-list-formats
* updated prov other tin npi names to list
* prompt calls code made common for cleints and saas
* final format fixes
* removed pipes
* crosswalk fixes
* Merged main into feature/standardize-list-formats
* dubug dynamic fields
* dynamic format fixes
* dyanmic fields fixes
* removed pipes from the whole code
* fixed provider TIN NPI Name format issues
* bill code issue fix
* pipeline error fixes
* Merge branch 'feature/standardize-list-formats' of https://bitbucket.org/aarete/doczy.ai into feature/standardize-list-formats
* Merged main into feature/standardize-list-formats
* Black format
* pipeline errors fixed
* pipeline fixes page funcs and tinnpi funcs
* qa_qc_utils fix
Approved-by: Katon Minhas
Bugfix/generic dynamic jan26
* Refactor: Extract dynamic primary metrics logic to separate module
- Created src/testbed/dynamic_primary_metrics.py with all dynamic primary field analysis logic
- Moved tally-based analysis functions to new module for better modularity
- Updated testbed_utils.py to import and use functions from dynamic_primary_metrics
- Simplified testbed_metrics_dynamic_only.py to use new module
- Removed unused analyze_dynamic_primary_fields function (400+ lines)
- Added field filtering in match_rows to handle missing columns gracefully
- Minimal changes to testbed_utils.py (~100 lines vs 845 before)
* feat: Add debug print blocks and prompt changes for dynamic primary fields in 1:1 processing
This commit adds prompt changes for when we pass dynamic primary to 1:1 and comprehensive debug logging for dynamic primary fields when they are escalated to 1:1 processing via Hybrid Smart Chunking (HSC) or Full Context methods.
Changes:
- Added debug print blocks in hybrid_smart_chunking_funcs.py to log:
* Retrieval question, context chunks, and final prompt for HSC processing
* Raw LLM output (reasoning + answer) for dynamic primary fields
- Added debug print blocks in prompt_calls.py to log:
* Contract context, final prompt for Full Context processing
* Raw LLM output (reasoning + answer) for dynamic primary fields
- Added FULL_CONTEXT_DYNAMIC_PRIMARY_INSTRUCTION() to prompt_templates.py:
* Provides specific guidance for dynamic primary fields in full context
* Emphasizes extracting values only if they refer to entire contract
* Includes pipe-delimited formatting instructions
- Updated dynamic_fun…
* Ran black, for formatting
* Refine dynamic primary field prompts and add Duals auto-detection
- Enhanced DYNAMIC_ASSIGNMENT_INSTRUCTION with structural boundary rules
- Added exhibit header and subsection context hierarchy for LOB assignment
- Implemented Duals auto-detection in dynamic_primary discovery (Medicare+Medicaid -> Duals)
- Added Duals value formatting instruction to prevent oversimplification
- Commented out DUAL_LOB_CHECK for performance testing
- Removed verbose debug print blocks from HSC and Full Context processing
- Added focused debug print for dynamic assignment raw LLM output
* Remove debug print blocks from pipeline files
- Removed all debug print blocks from file_processing.py (8 blocks)
- Removed all debug print blocks from dynamic_funcs.py (8 blocks)
- Removed debug print block from prompt_calls.py (dynamic assignment)
- DUAL_LOB_CHECK remains commented as requested
- Total: 188 lines of debug code removed
* Merge remote-tracking branch 'origin/main' into bugfix/generic_dynamic_jan26
* Ran black for CI
* Remove update_lob_for_duals
* Merge branch 'DEV' into bugfix/generic_dynamic_jan26
Approved-by: Katon Minhas
Bugfix/row count
* Updated lesser of statements filter; Updated lesser of prompt related to GLOBAL and STANDALONE
* Updated validate reimbursement prompt
* Update lesser of check prompt related to GLOBAL assignment
* Restructured VALIDATE_REIMBURSEMENTS_PROMPT
* Updated reimbursement primary; validate reimbursement prompt
* Updated reimb primary prompt and validate reimb prompt
* Code cleaned up. Updated unittest cases
* Updated validate reimbursement prompt
* Added logging
* Update on validate reimbursement prompt; Code clean up
* Fix Black version conflict in dependencies
* Add Black back to dependencies
* Apply black formatting and update uv.lock
* Resolve conflict
* Merge branch 'DEV' into bugfix/row_count
Approved-by: Katon Minhas
Feature/update caching
* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT
- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix postprocessing_funcs to remove invalid columns
* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid
* Apply black formatting to 11 files
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix import paths in testbed files
* Merge main into feature/update-caching
* Add SPLIT_REIMB_DATES_INSTRUCTION for prompt caching
* Merged main into feature/update-caching
* Merge remote-tracking branch 'origin/main' into feature/update-caching
* Merge branch 'feature/update-caching' of https://bitbucket.org/aarete/doczy.ai into feature/update-caching
* Add CODE_* instruction functions for prompt caching
* Merged main into feature/update-caching
* Merged main into feature/update-caching
* Maximize prompt caching by moving static content to instructions
- Refactored 12+ prompt functions to move static rules/examples to _INSTRUCTION()
- VALIDATE_REIMBURSEMENTS: Moved 45+ lines of validation rules to instruction
- DATE_FIX: Moved conversion rules and examples to instruction
- EXHIBIT_LEVEL, EXHIBIT_TITLE_MATCH, EXHIBIT_HEADER, EXHIBIT_LINKAGE: Moved rules
- FEE_SCHEDULE_BREAKOUT, GROUPER_BREAKOUT: Moved extraction rules
- CHECK_PROVIDER_NAME_MATCH: Moved matching rules to instruction
- LOB_RELATIONSHIP: Consolidated split instructions
- DERIVED_TERM_DATE: Moved calculation logic to instruction
- DYNAMIC_PRIMARY: Created separate HEADER and TEXT instructions
- Created new REIMB_DATES_ASSIGNMENT_INSTRUCTION with all date rules
- Updated get_cacheable_instructions() with new instruction functions
- Updated all callers in saas, clover, bcbs_promise pipelines
This enables ~90% cost savings via Claude API prompt caching by ensuring
static instruction text is cached and only dynamic content varies.
* Remove Dynamic Primary Header
* Standardize prompt caching across all LLM prompts for ~90% cost savings
Move static content to _INSTRUCTION() functions for caching:
- METHODOLOGY_BREAKOUT: 8 field definitions with resolved valid_values
- FEE_SCHEDULE_BREAKOUT: 3 field definitions
- GROUPER_BREAKOUT: 11 field definitions
- OUTLIER_BREAKOUT: 8 field definitions
- CODE_EXPLICIT: 8 field definitions
- EXHIBIT_HEADER: static header markers list
- CARVEOUT_CHECK: carveout and special case definitions
- SPECIAL_CASE_ASSIGNMENT: task rules and output format
Update callers to use cache=True, instruction, and usage_label params.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Apply black formatting to prompt_templates.py
* Update unit tests for prompt caching changes
- Fix function signatures in tests (METHODOLOGY_BREAKOUT, FEE_SCHEDULE_BREAKOUT, GROUPER_BREAKOUT, CODE_EXPLICIT)
- Add tests for CARVEOUT_CHECK and EXHIBIT_HEADER prompts
- Add TestInstructionFieldContent class to verify instruction functions contain field definitions
- Add tests for CARVEOUT_CHECK_INSTRUCTION and CODE_EXPLICIT_INSTRUCTION
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Refactor instructions to load fields from investment_prompts.json and Constants
Address PR review feedback:
- Add _get_constants() helper for lazy Constants instantiation
- Add _get_fields_text() helper to load fields from investment_prompts.json
- METHODOLOGY_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- FEE_SCHEDULE_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- GROUPER_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- OUTLIER_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- EXHIBIT_HEADER_INSTRUCTION: Load markers from Constants.EXHIBIT_HEADER_MARKERS
- CARVEOUT_CHECK_INSTRUCTION: Load carveouts from Constants.VALID_CARVEOUTS,
special cases from investment_prompts.json
- CODE_EXPLICIT_INSTRUCTION: Keep hardcoded (no code_explicit field_type in JSON)
Update tests to match actual field names from investment_prompts.json.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Merge main into feature/update-caching
* Remove deprecated dynamic_primary_header
* Update CODE_EXPLICIT_INSTRUCTION to load fields from investment_prompts.json
Use field_type=code_primary_breakout to load code fields dynamically.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Merged main into feature/update-caching
* Merge main into feature/update-caching
* Merge branch 'feature/update-caching' of https://bitbucket.org/aarete/doczy.ai into feature/update-caching
Approved-by: Katon Minhas
Bugfix/daip2 1440 contract amendment num to main
* update full context additional instruction for amendment num
* uv formatting done
* remove print statements
* Merged main into bugfix/daip2-1440-contract-amendment-num
* added filename amendment num
* updated pipelines and removed print statements
* Merged main into bugfix/daip2-1440-contract-amendment-num
* code moved to hybrid smart chunking
* Merged main into bugfix/daip2-1440-contract-amendment-num
* Merged main into bugfix/daip2-1440-contract-amendment-num
* removed categorical values from amendment number
Approved-by: Katon Minhas
Feature/chc client
* Add CHC client pipeline with Claims Coding/Editing Determinations extraction
- Add CHC client to pipelines with regex-based extraction
- Create extraction/claims_coding_funcs.py for section extraction
- Add chc_prompts.json with non_prompt field type
- Add CHC_PROMPTS_PATH to config
- Add tests for extraction function (11 test cases)
* Add black[jupyter] for code formatting
* Format testing notebook with black
* Remove unused prompt field from chc_prompts.json
* Merged main into feature/chc-client
Approved-by: Katon Minhas
ppanchigar