ppanchigar
3c89aecfaf
- Removed all debug print statements from prompt_calls.py and one_to_n_funcs.py - Updated PRD to reflect Phase 4 completion and recent bug fixes - Documented REIMB_TERM normalization fixes and cleanup work - Updated Phase 6 status to 25% complete (debug removal done)
3113 lines
108 KiB
Markdown
3113 lines
108 KiB
Markdown
# PRD: Standardize LLM Output Formats - Eliminate Pipe-Delimited Parsing
|
|
|
|
**Version:** 1.2
|
|
**Date:** February 2, 2026
|
|
**Last Updated:** February 3, 2026 (Commit: a633f67c)
|
|
**Status:** In Progress - Phase 4 Complete, Bug Fixes Applied
|
|
**Author:** AI Assistant
|
|
|
|
---
|
|
|
|
## Implementation Progress
|
|
|
|
**Overall Status**: 🟢 **Phase 4 of 6 Complete** (Foundation + Templates + Calling Layer + Downstream)
|
|
|
|
| Phase | Status | Completion | Details |
|
|
|-------|--------|------------|---------|
|
|
| Phase 1: Foundation | ✅ Complete | 100% | JSON parsers, deprecation warnings, unit tests |
|
|
| Phase 2: Prompt Templates | ✅ Complete | 100% | All 35 templates return (prompt, parser) tuples |
|
|
| Phase 3: Prompt Calling Layer | ✅ Complete | 100% | All prompt_calls.py updated; code_funcs.py fixed |
|
|
| Phase 4: Downstream Consumers | ✅ Complete | 100% | All LLM output parsing updated; testbed/QC files verified (internal data - out of scope) |
|
|
| Phase 5: Testing | 🔲 Not Started | 0% | Integration tests, golden tests |
|
|
| Phase 6: Cleanup & Documentation | 🟡 In Progress | 25% | Debug statements removed, defensive checks cleaned up |
|
|
|
|
**Latest Commits** (after PRD update):
|
|
- `a633f67c` - Fixed REIMB_TERM being converted to list in lesser_of_distribution; removed debug statements
|
|
- `4c0d2930` - Updated PRD: Phase 4 now 100% complete
|
|
- `13f3cbea` - Completed Phase 4: Fixed model_evaluation_utils.py LLM output parsing
|
|
- `a7a32b90` - Completed Phase 3: Fixed remaining universal_json_load calls in code_funcs.py
|
|
- `2e307347` - Updated investment_prompts.json to use JSON format instead of pipes
|
|
- `df103579` - Updated postprocessing_funcs.py
|
|
- `0d9f9314` - Updated tin_npi_funcs.py and page_funcs.py
|
|
- `8399dbdf` - Updated crosswalk reverse mapping
|
|
- `6c3f3f6f` - Updated code_funcs.py (mostly complete)
|
|
- `dba61146` - Updated all prompt_calls.py files (saas, clover, bcbs_promise)
|
|
- `98554afc` - All prompt templates now return `(prompt, parser)` tuples
|
|
- `5d1024e` - Updated prompt instruction functions with JSON format instructions
|
|
|
|
**Remaining Work**:
|
|
- **Phase 3**: ✅ **COMPLETE** - All `universal_json_load` calls replaced (Commit: a7a32b90)
|
|
- **Phase 4**: ✅ **COMPLETE** - All LLM output parsing updated (Commit: 13f3cbea)
|
|
- Fixed `model_evaluation_utils.py` to use JSON parsers
|
|
- Verified testbed/QC files process internal data (out of scope per PRD)
|
|
- `aarete_derived.py` has dual-format support (intentional for backward compatibility during transition)
|
|
|
|
**Recent Bug Fixes** (Commit: a633f67c):
|
|
- Fixed `REIMB_TERM` being converted to list in `lesser_of_distribution` - parser now extracts string from list when needed
|
|
- Normalized `SERVICE_TERM` and `REIMB_TERM` upstream in `prompt_reimbursement_primary` to ensure strings
|
|
- Removed defensive normalization checks that are no longer needed (CODE_EXPLICIT, CODE_CATEGORY, CODE_IMPLICIT, FILL_BILL_TYPE)
|
|
- Removed all debug print statements from `prompt_calls.py` and `one_to_n_funcs.py`
|
|
|
|
---
|
|
|
|
## Executive Summary
|
|
|
|
This PRD outlines a comprehensive refactor to eliminate pipe-delimited string parsing from the LLM output processing pipeline. All LLM responses will be migrated to use strict JSON formatting (dictionaries or lists), with explicit parser functions that replace the current `universal_json_load` implementation.
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Background and Motivation](#1-background-and-motivation)
|
|
2. [Non-Goals](#2-non-goals)
|
|
3. [Current State Analysis](#3-current-state-analysis)
|
|
4. [Technical Design](#4-technical-design)
|
|
5. [File-by-File Change List](#5-file-by-file-change-list)
|
|
6. [Migration Plan](#6-migration-plan)
|
|
7. [Testing Plan](#7-testing-plan)
|
|
8. [Risks and Mitigations](#8-risks-and-mitigations)
|
|
9. [Work Items Checklist](#9-work-items-checklist)
|
|
|
|
---
|
|
|
|
## 1. Background and Motivation
|
|
|
|
### 1.1 Problem Statement
|
|
|
|
The current system uses pipe-delimited strings (`|value|`) as a machine-readable format for LLM outputs in certain prompts, while using JSON dictionaries and lists in others. This creates several problems:
|
|
|
|
1. **Inconsistent parsing logic**: Two different parsing paradigms (delimiter extraction vs JSON parsing) exist side-by-side
|
|
2. **Fragility**: Pipe delimiters can appear in content text, breaking parsers
|
|
3. **Type ambiguity**: Pipe-delimited outputs lose type information (everything becomes a string)
|
|
4. **Maintenance overhead**: Downstream code must handle both pipe-separated values and JSON-parsed values
|
|
5. **Testing complexity**: Multiple parsing paths mean more edge cases to test
|
|
|
|
### 1.2 Benefits of Standardization
|
|
|
|
1. **Single parsing paradigm**: All LLM outputs use JSON, simplifying code
|
|
2. **Type safety**: JSON preserves types (strings, numbers, booleans, nulls, arrays, objects)
|
|
3. **Reliability**: JSON parsers are battle-tested and handle escaping properly
|
|
4. **Consistency**: Uniform approach across all prompt templates
|
|
5. **Maintainability**: Easier to reason about and extend
|
|
|
|
### 1.3 Why Now?
|
|
|
|
- The codebase has grown to 13+ files with pipe-split operations
|
|
- New features are being added that require consistent list handling
|
|
- Recent bugs related to pipe delimiter parsing in content text
|
|
|
|
---
|
|
|
|
## 2. Non-Goals
|
|
|
|
### 2.1 Out of Scope
|
|
|
|
1. **Changing internal data representation**: Internal pipe-delimited storage in dataframes/databases is out of scope (only parsing LLM outputs changes)
|
|
2. **Modifying existing output formats for end users**: This is an internal refactor; external APIs/files remain unchanged
|
|
3. **Reprocessing historical data**: Existing processed files don't need reprocessing
|
|
4. **Changing prompt caching strategy**: We maintain existing caching patterns
|
|
5. **Modifying crosswalk/mapping logic**: Crosswalks can still use pipe delimiters internally after parsing
|
|
|
|
### 2.2 Future Work (Not This PRD)
|
|
|
|
1. Eliminating pipe delimiters from internal storage (separate initiative)
|
|
2. Standardizing all multi-value fields to JSON arrays in the database
|
|
3. Updating external API response formats
|
|
|
|
---
|
|
|
|
## 3. Current State Analysis
|
|
|
|
### 3.0 Previous Implementation Attempt
|
|
|
|
#### 3.0.1 Commit faf7072 (Reverted)
|
|
|
|
A previous attempt at standardizing list formats was merged (commit `faf707265d1e0a5bb867e1163f6da6cef0a793e8`) but subsequently reverted (commit `02d10e711e7b1c43e0dce11586c8772d7805b4ce`).
|
|
|
|
**What Was Implemented**:
|
|
1. Added `JSON_LIST_FORMAT_INSTRUCTIONS` and `JSON_DICT_FORMAT_INSTRUCTIONS` constants
|
|
2. Updated prompt templates to reference these variables
|
|
3. Changed parser calls from `universal_json_load` to direct JSON parsing
|
|
4. Updated downstream code to handle JSON list formats
|
|
|
|
**Why It Was Reverted**:
|
|
- Likely caused pipeline failures or output format mismatches
|
|
- Possible issues with LLM responses not conforming to expected JSON formats
|
|
- Downstream code may not have fully handled transition from pipe-delimited to lists
|
|
- Testing may have been incomplete before merge
|
|
|
|
**Lessons Learned**:
|
|
1. **Need comprehensive testing**: Must test full pipeline end-to-end before merge
|
|
2. **Need backward compatibility**: Should support dual formats during transition
|
|
3. **Need clear format specifications**: Ambiguity about when to use dict vs list vs dict-with-lists
|
|
4. **Need LLM prompt validation**: Must verify LLMs actually produce valid JSON with new instructions
|
|
5. **Need gradual rollout**: Should phase implementation rather than big-bang change
|
|
|
|
**This PRD Addresses These Issues**:
|
|
- ✅ Comprehensive prompt-output mapping table (Section 4.2.2)
|
|
- ✅ Four distinct format types with clear usage guidelines (Section 4.2.3)
|
|
- ✅ Phased migration plan with backward compatibility (Section 6)
|
|
- ✅ Extensive testing plan including golden tests (Section 7)
|
|
- ✅ Dual-format support in downstream consumers (Section 5.4)
|
|
- ✅ Rollback strategy (Section 6.3)
|
|
|
|
---
|
|
|
|
### 3.1 Pipe-Delimited Usage
|
|
|
|
#### 3.1.1 Prompt Template Constant
|
|
|
|
**File**: `src/prompts/prompt_templates.py`
|
|
|
|
```python
|
|
PIPE_FORMAT_INSTRUCTIONS = "Briefly explain your answer, then enclose your final answer in |pipes|. If the requested information doesn't apply to this context, return |N/A|. If the information should exist but cannot be clearly identified, return |UNKNOWN|."
|
|
```
|
|
|
|
**Used in**: Legacy prompts (to be removed)
|
|
|
|
#### 3.1.2 Extraction Function
|
|
|
|
**File**: `src/utils/string_utils.py`
|
|
|
|
```python
|
|
def extract_text_from_delimiters(
|
|
raw_output: str, delimiter: Delimiter, match_index: int = -1
|
|
) -> str
|
|
```
|
|
|
|
**Called from**:
|
|
- `src/pipelines/saas/prompts/prompt_calls.py::prompt_dynamic_primary()` (line 93-95)
|
|
- `src/pipelines/saas/prompts/prompt_calls.py::prompt_carveout_check()` (line 251-253)
|
|
- `src/pipelines/clients/clover/prompts/prompt_calls.py::prompt_dynamic_primary()` (line 94-96)
|
|
- `src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py::prompt_dynamic_primary()` (line 88-90)
|
|
|
|
### 3.2 Universal JSON Load Usage
|
|
|
|
**Implementation**: `src/utils/string_utils.py::universal_json_load()`
|
|
|
|
**Call Sites** (11+ occurrences in `src/pipelines/saas/prompts/prompt_calls.py`):
|
|
|
|
1. `prompt_exhibit_level()` - line 36
|
|
2. `prompt_exhibit_level_breakout()` - line 74
|
|
3. `prompt_reimbursement_primary()` - line 129
|
|
4. `prompt_methodology_breakout()` - line 161
|
|
5. `prompt_fee_schedule_breakout()` - line 192
|
|
6. `prompt_grouper_breakout()` - line 222
|
|
7. `prompt_special_case_breakout()` - line 286
|
|
8. `prompt_full_context()` - line 390
|
|
9. `prompt_dynamic()` - line 465
|
|
10. `prompt_dynamic_field_safe()` - line 623
|
|
|
|
**Also used in**:
|
|
- `src/pipelines/clients/clover/prompts/prompt_calls.py`
|
|
- `src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py`
|
|
- `src/codes/code_funcs.py`
|
|
- `src/pipelines/shared/extraction/tin_npi_funcs.py`
|
|
- `src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py`
|
|
- Test files
|
|
|
|
### 3.3 Downstream Pipe Splitting
|
|
|
|
#### 3.3.1 aarete_derived.py
|
|
|
|
**File**: `src/pipelines/shared/postprocessing/aarete_derived.py`
|
|
|
|
**Occurrences**: 4 instances (lines 45, 67, 93, 155)
|
|
|
|
**Pattern**:
|
|
```python
|
|
if "|" in value:
|
|
value_list = value.split("|")
|
|
```
|
|
|
|
**Context**: Parsing LOB, PROGRAM, and PRODUCT fields that may contain pipe-delimited values
|
|
|
|
#### 3.3.2 crosswalk_utils.py
|
|
|
|
**File**: `src/utils/crosswalk_utils.py`
|
|
|
|
**Occurrence**: Line 40
|
|
|
|
**Pattern**:
|
|
```python
|
|
elif "|" in val:
|
|
values = [v.strip() for v in val.split("|")]
|
|
```
|
|
|
|
**Context**: Parsing values before applying crosswalk mappings
|
|
|
|
#### 3.3.3 Other Files with Pipe Splits
|
|
|
|
**Total files identified**: 13
|
|
|
|
- `src/pipelines/shared/extraction/dynamic_funcs.py` - LOB parsing (line 60)
|
|
- `src/pipelines/shared/postprocessing/postprocessing_funcs.py`
|
|
- `src/testbed/dynamic_primary_metrics.py`
|
|
- `src/testbed/testbed_utils.py`
|
|
- `src/codes/code_funcs.py`
|
|
- `src/qc_qa/pipeline.py`
|
|
- `src/utils/qa_qc_utils.py`
|
|
- `src/parent_child/qc.py`
|
|
- `src/pipelines/shared/extraction/tin_npi_funcs.py`
|
|
- `src/testbed/model_evaluation_utils.py`
|
|
|
|
### 3.4 Prompt Templates Mentioning Pipes
|
|
|
|
#### 3.4.1 Instructions with Pipe References
|
|
|
|
1. **EXHIBIT_LEVEL_INSTRUCTION** (line 200)
|
|
- "If there are multiple, return them in a pipe-separated list ("|")"
|
|
|
|
2. **DYNAMIC_PRIMARY_TEXT_INSTRUCTION** (line 236)
|
|
- "Briefly explain your answer before putting the final answer in |pipes|"
|
|
|
|
3. **CARVEOUT_CHECK_INSTRUCTION** (line 1672)
|
|
- "enclose your final answer in |pipes|"
|
|
|
|
4. **FULL_CONTEXT_ADDITIONAL_INSTRUCTIONS** (lines 1278, 1280)
|
|
- "return them in a pipe (|) separated list"
|
|
|
|
5. **DYNAMIC_ASSIGNMENT_INSTRUCTION** (line 391)
|
|
- References to pipe-delimited format for DUALS value
|
|
|
|
6. **REIMBURSEMENT_PRIMARY_INSTRUCTION** (line ~460)
|
|
- JSON list output (already correct format)
|
|
|
|
---
|
|
|
|
## 4. Technical Design
|
|
|
|
### 4.1 JSON Output Instruction Variables
|
|
|
|
#### 4.1.1 Standardized Format Instructions
|
|
|
|
**Approach**: Instead of embedding output format instructions directly in each prompt, we define module-level constants that are referenced by all prompts. This ensures consistency and makes updates easier.
|
|
|
|
**File**: `src/prompts/prompt_templates.py` (additions)
|
|
|
|
```python
|
|
# JSON format instruction constants
|
|
JSON_DICT_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted JSON dictionary.
|
|
If the requested information doesn't apply to this context, return 'N/A'.
|
|
If the information should exist but cannot be clearly identified, return 'UNKNOWN'.
|
|
|
|
Example format:
|
|
{
|
|
"FIELD_NAME": "value",
|
|
"ANOTHER_FIELD": "value or N/A"
|
|
}
|
|
"""
|
|
|
|
JSON_LIST_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted valid JSON list format.
|
|
If the requested information doesn't apply to this context, return ["N/A"].
|
|
If the information should exist but cannot be clearly identified, return ["UNKNOWN"].
|
|
|
|
Formatting examples:
|
|
- Single string value: ["value"]
|
|
- Multiple string values: ["value1", "value2"]
|
|
- Date value: ["YYYY-MM-DD"]
|
|
- Not applicable: ["N/A"]
|
|
- Information unclear or missing: ["UNKNOWN"]
|
|
"""
|
|
|
|
JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted JSON dictionary where each field value is a JSON list.
|
|
For fields with multiple values, return them as an array: {"FIELD": ["value1", "value2"]}.
|
|
For fields with a single value, still return as an array: {"FIELD": ["value"]}.
|
|
If the requested information doesn't apply, return ["N/A"]: {"FIELD": ["N/A"]}.
|
|
If the information should exist but cannot be identified, return ["UNKNOWN"]: {"FIELD": ["UNKNOWN"]}.
|
|
|
|
Example format:
|
|
{
|
|
"LOB": ["Medicare", "Medicaid"],
|
|
"NETWORK": ["N/A"],
|
|
"FACILITY_TYPE": ["Inpatient"]
|
|
}
|
|
"""
|
|
|
|
JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS = """Return your answer as a valid JSON array of objects. Each object represents one entity (e.g., one reimbursement term).
|
|
If no entities are found, return an empty array: []
|
|
|
|
Example format:
|
|
[
|
|
{
|
|
"SERVICE_TERM": "Inpatient Services",
|
|
"REIMB_TERM": "110% of Medicare"
|
|
},
|
|
{
|
|
"SERVICE_TERM": "Outpatient Services",
|
|
"REIMB_TERM": "$500 per visit"
|
|
}
|
|
]
|
|
|
|
For no results: []
|
|
"""
|
|
```
|
|
|
|
**Usage Pattern with Prompt Caching**:
|
|
|
|
Since we use prompt caching, format instructions belong in the **cached INSTRUCTION() functions**, not in the dynamic prompt content:
|
|
|
|
```python
|
|
# INSTRUCTION function (cached) - includes format instructions
|
|
def EXHIBIT_LEVEL_INSTRUCTION() -> str:
|
|
"""Static instruction for EXHIBIT_LEVEL prompt caching."""
|
|
return f"""[OBJECTIVE]
|
|
Extract attributes from a section of a contract.
|
|
|
|
[GENERAL INSTRUCTIONS]
|
|
- For each field, return ALL correct answers found
|
|
- Return each field value as a JSON list, even if only one value
|
|
- If a list of valid values is provided, ONLY answer with EXACT values from that list
|
|
- If no correct answers, return ["N/A"] for that field
|
|
|
|
[OUTPUT FORMAT]
|
|
{JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS}"""
|
|
|
|
|
|
# Dynamic prompt function (NOT cached) - no format instructions
|
|
def EXHIBIT_LEVEL(context, fields):
|
|
"""Returns ONLY dynamic content.
|
|
Call EXHIBIT_LEVEL_INSTRUCTION() separately for caching.
|
|
"""
|
|
return f"""[ATTRIBUTES]
|
|
{fields}
|
|
|
|
[CONTEXT]
|
|
{context.replace('"', "'")}"""
|
|
```
|
|
|
|
**Caching Architecture**:
|
|
```
|
|
┌─────────────────────────────────────────┐
|
|
│ CACHED (sent once, reused) │
|
|
│ - INSTRUCTION() function output │
|
|
│ - Includes JSON format instructions │
|
|
│ - Never changes │
|
|
└─────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────┐
|
|
│ DYNAMIC (sent every request) │
|
|
│ - Prompt function output │
|
|
│ - Context, fields, specific data │
|
|
│ - NO format instructions │
|
|
└─────────────────────────────────────────┘
|
|
```
|
|
|
|
**Benefits**:
|
|
- Single source of truth for output formats
|
|
- Optimal caching (format instructions cached, not sent repeatedly)
|
|
- Reduced token usage per request
|
|
- Easy to update all prompts by changing one variable
|
|
- Self-documenting code (variable names indicate format type)
|
|
|
|
---
|
|
|
|
### 4.2 Prompt Output Format Mapping
|
|
|
|
This section documents the expected output format for each prompt template in the system.
|
|
|
|
#### 4.2.1 Output Format Classification
|
|
|
|
**Format Types**:
|
|
1. **JSON Dict** - Single dictionary with field:value pairs
|
|
2. **JSON List** - Simple array of strings
|
|
3. **JSON Dict with Lists** - Dictionary where each value is an array
|
|
4. **JSON List of Dicts** - Array of dictionaries (multiple entities)
|
|
|
|
**Cardinality Types**:
|
|
- **Exhibit-level** - Single result per exhibit (one dict/set of values)
|
|
- **Reimbursement-level** - Multiple results per exhibit (list of entities)
|
|
- **Single-value** - Returns one atomic value
|
|
|
|
#### 4.2.2 Comprehensive Prompt Mapping
|
|
|
|
| Prompt Template | Output Format | Cardinality | Multi-Value Fields? | Format Instruction Variable | Example |
|
|
|-----------------|---------------|-------------|---------------------|----------------------------|---------|
|
|
| `EXHIBIT_LEVEL` | JSON Dict with Lists | Exhibit-level | Yes (LOB, NETWORK, etc. can have multiple values) | `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS` | `{"LOB": ["Medicare", "Medicaid"], "FACILITY_TYPE": ["Inpatient"]}` |
|
|
| `REIMBURSEMENT_PRIMARY` | JSON List of Dicts | Reimbursement-level (many rows) | No (each field is single value per row) | `JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS` | `[{"SERVICE_TERM": "...", "REIMB_TERM": "..."}, ...]` |
|
|
| `DYNAMIC_PRIMARY_TEXT` | JSON List | Single-value returning list | Yes (returns all found values) | `JSON_LIST_FORMAT_INSTRUCTIONS` | `["Medicare", "Medicaid", "Duals"]` |
|
|
| `DYNAMIC_ASSIGNMENT` | JSON Dict with Lists | Single assignment per call | Yes (can assign multiple values) | `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS` | `{"LOB": ["Medicare", "Duals"]}` |
|
|
| `CARVEOUT_CHECK` | JSON Dict | Single-value | No | `JSON_DICT_FORMAT_INSTRUCTIONS` | `{"CASE": "OUTLIER_TERM"}` |
|
|
| `METHODOLOGY_BREAKOUT` | JSON Dict | Reimbursement-level (one per term) | No | `JSON_DICT_FORMAT_INSTRUCTIONS` | `{"REIMBURSEMENT_METHOD": "Percent of Charges", "PERCENT": "110"}` |
|
|
| `FEE_SCHEDULE_BREAKOUT` | JSON Dict | Reimbursement-level (one per term) | No | `JSON_DICT_FORMAT_INSTRUCTIONS` | `{"FEE_SCHEDULE_NAME": "Medicare", "FEE_SCHEDULE_YEAR": "2024"}` |
|
|
| `GROUPER_BREAKOUT` | JSON Dict | Reimbursement-level (one per term) | No | `JSON_DICT_FORMAT_INSTRUCTIONS` | `{"GROUPER_TYPE": "DRG", "GROUPER_VERSION": "MS-DRG v40"}` |
|
|
| `FACILITY_ADJUSTMENT_BREAKOUT` | JSON Dict | Exhibit-level | No | `JSON_DICT_FORMAT_INSTRUCTIONS` | `{"ADJUSTMENT_TYPE": "Stop Loss", "ADJUSTMENT_VALUE": "100000"}` |
|
|
| `LESSER_OF_CHECK` | JSON Dict with Lists | Single check per call | Yes (scope and target can be lists) | `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS` | `{"scope": ["GLOBAL"], "target_exhibit": ["ALL"]}` |
|
|
| `LESSER_OF_DISTRIBUTION` | JSON List | Single value (returns statement) | Yes (returns list of applicable statements) | `JSON_LIST_FORMAT_INSTRUCTIONS` | `["Lab Test paid at lesser of $100 or charges"]` |
|
|
| `LOB_RELATIONSHIP` | JSON Dict | Single relationship per call | No | `JSON_DICT_FORMAT_INSTRUCTIONS` | `{"relationship": "Exclusive"}` |
|
|
| `REIMB_DATES_ASSIGNMENT` | JSON List | Single value (returns date range) | Yes (can be multiple date ranges) | `JSON_LIST_FORMAT_INSTRUCTIONS` | `["January 1, 2024 through December 31, 2024"]` |
|
|
| `ONE_TO_ONE_SINGLE_FIELD` (HSC) | JSON Dict with Lists | Contract-level (one per contract) | Yes (contract-level fields can have multiple values) | `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS` | `{"PAYER_NAME": ["Acme Health"]}` |
|
|
| `ONE_TO_ONE_MULTI_FIELD` (Full Context) | JSON Dict with Lists | Contract-level (one per contract) | Yes | `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS` | `{"PAYER_NAME": ["Acme"], "EFFECTIVE_DATE": ["2024-01-01"]}` |
|
|
| `TIN_NPI_TEMPLATE` | JSON Dict with Lists | Contract-level | Yes (multiple TINs/NPIs) | `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS` | `{"PROVIDER_TIN": ["123456789", "987654321"]}` |
|
|
| `EXHIBIT_TITLE_MATCH` | JSON List | Single-value (YES/NO) | No | `JSON_LIST_FORMAT_INSTRUCTIONS` | `["YES"]` or `["NO"]` |
|
|
| `VALIDATE_REIMBURSEMENTS` | JSON List of Dicts | Validation results (many rows) | No | `JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS` | `[{"valid": true, "reason": "..."}, ...]` |
|
|
| `EXHIBIT_LINKAGE` | JSON Dict | Single-value | No | `JSON_DICT_FORMAT_INSTRUCTIONS` | `{"is_same_exhibit": "YES"}` |
|
|
| Special Case Breakouts (STOP_LOSS, OUTLIER, etc.) | JSON Dict | Reimbursement-level (one per term) | No | `JSON_DICT_FORMAT_INSTRUCTIONS` | `{"STOP_LOSS_AMOUNT": "100000", "STOP_LOSS_UNIT": "dollars"}` |
|
|
|
|
#### 4.2.3 Format Selection Logic
|
|
|
|
**When to use each format**:
|
|
|
|
1. **`JSON_DICT_FORMAT_INSTRUCTIONS`** - Use when:
|
|
- Extracting a single entity with multiple fields
|
|
- Each field has exactly one value (no arrays)
|
|
- Examples: Methodology breakout, carveout check, special case breakouts
|
|
|
|
2. **`JSON_LIST_FORMAT_INSTRUCTIONS`** - Use when:
|
|
- Returning a simple list of values
|
|
- Returning a single answer that could be multi-valued (like dynamic primary)
|
|
- Returning YES/NO or similar single responses
|
|
- Examples: Dynamic primary discovery, exhibit title match, date assignment
|
|
|
|
3. **`JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`** - Use when:
|
|
- Extracting multiple fields where each field can have multiple values
|
|
- Exhibit-level extraction (LOB, NETWORK, etc.)
|
|
- Contract-level extraction (PAYER_NAME, etc.)
|
|
- Dynamic assignment (assigning multiple values to a field)
|
|
- Examples: Exhibit level, one-to-one fields, TIN/NPI, dynamic assignment
|
|
|
|
4. **`JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS`** - Use when:
|
|
- Extracting multiple entities, each with multiple fields
|
|
- Reimbursement primary extraction (many service/reimb pairs)
|
|
- Validation results (multiple validation outcomes)
|
|
- Examples: Reimbursement primary, validation
|
|
|
|
---
|
|
|
|
### 4.3 New Parser Architecture
|
|
|
|
#### 4.3.1 Parser Function Specifications
|
|
|
|
**File**: `src/utils/json_parsers.py` (new file)
|
|
|
|
```python
|
|
"""
|
|
JSON parsing utilities for LLM outputs.
|
|
Replaces universal_json_load with explicit dict/list parsers.
|
|
"""
|
|
|
|
import json
|
|
import logging
|
|
from typing import Any, Dict, List, Union
|
|
|
|
|
|
class JSONParseError(ValueError):
|
|
"""Raised when JSON parsing fails after all attempts."""
|
|
pass
|
|
|
|
|
|
def parse_json_dict(raw_llm_output: str) -> Dict[str, Any]:
|
|
"""
|
|
Extract and parse a JSON dictionary from raw LLM output.
|
|
|
|
Strategy:
|
|
1. Locate the first complete JSON object {...}
|
|
2. Parse with json.loads()
|
|
3. Validate it's a dictionary
|
|
4. Return parsed dict
|
|
|
|
Args:
|
|
raw_llm_output: Raw string output from LLM (may include reasoning text)
|
|
|
|
Returns:
|
|
Parsed dictionary
|
|
|
|
Raises:
|
|
JSONParseError: If no valid JSON dictionary found
|
|
|
|
Example:
|
|
>>> raw = "Here's my reasoning...\n{\"field\": \"value\", \"count\": 5}"
|
|
>>> parse_json_dict(raw)
|
|
{'field': 'value', 'count': 5}
|
|
"""
|
|
if not isinstance(raw_llm_output, str):
|
|
raise TypeError(f"Expected string input, got {type(raw_llm_output).__name__}")
|
|
|
|
# Find first JSON object
|
|
found = _extract_json_objects(raw_llm_output)
|
|
|
|
if not found:
|
|
raise JSONParseError(f"No valid JSON dictionary found in: {raw_llm_output[:100]}...")
|
|
|
|
# Return last valid object (convention: final answer comes last)
|
|
result = found[-1]
|
|
|
|
if not isinstance(result, dict):
|
|
raise JSONParseError(f"Expected dictionary, got {type(result).__name__}")
|
|
|
|
return result
|
|
|
|
|
|
def parse_json_list(raw_llm_output: str) -> List[Any]:
|
|
"""
|
|
Extract and parse a JSON list from raw LLM output.
|
|
|
|
Strategy:
|
|
1. Locate the first complete JSON array [...]
|
|
2. Parse with json.loads()
|
|
3. Validate it's a list
|
|
4. Return parsed list
|
|
|
|
Args:
|
|
raw_llm_output: Raw string output from LLM (may include reasoning text)
|
|
|
|
Returns:
|
|
Parsed list
|
|
|
|
Raises:
|
|
JSONParseError: If no valid JSON list found
|
|
|
|
Example:
|
|
>>> raw = "Here are the items:\n[\"item1\", \"item2\", \"item3\"]"
|
|
>>> parse_json_list(raw)
|
|
['item1', 'item2', 'item3']
|
|
"""
|
|
if not isinstance(raw_llm_output, str):
|
|
raise TypeError(f"Expected string input, got {type(raw_llm_output).__name__}")
|
|
|
|
# Find first JSON array
|
|
found = _extract_json_objects(raw_llm_output)
|
|
|
|
if not found:
|
|
raise JSONParseError(f"No valid JSON list found in: {raw_llm_output[:100]}...")
|
|
|
|
# Return last valid array (convention: final answer comes last)
|
|
result = found[-1]
|
|
|
|
if not isinstance(result, list):
|
|
raise JSONParseError(f"Expected list, got {type(result).__name__}")
|
|
|
|
return result
|
|
|
|
|
|
def _extract_json_objects(s: str) -> List[Union[Dict, List]]:
|
|
"""
|
|
Extract all top-level JSON objects/arrays from a string.
|
|
Returns list of parsed objects (dicts or lists).
|
|
|
|
This is adapted from the existing universal_json_load logic
|
|
but more explicit about return types.
|
|
"""
|
|
found = []
|
|
i = 0
|
|
|
|
while i < len(s):
|
|
if s[i] in "{[":
|
|
start = i
|
|
stack = [s[i]]
|
|
j = i + 1
|
|
|
|
while j < len(s):
|
|
# Skip over string literals to avoid counting brackets inside strings
|
|
if s[j] == '"':
|
|
j += 1
|
|
while j < len(s):
|
|
if s[j] == "\\" and j + 1 < len(s):
|
|
j += 2 # Skip escaped character
|
|
continue
|
|
if s[j] == '"':
|
|
break
|
|
j += 1
|
|
elif s[j] in "{[":
|
|
stack.append(s[j])
|
|
elif s[j] in "}]":
|
|
if not stack:
|
|
break
|
|
open_bracket = stack.pop()
|
|
if (open_bracket == "{" and s[j] != "}") or (
|
|
open_bracket == "[" and s[j] != "]"
|
|
):
|
|
break
|
|
if not stack:
|
|
candidate = s[start : j + 1]
|
|
try:
|
|
obj = json.loads(candidate)
|
|
found.append(obj)
|
|
except json.JSONDecodeError:
|
|
pass
|
|
i = j # Move past this top-level object
|
|
break
|
|
j += 1
|
|
i += 1
|
|
|
|
return found
|
|
|
|
|
|
# Backward compatibility alias (deprecated)
|
|
def universal_json_load(s: str) -> Union[Dict, List]:
|
|
"""
|
|
DEPRECATED: Use parse_json_dict() or parse_json_list() instead.
|
|
|
|
This function is maintained for backward compatibility during migration.
|
|
It will be removed in a future version.
|
|
"""
|
|
logging.warning(
|
|
"universal_json_load is deprecated. Use parse_json_dict() or parse_json_list() instead."
|
|
)
|
|
found = _extract_json_objects(s)
|
|
if not found:
|
|
raise ValueError(f"No valid JSON objects found in string: {s[:100]}...")
|
|
return found[-1]
|
|
```
|
|
|
|
#### 4.1.2 Prompt Template Parser Coupling (IMPLEMENTED)
|
|
|
|
**Status**: ✅ **COMPLETED** (Commit: 98554afc)
|
|
|
|
**Enhancement to Prompt System**: All prompt template functions now return a tuple of `(prompt_string, parser_function)`. This architectural change eliminates ambiguity about which parser to use for each prompt.
|
|
|
|
**File**: `src/prompts/prompt_templates.py` (modification)
|
|
|
|
**Implementation Approach**:
|
|
|
|
Instead of creating separate `_WITH_PARSER` functions, we modified all existing prompt template functions to directly return tuples. This provides a cleaner API and ensures every prompt has its parser explicitly defined.
|
|
|
|
**Helper Parser Functions** (added to `prompt_templates.py`):
|
|
|
|
```python
|
|
from typing import Callable, Tuple, Any
|
|
from src.utils import json_utils
|
|
|
|
def _json_dict_parser(raw_output: str) -> dict:
|
|
"""Parse JSON dictionary from LLM output."""
|
|
return json_utils.parse_json_dict(raw_output)
|
|
|
|
def _json_list_parser(raw_output: str) -> list:
|
|
"""Parse JSON list from LLM output."""
|
|
return json_utils.parse_json_list(raw_output)
|
|
```
|
|
|
|
**Updated Function Signatures** (examples):
|
|
|
|
```python
|
|
def EXHIBIT_LEVEL(context, fields) -> Tuple[str, Callable[[str], dict]]:
|
|
"""Returns ONLY dynamic content for exhibit level extraction.
|
|
Call EXHIBIT_LEVEL_INSTRUCTION() separately for the cached instruction.
|
|
|
|
Returns:
|
|
Tuple of (prompt_string, parser_function) where parser expects JSON dict output.
|
|
"""
|
|
prompt = f"""[ATTRIBUTES]
|
|
Here are the attributes to be included in the dictionary, and instructions on how to correctly answer:
|
|
{fields}
|
|
|
|
[CONTEXT]
|
|
Here is the text to analyze:
|
|
{context.replace('"', "'")}"""
|
|
|
|
return (prompt, _json_dict_parser)
|
|
|
|
|
|
def DYNAMIC_PRIMARY_TEXT(context, field_name, field_prompt) -> Tuple[str, Callable[[str], list]]:
|
|
"""Returns ONLY dynamic content for text-based dynamic primary extraction.
|
|
Call DYNAMIC_PRIMARY_TEXT_INSTRUCTION() separately for the cached instruction.
|
|
|
|
Returns:
|
|
Tuple of (prompt_string, parser_function) where parser expects JSON list output.
|
|
"""
|
|
prompt = f"""[ATTRIBUTE TO EXTRACT]
|
|
{field_name} : {field_prompt}
|
|
|
|
[CONTEXT]
|
|
Here is the text to analyze:
|
|
{context.replace('"', "'")}"""
|
|
|
|
return (prompt, _json_list_parser)
|
|
|
|
|
|
def CODE_EXPLICIT(service, methodology) -> Tuple[str, Callable[[str], dict]]:
|
|
"""Returns ONLY dynamic content for code explicit extraction.
|
|
Call CODE_EXPLICIT_INSTRUCTION() separately for the cached instruction.
|
|
Note: Field definitions are now included in CODE_EXPLICIT_INSTRUCTION() for caching.
|
|
|
|
Returns:
|
|
Tuple of (prompt_string, parser_function) where parser expects JSON dict output.
|
|
"""
|
|
prompt = f"""[CONTEXT]
|
|
Here is the Service and Methodology to analyze:
|
|
Service: {service.replace('"', "'")}
|
|
Methodology: {methodology.replace('"', "'")}"""
|
|
|
|
return (prompt, _json_dict_parser)
|
|
```
|
|
|
|
**Complete List of Updated Functions** (24 total):
|
|
|
|
1. `LESSER_OF_CHECK` → `(prompt, _json_dict_parser)`
|
|
2. `EXHIBIT_TITLE_MATCH` → `(prompt, _json_list_parser)`
|
|
3. `EXHIBIT_LEVEL` → `(prompt, _json_dict_parser)`
|
|
4. `DYNAMIC_PRIMARY_TEXT` → `(prompt, _json_list_parser)`
|
|
5. `REIMB_DATES_ASSIGNMENT` → `(prompt, _json_dict_parser)`
|
|
6. `DYNAMIC_ASSIGNMENT` → `(prompt, _json_dict_parser)`
|
|
7. `REIMBURSEMENT_PRIMARY` → `(prompt, _json_list_parser)`
|
|
8. `LESSER_OF_DISTRIBUTION` → `(prompt, _json_list_parser)`
|
|
9. `METHODOLOGY_BREAKOUT` → `(prompt, _json_list_parser)`
|
|
10. `FEE_SCHEDULE_BREAKOUT` → `(prompt, _json_dict_parser)`
|
|
11. `GROUPER_BREAKOUT` → `(prompt, _json_dict_parser)`
|
|
12. `SPECIAL_CASE_BREAKOUT` → `(prompt, _json_dict_parser)`
|
|
13. `OUTLIER_BREAKOUT` → `(prompt, _json_dict_parser)`
|
|
14. All breakout variants (RATE_ESCALATOR, TRIGGER_CAP, PREMIUM, ADDITION, STOP_LOSS, FACILITY_ADJUSTMENT, SEQUESTRATION, DISCOUNT)
|
|
15. `CODE_EXPLICIT` → `(prompt, _json_dict_parser)`
|
|
16. `CODE_CATEGORY` → `(prompt, _json_list_parser)`
|
|
17. `CODE_IMPLICIT_SPECIAL` → `(prompt, _json_list_parser)`
|
|
18. `CODE_IMPLICIT` → `(prompt, _json_list_parser)`
|
|
19. `CODE_LAST_CHECK` → `(prompt, _json_list_parser)`
|
|
20. `FILL_BILL_TYPE` → `(prompt, _json_list_parser)`
|
|
21. `ONE_TO_ONE_SINGLE_FIELD_TEMPLATE` → `(prompt, _json_dict_parser)`
|
|
22. `ONE_TO_ONE_MULTI_FIELD_TEMPLATE` → `(prompt, _json_dict_parser)`
|
|
23. `TIN_NPI_TEMPLATE` → `(prompt, _json_list_parser)`
|
|
24. `CHECK_PROVIDER_NAME_MATCH_PROMPT` → `(prompt, _json_list_parser)`
|
|
25. `EXTRACT_AMENDMENT_NUM_FROM_FILENAME` → `(prompt, _json_dict_parser)`
|
|
26. `EXHIBIT_HEADER` → `(prompt, _json_list_parser)`
|
|
27. `EXHIBIT_LINKAGE` → `(prompt, _json_list_parser)`
|
|
28. `DATE_FIX_PROMPT` → `(prompt, _json_list_parser)`
|
|
29. `EFFECTIVE_DATE_FIX_PROMPT` → `(prompt, _json_dict_parser)`
|
|
30. `SPLIT_REIMB_DATES` → `(prompt, _json_dict_parser)`
|
|
31. `DERIVED_TERM_DATE_PROMPT` → `(prompt, _json_list_parser)`
|
|
32. `VALIDATE_REIMBURSEMENTS_PROMPT` → `(prompt, _json_list_parser)`
|
|
33. `CARVEOUT_CHECK` → `(prompt, _json_list_parser)`
|
|
34. `LOB_RELATIONSHIP` → `(prompt, _json_list_parser)`
|
|
35. `SPECIAL_CASE_ASSIGNMENT` → `(prompt, _json_list_parser)`
|
|
|
|
**Usage Pattern** (example in `prompt_calls.py`):
|
|
|
|
```python
|
|
def prompt_exhibit_level(
|
|
exhibit_text: str,
|
|
exhibit_level_fields: FieldSet,
|
|
constants: Constants,
|
|
filename: str,
|
|
):
|
|
# Get prompt and parser together
|
|
prompt_template, parser = prompt_templates.EXHIBIT_LEVEL(
|
|
exhibit_text, exhibit_level_fields.print_prompt_dict(constants)
|
|
)
|
|
|
|
# Invoke LLM
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
prompt_template,
|
|
"sonnet_latest",
|
|
filename,
|
|
max_tokens=8192,
|
|
cache=True,
|
|
instruction=prompt_templates.EXHIBIT_LEVEL_INSTRUCTION(),
|
|
usage_label="EXHIBIT_LEVEL",
|
|
)
|
|
|
|
# Parse with returned parser (no ambiguity about which parser to use)
|
|
llm_answer_final = parser(llm_answer_raw)
|
|
|
|
return llm_answer_final
|
|
```
|
|
|
|
**Benefits of This Approach**:
|
|
|
|
1. **No Ambiguity**: The prompt template itself declares which parser should be used
|
|
2. **Type Safety**: Return type hints make it clear what parser function expects
|
|
3. **Self-Documenting**: Reading the prompt function immediately shows the expected output format
|
|
4. **Refactor-Friendly**: Changing a prompt's output format requires updating one function
|
|
5. **Testing**: Easy to test that prompt returns correct parser for its format
|
|
|
|
**Downstream Impact**:
|
|
|
|
All calling code in `prompt_calls.py` files must be updated to:
|
|
1. Unpack the tuple: `prompt, parser = prompt_templates.SOME_PROMPT(...)`
|
|
2. Use the returned parser: `result = parser(llm_raw_output)`
|
|
3. Remove hardcoded parser calls (e.g., no more `string_utils.extract_text_from_delimiters`)
|
|
|
|
**Migration Status**:
|
|
- ✅ All 35 prompt template functions updated
|
|
- ⏳ Downstream calling code update in progress (see Phase 2 below)
|
|
|
|
### 4.2 JSON Schema Expectations
|
|
|
|
#### 4.2.1 Dictionary Outputs
|
|
|
|
**Used for**: Single-entity extractions (exhibit level, methodology breakout, etc.)
|
|
|
|
**Schema**:
|
|
```json
|
|
{
|
|
"FIELD_NAME_1": "value or N/A",
|
|
"FIELD_NAME_2": ["value1", "value2"],
|
|
"FIELD_NAME_3": 123,
|
|
"FIELD_NAME_4": true
|
|
}
|
|
```
|
|
|
|
**Example Prompt Update**:
|
|
|
|
**OLD** (pipe-delimited):
|
|
```
|
|
[OUTPUT FORMAT]
|
|
Briefly explain your answer, then enclose your final answer in |pipes|.
|
|
```
|
|
|
|
**NEW** (JSON):
|
|
```
|
|
[OUTPUT FORMAT]
|
|
First provide your reasoning, then output a JSON dictionary with the field name as the key.
|
|
|
|
Example:
|
|
My reasoning: The text mentions "Medicare Advantage"...
|
|
|
|
{"LOB": ["Medicare", "Medicare Advantage"]}
|
|
```
|
|
|
|
#### 4.2.2 List Outputs
|
|
|
|
**Used for**: Multi-entity extractions (reimbursement terms, etc.)
|
|
|
|
**Schema**:
|
|
```json
|
|
[
|
|
{
|
|
"SERVICE_TERM": "...",
|
|
"REIMB_TERM": "...",
|
|
"OTHER_FIELD": "..."
|
|
},
|
|
{
|
|
"SERVICE_TERM": "...",
|
|
"REIMB_TERM": "...",
|
|
"OTHER_FIELD": "..."
|
|
}
|
|
]
|
|
```
|
|
|
|
**Special Case**: Empty list for "no results found"
|
|
```json
|
|
[]
|
|
```
|
|
|
|
Or explicit marker:
|
|
```json
|
|
{"NO_RESULTS": true}
|
|
```
|
|
|
|
#### 4.2.3 Multi-Value Fields
|
|
|
|
**OLD** (pipe-delimited in JSON):
|
|
```json
|
|
{
|
|
"LOB": "Medicare|Medicaid|Duals"
|
|
}
|
|
```
|
|
|
|
**NEW** (JSON array):
|
|
```json
|
|
{
|
|
"LOB": ["Medicare", "Medicaid", "Duals"]
|
|
}
|
|
```
|
|
|
|
**Prompt instruction change**:
|
|
|
|
**OLD**:
|
|
> "If there are multiple values, return them in a pipe-separated list ("|")"
|
|
|
|
**NEW**:
|
|
> "If there are multiple values, return them as a JSON array. For example: `\"LOB\": [\"Medicare\", \"Medicaid\"]`"
|
|
|
|
---
|
|
|
|
## 5. File-by-File Change List
|
|
|
|
### 5.1 Core Utilities
|
|
|
|
#### 5.1.1 Create: `src/utils/json_parsers.py`
|
|
|
|
**Change Type**: NEW FILE
|
|
|
|
**Content**: See section 4.1.1 for full implementation
|
|
|
|
**Key Functions**:
|
|
- `parse_json_dict()`
|
|
- `parse_json_list()`
|
|
- `_extract_json_objects()` (helper)
|
|
- `universal_json_load()` (deprecated alias)
|
|
|
|
---
|
|
|
|
#### 5.1.2 Modify: `src/utils/string_utils.py`
|
|
|
|
**Change Type**: DEPRECATION
|
|
|
|
**Changes**:
|
|
1. Add deprecation notice to `universal_json_load()`
|
|
2. Import and delegate to new parsers
|
|
3. Keep `extract_text_from_delimiters()` but mark as deprecated
|
|
|
|
**Code**:
|
|
```python
|
|
# Add to imports
|
|
from src.utils.json_parsers import parse_json_dict, parse_json_list, JSONParseError
|
|
|
|
# Update universal_json_load
|
|
def universal_json_load(s: str):
|
|
"""
|
|
DEPRECATED: Use parse_json_dict() or parse_json_list() from json_parsers module.
|
|
|
|
This function is maintained for backward compatibility.
|
|
Will be removed in future version.
|
|
"""
|
|
import warnings
|
|
warnings.warn(
|
|
"universal_json_load is deprecated. Use json_parsers.parse_json_dict() "
|
|
"or json_parsers.parse_json_list() instead.",
|
|
DeprecationWarning,
|
|
stacklevel=2
|
|
)
|
|
from src.utils.json_parsers import _extract_json_objects
|
|
found = _extract_json_objects(s)
|
|
if not found:
|
|
raise ValueError(f"No valid JSON objects found in string: {s[:100]}...")
|
|
return found[-1]
|
|
|
|
|
|
def extract_text_from_delimiters(
|
|
raw_output: str, delimiter: Delimiter, match_index: int = -1
|
|
) -> str:
|
|
"""
|
|
DEPRECATED: Extracting text from delimiters is being phased out.
|
|
Use JSON parsing instead.
|
|
|
|
... (rest of implementation unchanged)
|
|
"""
|
|
import warnings
|
|
warnings.warn(
|
|
"extract_text_from_delimiters is deprecated. "
|
|
"Use JSON-based prompt formats instead.",
|
|
DeprecationWarning,
|
|
stacklevel=2
|
|
)
|
|
# ... existing implementation ...
|
|
```
|
|
|
|
---
|
|
|
|
#### 5.1.3 Modify: `src/constants/delimiters.py`
|
|
|
|
**Change Type**: DOCUMENTATION
|
|
|
|
**Changes**:
|
|
1. Add comment noting PIPE delimiter is deprecated for LLM parsing
|
|
2. Keep enum for backward compatibility
|
|
|
|
**Code**:
|
|
```python
|
|
from enum import Enum
|
|
|
|
|
|
class Delimiter(Enum):
|
|
"""
|
|
Delimiter types for parsing.
|
|
|
|
NOTE: PIPE delimiter is deprecated for LLM output parsing.
|
|
Use JSON formats instead. This enum is maintained for
|
|
backward compatibility only.
|
|
"""
|
|
PIPE = "|" # DEPRECATED for LLM outputs
|
|
BACKTICK = "`"
|
|
TRIPLE_BACKTICK = "```"
|
|
```
|
|
|
|
---
|
|
|
|
### 5.2 Prompt Templates
|
|
|
|
#### 5.2.1 Modify: `src/prompts/prompt_templates.py`
|
|
|
|
**Change Type**: MAJOR REFACTOR
|
|
|
|
**Changes**:
|
|
|
|
1. **Add JSON Format Instruction Constants** (at module level, after existing constants):
|
|
```python
|
|
# JSON format instruction constants - referenced by all prompt templates
|
|
JSON_DICT_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted JSON dictionary.
|
|
If the requested information doesn't apply to this context, return 'N/A'.
|
|
If the information should exist but cannot be clearly identified, return 'UNKNOWN'.
|
|
|
|
Example format:
|
|
{
|
|
"FIELD_NAME": "value",
|
|
"ANOTHER_FIELD": "value or N/A"
|
|
}
|
|
"""
|
|
|
|
JSON_LIST_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted valid JSON list format.
|
|
If the requested information doesn't apply to this context, return ["N/A"].
|
|
If the information should exist but cannot be clearly identified, return ["UNKNOWN"].
|
|
|
|
Formatting examples:
|
|
- Single string value: ["value"]
|
|
- Multiple string values: ["value1", "value2"]
|
|
- Date value: ["YYYY-MM-DD"]
|
|
- Not applicable: ["N/A"]
|
|
- Information unclear or missing: ["UNKNOWN"]
|
|
"""
|
|
|
|
JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted JSON dictionary where each field value is a JSON list.
|
|
For fields with multiple values, return them as an array: {"FIELD": ["value1", "value2"]}.
|
|
For fields with a single value, still return as an array: {"FIELD": ["value"]}.
|
|
If the requested information doesn't apply, return ["N/A"]: {"FIELD": ["N/A"]}.
|
|
If the information should exist but cannot be identified, return ["UNKNOWN"]: {"FIELD": ["UNKNOWN"]}.
|
|
|
|
Example format:
|
|
{
|
|
"LOB": ["Medicare", "Medicaid"],
|
|
"NETWORK": ["N/A"],
|
|
"FACILITY_TYPE": ["Inpatient"]
|
|
}
|
|
"""
|
|
|
|
JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS = """Return your answer as a valid JSON array of objects. Each object represents one entity (e.g., one reimbursement term).
|
|
If no entities are found, return an empty array: []
|
|
|
|
Example format:
|
|
[
|
|
{
|
|
"SERVICE_TERM": "Inpatient Services",
|
|
"REIMB_TERM": "110% of Medicare"
|
|
},
|
|
{
|
|
"SERVICE_TERM": "Outpatient Services",
|
|
"REIMB_TERM": "$500 per visit"
|
|
}
|
|
]
|
|
|
|
For no results: []
|
|
"""
|
|
|
|
# Keep PIPE_FORMAT_INSTRUCTIONS temporarily for backward compatibility
|
|
# Mark as deprecated
|
|
PIPE_FORMAT_INSTRUCTIONS = "DEPRECATED: Use JSON format instructions instead."
|
|
```
|
|
|
|
2. **Update EXHIBIT_LEVEL() and EXHIBIT_LEVEL_INSTRUCTION()**:
|
|
|
|
**Dynamic prompt (NO format instructions)**:
|
|
```python
|
|
def EXHIBIT_LEVEL(context, fields):
|
|
"""Returns ONLY dynamic content for exhibit level extraction.
|
|
Call EXHIBIT_LEVEL_INSTRUCTION() separately for the cached instruction.
|
|
"""
|
|
return f"""[ATTRIBUTES]
|
|
Here are the attributes to be included in the dictionary, and instructions on how to correctly answer:
|
|
{fields}
|
|
|
|
[CONTEXT]
|
|
Here is the text to analyze:
|
|
{context.replace('"', "'")}"""
|
|
```
|
|
|
|
**Cached instruction (WITH format instructions)**:
|
|
```python
|
|
def EXHIBIT_LEVEL_INSTRUCTION() -> str:
|
|
"""Static instruction for EXHIBIT_LEVEL prompt caching.
|
|
Contains all static instructions including output format.
|
|
"""
|
|
return f"""[OBJECTIVE]
|
|
Extract attributes from a section of a contract.
|
|
|
|
[GENERAL INSTRUCTIONS]
|
|
- For each field, return ALL correct answers found
|
|
- Return each field value as a JSON list, even if only one value: {{"FIELD": ["value"]}}
|
|
- If multiple values exist, return all: {{"FIELD": ["value1", "value2"]}}
|
|
- If a list of valid values is provided, ONLY answer with EXACT values from that list
|
|
- If no correct answers for a field, return ["N/A"] for that field
|
|
- Be consistent and deterministic; avoid creative paraphrasing
|
|
|
|
[OUTPUT FORMAT]
|
|
{JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS}"""
|
|
```
|
|
|
|
3. **Update DYNAMIC_PRIMARY_TEXT() and DYNAMIC_PRIMARY_TEXT_INSTRUCTION()**:
|
|
|
|
**Dynamic prompt (NO format instructions)**:
|
|
```python
|
|
def DYNAMIC_PRIMARY_TEXT(context, field_name, field_prompt):
|
|
"""Returns ONLY dynamic content for text-based dynamic primary extraction.
|
|
Call DYNAMIC_PRIMARY_TEXT_INSTRUCTION() separately for the cached instruction.
|
|
"""
|
|
return f"""[ATTRIBUTE TO EXTRACT]
|
|
{field_name} : {field_prompt}
|
|
|
|
[CONTEXT]
|
|
Here is the text to analyze:
|
|
{context.replace('"', "'")}"""
|
|
```
|
|
|
|
**Cached instruction (WITH format instructions)**:
|
|
```python
|
|
def DYNAMIC_PRIMARY_TEXT_INSTRUCTION() -> str:
|
|
"""Static instruction for DYNAMIC_PRIMARY_TEXT prompt caching.
|
|
Contains all static instructions including output format.
|
|
"""
|
|
return f"""[OBJECTIVE]
|
|
Extract attribute values for dynamic fields from contract text, paying close attention to detail.
|
|
|
|
[GENERAL INSTRUCTIONS]
|
|
- Return all valid, explicitly stated values as a JSON list
|
|
- If multiple values found, include all: ["value1", "value2", "value3"]
|
|
- If only one value found, still return as a list: ["value1"]
|
|
- If no valid values appear explicitly, return: ["N/A"]
|
|
- Make appropriate and obvious assumptions, but don't take huge leaps of logic
|
|
Example: "CHIP-P" can be assumed to be "CHIP Perinate", but NOT "Medicaid"
|
|
|
|
[OUTPUT FORMAT]
|
|
{JSON_LIST_FORMAT_INSTRUCTIONS}"""
|
|
```
|
|
|
|
4. **Update CARVEOUT_CHECK() and CARVEOUT_CHECK_INSTRUCTION()**:
|
|
|
|
**Dynamic prompt (NO format instructions)**:
|
|
```python
|
|
def CARVEOUT_CHECK(service_term: str, reimb_term: str):
|
|
"""Returns prompt for carveout check.
|
|
Call CARVEOUT_CHECK_INSTRUCTION() separately for the cached instruction.
|
|
"""
|
|
return f"""[REIMBURSEMENT TERM]
|
|
Service Term: {service_term}
|
|
Reimbursement Term: {reimb_term}
|
|
|
|
Determine which case from the provided list this reimbursement term falls under."""
|
|
```
|
|
|
|
**Cached instruction (WITH format instructions)**:
|
|
```python
|
|
def CARVEOUT_CHECK_INSTRUCTION() -> str:
|
|
"""Static instruction for CARVEOUT_CHECK prompt caching.
|
|
Contains all static instructions including output format and case definitions.
|
|
"""
|
|
# Build carveout_text and special_case_text from constants
|
|
constants = _get_constants()
|
|
carveout_definitions = constants.get_constant("CARVEOUT_DEFINITIONS")
|
|
special_case_definitions = constants.get_constant("SPECIAL_CASE_DEFINITIONS")
|
|
# ... format definitions ...
|
|
|
|
return f"""[OBJECTIVE]
|
|
Examine the Reimbursement Term and identify which case it falls under.
|
|
|
|
[CASE DEFINITIONS]
|
|
Here are the cases, with their definitions:
|
|
Carveout cases: {carveout_text}
|
|
|
|
Special cases: {special_case_text}
|
|
|
|
[INSTRUCTIONS]
|
|
- Determine which case description corresponds to the specific reimbursement method
|
|
- The Service and Reimbursement will always match one of the listed cases
|
|
- If either is an explicit match, choose that case
|
|
- If multiple cases apply, choose the BEST fit
|
|
- Always return one value from the combined list; do not return 'N/A' or 'UNKNOWN'
|
|
|
|
[OUTPUT FORMAT]
|
|
{JSON_DICT_FORMAT_INSTRUCTIONS}
|
|
|
|
Return as: {{"CASE": "CASE_NAME"}}"""
|
|
```
|
|
|
|
5. **Update REIMBURSEMENT_PRIMARY() and REIMBURSEMENT_PRIMARY_INSTRUCTION()**:
|
|
|
|
**Dynamic prompt (NO format instructions)**:
|
|
```python
|
|
def REIMBURSEMENT_PRIMARY(context):
|
|
"""Returns ONLY dynamic content for reimbursement primary extraction.
|
|
Call REIMBURSEMENT_PRIMARY_INSTRUCTION() separately for the cached instruction.
|
|
"""
|
|
return f"""[CONTEXT]
|
|
Here is the contract text to analyze:
|
|
{context}"""
|
|
```
|
|
|
|
**Cached instruction (WITH format instructions)**:
|
|
```python
|
|
def REIMBURSEMENT_PRIMARY_INSTRUCTION() -> str:
|
|
"""Static instruction for REIMBURSEMENT_PRIMARY prompt caching.
|
|
Contains all static instructions including output format.
|
|
"""
|
|
return f"""[OBJECTIVE]
|
|
Extract all reimbursement terms from the contract text.
|
|
|
|
[INSTRUCTIONS]
|
|
- Identify every service and its associated reimbursement rate/method
|
|
- Each service-reimbursement pair becomes one object in the output array
|
|
- Be comprehensive - capture all reimbursement terms on the page
|
|
|
|
[OUTPUT FORMAT]
|
|
{JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS}
|
|
|
|
Expected fields per reimbursement term:
|
|
- SERVICE_TERM: The service or procedure description
|
|
- REIMB_TERM: The reimbursement rate or method
|
|
|
|
If no reimbursement terms found, return: []"""
|
|
```
|
|
|
|
6. **Update other prompt functions** - General pattern:
|
|
|
|
**CRITICAL PATTERN**: Format instructions go in INSTRUCTION() functions (cached), NOT in dynamic prompts.
|
|
|
|
**Example: METHODOLOGY_BREAKOUT**:
|
|
```python
|
|
# Dynamic prompt (NO format instructions)
|
|
def METHODOLOGY_BREAKOUT(service_term, reimb_term):
|
|
return f"""[CONTEXT]
|
|
Service: {service_term}
|
|
Reimbursement: {reimb_term}
|
|
|
|
Extract methodology components."""
|
|
|
|
# Cached instruction (WITH format instructions)
|
|
def METHODOLOGY_BREAKOUT_INSTRUCTION():
|
|
return f"""[OBJECTIVE]
|
|
Break down reimbursement methodology into components.
|
|
|
|
[INSTRUCTIONS]
|
|
... (domain logic) ...
|
|
|
|
[OUTPUT FORMAT]
|
|
{JSON_DICT_FORMAT_INSTRUCTIONS}
|
|
|
|
Expected fields: REIMBURSEMENT_METHOD, PERCENT, BASIS, FEE_SCHEDULE, etc."""
|
|
```
|
|
|
|
**Example: DYNAMIC_ASSIGNMENT**:
|
|
```python
|
|
# Dynamic prompt (NO format instructions)
|
|
def DYNAMIC_ASSIGNMENT(service_term, reimb_term, field_name, field_prompt, exhibit_text, page_num):
|
|
return f"""[CONTEXT]
|
|
{exhibit_text}
|
|
|
|
[REIMBURSEMENT_TERM]
|
|
Service: {service_term}
|
|
Reimbursement: {reimb_term}
|
|
|
|
[FIELD TO ASSIGN]
|
|
{field_name}: {field_prompt}"""
|
|
|
|
# Cached instruction (WITH format instructions)
|
|
def DYNAMIC_ASSIGNMENT_INSTRUCTION():
|
|
return f"""[OBJECTIVE]
|
|
Identify which field value applies to a given Reimbursement Term.
|
|
|
|
[INSTRUCTIONS]
|
|
... (domain logic about context, boundaries, etc.) ...
|
|
|
|
[OUTPUT FORMAT]
|
|
{JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS}
|
|
|
|
Return as: {{"FIELD_NAME": ["value1", "value2"]}}"""
|
|
```
|
|
|
|
**Example: ONE_TO_ONE_MULTI_FIELD_TEMPLATE**:
|
|
```python
|
|
# Dynamic prompt (NO format instructions)
|
|
def ONE_TO_ONE_MULTI_FIELD_TEMPLATE(context, questions):
|
|
return f"""[CONTRACT TEXT]
|
|
{context}
|
|
|
|
[FIELDS TO EXTRACT]
|
|
{questions}"""
|
|
|
|
# Cached instruction (WITH format instructions)
|
|
def ONE_TO_ONE_MULTI_FIELD_INSTRUCTION():
|
|
return f"""[OBJECTIVE]
|
|
Extract contract-level fields from full contract text.
|
|
|
|
[INSTRUCTIONS]
|
|
... (domain logic) ...
|
|
|
|
[OUTPUT FORMAT]
|
|
{JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS}
|
|
|
|
Each field value should be a JSON list, even if single value."""
|
|
```
|
|
|
|
7. **Comprehensive List of Prompt Functions to Update**:
|
|
|
|
**CRITICAL**: Format instructions are added to `_INSTRUCTION()` functions (cached), NOT to dynamic prompt functions.
|
|
|
|
All `_INSTRUCTION()` functions must be updated to include the appropriate JSON format instruction variable at the end. Here's the complete list organized by format type:
|
|
|
|
**INSTRUCTION Functions to Update with `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`**:
|
|
- `EXHIBIT_LEVEL_INSTRUCTION()` ✓
|
|
- `DYNAMIC_ASSIGNMENT_INSTRUCTION()`
|
|
- `ONE_TO_ONE_SINGLE_FIELD_INSTRUCTION()`
|
|
- `ONE_TO_ONE_MULTI_FIELD_INSTRUCTION()`
|
|
- `TIN_NPI_TEMPLATE_INSTRUCTION()`
|
|
- `LESSER_OF_CHECK_INSTRUCTION()`
|
|
|
|
**INSTRUCTION Functions to Update with `JSON_DICT_FORMAT_INSTRUCTIONS`**:
|
|
- `CARVEOUT_CHECK_INSTRUCTION()` ✓
|
|
- `METHODOLOGY_BREAKOUT_INSTRUCTION()`
|
|
- `FEE_SCHEDULE_BREAKOUT_INSTRUCTION()`
|
|
- `GROUPER_BREAKOUT_INSTRUCTION()`
|
|
- `FACILITY_ADJUSTMENT_BREAKOUT_INSTRUCTION()`
|
|
- `LOB_RELATIONSHIP_INSTRUCTION()`
|
|
- `EXHIBIT_LINKAGE_INSTRUCTION()`
|
|
- All special case breakout instructions (STOP_LOSS, OUTLIER, etc.)
|
|
|
|
**INSTRUCTION Functions to Update with `JSON_LIST_FORMAT_INSTRUCTIONS`**:
|
|
- `DYNAMIC_PRIMARY_TEXT_INSTRUCTION()` ✓
|
|
- `LESSER_OF_DISTRIBUTION_INSTRUCTION()`
|
|
- `REIMB_DATES_ASSIGNMENT_INSTRUCTION()`
|
|
- `EXHIBIT_TITLE_MATCH_INSTRUCTION()`
|
|
- `CODE_IMPLICIT_SPECIAL_INSTRUCTION()`
|
|
- `CODE_PLACEHOLDER_INSTRUCTION()`
|
|
|
|
**INSTRUCTION Functions to Update with `JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS`**:
|
|
- `REIMBURSEMENT_PRIMARY_INSTRUCTION()` ✓
|
|
- `VALIDATE_REIMBURSEMENTS_INSTRUCTION()`
|
|
|
|
**Pattern for Updates**:
|
|
```python
|
|
# OLD (no format instructions in INSTRUCTION function):
|
|
def SOME_PROMPT_INSTRUCTION():
|
|
return """[OBJECTIVE]
|
|
Do something...
|
|
|
|
[INSTRUCTIONS]
|
|
Follow these rules..."""
|
|
|
|
# NEW (format instructions added to INSTRUCTION function):
|
|
def SOME_PROMPT_INSTRUCTION():
|
|
return f"""[OBJECTIVE]
|
|
Do something...
|
|
|
|
[INSTRUCTIONS]
|
|
Follow these rules...
|
|
|
|
[OUTPUT FORMAT]
|
|
{JSON_DICT_FORMAT_INSTRUCTIONS}"""
|
|
|
|
# Dynamic prompt remains clean (NO format instructions)
|
|
def SOME_PROMPT(context):
|
|
return f"""[CONTEXT]
|
|
{context}"""
|
|
```
|
|
|
|
**Instruction Functions** (cached - ADD format instruction variables):
|
|
|
|
**CRITICAL**: These are the functions that need updates. Format instructions must be ADDED to these cached instruction functions, NOT to the dynamic prompt functions.
|
|
|
|
Complete list of `_INSTRUCTION()` functions to update:
|
|
- `EXHIBIT_LEVEL_INSTRUCTION()` ✓ - Add `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- `DYNAMIC_PRIMARY_TEXT_INSTRUCTION()` ✓ - Add `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- `CARVEOUT_CHECK_INSTRUCTION()` ✓ - Add `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
- `REIMBURSEMENT_PRIMARY_INSTRUCTION()` ✓ - Add `JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS`
|
|
- `METHODOLOGY_BREAKOUT_INSTRUCTION()` - Add `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
- `FEE_SCHEDULE_BREAKOUT_INSTRUCTION()` - Add `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
- `GROUPER_BREAKOUT_INSTRUCTION()` - Add `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
- `DYNAMIC_ASSIGNMENT_INSTRUCTION()` - Add `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- `REIMB_DATES_ASSIGNMENT_INSTRUCTION()` - Add `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- `LESSER_OF_DISTRIBUTION_INSTRUCTION()` - Add `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- `LESSER_OF_CHECK_INSTRUCTION()` - Add `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- `LOB_RELATIONSHIP_INSTRUCTION()` - Add `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
- `ONE_TO_ONE_SINGLE_FIELD_INSTRUCTION()` - Add `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- `ONE_TO_ONE_MULTI_FIELD_INSTRUCTION()` - Add `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- `TIN_NPI_TEMPLATE_INSTRUCTION()` - Add `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- `EXHIBIT_LINKAGE_INSTRUCTION()` - Add `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
- `VALIDATE_REIMBURSEMENTS_INSTRUCTION()` - Add `JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS`
|
|
- `EXHIBIT_TITLE_MATCH_INSTRUCTION()` - Add `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- All special case breakout instructions - Add `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
|
|
**Dynamic Prompt Functions** (do NOT add format instructions to these - they should remain clean)
|
|
|
|
---
|
|
|
|
### 5.3 Prompt Calling Layer
|
|
|
|
#### 5.3.1 Modify: `src/pipelines/saas/prompts/prompt_calls.py`
|
|
|
|
**Change Type**: MAJOR REFACTOR
|
|
|
|
**Changes**:
|
|
|
|
1. **Update Imports**:
|
|
```python
|
|
from src.utils import json_parsers # Add this
|
|
# Remove references to extract_text_from_delimiters for new code
|
|
```
|
|
|
|
2. **Update `prompt_exhibit_level()`**:
|
|
```python
|
|
def prompt_exhibit_level(
|
|
exhibit_text: str,
|
|
exhibit_level_fields: FieldSet,
|
|
constants: Constants,
|
|
filename: str,
|
|
):
|
|
logging.debug(f"Running exhibit Level Fields for {filename}:")
|
|
logging.debug(exhibit_level_fields.print_prompt_dict(constants))
|
|
if not exhibit_level_fields.contains_fields():
|
|
return {}
|
|
|
|
prompt = prompt_templates.EXHIBIT_LEVEL(
|
|
exhibit_text, exhibit_level_fields.print_prompt_dict(constants)
|
|
)
|
|
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
prompt,
|
|
"sonnet_latest",
|
|
filename,
|
|
max_tokens=8192,
|
|
cache=True,
|
|
instruction=prompt_templates.EXHIBIT_LEVEL_INSTRUCTION(),
|
|
usage_label="EXHIBIT_LEVEL",
|
|
)
|
|
logging.debug(f"LLM raw output for {filename}: {llm_answer_raw}")
|
|
|
|
# OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
|
|
# NEW:
|
|
llm_answer_final = json_parsers.parse_json_dict(llm_answer_raw)
|
|
|
|
return llm_answer_final
|
|
```
|
|
|
|
3. **Update `prompt_exhibit_level_breakout()`**:
|
|
```python
|
|
def prompt_exhibit_level_breakout(
|
|
exhibit_level_answers: dict[str, str], filename: str
|
|
) -> dict[str, str]:
|
|
if not string_utils.is_empty(
|
|
exhibit_level_answers.get("FACILITY_ADJUSTMENT_TERM", "")
|
|
):
|
|
prompt = prompt_templates.FACILITY_ADJUSTMENT_BREAKOUT(
|
|
exhibit_level_answers["FACILITY_ADJUSTMENT_TERM"]
|
|
)
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
prompt=prompt, model_id="sonnet_latest", filename=filename
|
|
)
|
|
|
|
# OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
|
|
# NEW:
|
|
llm_answer_final = json_parsers.parse_json_dict(llm_answer_raw)
|
|
|
|
exhibit_level_answers.update(llm_answer_final)
|
|
|
|
return exhibit_level_answers
|
|
```
|
|
|
|
4. **Update `prompt_dynamic_primary()`**:
|
|
|
|
**CRITICAL CHANGE**: This function currently uses `extract_text_from_delimiters()` with PIPE delimiter. It must transition to JSON list output.
|
|
|
|
```python
|
|
def prompt_dynamic_primary(
|
|
exhibit_text: str, field: Field, constants: Constants, filename: str, TEMPLATE
|
|
):
|
|
prompt = TEMPLATE(exhibit_text, field.field_name, field.get_prompt(constants))
|
|
logging.debug(f"Dynamic primary prompt for {filename}; {field}: {prompt}")
|
|
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
prompt,
|
|
"sonnet_latest",
|
|
filename,
|
|
cache=True,
|
|
instruction=prompt_templates.DYNAMIC_PRIMARY_TEXT_INSTRUCTION(), # Updated instruction
|
|
)
|
|
logging.debug(f"Claude answer for {filename}; {field}: {llm_answer_raw}")
|
|
|
|
# OLD:
|
|
# llm_answer_final = string_utils.extract_text_from_delimiters(
|
|
# llm_answer_raw, string_utils.Delimiter.PIPE
|
|
# )
|
|
# if "," in llm_answer_final:
|
|
# llm_answer_final = "|".join(
|
|
# [item.strip() for item in llm_answer_final.split(",")]
|
|
# )
|
|
|
|
# NEW: Parse as JSON list, then join with pipes for backward compatibility
|
|
# (downstream code still expects pipe-delimited for now)
|
|
try:
|
|
answer_list = json_parsers.parse_json_list(llm_answer_raw)
|
|
llm_answer_final = "|".join(str(item).strip() for item in answer_list)
|
|
except json_parsers.JSONParseError as e:
|
|
logging.error(f"Failed to parse dynamic primary answer: {e}")
|
|
llm_answer_final = "N/A"
|
|
|
|
return llm_answer_final
|
|
```
|
|
|
|
**NOTE**: This function converts JSON list back to pipe-delimited for backward compatibility during Phase 3. In Phase 4, once downstream consumers (particularly `dynamic_funcs.py`) are updated to handle JSON lists natively, this conversion will be removed.
|
|
|
|
**Phase 3 (Temporary)**:
|
|
- LLM outputs: `["Medicare", "Medicaid"]`
|
|
- Parser extracts: `["Medicare", "Medicaid"]`
|
|
- Function returns: `"Medicare|Medicaid"` (converted for compatibility)
|
|
- Downstream code sees: pipe-delimited string (existing behavior)
|
|
|
|
**Phase 4 (Final)**:
|
|
- LLM outputs: `["Medicare", "Medicaid"]`
|
|
- Parser extracts: `["Medicare", "Medicaid"]`
|
|
- Function returns: `["Medicare", "Medicaid"]` (no conversion)
|
|
- Downstream code handles: JSON list natively
|
|
|
|
See Section 5.4.3 for downstream updates to `dynamic_funcs.py`.
|
|
|
|
5. **Update `prompt_reimbursement_primary()`**:
|
|
```python
|
|
def prompt_reimbursement_primary(
|
|
page_text: str,
|
|
filename: str,
|
|
) -> list[dict[str, str]]:
|
|
|
|
prompt = prompt_templates.REIMBURSEMENT_PRIMARY(page_text)
|
|
|
|
logging.debug(
|
|
f"""Running reimbursement primary prompt for {filename} with prompt: {prompt}"""
|
|
)
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
prompt,
|
|
"sonnet_latest",
|
|
filename,
|
|
max_tokens=25000,
|
|
cache=True,
|
|
instruction=prompt_templates.REIMBURSEMENT_PRIMARY_INSTRUCTION(),
|
|
usage_label="REIMBURSEMENT_PRIMARY",
|
|
)
|
|
logging.debug(f"""LLM raw output for {filename}: {llm_answer_raw}""")
|
|
|
|
# Check for the special "no results" case (could be empty array or special marker)
|
|
try:
|
|
# OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
|
|
# NEW:
|
|
llm_answer_final = json_parsers.parse_json_list(llm_answer_raw)
|
|
|
|
# Handle empty list or special no-results marker
|
|
if not llm_answer_final:
|
|
return []
|
|
|
|
# Check for special marker dict
|
|
if len(llm_answer_final) == 1 and isinstance(llm_answer_final[0], dict):
|
|
if llm_answer_final[0].get("NO_RESULTS") or "NO_REIMBURSEMENT_TERMS_FOUND" in str(llm_answer_final[0]):
|
|
return []
|
|
|
|
return llm_answer_final
|
|
|
|
except json_parsers.JSONParseError as e:
|
|
logging.error(f"Error parsing LLM response: {e}")
|
|
logging.error(f"Raw LLM output: {llm_answer_raw}")
|
|
raise
|
|
```
|
|
|
|
6. **Update `prompt_methodology_breakout()`**:
|
|
```python
|
|
def prompt_methodology_breakout(
|
|
service_term: str,
|
|
reimb_term: str,
|
|
filename: str,
|
|
):
|
|
prompt = prompt_templates.METHODOLOGY_BREAKOUT(service_term, reimb_term)
|
|
logging.debug(f"Methodology Breakout Prompt for {filename}: {prompt}")
|
|
|
|
llm_response = llm_utils.invoke_claude(
|
|
prompt,
|
|
"sonnet_latest",
|
|
filename,
|
|
cache=True,
|
|
instruction=prompt_templates.METHODOLOGY_BREAKOUT_INSTRUCTION(),
|
|
usage_label="METHODOLOGY_BREAKOUT",
|
|
)
|
|
logging.debug(f"LLM Response for {filename}: {llm_response}")
|
|
|
|
try:
|
|
# OLD: methodology_breakout_answers = string_utils.universal_json_load(llm_response)
|
|
# NEW:
|
|
methodology_breakout_answers = json_parsers.parse_json_dict(llm_response)
|
|
except json_parsers.JSONParseError as e:
|
|
logging.error(f"Error parsing LLM response: {e}")
|
|
methodology_breakout_answers = {}
|
|
|
|
return methodology_breakout_answers
|
|
```
|
|
|
|
7. **Update `prompt_fee_schedule_breakout()`**:
|
|
```python
|
|
def prompt_fee_schedule_breakout(
|
|
methodology_breakout_dict: dict,
|
|
reimbursement_method: str,
|
|
filename: str,
|
|
):
|
|
prompt = prompt_templates.FEE_SCHEDULE_BREAKOUT(
|
|
reimbursement_method,
|
|
methodology_breakout_dict.get("FEE_SCHEDULE"),
|
|
)
|
|
logging.debug(f"Fee Schedule Breakout Prompt for {filename}: {prompt}")
|
|
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
prompt,
|
|
"sonnet_latest",
|
|
filename,
|
|
cache=True,
|
|
instruction=prompt_templates.FEE_SCHEDULE_BREAKOUT_INSTRUCTION(),
|
|
usage_label="FEE_SCHEDULE_BREAKOUT",
|
|
)
|
|
logging.debug(f"LLM Response for {filename}: {llm_answer_raw}")
|
|
|
|
try:
|
|
# OLD: fs_breakout_dict = string_utils.universal_json_load(llm_answer_raw)
|
|
# NEW:
|
|
fs_breakout_dict = json_parsers.parse_json_dict(llm_answer_raw)
|
|
except json_parsers.JSONParseError as e:
|
|
logging.error(f"Error parsing fee schedule breakout: {e}")
|
|
fs_breakout_dict = {}
|
|
|
|
return fs_breakout_dict
|
|
```
|
|
|
|
8. **Update `prompt_grouper_breakout()`**:
|
|
```python
|
|
def prompt_grouper_breakout(
|
|
service: str,
|
|
reimbursement_method: str,
|
|
filename: str,
|
|
):
|
|
prompt = prompt_templates.GROUPER_BREAKOUT(service, reimbursement_method)
|
|
logging.debug(f"Grouper Breakout Prompt for {filename}: {prompt}")
|
|
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
prompt,
|
|
"sonnet_latest",
|
|
filename,
|
|
cache=True,
|
|
instruction=prompt_templates.GROUPER_BREAKOUT_INSTRUCTION(),
|
|
usage_label="GROUPER_BREAKOUT",
|
|
)
|
|
logging.debug(f"LLM Response for {filename}: {llm_answer_raw}")
|
|
|
|
try:
|
|
# OLD: grouper_breakout_dict = string_utils.universal_json_load(llm_answer_raw)
|
|
# NEW:
|
|
grouper_breakout_dict = json_parsers.parse_json_dict(llm_answer_raw)
|
|
except json_parsers.JSONParseError as e:
|
|
logging.error(f"Error parsing grouper breakout: {e}")
|
|
grouper_breakout_dict = {}
|
|
|
|
return grouper_breakout_dict
|
|
```
|
|
|
|
9. **Update `prompt_carveout_check()`**:
|
|
|
|
**CRITICAL CHANGE**: Currently uses `extract_text_from_delimiters()`. Must parse JSON dict.
|
|
|
|
```python
|
|
def prompt_carveout_check(
|
|
service_term: str,
|
|
reimb_term: str,
|
|
filename: str,
|
|
):
|
|
carveout_prompt = prompt_templates.CARVEOUT_CHECK(service_term, reimb_term)
|
|
logging.debug(
|
|
f"Running carveout check for {filename} with prompt:\n{carveout_prompt}"
|
|
)
|
|
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
carveout_prompt,
|
|
"sonnet_latest",
|
|
filename,
|
|
cache=True,
|
|
instruction=prompt_templates.CARVEOUT_CHECK_INSTRUCTION(),
|
|
usage_label="CARVEOUT_CHECK",
|
|
)
|
|
|
|
# OLD:
|
|
# carveout_answer = string_utils.extract_text_from_delimiters(
|
|
# llm_answer_raw, Delimiter.PIPE
|
|
# )
|
|
|
|
# NEW:
|
|
try:
|
|
answer_dict = json_parsers.parse_json_dict(llm_answer_raw)
|
|
carveout_answer = answer_dict.get("CASE", "N/A")
|
|
except json_parsers.JSONParseError as e:
|
|
logging.error(f"Failed to parse carveout check answer: {e}")
|
|
carveout_answer = "N/A"
|
|
|
|
return carveout_answer
|
|
```
|
|
|
|
10. **Update `prompt_special_case_breakout()`**:
|
|
```python
|
|
def prompt_special_case_breakout(breakout_template, term_answer, filename):
|
|
prompt = breakout_template(term_answer)
|
|
|
|
if not isinstance(prompt, str):
|
|
logging.warning(
|
|
f"Expected prompt string, got {type(prompt)} in prompt_special_case_breakout; coercing to str."
|
|
)
|
|
prompt = str(prompt)
|
|
|
|
template_name = getattr(breakout_template, "__name__", "SPECIAL_CASE_BREAKOUT")
|
|
instruction_func_name = f"{template_name}_INSTRUCTION"
|
|
instruction_func = getattr(prompt_templates, instruction_func_name, None)
|
|
|
|
if instruction_func and callable(instruction_func):
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
prompt,
|
|
"sonnet_latest",
|
|
filename,
|
|
cache=True,
|
|
instruction=instruction_func(),
|
|
usage_label=template_name,
|
|
)
|
|
else:
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
prompt, "sonnet_latest", filename, usage_label="SPECIAL_CASE_BREAKOUT"
|
|
)
|
|
|
|
# OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
|
|
# NEW:
|
|
llm_answer_final = json_parsers.parse_json_dict(llm_answer_raw)
|
|
|
|
return llm_answer_final
|
|
```
|
|
|
|
11. **Update `prompt_full_context()`**:
|
|
```python
|
|
def prompt_full_context(
|
|
contract_text: str,
|
|
full_context_fields: FieldSet,
|
|
constants: Constants,
|
|
filename: str,
|
|
):
|
|
# ... (setup code unchanged)
|
|
|
|
try:
|
|
claude_answer_raw = llm_utils.invoke_claude(
|
|
full_context_prompt,
|
|
"sonnet_latest",
|
|
filename,
|
|
8192,
|
|
cache=True,
|
|
instruction=prompt_templates.ONE_TO_ONE_MULTI_FIELD_INSTRUCTION(),
|
|
)
|
|
|
|
# OLD: full_context_answers_dict = string_utils.universal_json_load(claude_answer_raw)
|
|
# NEW:
|
|
full_context_answers_dict = json_parsers.parse_json_dict(claude_answer_raw)
|
|
|
|
except Exception as e:
|
|
logging.error(f"Error processing high accuracy fields: {str(e)}")
|
|
full_context_answers_dict = {}
|
|
for field in full_context_fields.list_fields():
|
|
full_context_answers_dict[field] = f"Error: {str(e)}"
|
|
|
|
return full_context_answers_dict
|
|
```
|
|
|
|
12. **Update `prompt_dynamic()`**:
|
|
```python
|
|
def prompt_dynamic(
|
|
text: str, field_prompts: dict, filename: str
|
|
):
|
|
# ... (setup code unchanged)
|
|
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
prompt,
|
|
"sonnet_latest",
|
|
filename,
|
|
cache=True,
|
|
instruction=prompt_templates.EXHIBIT_LEVEL_INSTRUCTION(),
|
|
)
|
|
logging.debug(f"Dynamic answer for {filename}: {llm_answer_raw}")
|
|
|
|
# OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
|
|
# NEW:
|
|
llm_answer_final = json_parsers.parse_json_dict(llm_answer_raw)
|
|
|
|
return llm_answer_final
|
|
```
|
|
|
|
13. **Update `prompt_dynamic_field_safe()`**:
|
|
```python
|
|
def prompt_dynamic_field_safe(
|
|
# ... parameters
|
|
):
|
|
# ... (prompt construction unchanged)
|
|
|
|
llm_answer_raw = llm_utils.invoke_claude(
|
|
# ... parameters
|
|
)
|
|
|
|
try:
|
|
# OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
|
|
# NEW:
|
|
llm_answer_final = json_parsers.parse_json_dict(llm_answer_raw)
|
|
return llm_answer_final
|
|
except json_parsers.JSONParseError:
|
|
logging.error(
|
|
f"Failed to parse Dynamic Field: {field_name}. Response: {llm_answer_raw}"
|
|
)
|
|
return {}
|
|
```
|
|
|
|
**Summary**: 11 call sites updated in this file
|
|
|
|
---
|
|
|
|
#### 5.3.2 Modify: `src/pipelines/clients/clover/prompts/prompt_calls.py`
|
|
|
|
**Change Type**: SAME AS SAAS
|
|
|
|
**Changes**: Apply identical updates as in `src/pipelines/saas/prompts/prompt_calls.py`
|
|
|
|
**Functions to Update**:
|
|
- `prompt_exhibit_level_breakout()`
|
|
- `prompt_dynamic_primary()`
|
|
- `prompt_reimbursement_primary()`
|
|
- All other functions that use `universal_json_load` or `extract_text_from_delimiters`
|
|
|
|
---
|
|
|
|
#### 5.3.3 Modify: `src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py`
|
|
|
|
**Change Type**: SAME AS SAAS
|
|
|
|
**Changes**: Apply identical updates as in `src/pipelines/saas/prompts/prompt_calls.py`
|
|
|
|
---
|
|
|
|
### 5.4 Downstream Pipe Split Consumers
|
|
|
|
#### 5.4.1 Modify: `src/pipelines/shared/postprocessing/aarete_derived.py`
|
|
|
|
**Change Type**: LOGIC UPDATE
|
|
|
|
**Current Issue**: Code expects pipe-delimited strings from upstream
|
|
|
|
**Changes**:
|
|
|
|
**Location 1** (lines 42-49):
|
|
```python
|
|
# OLD:
|
|
all_lob_values = set()
|
|
existing_lob = answer_dict.get("AARETE_DERIVED_LOB", "")
|
|
if not string_utils.is_empty(existing_lob):
|
|
if "|" in existing_lob:
|
|
all_lob_values.update(
|
|
v.strip()
|
|
for v in existing_lob.split("|")
|
|
if v.strip() and v.strip() != "N/A"
|
|
)
|
|
elif existing_lob.strip() and existing_lob.strip() != "N/A":
|
|
all_lob_values.add(existing_lob.strip())
|
|
|
|
# NEW (handles both list and pipe-delimited for transition):
|
|
all_lob_values = set()
|
|
existing_lob = answer_dict.get("AARETE_DERIVED_LOB", "")
|
|
|
|
if not string_utils.is_empty(existing_lob):
|
|
# Handle JSON list format (new)
|
|
if isinstance(existing_lob, list):
|
|
all_lob_values.update(
|
|
str(v).strip()
|
|
for v in existing_lob
|
|
if str(v).strip() and str(v).strip() != "N/A"
|
|
)
|
|
# Handle pipe-delimited format (legacy)
|
|
elif isinstance(existing_lob, str):
|
|
if "|" in existing_lob:
|
|
all_lob_values.update(
|
|
v.strip()
|
|
for v in existing_lob.split("|")
|
|
if v.strip() and v.strip() != "N/A"
|
|
)
|
|
elif existing_lob.strip() and existing_lob.strip() != "N/A":
|
|
all_lob_values.add(existing_lob.strip())
|
|
```
|
|
|
|
**Apply similar logic to**:
|
|
- Lines 64-74 (program_lob_values)
|
|
- Lines 90-100 (product_lob_values)
|
|
- Lines 154-159 (from_field_value_list parsing)
|
|
|
|
**Helper Function** (add to file):
|
|
```python
|
|
def _parse_multi_value_field(value: Any) -> list[str]:
|
|
"""
|
|
Parse a multi-value field that may be in JSON list or pipe-delimited format.
|
|
|
|
Args:
|
|
value: Field value (list, str, or other)
|
|
|
|
Returns:
|
|
List of parsed values
|
|
"""
|
|
if string_utils.is_empty(value):
|
|
return []
|
|
|
|
# JSON list format (preferred)
|
|
if isinstance(value, list):
|
|
return [str(v).strip() for v in value if not string_utils.is_empty(v)]
|
|
|
|
# Pipe-delimited format (legacy)
|
|
if isinstance(value, str):
|
|
if "|" in value:
|
|
return [v.strip() for v in value.split("|") if v.strip()]
|
|
elif "," in value:
|
|
return [v.strip() for v in value.split(",") if v.strip()]
|
|
else:
|
|
return [value.strip()] if value.strip() else []
|
|
|
|
# Fallback: convert to string
|
|
return [str(value).strip()] if str(value).strip() else []
|
|
```
|
|
|
|
**Then refactor all pipe-split logic**:
|
|
```python
|
|
# Instead of:
|
|
if "|" in existing_lob:
|
|
all_lob_values.update(v.strip() for v in existing_lob.split("|") ...)
|
|
|
|
# Use:
|
|
values = _parse_multi_value_field(existing_lob)
|
|
all_lob_values.update(v for v in values if v != "N/A")
|
|
```
|
|
|
|
---
|
|
|
|
#### 5.4.2 Modify: `src/utils/crosswalk_utils.py`
|
|
|
|
**Change Type**: LOGIC UPDATE
|
|
|
|
**Location**: Lines 33-42
|
|
|
|
**Changes**:
|
|
```python
|
|
def apply_crosswalk(val: str, mapping: dict[str, str], default: str = "N/A") -> str:
|
|
# ... (initial checks unchanged)
|
|
|
|
try:
|
|
if "[" in val and "]" in val:
|
|
# Try to parse as literal, but handle any structure returned
|
|
parsed_val = ast.literal_eval(val)
|
|
values = string_utils.flatten_to_strings(parsed_val)
|
|
# NEW: Check if it's already a list object (JSON-parsed)
|
|
elif isinstance(val, list):
|
|
values = string_utils.flatten_to_strings(val)
|
|
# Legacy string parsing
|
|
elif "," in val:
|
|
values = [v.strip() for v in val.split(",")]
|
|
elif "|" in val:
|
|
values = [v.strip() for v in val.split("|")]
|
|
else:
|
|
values = [val.strip()]
|
|
except (ValueError, SyntaxError):
|
|
# If parsing fails, treat val as a regular string
|
|
values = [val.strip()]
|
|
|
|
# ... (rest unchanged)
|
|
```
|
|
|
|
---
|
|
|
|
#### 5.4.3 Modify: `src/pipelines/shared/extraction/dynamic_funcs.py`
|
|
|
|
**Change Type**: LOGIC UPDATE
|
|
|
|
**Location**: Lines 57-63
|
|
|
|
**Changes**:
|
|
```python
|
|
def dynamic_primary(
|
|
# ... parameters
|
|
):
|
|
# ... (existing code)
|
|
|
|
if not string_utils.is_empty(exhibit_text_answer):
|
|
# Special handling for LOB
|
|
if field.field_name == "LOB":
|
|
# OLD: Parse pipe-delimited or comma-separated
|
|
# values_list = [
|
|
# val.strip()
|
|
# for val in exhibit_text_answer.replace(",", "|").split("|")
|
|
# if val.strip()
|
|
# ]
|
|
|
|
# NEW: Handle both JSON list and pipe-delimited
|
|
if isinstance(exhibit_text_answer, list):
|
|
values_list = [str(val).strip() for val in exhibit_text_answer if val]
|
|
elif isinstance(exhibit_text_answer, str):
|
|
# Legacy: pipe or comma-delimited
|
|
values_list = [
|
|
val.strip()
|
|
for val in exhibit_text_answer.replace(",", "|").split("|")
|
|
if val.strip()
|
|
]
|
|
else:
|
|
values_list = [str(exhibit_text_answer)]
|
|
|
|
values_lower = [val.lower() for val in values_list]
|
|
|
|
# Check if both Medicare and Medicaid are present
|
|
has_medicare = any("medicare" in val for val in values_lower)
|
|
has_medicaid = any("medicaid" in val for val in values_lower)
|
|
|
|
if has_medicare and has_medicaid:
|
|
if "duals" not in values_lower:
|
|
# OLD: exhibit_text_answer = exhibit_text_answer + " | Duals"
|
|
# NEW: Add to list or append with pipe
|
|
if isinstance(exhibit_text_answer, list):
|
|
exhibit_text_answer.append("Duals")
|
|
else:
|
|
exhibit_text_answer = exhibit_text_answer + " | Duals"
|
|
|
|
logging.debug(
|
|
f"Added 'Duals' to LOB valid values because both Medicare and Medicaid were detected"
|
|
)
|
|
|
|
# Update valid values (convert list to pipe-delimited for Field.update_valid_values)
|
|
if isinstance(exhibit_text_answer, list):
|
|
valid_values_str = "|".join(str(v) for v in exhibit_text_answer) + " | N/A"
|
|
else:
|
|
valid_values_str = str(exhibit_text_answer) + " | N/A"
|
|
|
|
field.update_valid_values(valid_values_str)
|
|
dynamic_reimbursement_fields.add_field(field)
|
|
dynamic_primary_fields.remove_field(field.field_name)
|
|
|
|
# ... (rest unchanged)
|
|
```
|
|
|
|
---
|
|
|
|
#### 5.4.4 Modify: `src/pipelines/shared/postprocessing/postprocessing_funcs.py`
|
|
|
|
**Change Type**: REVIEW & UPDATE
|
|
|
|
**Action**: Search for all `.split("|")` occurrences and apply similar dual-format handling
|
|
|
|
---
|
|
|
|
#### 5.4.5 Modify: Other Files
|
|
|
|
**Files to update** (apply similar dual-format parsing):
|
|
- `src/testbed/dynamic_primary_metrics.py`
|
|
- `src/testbed/testbed_utils.py`
|
|
- `src/codes/code_funcs.py`
|
|
- `src/qc_qa/pipeline.py`
|
|
- `src/utils/qa_qc_utils.py`
|
|
- `src/parent_child/qc.py`
|
|
- `src/pipelines/shared/extraction/tin_npi_funcs.py`
|
|
- `src/testbed/model_evaluation_utils.py`
|
|
|
|
**Pattern**: Add helper function to each file or use shared utility:
|
|
|
|
```python
|
|
def _parse_multi_value_field(value):
|
|
"""Parse field that may be list or pipe-delimited string."""
|
|
if isinstance(value, list):
|
|
return value
|
|
if isinstance(value, str) and "|" in value:
|
|
return value.split("|")
|
|
return [value] if value else []
|
|
```
|
|
|
|
---
|
|
|
|
### 5.5 Other Files
|
|
|
|
#### 5.5.1 Modify: `src/codes/code_funcs.py`
|
|
|
|
**Change Type**: UPDATE `universal_json_load` USAGE
|
|
|
|
**Action**: Replace `string_utils.universal_json_load()` with `json_parsers.parse_json_dict()` or `parse_json_list()`
|
|
|
|
---
|
|
|
|
#### 5.5.2 Modify: `src/pipelines/shared/extraction/tin_npi_funcs.py`
|
|
|
|
**Change Type**: UPDATE `universal_json_load` USAGE
|
|
|
|
**Action**: Replace with appropriate parser
|
|
|
|
---
|
|
|
|
#### 5.5.3 Modify: `src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py`
|
|
|
|
**Change Type**: UPDATE `universal_json_load` USAGE
|
|
|
|
**Action**: Replace with `json_parsers.parse_json_dict()`
|
|
|
|
---
|
|
|
|
### 5.6 Test Files
|
|
|
|
#### 5.6.1 Create: `src/tests/test_json_parsers.py`
|
|
|
|
**Change Type**: NEW FILE
|
|
|
|
**Content**:
|
|
```python
|
|
"""
|
|
Unit tests for JSON parsing utilities.
|
|
"""
|
|
|
|
import pytest
|
|
from src.utils.json_parsers import (
|
|
parse_json_dict,
|
|
parse_json_list,
|
|
JSONParseError,
|
|
)
|
|
|
|
|
|
class TestParseJsonDict:
|
|
"""Tests for parse_json_dict function."""
|
|
|
|
def test_simple_dict(self):
|
|
"""Test parsing simple dictionary."""
|
|
raw = '{"field": "value", "count": 5}'
|
|
result = parse_json_dict(raw)
|
|
assert result == {"field": "value", "count": 5}
|
|
|
|
def test_dict_with_reasoning(self):
|
|
"""Test parsing dictionary with preceding reasoning text."""
|
|
raw = """
|
|
My reasoning for this answer is based on the contract text...
|
|
|
|
{"LOB": ["Medicare", "Medicaid"], "NETWORK": "N/A"}
|
|
"""
|
|
result = parse_json_dict(raw)
|
|
assert result == {"LOB": ["Medicare", "Medicaid"], "NETWORK": "N/A"}
|
|
|
|
def test_nested_dict(self):
|
|
"""Test parsing nested dictionary."""
|
|
raw = '{"outer": {"inner": "value"}, "list": [1, 2, 3]}'
|
|
result = parse_json_dict(raw)
|
|
assert result == {"outer": {"inner": "value"}, "list": [1, 2, 3]}
|
|
|
|
def test_multiple_dicts_returns_last(self):
|
|
"""Test that when multiple dicts present, last one is returned."""
|
|
raw = '{"first": 1} some text {"second": 2}'
|
|
result = parse_json_dict(raw)
|
|
assert result == {"second": 2}
|
|
|
|
def test_no_dict_raises_error(self):
|
|
"""Test that missing dict raises JSONParseError."""
|
|
raw = "No JSON here"
|
|
with pytest.raises(JSONParseError):
|
|
parse_json_dict(raw)
|
|
|
|
def test_list_instead_of_dict_raises_error(self):
|
|
"""Test that list input raises JSONParseError."""
|
|
raw = '["item1", "item2"]'
|
|
with pytest.raises(JSONParseError):
|
|
parse_json_dict(raw)
|
|
|
|
def test_malformed_json_raises_error(self):
|
|
"""Test that malformed JSON raises JSONParseError."""
|
|
raw = '{"incomplete": '
|
|
with pytest.raises(JSONParseError):
|
|
parse_json_dict(raw)
|
|
|
|
def test_dict_with_trailing_text(self):
|
|
"""Test parsing dict with trailing text."""
|
|
raw = '{"field": "value"} and some trailing text'
|
|
result = parse_json_dict(raw)
|
|
assert result == {"field": "value"}
|
|
|
|
|
|
class TestParseJsonList:
|
|
"""Tests for parse_json_list function."""
|
|
|
|
def test_simple_list(self):
|
|
"""Test parsing simple list."""
|
|
raw = '["item1", "item2", "item3"]'
|
|
result = parse_json_list(raw)
|
|
assert result == ["item1", "item2", "item3"]
|
|
|
|
def test_list_with_reasoning(self):
|
|
"""Test parsing list with preceding reasoning text."""
|
|
raw = """
|
|
The values found are:
|
|
|
|
["Medicare", "Medicaid", "Duals"]
|
|
"""
|
|
result = parse_json_list(raw)
|
|
assert result == ["Medicare", "Medicaid", "Duals"]
|
|
|
|
def test_empty_list(self):
|
|
"""Test parsing empty list."""
|
|
raw = "[]"
|
|
result = parse_json_list(raw)
|
|
assert result == []
|
|
|
|
def test_list_of_dicts(self):
|
|
"""Test parsing list of dictionaries."""
|
|
raw = '[{"name": "A", "value": 1}, {"name": "B", "value": 2}]'
|
|
result = parse_json_list(raw)
|
|
assert result == [{"name": "A", "value": 1}, {"name": "B", "value": 2}]
|
|
|
|
def test_multiple_lists_returns_last(self):
|
|
"""Test that when multiple lists present, last one is returned."""
|
|
raw = '["first"] some text ["second", "list"]'
|
|
result = parse_json_list(raw)
|
|
assert result == ["second", "list"]
|
|
|
|
def test_no_list_raises_error(self):
|
|
"""Test that missing list raises JSONParseError."""
|
|
raw = "No JSON here"
|
|
with pytest.raises(JSONParseError):
|
|
parse_json_list(raw)
|
|
|
|
def test_dict_instead_of_list_raises_error(self):
|
|
"""Test that dict input raises JSONParseError."""
|
|
raw = '{"field": "value"}'
|
|
with pytest.raises(JSONParseError):
|
|
parse_json_list(raw)
|
|
|
|
|
|
class TestBackwardCompatibility:
|
|
"""Tests for deprecated universal_json_load."""
|
|
|
|
def test_universal_json_load_still_works(self):
|
|
"""Test that deprecated function still works."""
|
|
from src.utils.json_parsers import universal_json_load
|
|
|
|
raw = '{"field": "value"}'
|
|
with pytest.warns(DeprecationWarning):
|
|
result = universal_json_load(raw)
|
|
assert result == {"field": "value"}
|
|
|
|
|
|
class TestEdgeCases:
|
|
"""Tests for edge cases and special scenarios."""
|
|
|
|
def test_json_with_escaped_quotes(self):
|
|
"""Test parsing JSON with escaped quotes in strings."""
|
|
raw = '{"text": "He said \\"hello\\""}'
|
|
result = parse_json_dict(raw)
|
|
assert result == {"text": 'He said "hello"'}
|
|
|
|
def test_json_with_newlines(self):
|
|
"""Test parsing JSON with newlines in strings."""
|
|
raw = '{"multiline": "line1\\nline2\\nline3"}'
|
|
result = parse_json_dict(raw)
|
|
assert "line1\nline2\nline3" in str(result.values())
|
|
|
|
def test_non_string_input_raises_error(self):
|
|
"""Test that non-string input raises TypeError."""
|
|
with pytest.raises(TypeError):
|
|
parse_json_dict(123)
|
|
|
|
with pytest.raises(TypeError):
|
|
parse_json_list(None)
|
|
|
|
def test_unicode_handling(self):
|
|
"""Test parsing JSON with unicode characters."""
|
|
raw = '{"text": "café"}'
|
|
result = parse_json_dict(raw)
|
|
assert result == {"text": "café"}
|
|
```
|
|
|
|
---
|
|
|
|
#### 5.6.2 Update: `src/tests/test_string_utils.py`
|
|
|
|
**Change Type**: ADD DEPRECATION TESTS
|
|
|
|
**Add**:
|
|
```python
|
|
def test_universal_json_load_deprecated():
|
|
"""Test that universal_json_load shows deprecation warning."""
|
|
with pytest.warns(DeprecationWarning):
|
|
result = string_utils.universal_json_load('{"test": "value"}')
|
|
assert result == {"test": "value"}
|
|
|
|
|
|
def test_extract_text_from_delimiters_deprecated():
|
|
"""Test that extract_text_from_delimiters shows deprecation warning."""
|
|
from src.constants.delimiters import Delimiter
|
|
|
|
with pytest.warns(DeprecationWarning):
|
|
result = string_utils.extract_text_from_delimiters(
|
|
"text |value| more text", Delimiter.PIPE
|
|
)
|
|
assert result == "value"
|
|
```
|
|
|
|
---
|
|
|
|
#### 5.6.3 Update: Existing Test Files
|
|
|
|
**Files to update**:
|
|
- `src/tests/test_one_to_n.py`
|
|
- `src/tests/test_code_funcs.py`
|
|
- `src/tests/test_dynamic.py`
|
|
|
|
**Action**: Update any tests that check for pipe-delimited outputs to check for JSON list/dict outputs instead
|
|
|
|
---
|
|
|
|
## 6. Migration Plan
|
|
|
|
### 6.0 Lessons from Previous Attempt
|
|
|
|
The previous implementation (commit faf7072) was reverted. This PRD incorporates lessons learned:
|
|
|
|
**Previous Issues → Current Mitigations**:
|
|
|
|
| Issue | Mitigation in This PRD |
|
|
|-------|----------------------|
|
|
| Big-bang deployment caused failures | Phased rollout (6 phases over 5-6 weeks) |
|
|
| No backward compatibility | Dual-format support in all downstream consumers |
|
|
| Unclear format specifications | Comprehensive prompt-output mapping table (4.2.2) |
|
|
| Insufficient testing | Extensive test plan including golden tests (Section 7) |
|
|
| LLM responses didn't match expectations | Clear format instructions with examples in prompts |
|
|
| Hard to rollback | Explicit rollback plan with safe revert points (6.3) |
|
|
| Ambiguous prompt instructions | Four standardized format instruction variables (4.1.1) |
|
|
|
|
**Key Improvements**:
|
|
1. **Format Specification**: Four distinct format types (`JSON_DICT`, `JSON_LIST`, `JSON_DICT_WITH_LISTS`, `JSON_LIST_OF_DICTS`) with clear usage guidelines
|
|
2. **Caching Optimization**: Format instructions in cached `_INSTRUCTION()` functions, not in dynamic prompts (saves tokens)
|
|
3. **Backward Compatibility**: All pipe-split consumers support both formats during transition
|
|
4. **Testing Strategy**: Golden tests compare semantic equivalence (not byte-for-byte)
|
|
5. **Rollback Safety**: Can revert at any phase without breaking production
|
|
6. **Documentation**: Comprehensive prompt mapping table eliminates ambiguity
|
|
|
|
---
|
|
|
|
### 6.1 Migration Strategy
|
|
|
|
**Approach**: **Phased rollout with backward compatibility**
|
|
|
|
#### Phase 1: Foundation (Week 1)
|
|
1. Create `src/utils/json_parsers.py` with new parsers
|
|
2. Add deprecation warnings to `universal_json_load` and `extract_text_from_delimiters`
|
|
3. Write unit tests for new parsers
|
|
4. Document migration guide
|
|
|
|
#### Phase 2: Prompt Templates (Week 2)
|
|
1. Update all prompt instruction functions to use JSON output format
|
|
2. Update examples in prompts to show JSON structure
|
|
3. Test prompts with updated instructions (manual QA)
|
|
|
|
#### Phase 3: Prompt Calling Layer (Week 2-3)
|
|
1. Update `src/pipelines/saas/prompts/prompt_calls.py`
|
|
- Replace `universal_json_load` → `parse_json_dict` / `parse_json_list`
|
|
- Replace `extract_text_from_delimiters` → `parse_json_dict` / `parse_json_list`
|
|
- Keep pipe-to-JSON conversion in `prompt_dynamic_primary()` temporarily
|
|
2. Update client-specific prompt_calls files (clover, bcbs_promise)
|
|
3. Update other files using `universal_json_load`
|
|
|
|
#### Phase 4: Downstream Consumers (Week 3-4)
|
|
1. Update `aarete_derived.py` to handle both JSON lists and pipe-delimited
|
|
2. Update `crosswalk_utils.py` to handle both formats
|
|
3. Update `dynamic_funcs.py` to handle both formats
|
|
4. Add `_parse_multi_value_field()` helper to shared utilities
|
|
5. Update all other files with pipe-split operations
|
|
|
|
#### Phase 5: Testing & Validation (Week 4-5)
|
|
1. Run full integration tests
|
|
2. Run on sample contracts and compare outputs
|
|
3. Golden test validation (ensure outputs match expected format)
|
|
4. Performance testing
|
|
|
|
#### Phase 6: Cleanup (Week 5-6)
|
|
1. Remove backward compatibility code from Phase 4 (dual-format handling)
|
|
2. Finalize deprecation of `universal_json_load` (plan removal date)
|
|
3. Update documentation
|
|
4. Code review and final QA
|
|
|
|
### 6.2 Backward Compatibility
|
|
|
|
#### 6.2.1 During Migration
|
|
|
|
**Dual-Format Support**: All downstream consumers will temporarily support both formats:
|
|
- JSON lists: `["value1", "value2"]`
|
|
- Pipe-delimited strings: `"value1|value2"`
|
|
|
|
**Example**:
|
|
```python
|
|
def _parse_multi_value_field(value):
|
|
"""Handles both JSON list and pipe-delimited during migration."""
|
|
if isinstance(value, list):
|
|
return value # New format
|
|
if isinstance(value, str) and "|" in value:
|
|
return value.split("|") # Legacy format
|
|
return [value] if value else []
|
|
```
|
|
|
|
#### 6.2.2 After Migration
|
|
|
|
**Deprecation Timeline**:
|
|
- **Immediate**: Add deprecation warnings
|
|
- **+3 months**: Remove dual-format support
|
|
- **+6 months**: Remove `universal_json_load` and `extract_text_from_delimiters`
|
|
|
|
### 6.3 Rollback Plan
|
|
|
|
**If issues arise**:
|
|
|
|
1. **Prompt level**: Revert prompt instruction changes (restore pipe format instructions)
|
|
2. **Parser level**: Keep using `universal_json_load` (backward compatible)
|
|
3. **Downstream level**: Dual-format support allows partial rollback
|
|
|
|
**Rollback is safe because**:
|
|
- No breaking changes to external APIs
|
|
- No database schema changes
|
|
- Dual-format support in downstream code
|
|
|
|
---
|
|
|
|
## 7. Testing Plan
|
|
|
|
### 7.1 Unit Tests
|
|
|
|
#### 7.1.1 Parser Unit Tests
|
|
|
|
**File**: `src/tests/test_json_parsers.py`
|
|
|
|
**Coverage**:
|
|
- ✅ Parse simple dict
|
|
- ✅ Parse dict with reasoning text
|
|
- ✅ Parse nested dict
|
|
- ✅ Parse simple list
|
|
- ✅ Parse list with reasoning text
|
|
- ✅ Parse empty list
|
|
- ✅ Parse list of dicts
|
|
- ✅ Error handling: No JSON found
|
|
- ✅ Error handling: Wrong type (dict vs list)
|
|
- ✅ Error handling: Malformed JSON
|
|
- ✅ Edge cases: Escaped quotes, newlines, unicode
|
|
|
|
#### 7.1.2 Downstream Consumer Tests
|
|
|
|
**Files to add tests to**:
|
|
- `src/tests/test_aarete_derived.py` (new or update existing)
|
|
- `src/tests/test_crosswalk_utils.py` (new or update existing)
|
|
- `src/tests/test_dynamic_funcs.py` (update existing)
|
|
|
|
**Test cases**:
|
|
- Parsing JSON list inputs
|
|
- Parsing pipe-delimited inputs (backward compatibility)
|
|
- Mixed scenarios
|
|
- Empty values
|
|
- Edge cases (N/A handling, etc.)
|
|
|
|
### 7.2 Integration Tests
|
|
|
|
#### 7.2.1 Prompt-to-Parser Integration
|
|
|
|
**Test**: End-to-end prompt execution with new format
|
|
|
|
**Files**: `src/tests/test_integration_prompts.py` (new)
|
|
|
|
**Test cases**:
|
|
```python
|
|
def test_exhibit_level_json_output():
|
|
"""Test that exhibit level prompt returns valid JSON dict."""
|
|
# Mock LLM response with JSON dict
|
|
# Call prompt_exhibit_level()
|
|
# Assert result is dict with expected fields
|
|
|
|
|
|
def test_reimbursement_primary_json_output():
|
|
"""Test that reimbursement primary prompt returns valid JSON list."""
|
|
# Mock LLM response with JSON list
|
|
# Call prompt_reimbursement_primary()
|
|
# Assert result is list of dicts
|
|
|
|
|
|
def test_dynamic_primary_json_output():
|
|
"""Test that dynamic primary prompt returns valid JSON list."""
|
|
# Mock LLM response with JSON list
|
|
# Call prompt_dynamic_primary()
|
|
# Assert result is correctly parsed
|
|
```
|
|
|
|
#### 7.2.2 Full Pipeline Test
|
|
|
|
**Test**: Run complete extraction pipeline on sample contract
|
|
|
|
**File**: `src/tests/test_end_to_end.py`
|
|
|
|
**Test case**:
|
|
```python
|
|
def test_full_pipeline_with_json_parsers():
|
|
"""
|
|
Test complete extraction pipeline with JSON parsers.
|
|
Ensures all components work together.
|
|
"""
|
|
# Load sample contract
|
|
# Run full extraction (1:N and 1:1)
|
|
# Validate output structure
|
|
# Compare to golden output (allow for equivalent formats)
|
|
```
|
|
|
|
### 7.3 Golden Tests
|
|
|
|
#### 7.3.1 Output Format Validation
|
|
|
|
**Approach**: Compare outputs before and after migration
|
|
|
|
**Process**:
|
|
1. Run extraction on test contracts with OLD system (pipe-delimited)
|
|
2. Store outputs as "golden" files
|
|
3. Run extraction on same contracts with NEW system (JSON)
|
|
4. Compare outputs semantically (not byte-for-byte)
|
|
- Allow `["A", "B"]` to match `"A|B"`
|
|
- Allow different whitespace/formatting
|
|
- Ensure same fields and values extracted
|
|
|
|
**Tool**: Create comparison script `scripts/compare_outputs.py`
|
|
|
|
### 7.4 LLM Prompt Testing
|
|
|
|
#### 7.4.1 Manual QA
|
|
|
|
**Process**:
|
|
1. Test new prompt instructions with real LLM
|
|
2. Verify JSON format compliance
|
|
3. Check for any formatting errors
|
|
4. Validate multi-value fields use arrays
|
|
|
|
**Sample prompts to test**:
|
|
- EXHIBIT_LEVEL
|
|
- DYNAMIC_PRIMARY_TEXT
|
|
- CARVEOUT_CHECK
|
|
- REIMBURSEMENT_PRIMARY
|
|
- All breakout prompts
|
|
|
|
#### 7.4.2 Automated Prompt Validation
|
|
|
|
**Script**: `scripts/validate_prompts.py`
|
|
|
|
**Checks**:
|
|
- All instruction functions updated
|
|
- No lingering pipe format instructions
|
|
- All examples use JSON format
|
|
- Multi-value instructions mention arrays
|
|
|
|
### 7.5 Performance Testing
|
|
|
|
#### 7.5.1 Parsing Performance
|
|
|
|
**Test**: Compare parsing speed
|
|
|
|
```python
|
|
def test_parsing_performance():
|
|
"""Compare universal_json_load vs new parsers."""
|
|
sample_json = '{"field": "value", ...}' # Large JSON
|
|
|
|
# Time old parser
|
|
# Time new parser
|
|
# Assert new parser is comparable or faster
|
|
```
|
|
|
|
#### 7.5.2 End-to-End Performance
|
|
|
|
**Test**: Measure total pipeline time with new parsers
|
|
|
|
**Expectation**: No significant performance degradation (<5% slower acceptable)
|
|
|
|
---
|
|
|
|
## 8. Risks and Mitigations
|
|
|
|
### 8.1 Risk: Malformed JSON from LLM
|
|
|
|
**Description**: LLM may sometimes return malformed JSON or JSON with trailing text
|
|
|
|
**Likelihood**: Medium
|
|
**Impact**: High (pipeline failure)
|
|
|
|
**Mitigation**:
|
|
1. **Robust parser**: `_extract_json_objects()` handles text before/after JSON
|
|
2. **Error handling**: All parser calls wrapped in try/except
|
|
3. **Fallback values**: Return empty dict/list on parse failure
|
|
4. **Logging**: Log all parse failures for analysis
|
|
5. **Prompt clarity**: Clear instructions and examples reduce formatting errors
|
|
|
|
**Monitoring**:
|
|
- Track parse failure rate
|
|
- Alert if >5% of prompts fail to parse
|
|
- Review and improve prompts with high failure rates
|
|
|
|
---
|
|
|
|
### 8.2 Risk: Partial Outputs
|
|
|
|
**Description**: LLM may return incomplete JSON (e.g., context length limits)
|
|
|
|
**Likelihood**: Low
|
|
**Impact**: Medium (missing data)
|
|
|
|
**Mitigation**:
|
|
1. **Parser handles incomplete JSON**: Returns what's parseable
|
|
2. **Validation**: Check for required fields after parsing
|
|
3. **Token limits**: Ensure prompts stay well under limits
|
|
4. **Retry logic**: Retry with shorter context if parsing fails
|
|
|
|
---
|
|
|
|
### 8.3 Risk: Model Verbosity Around JSON
|
|
|
|
**Description**: LLM may add extra text or formatting around JSON
|
|
|
|
**Likelihood**: Medium
|
|
**Impact**: Low (parser handles it)
|
|
|
|
**Mitigation**:
|
|
1. **Parser designed for this**: `_extract_json_objects()` finds JSON anywhere in text
|
|
2. **Clear instructions**: Prompts emphasize clean JSON output
|
|
3. **Examples**: Show clean JSON without extra formatting
|
|
|
|
---
|
|
|
|
### 8.4 Risk: Parsing Edge Cases
|
|
|
|
**Description**: Special characters, nested structures, etc. may cause issues
|
|
|
|
**Likelihood**: Medium
|
|
**Impact**: Medium
|
|
|
|
**Mitigation**:
|
|
1. **Comprehensive unit tests**: Cover edge cases
|
|
2. **String escaping**: JSON parser handles escaping correctly
|
|
3. **Validation**: Post-parse validation catches issues
|
|
4. **Error logging**: Track and fix edge cases as discovered
|
|
|
|
---
|
|
|
|
### 8.5 Risk: Backward Compatibility Breakage
|
|
|
|
**Description**: Changes may break existing code or data pipelines
|
|
|
|
**Likelihood**: Low
|
|
**Impact**: High
|
|
|
|
**Mitigation**:
|
|
1. **Dual-format support**: Downstream code handles both formats during transition
|
|
2. **Deprecation warnings**: Clear warnings before removal
|
|
3. **Phased rollout**: Test each phase before proceeding
|
|
4. **Rollback plan**: Can revert to old system if needed
|
|
|
|
---
|
|
|
|
### 8.6 Risk: Inconsistent Multi-Value Formats
|
|
|
|
**Description**: Some fields may still use pipes internally after parsing
|
|
|
|
**Likelihood**: Medium
|
|
**Impact**: Medium (confusion, bugs)
|
|
|
|
**Mitigation**:
|
|
1. **Clear convention**: Document when to use lists vs pipe-delimited
|
|
2. **Helper functions**: Provide `_parse_multi_value_field()` utility
|
|
3. **Code review**: Ensure consistency across codebase
|
|
4. **Future refactor**: Plan to eliminate internal pipe usage (out of scope)
|
|
|
|
---
|
|
|
|
### 8.7 Risk: Performance Degradation
|
|
|
|
**Description**: New parsing may be slower than old regex-based approach
|
|
|
|
**Likelihood**: Low
|
|
**Impact**: Low
|
|
|
|
**Mitigation**:
|
|
1. **Performance tests**: Benchmark before and after
|
|
2. **Optimize if needed**: Profile and optimize hot paths
|
|
3. **Expectation**: JSON parsing is well-optimized in Python
|
|
|
|
---
|
|
|
|
## 9. Work Items Checklist
|
|
|
|
### 9.1 Implementation Tasks
|
|
|
|
#### Phase 1: Foundation ✅ **COMPLETED**
|
|
- [x] Create `src/utils/json_parsers.py` → **Renamed to `json_utils.py`**
|
|
- [x] Implement `parse_json_dict()`
|
|
- [x] Implement `parse_json_list()`
|
|
- [x] Implement `parse_json_dict_or_list()` helper
|
|
- [x] Add `universal_json_load()` deprecated alias (in string_utils.py)
|
|
- [x] Update `src/utils/string_utils.py`
|
|
- [x] Add deprecation warning to `universal_json_load()`
|
|
- [x] Add deprecation warning to `extract_text_from_delimiters()`
|
|
- [x] Create `src/tests/test_json_utils.py` (formerly `test_json_parsers.py`)
|
|
- [x] Write 20+ unit tests for parsers
|
|
- [x] Test edge cases
|
|
- [x] Test error handling
|
|
- [x] **ARCHITECTURE ENHANCEMENT**: Update all prompt templates to return `(prompt, parser)` tuples
|
|
- [x] Add `_json_dict_parser()` and `_json_list_parser()` helper functions
|
|
- [x] Update 35 prompt template functions with tuple returns
|
|
- [x] Add type hints: `-> Tuple[str, Callable[[str], dict/list]]`
|
|
- [x] Update docstrings with Returns section
|
|
|
|
#### Phase 2: Prompt Templates ✅ **COMPLETED**
|
|
- [x] Update `src/prompts/prompt_templates.py`
|
|
- [x] Add four JSON format instruction constants (after `PIPE_FORMAT_INSTRUCTIONS`)
|
|
- [x] Add `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
- [x] Add `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- [x] Add `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- [x] Add `JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS`
|
|
- [x] Deprecate `PIPE_FORMAT_INSTRUCTIONS` (marked as legacy)
|
|
- [x] Update all `_INSTRUCTION()` functions (cached) to include format instructions:
|
|
- [x] `EXHIBIT_LEVEL_INSTRUCTION()` + `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- [x] `DYNAMIC_PRIMARY_TEXT_INSTRUCTION()` + `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- [x] `CARVEOUT_CHECK_INSTRUCTION()` + `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- [x] `REIMBURSEMENT_PRIMARY_INSTRUCTION()` (already JSON)
|
|
- [x] `METHODOLOGY_BREAKOUT_INSTRUCTION()` (already JSON)
|
|
- [x] `FEE_SCHEDULE_BREAKOUT_INSTRUCTION()` (already JSON)
|
|
- [x] `GROUPER_BREAKOUT_INSTRUCTION()` (already JSON)
|
|
- [x] `DYNAMIC_ASSIGNMENT_INSTRUCTION()` + `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- [x] `REIMB_DATES_ASSIGNMENT_INSTRUCTION()` + `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
- [x] `LESSER_OF_DISTRIBUTION_INSTRUCTION()` + `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- [x] `LESSER_OF_CHECK_INSTRUCTION()` + `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- [x] `LOB_RELATIONSHIP_INSTRUCTION()` + `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- [x] `ONE_TO_ONE_SINGLE_FIELD_INSTRUCTION()` (already uses dict)
|
|
- [x] `ONE_TO_ONE_MULTI_FIELD_INSTRUCTION()` + `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
- [x] `TIN_NPI_TEMPLATE_INSTRUCTION()` (already uses list)
|
|
- [x] `EXHIBIT_LINKAGE_INSTRUCTION()` + `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- [x] `VALIDATE_REIMBURSEMENTS_INSTRUCTION()` + `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- [x] `EXHIBIT_TITLE_MATCH_INSTRUCTION()` + `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
- [x] All special case breakout `_INSTRUCTION()` functions (already JSON)
|
|
- [x] Verified dynamic prompt functions remain clean (NO format instructions added)
|
|
- [x] **All 35 prompt template functions now return `(prompt, parser)` tuples**
|
|
- [x] Update `src/prompts/investment_prompts.json` ✅ **COMPLETE**
|
|
- [x] `EFFECTIVE_DT` field: Replaced pipe format examples and "ENCLOSED in |pipes|" instruction with JSON dictionary format
|
|
- [x] `PROVIDER_NAME` field: Replaced "separated by ' | '" instruction with JSON list format for multiple values
|
|
- [x] All pipe references removed from field prompts
|
|
- [ ] Manual QA testing
|
|
- [ ] Test each updated prompt with real LLM
|
|
- [ ] Verify JSON format compliance
|
|
- [ ] Fix any formatting issues
|
|
|
|
**Commits**:
|
|
- Phase 1: Multiple commits (json_utils.py, test files, deprecation warnings)
|
|
- Phase 2 Templates: Commit 5d1024e (by user)
|
|
- Phase 2 Architecture: Commit 98554afc (tuple returns)
|
|
|
|
#### Phase 3: Prompt Calling Layer ✅ **MOSTLY COMPLETE** (95%)
|
|
- [x] Update `src/pipelines/saas/prompts/prompt_calls.py` ✅ **COMPLETE** (Commit: dba61146)
|
|
- [x] Update `prompt_exhibit_level()` - Uses tuple unpacking
|
|
- [x] Update `prompt_exhibit_level_breakout()` - Uses tuple unpacking
|
|
- [x] Update `prompt_dynamic_primary()` - Uses tuple unpacking
|
|
- [x] Update `prompt_reimbursement_primary()` - Uses tuple unpacking
|
|
- [x] Update `prompt_methodology_breakout()` - Uses tuple unpacking
|
|
- [x] Update `prompt_fee_schedule_breakout()` - Uses tuple unpacking
|
|
- [x] Update `prompt_grouper_breakout()` - Uses tuple unpacking
|
|
- [x] Update `prompt_carveout_check()` - Uses tuple unpacking
|
|
- [x] Update `prompt_special_case_breakout()` - Uses tuple unpacking
|
|
- [x] Update `prompt_full_context()` - Uses tuple unpacking
|
|
- [x] Update `prompt_dynamic()` - Uses tuple unpacking
|
|
- [x] Update `prompt_dynamic_field_safe()` - Uses tuple unpacking
|
|
- [x] All 24+ functions updated to use `prompt, parser = template_func(...)` pattern
|
|
- [x] Update `src/pipelines/clients/clover/prompts/prompt_calls.py` ✅ **COMPLETE** (Commit: dba61146)
|
|
- [x] All functions updated to use tuple unpacking pattern (25 instances)
|
|
- [x] Update `src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py` ✅ **COMPLETE** (Commit: dba61146)
|
|
- [x] All functions updated to use tuple unpacking pattern (25 instances)
|
|
- [x] Update `src/codes/code_funcs.py` ✅ **COMPLETE** (Commit: a7a32b90)
|
|
- [x] Most functions updated (Commit: 6c3f3f6f)
|
|
- [x] Line 498: Changed to use `_parser` from FILL_BILL_TYPE tuple
|
|
- [x] Line 856: Changed to use `grouper_parser` from GROUPER_BREAKOUT tuple
|
|
- [x] Removed unused `GROUPER_QUESTIONS` variable
|
|
- [x] Update `src/pipelines/shared/extraction/tin_npi_funcs.py` ✅ **COMPLETE** (Commit: 0d9f9314)
|
|
- [x] All `universal_json_load` calls replaced
|
|
- [x] Update `src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py` ✅ **COMPLETE** (Commit: dba61146)
|
|
- [x] All `universal_json_load` calls replaced
|
|
|
|
#### Phase 4: Downstream Consumers ✅ **COMPLETE** (100%)
|
|
- [x] Create shared utility function (if needed)
|
|
- [x] Dual-format support implemented inline where needed
|
|
- [x] Update `src/pipelines/shared/postprocessing/aarete_derived.py` ✅ **COMPLETE** (Dual-format support)
|
|
- [x] Line 42-43: LOB parsing with dual-format support (checks for "|" before splitting)
|
|
- [x] Line 65-66: PROGRAM parsing with dual-format support
|
|
- [x] Line 89-90: PRODUCT parsing with dual-format support
|
|
- [x] All pipe-split operations have dual-format support (handles both JSON lists and pipe-delimited)
|
|
- **Note**: This file intentionally maintains backward compatibility by checking for "|" before splitting
|
|
- [x] Update `src/utils/crosswalk_utils.py` ✅ **COMPLETE** (Commit: 8399dbdf)
|
|
- [x] `apply_crosswalk()` updated to handle JSON lists
|
|
- [x] Update `src/pipelines/shared/extraction/dynamic_funcs.py` ✅ **COMPLETE** (Commit: dba61146)
|
|
- [x] `dynamic_primary()` updated - no pipe splitting found
|
|
- [x] Handles JSON lists properly
|
|
- [x] Update `src/pipelines/shared/postprocessing/postprocessing_funcs.py` ✅ **COMPLETE** (Commit: df103579)
|
|
- [x] All pipe-split operations removed/updated
|
|
- [x] 113 lines changed, significant cleanup
|
|
- [x] Update remaining files with LLM output parsing ✅ **COMPLETE** (Commit: 13f3cbea)
|
|
- [x] `src/testbed/model_evaluation_utils.py` - Fixed DYNAMIC_PRIMARY_TEXT and REIMBURSEMENT_PRIMARY to use JSON parsers
|
|
- [x] Verified other testbed/QC files (`dynamic_primary_metrics.py`, `testbed_utils.py`, `qc_qa/pipeline.py`, `qa_qc_utils.py`, `parent_child/qc.py`) - These process internal data formats (out of scope per PRD non-goals)
|
|
- [ ] `src/utils/qa_qc_utils.py` - Status unknown
|
|
- [ ] `src/parent_child/qc.py` - Status unknown
|
|
- [ ] `src/testbed/model_evaluation_utils.py` - Status unknown (has universal_json_load import)
|
|
|
|
#### Phase 5: Testing & Validation
|
|
- [ ] Create integration test file
|
|
- [ ] `src/tests/test_integration_prompts.py`
|
|
- [ ] Test exhibit level integration
|
|
- [ ] Test reimbursement primary integration
|
|
- [ ] Test dynamic primary integration
|
|
- [ ] Update existing test files
|
|
- [ ] `src/tests/test_one_to_n.py`
|
|
- [ ] `src/tests/test_code_funcs.py`
|
|
- [ ] `src/tests/test_dynamic.py`
|
|
- [ ] Create golden test comparison
|
|
- [ ] Script: `scripts/compare_outputs.py`
|
|
- [ ] Run OLD system on test contracts
|
|
- [ ] Run NEW system on test contracts
|
|
- [ ] Compare outputs semantically
|
|
- [ ] Run full test suite
|
|
- [ ] Unit tests pass
|
|
- [ ] Integration tests pass
|
|
- [ ] Golden tests pass
|
|
- [ ] Manual testing
|
|
- [ ] Test on sample contracts
|
|
- [ ] Verify output quality
|
|
- [ ] Check for any regressions
|
|
|
|
#### Phase 6: Cleanup & Documentation
|
|
- [ ] Remove backward compatibility code
|
|
- [ ] Remove dual-format support from downstream consumers
|
|
- [ ] Direct JSON list handling only
|
|
- [ ] Update documentation
|
|
- [ ] Add migration guide to docs
|
|
- [ ] Update developer guide with new parsing patterns
|
|
- [ ] Document JSON output schemas
|
|
- [ ] Code review
|
|
- [ ] Review all changes
|
|
- [ ] Check for missed occurrences
|
|
- [ ] Verify consistency
|
|
- [ ] Final QA
|
|
- [ ] Run on production-like data
|
|
- [ ] Performance validation
|
|
- [ ] Error rate monitoring
|
|
|
|
### 9.2 Documentation Tasks
|
|
|
|
- [ ] Create migration guide (`docs/MIGRATION_GUIDE_JSON_PARSING.md`)
|
|
- [ ] Update developer onboarding docs
|
|
- [ ] Add JSON schema examples to prompt documentation
|
|
- [ ] Document new parser usage patterns
|
|
- [ ] Update architecture diagrams
|
|
- [ ] Create deprecation announcement
|
|
|
|
### 9.3 Monitoring & Rollout Tasks
|
|
|
|
- [ ] Set up monitoring for parse failures
|
|
- [ ] Create dashboard for JSON parsing metrics
|
|
- [ ] Define rollback procedures
|
|
- [ ] Create rollout checklist
|
|
- [ ] Schedule phased deployment
|
|
- [ ] Plan deprecation timeline communication
|
|
|
|
---
|
|
|
|
## Implementation Status Summary (Last Updated: Feb 3, 2026 - Commit: a633f67c)
|
|
|
|
### ✅ Completed Work
|
|
|
|
**Phase 1: Foundation** (100% Complete)
|
|
- ✅ Created `src/utils/json_utils.py` with `parse_json_dict()` and `parse_json_list()`
|
|
- ✅ Added deprecation warnings to `string_utils.py`
|
|
- ✅ Created comprehensive unit tests in `src/tests/test_json_utils.py`
|
|
|
|
**Phase 2: Prompt Templates** (100% Complete)
|
|
- ✅ All 35 prompt template functions return `(prompt, parser)` tuples
|
|
- ✅ All `_INSTRUCTION()` functions updated with JSON format instructions
|
|
- ✅ Removed all pipe-delimited format references from prompts
|
|
|
|
**Phase 3: Prompt Calling Layer** (95% Complete)
|
|
- ✅ `src/pipelines/saas/prompts/prompt_calls.py` - All 24+ functions updated
|
|
- ✅ `src/pipelines/clients/clover/prompts/prompt_calls.py` - All functions updated
|
|
- ✅ `src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py` - All functions updated
|
|
- ✅ `src/pipelines/shared/extraction/tin_npi_funcs.py` - All updated
|
|
- ✅ `src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py` - All updated
|
|
- ⚠️ `src/codes/code_funcs.py` - 2 remaining `universal_json_load` calls (lines 498, 856)
|
|
|
|
**Phase 4: Downstream Consumers** (100% Complete)
|
|
- ✅ `src/utils/crosswalk_utils.py` - Updated to handle JSON lists
|
|
- ✅ `src/pipelines/shared/extraction/dynamic_funcs.py` - Updated
|
|
- ✅ `src/pipelines/shared/postprocessing/postprocessing_funcs.py` - Major cleanup (113 lines changed)
|
|
- ✅ `src/pipelines/shared/postprocessing/aarete_derived.py` - Dual-format support implemented (intentional backward compatibility)
|
|
- ✅ `src/testbed/model_evaluation_utils.py` - Updated to use JSON parsers (Commit: 13f3cbea)
|
|
|
|
### ⏳ Remaining Work
|
|
|
|
**Phase 3: Prompt Calling Layer** ✅ **COMPLETE** (100%)
|
|
- [x] Fixed all `universal_json_load` calls in `src/codes/code_funcs.py` (Commit: a7a32b90)
|
|
|
|
**Phase 4: Downstream Consumers** ✅ **COMPLETE** (100%)
|
|
- [x] Fixed `src/testbed/model_evaluation_utils.py` - Updated LLM output parsing to use JSON parsers (Commit: 13f3cbea)
|
|
- [x] Verified remaining files - Process internal data formats (out of scope per PRD):
|
|
- `src/testbed/dynamic_primary_metrics.py` - Internal data processing
|
|
- `src/testbed/testbed_utils.py` - Internal data processing
|
|
- `src/qc_qa/pipeline.py` - Internal data processing
|
|
- `src/utils/qa_qc_utils.py` - Internal data processing
|
|
- `src/parent_child/qc.py` - Internal data processing
|
|
|
|
**Phase 5: Testing & Validation** (0% complete)
|
|
- [ ] Create integration tests
|
|
- [ ] Update existing test files
|
|
- [ ] Create golden test comparison
|
|
- [ ] Run full test suite
|
|
- [ ] Manual QA testing
|
|
|
|
**Phase 6: Cleanup & Documentation** (Partial - 25% complete)
|
|
- [x] Removed all debug print statements from `prompt_calls.py` and `one_to_n_funcs.py` (Commit: a633f67c)
|
|
- [x] Removed defensive normalization checks that are no longer needed (Commit: a633f67c)
|
|
- [ ] Remove backward compatibility code (if desired)
|
|
- [ ] Update documentation
|
|
- [ ] Code review
|
|
- [ ] Final QA
|
|
|
|
### 📊 Statistics
|
|
|
|
- **Files Modified**: 26 files across the codebase
|
|
- **Lines Changed**: ~855 insertions, ~1204 deletions (net reduction of ~350 lines)
|
|
- **Prompt Functions Updated**: 35 template functions + all calling code
|
|
- **Commits**: 20+ commits implementing the changes
|
|
- **Test Coverage**: Unit tests for JSON parsers complete
|
|
|
|
### 🎯 Key Achievements
|
|
|
|
1. **Architectural Improvement**: All prompt templates now return `(prompt, parser)` tuples, eliminating ambiguity
|
|
2. **Code Reduction**: Net reduction of ~400 lines through cleanup
|
|
3. **Consistency**: All prompt calling code uses the same pattern
|
|
4. **Backward Compatibility**: `aarete_derived.py` maintains dual-format support during transition
|
|
5. **Data Normalization**: SERVICE_TERM and REIMB_TERM normalized upstream to ensure consistent string format
|
|
6. **Bug Fixes**: Fixed REIMB_TERM list conversion in lesser_of_distribution, removed unnecessary defensive checks
|
|
|
|
---
|
|
|
|
## Appendix A: JSON Schema Examples
|
|
|
|
### A.1 Exhibit Level Response
|
|
|
|
```json
|
|
{
|
|
"LOB": ["Medicare", "Medicaid"],
|
|
"FACILITY_TYPE": "Inpatient",
|
|
"NETWORK": "N/A",
|
|
"EXHIBIT_NOTE": "This exhibit applies to all inpatient services"
|
|
}
|
|
```
|
|
|
|
### A.2 Reimbursement Primary Response
|
|
|
|
```json
|
|
[
|
|
{
|
|
"SERVICE_TERM": "Inpatient Services",
|
|
"REIMB_TERM": "110% of Medicare"
|
|
},
|
|
{
|
|
"SERVICE_TERM": "Outpatient Services",
|
|
"REIMB_TERM": "$500 per visit"
|
|
}
|
|
]
|
|
```
|
|
|
|
### A.3 Dynamic Primary Response (LOB)
|
|
|
|
```json
|
|
["Medicare", "Medicare Advantage"]
|
|
```
|
|
|
|
### A.4 Carveout Check Response
|
|
|
|
```json
|
|
{
|
|
"CASE": "OUTLIER_TERM"
|
|
}
|
|
```
|
|
|
|
### A.5 Methodology Breakout Response
|
|
|
|
```json
|
|
{
|
|
"REIMBURSEMENT_METHOD": "Percent of Charges",
|
|
"PERCENT": "110",
|
|
"BASIS": "Medicare",
|
|
"FEE_SCHEDULE": "N/A"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Appendix B: Migration Guide Summary
|
|
|
|
### For Developers
|
|
|
|
**When adding new prompts**:
|
|
|
|
1. **Create both functions** - An `_INSTRUCTION()` function and a dynamic prompt function:
|
|
```python
|
|
# INSTRUCTION function (cached) - includes format instructions
|
|
def NEW_PROMPT_INSTRUCTION() -> str:
|
|
return f"""[OBJECTIVE]
|
|
Do something...
|
|
|
|
[INSTRUCTIONS]
|
|
Domain-specific rules...
|
|
|
|
[OUTPUT FORMAT]
|
|
{JSON_DICT_FORMAT_INSTRUCTIONS}""" # Choose appropriate format variable
|
|
|
|
# Dynamic prompt (NOT cached) - NO format instructions
|
|
def NEW_PROMPT(context, data):
|
|
return f"""[CONTEXT]
|
|
{context}
|
|
|
|
[DATA]
|
|
{data}"""
|
|
```
|
|
|
|
2. **Choose correct format instruction variable**:
|
|
- Single entity, single values → `JSON_DICT_FORMAT_INSTRUCTIONS`
|
|
- Single entity, multi-value fields → `JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`
|
|
- Multiple entities → `JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS`
|
|
- Simple list → `JSON_LIST_FORMAT_INSTRUCTIONS`
|
|
|
|
3. **Import and use parser**:
|
|
```python
|
|
from src.utils.json_parsers import parse_json_dict, parse_json_list
|
|
|
|
result = parse_json_dict(llm_output) # or parse_json_list
|
|
```
|
|
|
|
4. **Handle errors**: Wrap in try/except with JSONParseError
|
|
|
|
**When consuming LLM outputs**:
|
|
1. For single-entity extractions: Expect `dict`
|
|
2. For multi-entity extractions: Expect `list[dict]`
|
|
3. For multi-value fields in dicts: Expect each value as `list`
|
|
4. For simple multi-value: Expect `list`
|
|
|
|
**CRITICAL - Prompt Caching Optimization**:
|
|
- ✅ Format instructions in `_INSTRUCTION()` functions (cached)
|
|
- ❌ Format instructions in dynamic prompt functions (sent every time)
|
|
|
|
**During transition period**:
|
|
- Use helper: `_parse_multi_value_field()` to handle both formats
|
|
- Test with both JSON and pipe-delimited inputs
|
|
- Add logging for format detection
|
|
|
|
**After migration complete**:
|
|
- Assume all multi-value fields are lists
|
|
- No pipe splitting needed
|
|
- JSON parsing only
|
|
|
|
---
|
|
|
|
## Appendix C: Glossary
|
|
|
|
- **Pipe-delimited**: Format using `|` as separator, e.g., `"value1|value2|value3"`
|
|
- **JSON dict**: JSON object/dictionary, e.g., `{"key": "value"}`
|
|
- **JSON list**: JSON array, e.g., `["item1", "item2"]`
|
|
- **Universal JSON load**: Legacy parser that extracts JSON from mixed text
|
|
- **Parse JSON dict**: New parser specifically for dictionary outputs
|
|
- **Parse JSON list**: New parser specifically for list/array outputs
|
|
- **Multi-value field**: Field that can contain multiple values (LOB, PROGRAM, etc.)
|
|
- **Downstream consumer**: Code that processes parsed LLM outputs
|
|
- **Dual-format support**: Temporary ability to handle both JSON and pipe-delimited
|
|
|
|
---
|
|
|
|
**End of PRD**
|