Files

T

ppanchigar cbff139ce9 docs: update PRD with tuple-returning architecture and progress

- Updated section 4.1.2 with actual implementation details
- All 35 prompt templates now return (prompt, parser) tuples
- Added implementation progress table showing Phase 1 & 2 complete
- Documented benefits of tuple-returning approach
- Listed all updated functions and their parser types
- Updated work items checklist to reflect completed phases
- Added commit references: 98554afc (architecture), 5d1024e (templates)
- Marked Phase 3 as next step: update prompt_calls.py files

2026-02-02 17:16:56 -06:00

101 KiB

Raw Blame History

PRD: Standardize LLM Output Formats - Eliminate Pipe-Delimited Parsing

Version: 1.1
Date: February 2, 2026
Last Updated: February 2, 2026 (Commit: 98554afc) Status: In Progress - Phase 2 Complete
Author: AI Assistant

Implementation Progress

Overall Status: 🟡 Phase 2 of 6 Complete (Foundation + Prompt Templates)

Phase	Status	Completion	Details
Phase 1: Foundation	✅ Complete	100%	JSON parsers, deprecation warnings, unit tests
Phase 2: Prompt Templates	✅ Complete	100%	All 35 templates return (prompt, parser) tuples
Phase 3: Prompt Calling Layer	⏳ In Progress	0%	Update prompt_calls.py files
Phase 4: Downstream Consumers	🔲 Not Started	0%	aarete_derived, crosswalk, dynamic_funcs
Phase 5: Testing	🔲 Not Started	0%	Integration tests, golden tests
Phase 6: Cleanup & Documentation	🔲 Not Started	0%	Remove backward compat, docs

Latest Commits:

98554afc - All prompt templates now return (prompt, parser) tuples
5d1024e - Updated prompt instruction functions with JSON format instructions
Earlier - JSON parsers (json_utils.py), deprecation warnings, unit tests

Next Steps: Update prompt_calls.py files to use new (prompt, parser) pattern (Phase 3)

Executive Summary

This PRD outlines a comprehensive refactor to eliminate pipe-delimited string parsing from the LLM output processing pipeline. All LLM responses will be migrated to use strict JSON formatting (dictionaries or lists), with explicit parser functions that replace the current universal_json_load implementation.

Background and Motivation
Non-Goals
Current State Analysis
Technical Design
File-by-File Change List
Migration Plan
Testing Plan
Risks and Mitigations
Work Items Checklist

1. Background and Motivation

1.1 Problem Statement

The current system uses pipe-delimited strings (|value|) as a machine-readable format for LLM outputs in certain prompts, while using JSON dictionaries and lists in others. This creates several problems:

Inconsistent parsing logic: Two different parsing paradigms (delimiter extraction vs JSON parsing) exist side-by-side
Fragility: Pipe delimiters can appear in content text, breaking parsers
Type ambiguity: Pipe-delimited outputs lose type information (everything becomes a string)
Maintenance overhead: Downstream code must handle both pipe-separated values and JSON-parsed values
Testing complexity: Multiple parsing paths mean more edge cases to test

1.2 Benefits of Standardization

Single parsing paradigm: All LLM outputs use JSON, simplifying code
Type safety: JSON preserves types (strings, numbers, booleans, nulls, arrays, objects)
Reliability: JSON parsers are battle-tested and handle escaping properly
Consistency: Uniform approach across all prompt templates
Maintainability: Easier to reason about and extend

1.3 Why Now?

The codebase has grown to 13+ files with pipe-split operations
New features are being added that require consistent list handling
Recent bugs related to pipe delimiter parsing in content text

2. Non-Goals

2.1 Out of Scope

Changing internal data representation: Internal pipe-delimited storage in dataframes/databases is out of scope (only parsing LLM outputs changes)
Modifying existing output formats for end users: This is an internal refactor; external APIs/files remain unchanged
Reprocessing historical data: Existing processed files don't need reprocessing
Changing prompt caching strategy: We maintain existing caching patterns
Modifying crosswalk/mapping logic: Crosswalks can still use pipe delimiters internally after parsing

2.2 Future Work (Not This PRD)

Eliminating pipe delimiters from internal storage (separate initiative)
Standardizing all multi-value fields to JSON arrays in the database
Updating external API response formats

3. Current State Analysis

3.0 Previous Implementation Attempt

3.0.1 Commit `faf7072` (Reverted)

A previous attempt at standardizing list formats was merged (commit faf707265d1e0a5bb867e1163f6da6cef0a793e8) but subsequently reverted (commit 02d10e711e7b1c43e0dce11586c8772d7805b4ce).

What Was Implemented:

Added JSON_LIST_FORMAT_INSTRUCTIONS and JSON_DICT_FORMAT_INSTRUCTIONS constants
Updated prompt templates to reference these variables
Changed parser calls from universal_json_load to direct JSON parsing
Updated downstream code to handle JSON list formats

Why It Was Reverted:

Likely caused pipeline failures or output format mismatches
Possible issues with LLM responses not conforming to expected JSON formats
Downstream code may not have fully handled transition from pipe-delimited to lists
Testing may have been incomplete before merge

Lessons Learned:

Need comprehensive testing: Must test full pipeline end-to-end before merge
Need backward compatibility: Should support dual formats during transition
Need clear format specifications: Ambiguity about when to use dict vs list vs dict-with-lists
Need LLM prompt validation: Must verify LLMs actually produce valid JSON with new instructions
Need gradual rollout: Should phase implementation rather than big-bang change

This PRD Addresses These Issues:

✅ Comprehensive prompt-output mapping table (Section 4.2.2)
✅ Four distinct format types with clear usage guidelines (Section 4.2.3)
✅ Phased migration plan with backward compatibility (Section 6)
✅ Extensive testing plan including golden tests (Section 7)
✅ Dual-format support in downstream consumers (Section 5.4)
✅ Rollback strategy (Section 6.3)

3.1 Pipe-Delimited Usage

3.1.1 Prompt Template Constant

File: src/prompts/prompt_templates.py

PIPE_FORMAT_INSTRUCTIONS = "Briefly explain your answer, then enclose your final answer in |pipes|. If the requested information doesn't apply to this context, return |N/A|. If the information should exist but cannot be clearly identified, return |UNKNOWN|."

Used in: Legacy prompts (to be removed)

3.1.2 Extraction Function

File: src/utils/string_utils.py

def extract_text_from_delimiters(
    raw_output: str, delimiter: Delimiter, match_index: int = -1
) -> str

Called from:

src/pipelines/saas/prompts/prompt_calls.py::prompt_dynamic_primary() (line 93-95)
src/pipelines/saas/prompts/prompt_calls.py::prompt_carveout_check() (line 251-253)
src/pipelines/clients/clover/prompts/prompt_calls.py::prompt_dynamic_primary() (line 94-96)
src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py::prompt_dynamic_primary() (line 88-90)

3.2 Universal JSON Load Usage

Implementation: src/utils/string_utils.py::universal_json_load()

Call Sites (11+ occurrences in src/pipelines/saas/prompts/prompt_calls.py):

prompt_exhibit_level() - line 36
prompt_exhibit_level_breakout() - line 74
prompt_reimbursement_primary() - line 129
prompt_methodology_breakout() - line 161
prompt_fee_schedule_breakout() - line 192
prompt_grouper_breakout() - line 222
prompt_special_case_breakout() - line 286
prompt_full_context() - line 390
prompt_dynamic() - line 465
prompt_dynamic_field_safe() - line 623

Also used in:

src/pipelines/clients/clover/prompts/prompt_calls.py
src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py
src/codes/code_funcs.py
src/pipelines/shared/extraction/tin_npi_funcs.py
src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py
Test files

3.3 Downstream Pipe Splitting

3.3.1 aarete_derived.py

File: src/pipelines/shared/postprocessing/aarete_derived.py

Occurrences: 4 instances (lines 45, 67, 93, 155)

Pattern:

if "|" in value:
    value_list = value.split("|")

Context: Parsing LOB, PROGRAM, and PRODUCT fields that may contain pipe-delimited values

3.3.2 crosswalk_utils.py

File: src/utils/crosswalk_utils.py

Occurrence: Line 40

Pattern:

elif "|" in val:
    values = [v.strip() for v in val.split("|")]

Context: Parsing values before applying crosswalk mappings

3.3.3 Other Files with Pipe Splits

Total files identified: 13

src/pipelines/shared/extraction/dynamic_funcs.py - LOB parsing (line 60)
src/pipelines/shared/postprocessing/postprocessing_funcs.py
src/testbed/dynamic_primary_metrics.py
src/testbed/testbed_utils.py
src/codes/code_funcs.py
src/qc_qa/pipeline.py
src/utils/qa_qc_utils.py
src/parent_child/qc.py
src/pipelines/shared/extraction/tin_npi_funcs.py
src/testbed/model_evaluation_utils.py

3.4 Prompt Templates Mentioning Pipes

3.4.1 Instructions with Pipe References

EXHIBIT_LEVEL_INSTRUCTION (line 200)
- "If there are multiple, return them in a pipe-separated list ("|")"
DYNAMIC_PRIMARY_TEXT_INSTRUCTION (line 236)
- "Briefly explain your answer before putting the final answer in |pipes|"
CARVEOUT_CHECK_INSTRUCTION (line 1672)
- "enclose your final answer in |pipes|"
FULL_CONTEXT_ADDITIONAL_INSTRUCTIONS (lines 1278, 1280)
- "return them in a pipe (|) separated list"
DYNAMIC_ASSIGNMENT_INSTRUCTION (line 391)
- References to pipe-delimited format for DUALS value
REIMBURSEMENT_PRIMARY_INSTRUCTION (line ~460)
- JSON list output (already correct format)

4. Technical Design

4.1 JSON Output Instruction Variables

4.1.1 Standardized Format Instructions

Approach: Instead of embedding output format instructions directly in each prompt, we define module-level constants that are referenced by all prompts. This ensures consistency and makes updates easier.

File: src/prompts/prompt_templates.py (additions)

# JSON format instruction constants
JSON_DICT_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted JSON dictionary. 
If the requested information doesn't apply to this context, return 'N/A'. 
If the information should exist but cannot be clearly identified, return 'UNKNOWN'.

Example format:
{
  "FIELD_NAME": "value",
  "ANOTHER_FIELD": "value or N/A"
}
"""

JSON_LIST_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted valid JSON list format. 
If the requested information doesn't apply to this context, return ["N/A"]. 
If the information should exist but cannot be clearly identified, return ["UNKNOWN"]. 

Formatting examples:
- Single string value: ["value"]
- Multiple string values: ["value1", "value2"]
- Date value: ["YYYY-MM-DD"]
- Not applicable: ["N/A"]
- Information unclear or missing: ["UNKNOWN"]
"""

JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted JSON dictionary where each field value is a JSON list.
For fields with multiple values, return them as an array: {"FIELD": ["value1", "value2"]}.
For fields with a single value, still return as an array: {"FIELD": ["value"]}.
If the requested information doesn't apply, return ["N/A"]: {"FIELD": ["N/A"]}.
If the information should exist but cannot be identified, return ["UNKNOWN"]: {"FIELD": ["UNKNOWN"]}.

Example format:
{
  "LOB": ["Medicare", "Medicaid"],
  "NETWORK": ["N/A"],
  "FACILITY_TYPE": ["Inpatient"]
}
"""

JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS = """Return your answer as a valid JSON array of objects. Each object represents one entity (e.g., one reimbursement term).
If no entities are found, return an empty array: []

Example format:
[
  {
    "SERVICE_TERM": "Inpatient Services",
    "REIMB_TERM": "110% of Medicare"
  },
  {
    "SERVICE_TERM": "Outpatient Services",
    "REIMB_TERM": "$500 per visit"
  }
]

For no results: []
"""

Usage Pattern with Prompt Caching:

Since we use prompt caching, format instructions belong in the cached INSTRUCTION() functions, not in the dynamic prompt content:

# INSTRUCTION function (cached) - includes format instructions
def EXHIBIT_LEVEL_INSTRUCTION() -> str:
    """Static instruction for EXHIBIT_LEVEL prompt caching."""
    return f"""[OBJECTIVE]
Extract attributes from a section of a contract.

[GENERAL INSTRUCTIONS]
- For each field, return ALL correct answers found
- Return each field value as a JSON list, even if only one value
- If a list of valid values is provided, ONLY answer with EXACT values from that list
- If no correct answers, return ["N/A"] for that field

[OUTPUT FORMAT]
{JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS}"""


# Dynamic prompt function (NOT cached) - no format instructions
def EXHIBIT_LEVEL(context, fields):
    """Returns ONLY dynamic content.
    Call EXHIBIT_LEVEL_INSTRUCTION() separately for caching.
    """
    return f"""[ATTRIBUTES]
{fields}

[CONTEXT]
{context.replace('"', "'")}"""

Caching Architecture:

┌─────────────────────────────────────────┐
│ CACHED (sent once, reused)              │
│ - INSTRUCTION() function output         │
│ - Includes JSON format instructions     │
│ - Never changes                         │
└─────────────────────────────────────────┘
           ↓
┌─────────────────────────────────────────┐
│ DYNAMIC (sent every request)            │
│ - Prompt function output                │
│ - Context, fields, specific data        │
│ - NO format instructions                │
└─────────────────────────────────────────┘

Benefits:

Single source of truth for output formats
Optimal caching (format instructions cached, not sent repeatedly)
Reduced token usage per request
Easy to update all prompts by changing one variable
Self-documenting code (variable names indicate format type)

4.2 Prompt Output Format Mapping

This section documents the expected output format for each prompt template in the system.

4.2.1 Output Format Classification

Format Types:

JSON Dict - Single dictionary with field:value pairs
JSON List - Simple array of strings
JSON Dict with Lists - Dictionary where each value is an array
JSON List of Dicts - Array of dictionaries (multiple entities)

Cardinality Types:

Exhibit-level - Single result per exhibit (one dict/set of values)
Reimbursement-level - Multiple results per exhibit (list of entities)
Single-value - Returns one atomic value

4.2.2 Comprehensive Prompt Mapping

Prompt Template	Output Format	Cardinality	Multi-Value Fields?	Format Instruction Variable	Example
`EXHIBIT_LEVEL`	JSON Dict with Lists	Exhibit-level	Yes (LOB, NETWORK, etc. can have multiple values)	`JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`	`{"LOB": ["Medicare", "Medicaid"], "FACILITY_TYPE": ["Inpatient"]}`
`REIMBURSEMENT_PRIMARY`	JSON List of Dicts	Reimbursement-level (many rows)	No (each field is single value per row)	`JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS`	`[{"SERVICE_TERM": "...", "REIMB_TERM": "..."}, ...]`
`DYNAMIC_PRIMARY_TEXT`	JSON List	Single-value returning list	Yes (returns all found values)	`JSON_LIST_FORMAT_INSTRUCTIONS`	`["Medicare", "Medicaid", "Duals"]`
`DYNAMIC_ASSIGNMENT`	JSON Dict with Lists	Single assignment per call	Yes (can assign multiple values)	`JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`	`{"LOB": ["Medicare", "Duals"]}`
`CARVEOUT_CHECK`	JSON Dict	Single-value	No	`JSON_DICT_FORMAT_INSTRUCTIONS`	`{"CASE": "OUTLIER_TERM"}`
`METHODOLOGY_BREAKOUT`	JSON Dict	Reimbursement-level (one per term)	No	`JSON_DICT_FORMAT_INSTRUCTIONS`	`{"REIMBURSEMENT_METHOD": "Percent of Charges", "PERCENT": "110"}`
`FEE_SCHEDULE_BREAKOUT`	JSON Dict	Reimbursement-level (one per term)	No	`JSON_DICT_FORMAT_INSTRUCTIONS`	`{"FEE_SCHEDULE_NAME": "Medicare", "FEE_SCHEDULE_YEAR": "2024"}`
`GROUPER_BREAKOUT`	JSON Dict	Reimbursement-level (one per term)	No	`JSON_DICT_FORMAT_INSTRUCTIONS`	`{"GROUPER_TYPE": "DRG", "GROUPER_VERSION": "MS-DRG v40"}`
`FACILITY_ADJUSTMENT_BREAKOUT`	JSON Dict	Exhibit-level	No	`JSON_DICT_FORMAT_INSTRUCTIONS`	`{"ADJUSTMENT_TYPE": "Stop Loss", "ADJUSTMENT_VALUE": "100000"}`
`LESSER_OF_CHECK`	JSON Dict with Lists	Single check per call	Yes (scope and target can be lists)	`JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`	`{"scope": ["GLOBAL"], "target_exhibit": ["ALL"]}`
`LESSER_OF_DISTRIBUTION`	JSON List	Single value (returns statement)	Yes (returns list of applicable statements)	`JSON_LIST_FORMAT_INSTRUCTIONS`	`["Lab Test paid at lesser of $100 or charges"]`
`LOB_RELATIONSHIP`	JSON Dict	Single relationship per call	No	`JSON_DICT_FORMAT_INSTRUCTIONS`	`{"relationship": "Exclusive"}`
`REIMB_DATES_ASSIGNMENT`	JSON List	Single value (returns date range)	Yes (can be multiple date ranges)	`JSON_LIST_FORMAT_INSTRUCTIONS`	`["January 1, 2024 through December 31, 2024"]`
`ONE_TO_ONE_SINGLE_FIELD` (HSC)	JSON Dict with Lists	Contract-level (one per contract)	Yes (contract-level fields can have multiple values)	`JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`	`{"PAYER_NAME": ["Acme Health"]}`
`ONE_TO_ONE_MULTI_FIELD` (Full Context)	JSON Dict with Lists	Contract-level (one per contract)	Yes	`JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`	`{"PAYER_NAME": ["Acme"], "EFFECTIVE_DATE": ["2024-01-01"]}`
`TIN_NPI_TEMPLATE`	JSON Dict with Lists	Contract-level	Yes (multiple TINs/NPIs)	`JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS`	`{"PROVIDER_TIN": ["123456789", "987654321"]}`
`EXHIBIT_TITLE_MATCH`	JSON List	Single-value (YES/NO)	No	`JSON_LIST_FORMAT_INSTRUCTIONS`	`["YES"]` or `["NO"]`
`VALIDATE_REIMBURSEMENTS`	JSON List of Dicts	Validation results (many rows)	No	`JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS`	`[{"valid": true, "reason": "..."}, ...]`
`EXHIBIT_LINKAGE`	JSON Dict	Single-value	No	`JSON_DICT_FORMAT_INSTRUCTIONS`	`{"is_same_exhibit": "YES"}`
Special Case Breakouts (STOP_LOSS, OUTLIER, etc.)	JSON Dict	Reimbursement-level (one per term)	No	`JSON_DICT_FORMAT_INSTRUCTIONS`	`{"STOP_LOSS_AMOUNT": "100000", "STOP_LOSS_UNIT": "dollars"}`

4.2.3 Format Selection Logic

When to use each format:

JSON_DICT_FORMAT_INSTRUCTIONS - Use when:
- Extracting a single entity with multiple fields
- Each field has exactly one value (no arrays)
- Examples: Methodology breakout, carveout check, special case breakouts
JSON_LIST_FORMAT_INSTRUCTIONS - Use when:
- Returning a simple list of values
- Returning a single answer that could be multi-valued (like dynamic primary)
- Returning YES/NO or similar single responses
- Examples: Dynamic primary discovery, exhibit title match, date assignment
JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS - Use when:
- Extracting multiple fields where each field can have multiple values
- Exhibit-level extraction (LOB, NETWORK, etc.)
- Contract-level extraction (PAYER_NAME, etc.)
- Dynamic assignment (assigning multiple values to a field)
- Examples: Exhibit level, one-to-one fields, TIN/NPI, dynamic assignment
JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS - Use when:
- Extracting multiple entities, each with multiple fields
- Reimbursement primary extraction (many service/reimb pairs)
- Validation results (multiple validation outcomes)
- Examples: Reimbursement primary, validation

4.3 New Parser Architecture

4.3.1 Parser Function Specifications

File: src/utils/json_parsers.py (new file)

"""
JSON parsing utilities for LLM outputs.
Replaces universal_json_load with explicit dict/list parsers.
"""

import json
import logging
from typing import Any, Dict, List, Union


class JSONParseError(ValueError):
    """Raised when JSON parsing fails after all attempts."""
    pass


def parse_json_dict(raw_llm_output: str) -> Dict[str, Any]:
    """
    Extract and parse a JSON dictionary from raw LLM output.
    
    Strategy:
    1. Locate the first complete JSON object {...}
    2. Parse with json.loads()
    3. Validate it's a dictionary
    4. Return parsed dict
    
    Args:
        raw_llm_output: Raw string output from LLM (may include reasoning text)
    
    Returns:
        Parsed dictionary
    
    Raises:
        JSONParseError: If no valid JSON dictionary found
    
    Example:
        >>> raw = "Here's my reasoning...\n{\"field\": \"value\", \"count\": 5}"
        >>> parse_json_dict(raw)
        {'field': 'value', 'count': 5}
    """
    if not isinstance(raw_llm_output, str):
        raise TypeError(f"Expected string input, got {type(raw_llm_output).__name__}")
    
    # Find first JSON object
    found = _extract_json_objects(raw_llm_output)
    
    if not found:
        raise JSONParseError(f"No valid JSON dictionary found in: {raw_llm_output[:100]}...")
    
    # Return last valid object (convention: final answer comes last)
    result = found[-1]
    
    if not isinstance(result, dict):
        raise JSONParseError(f"Expected dictionary, got {type(result).__name__}")
    
    return result


def parse_json_list(raw_llm_output: str) -> List[Any]:
    """
    Extract and parse a JSON list from raw LLM output.
    
    Strategy:
    1. Locate the first complete JSON array [...]
    2. Parse with json.loads()
    3. Validate it's a list
    4. Return parsed list
    
    Args:
        raw_llm_output: Raw string output from LLM (may include reasoning text)
    
    Returns:
        Parsed list
    
    Raises:
        JSONParseError: If no valid JSON list found
    
    Example:
        >>> raw = "Here are the items:\n[\"item1\", \"item2\", \"item3\"]"
        >>> parse_json_list(raw)
        ['item1', 'item2', 'item3']
    """
    if not isinstance(raw_llm_output, str):
        raise TypeError(f"Expected string input, got {type(raw_llm_output).__name__}")
    
    # Find first JSON array
    found = _extract_json_objects(raw_llm_output)
    
    if not found:
        raise JSONParseError(f"No valid JSON list found in: {raw_llm_output[:100]}...")
    
    # Return last valid array (convention: final answer comes last)
    result = found[-1]
    
    if not isinstance(result, list):
        raise JSONParseError(f"Expected list, got {type(result).__name__}")
    
    return result


def _extract_json_objects(s: str) -> List[Union[Dict, List]]:
    """
    Extract all top-level JSON objects/arrays from a string.
    Returns list of parsed objects (dicts or lists).
    
    This is adapted from the existing universal_json_load logic
    but more explicit about return types.
    """
    found = []
    i = 0
    
    while i < len(s):
        if s[i] in "{[":
            start = i
            stack = [s[i]]
            j = i + 1
            
            while j < len(s):
                # Skip over string literals to avoid counting brackets inside strings
                if s[j] == '"':
                    j += 1
                    while j < len(s):
                        if s[j] == "\\" and j + 1 < len(s):
                            j += 2  # Skip escaped character
                            continue
                        if s[j] == '"':
                            break
                        j += 1
                elif s[j] in "{[":
                    stack.append(s[j])
                elif s[j] in "}]":
                    if not stack:
                        break
                    open_bracket = stack.pop()
                    if (open_bracket == "{" and s[j] != "}") or (
                        open_bracket == "[" and s[j] != "]"
                    ):
                        break
                    if not stack:
                        candidate = s[start : j + 1]
                        try:
                            obj = json.loads(candidate)
                            found.append(obj)
                        except json.JSONDecodeError:
                            pass
                        i = j  # Move past this top-level object
                        break
                j += 1
        i += 1
    
    return found


# Backward compatibility alias (deprecated)
def universal_json_load(s: str) -> Union[Dict, List]:
    """
    DEPRECATED: Use parse_json_dict() or parse_json_list() instead.
    
    This function is maintained for backward compatibility during migration.
    It will be removed in a future version.
    """
    logging.warning(
        "universal_json_load is deprecated. Use parse_json_dict() or parse_json_list() instead."
    )
    found = _extract_json_objects(s)
    if not found:
        raise ValueError(f"No valid JSON objects found in string: {s[:100]}...")
    return found[-1]

4.1.2 Prompt Template Parser Coupling (IMPLEMENTED)

Status: ✅ COMPLETED (Commit: 98554afc)

Enhancement to Prompt System: All prompt template functions now return a tuple of (prompt_string, parser_function). This architectural change eliminates ambiguity about which parser to use for each prompt.

File: src/prompts/prompt_templates.py (modification)

Implementation Approach:

Instead of creating separate _WITH_PARSER functions, we modified all existing prompt template functions to directly return tuples. This provides a cleaner API and ensures every prompt has its parser explicitly defined.

Helper Parser Functions (added to prompt_templates.py):

from typing import Callable, Tuple, Any
from src.utils import json_utils

def _json_dict_parser(raw_output: str) -> dict:
    """Parse JSON dictionary from LLM output."""
    return json_utils.parse_json_dict(raw_output)

def _json_list_parser(raw_output: str) -> list:
    """Parse JSON list from LLM output."""
    return json_utils.parse_json_list(raw_output)

Updated Function Signatures (examples):

def EXHIBIT_LEVEL(context, fields) -> Tuple[str, Callable[[str], dict]]:
    """Returns ONLY dynamic content for exhibit level extraction.
    Call EXHIBIT_LEVEL_INSTRUCTION() separately for the cached instruction.
    
    Returns:
        Tuple of (prompt_string, parser_function) where parser expects JSON dict output.
    """
    prompt = f"""[ATTRIBUTES]
Here are the attributes to be included in the dictionary, and instructions on how to correctly answer:
{fields}

[CONTEXT]
Here is the text to analyze:
{context.replace('"', "'")}"""
    
    return (prompt, _json_dict_parser)


def DYNAMIC_PRIMARY_TEXT(context, field_name, field_prompt) -> Tuple[str, Callable[[str], list]]:
    """Returns ONLY dynamic content for text-based dynamic primary extraction.
    Call DYNAMIC_PRIMARY_TEXT_INSTRUCTION() separately for the cached instruction.
    
    Returns:
        Tuple of (prompt_string, parser_function) where parser expects JSON list output.
    """
    prompt = f"""[ATTRIBUTE TO EXTRACT]
{field_name} : {field_prompt}

[CONTEXT]
Here is the text to analyze:
{context.replace('"', "'")}"""
    
    return (prompt, _json_list_parser)


def CODE_EXPLICIT(service, methodology) -> Tuple[str, Callable[[str], dict]]:
    """Returns ONLY dynamic content for code explicit extraction.
    Call CODE_EXPLICIT_INSTRUCTION() separately for the cached instruction.
    Note: Field definitions are now included in CODE_EXPLICIT_INSTRUCTION() for caching.
    
    Returns:
        Tuple of (prompt_string, parser_function) where parser expects JSON dict output.
    """
    prompt = f"""[CONTEXT]
Here is the Service and Methodology to analyze:
Service: {service.replace('"', "'")}
Methodology: {methodology.replace('"', "'")}"""
    
    return (prompt, _json_dict_parser)

Complete List of Updated Functions (24 total):

LESSER_OF_CHECK → (prompt, _json_dict_parser)
EXHIBIT_TITLE_MATCH → (prompt, _json_list_parser)
EXHIBIT_LEVEL → (prompt, _json_dict_parser)
DYNAMIC_PRIMARY_TEXT → (prompt, _json_list_parser)
REIMB_DATES_ASSIGNMENT → (prompt, _json_dict_parser)
DYNAMIC_ASSIGNMENT → (prompt, _json_dict_parser)
REIMBURSEMENT_PRIMARY → (prompt, _json_list_parser)
LESSER_OF_DISTRIBUTION → (prompt, _json_list_parser)
METHODOLOGY_BREAKOUT → (prompt, _json_list_parser)
FEE_SCHEDULE_BREAKOUT → (prompt, _json_dict_parser)
GROUPER_BREAKOUT → (prompt, _json_dict_parser)
SPECIAL_CASE_BREAKOUT → (prompt, _json_dict_parser)
OUTLIER_BREAKOUT → (prompt, _json_dict_parser)
All breakout variants (RATE_ESCALATOR, TRIGGER_CAP, PREMIUM, ADDITION, STOP_LOSS, FACILITY_ADJUSTMENT, SEQUESTRATION, DISCOUNT)
CODE_EXPLICIT → (prompt, _json_dict_parser)
CODE_CATEGORY → (prompt, _json_list_parser)
CODE_IMPLICIT_SPECIAL → (prompt, _json_list_parser)
CODE_IMPLICIT → (prompt, _json_list_parser)
CODE_LAST_CHECK → (prompt, _json_list_parser)
FILL_BILL_TYPE → (prompt, _json_list_parser)
ONE_TO_ONE_SINGLE_FIELD_TEMPLATE → (prompt, _json_dict_parser)
ONE_TO_ONE_MULTI_FIELD_TEMPLATE → (prompt, _json_dict_parser)
TIN_NPI_TEMPLATE → (prompt, _json_list_parser)
CHECK_PROVIDER_NAME_MATCH_PROMPT → (prompt, _json_list_parser)
EXTRACT_AMENDMENT_NUM_FROM_FILENAME → (prompt, _json_dict_parser)
EXHIBIT_HEADER → (prompt, _json_list_parser)
EXHIBIT_LINKAGE → (prompt, _json_list_parser)
DATE_FIX_PROMPT → (prompt, _json_list_parser)
EFFECTIVE_DATE_FIX_PROMPT → (prompt, _json_dict_parser)
SPLIT_REIMB_DATES → (prompt, _json_dict_parser)
DERIVED_TERM_DATE_PROMPT → (prompt, _json_list_parser)
VALIDATE_REIMBURSEMENTS_PROMPT → (prompt, _json_list_parser)
CARVEOUT_CHECK → (prompt, _json_list_parser)
LOB_RELATIONSHIP → (prompt, _json_list_parser)
SPECIAL_CASE_ASSIGNMENT → (prompt, _json_list_parser)

Usage Pattern (example in prompt_calls.py):

def prompt_exhibit_level(
    exhibit_text: str,
    exhibit_level_fields: FieldSet,
    constants: Constants,
    filename: str,
):
    # Get prompt and parser together
    prompt_template, parser = prompt_templates.EXHIBIT_LEVEL(
        exhibit_text, exhibit_level_fields.print_prompt_dict(constants)
    )
    
    # Invoke LLM
    llm_answer_raw = llm_utils.invoke_claude(
        prompt_template,
        "sonnet_latest",
        filename,
        max_tokens=8192,
        cache=True,
        instruction=prompt_templates.EXHIBIT_LEVEL_INSTRUCTION(),
        usage_label="EXHIBIT_LEVEL",
    )
    
    # Parse with returned parser (no ambiguity about which parser to use)
    llm_answer_final = parser(llm_answer_raw)
    
    return llm_answer_final

Benefits of This Approach:

No Ambiguity: The prompt template itself declares which parser should be used
Type Safety: Return type hints make it clear what parser function expects
Self-Documenting: Reading the prompt function immediately shows the expected output format
Refactor-Friendly: Changing a prompt's output format requires updating one function
Testing: Easy to test that prompt returns correct parser for its format

Downstream Impact:

All calling code in prompt_calls.py files must be updated to:

Unpack the tuple: prompt, parser = prompt_templates.SOME_PROMPT(...)
Use the returned parser: result = parser(llm_raw_output)
Remove hardcoded parser calls (e.g., no more string_utils.extract_text_from_delimiters)

Migration Status:

✅ All 35 prompt template functions updated
⏳ Downstream calling code update in progress (see Phase 2 below)

4.2 JSON Schema Expectations

4.2.1 Dictionary Outputs

Used for: Single-entity extractions (exhibit level, methodology breakout, etc.)

Schema:

{
  "FIELD_NAME_1": "value or N/A",
  "FIELD_NAME_2": ["value1", "value2"],
  "FIELD_NAME_3": 123,
  "FIELD_NAME_4": true
}

Example Prompt Update:

OLD (pipe-delimited):

[OUTPUT FORMAT]
Briefly explain your answer, then enclose your final answer in |pipes|.

NEW (JSON):

[OUTPUT FORMAT]
First provide your reasoning, then output a JSON dictionary with the field name as the key.

Example:
My reasoning: The text mentions "Medicare Advantage"...

{"LOB": ["Medicare", "Medicare Advantage"]}

4.2.2 List Outputs

Used for: Multi-entity extractions (reimbursement terms, etc.)

Schema:

[
  {
    "SERVICE_TERM": "...",
    "REIMB_TERM": "...",
    "OTHER_FIELD": "..."
  },
  {
    "SERVICE_TERM": "...",
    "REIMB_TERM": "...",
    "OTHER_FIELD": "..."
  }
]

Special Case: Empty list for "no results found"

[]

Or explicit marker:

{"NO_RESULTS": true}

4.2.3 Multi-Value Fields

OLD (pipe-delimited in JSON):

{
  "LOB": "Medicare|Medicaid|Duals"
}

NEW (JSON array):

{
  "LOB": ["Medicare", "Medicaid", "Duals"]
}

Prompt instruction change:

OLD:

"If there are multiple values, return them in a pipe-separated list ("|")"

NEW:

"If there are multiple values, return them as a JSON array. For example: \"LOB\": [\"Medicare\", \"Medicaid\"]"

5. File-by-File Change List

5.1 Core Utilities

5.1.1 Create: `src/utils/json_parsers.py`

Change Type: NEW FILE

Content: See section 4.1.1 for full implementation

Key Functions:

parse_json_dict()
parse_json_list()
_extract_json_objects() (helper)
universal_json_load() (deprecated alias)

5.1.2 Modify: `src/utils/string_utils.py`

Change Type: DEPRECATION

Changes:

Add deprecation notice to universal_json_load()
Import and delegate to new parsers
Keep extract_text_from_delimiters() but mark as deprecated

Code:

# Add to imports
from src.utils.json_parsers import parse_json_dict, parse_json_list, JSONParseError

# Update universal_json_load
def universal_json_load(s: str):
    """
    DEPRECATED: Use parse_json_dict() or parse_json_list() from json_parsers module.
    
    This function is maintained for backward compatibility.
    Will be removed in future version.
    """
    import warnings
    warnings.warn(
        "universal_json_load is deprecated. Use json_parsers.parse_json_dict() "
        "or json_parsers.parse_json_list() instead.",
        DeprecationWarning,
        stacklevel=2
    )
    from src.utils.json_parsers import _extract_json_objects
    found = _extract_json_objects(s)
    if not found:
        raise ValueError(f"No valid JSON objects found in string: {s[:100]}...")
    return found[-1]


def extract_text_from_delimiters(
    raw_output: str, delimiter: Delimiter, match_index: int = -1
) -> str:
    """
    DEPRECATED: Extracting text from delimiters is being phased out.
    Use JSON parsing instead.
    
    ... (rest of implementation unchanged)
    """
    import warnings
    warnings.warn(
        "extract_text_from_delimiters is deprecated. "
        "Use JSON-based prompt formats instead.",
        DeprecationWarning,
        stacklevel=2
    )
    # ... existing implementation ...

5.1.3 Modify: `src/constants/delimiters.py`

Change Type: DOCUMENTATION

Changes:

Add comment noting PIPE delimiter is deprecated for LLM parsing
Keep enum for backward compatibility

Code:

from enum import Enum


class Delimiter(Enum):
    """
    Delimiter types for parsing.
    
    NOTE: PIPE delimiter is deprecated for LLM output parsing.
    Use JSON formats instead. This enum is maintained for
    backward compatibility only.
    """
    PIPE = "|"  # DEPRECATED for LLM outputs
    BACKTICK = "`"
    TRIPLE_BACKTICK = "```"

5.2 Prompt Templates

5.2.1 Modify: `src/prompts/prompt_templates.py`

Change Type: MAJOR REFACTOR

Changes:

Add JSON Format Instruction Constants (at module level, after existing constants):

# JSON format instruction constants - referenced by all prompt templates
JSON_DICT_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted JSON dictionary. 
If the requested information doesn't apply to this context, return 'N/A'. 
If the information should exist but cannot be clearly identified, return 'UNKNOWN'.

Example format:
{
  "FIELD_NAME": "value",
  "ANOTHER_FIELD": "value or N/A"
}
"""

JSON_LIST_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted valid JSON list format. 
If the requested information doesn't apply to this context, return ["N/A"]. 
If the information should exist but cannot be clearly identified, return ["UNKNOWN"]. 

Formatting examples:
- Single string value: ["value"]
- Multiple string values: ["value1", "value2"]
- Date value: ["YYYY-MM-DD"]
- Not applicable: ["N/A"]
- Information unclear or missing: ["UNKNOWN"]
"""

JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS = """Briefly explain your answer, then put the final answer in a properly formatted JSON dictionary where each field value is a JSON list.
For fields with multiple values, return them as an array: {"FIELD": ["value1", "value2"]}.
For fields with a single value, still return as an array: {"FIELD": ["value"]}.
If the requested information doesn't apply, return ["N/A"]: {"FIELD": ["N/A"]}.
If the information should exist but cannot be identified, return ["UNKNOWN"]: {"FIELD": ["UNKNOWN"]}.

Example format:
{
  "LOB": ["Medicare", "Medicaid"],
  "NETWORK": ["N/A"],
  "FACILITY_TYPE": ["Inpatient"]
}
"""

JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS = """Return your answer as a valid JSON array of objects. Each object represents one entity (e.g., one reimbursement term).
If no entities are found, return an empty array: []

Example format:
[
  {
    "SERVICE_TERM": "Inpatient Services",
    "REIMB_TERM": "110% of Medicare"
  },
  {
    "SERVICE_TERM": "Outpatient Services",
    "REIMB_TERM": "$500 per visit"
  }
]

For no results: []
"""

# Keep PIPE_FORMAT_INSTRUCTIONS temporarily for backward compatibility
# Mark as deprecated
PIPE_FORMAT_INSTRUCTIONS = "DEPRECATED: Use JSON format instructions instead."

Update EXHIBIT_LEVEL() and EXHIBIT_LEVEL_INSTRUCTION():

Dynamic prompt (NO format instructions):

def EXHIBIT_LEVEL(context, fields):
    """Returns ONLY dynamic content for exhibit level extraction.
    Call EXHIBIT_LEVEL_INSTRUCTION() separately for the cached instruction.
    """
    return f"""[ATTRIBUTES]
Here are the attributes to be included in the dictionary, and instructions on how to correctly answer:
{fields}

[CONTEXT]
Here is the text to analyze:
{context.replace('"', "'")}"""

Cached instruction (WITH format instructions):

def EXHIBIT_LEVEL_INSTRUCTION() -> str:
    """Static instruction for EXHIBIT_LEVEL prompt caching.
    Contains all static instructions including output format.
    """
    return f"""[OBJECTIVE]
Extract attributes from a section of a contract.

[GENERAL INSTRUCTIONS]
- For each field, return ALL correct answers found
- Return each field value as a JSON list, even if only one value: {{"FIELD": ["value"]}}
- If multiple values exist, return all: {{"FIELD": ["value1", "value2"]}}
- If a list of valid values is provided, ONLY answer with EXACT values from that list
- If no correct answers for a field, return ["N/A"] for that field
- Be consistent and deterministic; avoid creative paraphrasing

[OUTPUT FORMAT]
{JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS}"""

Update DYNAMIC_PRIMARY_TEXT() and DYNAMIC_PRIMARY_TEXT_INSTRUCTION():

Dynamic prompt (NO format instructions):

def DYNAMIC_PRIMARY_TEXT(context, field_name, field_prompt):
    """Returns ONLY dynamic content for text-based dynamic primary extraction.
    Call DYNAMIC_PRIMARY_TEXT_INSTRUCTION() separately for the cached instruction.
    """
    return f"""[ATTRIBUTE TO EXTRACT]
{field_name} : {field_prompt}

[CONTEXT]
Here is the text to analyze:
{context.replace('"', "'")}"""

Cached instruction (WITH format instructions):

def DYNAMIC_PRIMARY_TEXT_INSTRUCTION() -> str:
    """Static instruction for DYNAMIC_PRIMARY_TEXT prompt caching.
    Contains all static instructions including output format.
    """
    return f"""[OBJECTIVE]
Extract attribute values for dynamic fields from contract text, paying close attention to detail.

[GENERAL INSTRUCTIONS]
- Return all valid, explicitly stated values as a JSON list
- If multiple values found, include all: ["value1", "value2", "value3"]
- If only one value found, still return as a list: ["value1"]
- If no valid values appear explicitly, return: ["N/A"]
- Make appropriate and obvious assumptions, but don't take huge leaps of logic
  Example: "CHIP-P" can be assumed to be "CHIP Perinate", but NOT "Medicaid"

[OUTPUT FORMAT]
{JSON_LIST_FORMAT_INSTRUCTIONS}"""

Update CARVEOUT_CHECK() and CARVEOUT_CHECK_INSTRUCTION():

Dynamic prompt (NO format instructions):

def CARVEOUT_CHECK(service_term: str, reimb_term: str):
    """Returns prompt for carveout check.
    Call CARVEOUT_CHECK_INSTRUCTION() separately for the cached instruction.
    """
    return f"""[REIMBURSEMENT TERM]
Service Term: {service_term}
Reimbursement Term: {reimb_term}

Determine which case from the provided list this reimbursement term falls under."""

Cached instruction (WITH format instructions):

def CARVEOUT_CHECK_INSTRUCTION() -> str:
    """Static instruction for CARVEOUT_CHECK prompt caching.
    Contains all static instructions including output format and case definitions.
    """
    # Build carveout_text and special_case_text from constants
    constants = _get_constants()
    carveout_definitions = constants.get_constant("CARVEOUT_DEFINITIONS")
    special_case_definitions = constants.get_constant("SPECIAL_CASE_DEFINITIONS")
    # ... format definitions ...
    
    return f"""[OBJECTIVE]
Examine the Reimbursement Term and identify which case it falls under.

[CASE DEFINITIONS]
Here are the cases, with their definitions:
Carveout cases: {carveout_text}

Special cases: {special_case_text}

[INSTRUCTIONS]
- Determine which case description corresponds to the specific reimbursement method
- The Service and Reimbursement will always match one of the listed cases
- If either is an explicit match, choose that case
- If multiple cases apply, choose the BEST fit
- Always return one value from the combined list; do not return 'N/A' or 'UNKNOWN'

[OUTPUT FORMAT]
{JSON_DICT_FORMAT_INSTRUCTIONS}

Return as: {{"CASE": "CASE_NAME"}}"""

Update REIMBURSEMENT_PRIMARY() and REIMBURSEMENT_PRIMARY_INSTRUCTION():

Dynamic prompt (NO format instructions):

def REIMBURSEMENT_PRIMARY(context):
    """Returns ONLY dynamic content for reimbursement primary extraction.
    Call REIMBURSEMENT_PRIMARY_INSTRUCTION() separately for the cached instruction.
    """
    return f"""[CONTEXT]
Here is the contract text to analyze:
{context}"""

Cached instruction (WITH format instructions):

def REIMBURSEMENT_PRIMARY_INSTRUCTION() -> str:
    """Static instruction for REIMBURSEMENT_PRIMARY prompt caching.
    Contains all static instructions including output format.
    """
    return f"""[OBJECTIVE]
Extract all reimbursement terms from the contract text.

[INSTRUCTIONS]
- Identify every service and its associated reimbursement rate/method
- Each service-reimbursement pair becomes one object in the output array
- Be comprehensive - capture all reimbursement terms on the page

[OUTPUT FORMAT]
{JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS}

Expected fields per reimbursement term:
- SERVICE_TERM: The service or procedure description
- REIMB_TERM: The reimbursement rate or method

If no reimbursement terms found, return: []"""

Update other prompt functions - General pattern:

CRITICAL PATTERN: Format instructions go in INSTRUCTION() functions (cached), NOT in dynamic prompts.

Example: METHODOLOGY_BREAKOUT:

# Dynamic prompt (NO format instructions)
def METHODOLOGY_BREAKOUT(service_term, reimb_term):
    return f"""[CONTEXT]
Service: {service_term}
Reimbursement: {reimb_term}

Extract methodology components."""

# Cached instruction (WITH format instructions)
def METHODOLOGY_BREAKOUT_INSTRUCTION():
    return f"""[OBJECTIVE]
Break down reimbursement methodology into components.

[INSTRUCTIONS]
... (domain logic) ...

[OUTPUT FORMAT]
{JSON_DICT_FORMAT_INSTRUCTIONS}

Expected fields: REIMBURSEMENT_METHOD, PERCENT, BASIS, FEE_SCHEDULE, etc."""

Example: DYNAMIC_ASSIGNMENT:

# Dynamic prompt (NO format instructions)
def DYNAMIC_ASSIGNMENT(service_term, reimb_term, field_name, field_prompt, exhibit_text, page_num):
    return f"""[CONTEXT]
{exhibit_text}

[REIMBURSEMENT_TERM]
Service: {service_term}
Reimbursement: {reimb_term}

[FIELD TO ASSIGN]
{field_name}: {field_prompt}"""

# Cached instruction (WITH format instructions)
def DYNAMIC_ASSIGNMENT_INSTRUCTION():
    return f"""[OBJECTIVE]
Identify which field value applies to a given Reimbursement Term.

[INSTRUCTIONS]
... (domain logic about context, boundaries, etc.) ...

[OUTPUT FORMAT]
{JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS}

Return as: {{"FIELD_NAME": ["value1", "value2"]}}"""

Example: ONE_TO_ONE_MULTI_FIELD_TEMPLATE:

# Dynamic prompt (NO format instructions)
def ONE_TO_ONE_MULTI_FIELD_TEMPLATE(context, questions):
    return f"""[CONTRACT TEXT]
{context}

[FIELDS TO EXTRACT]
{questions}"""

# Cached instruction (WITH format instructions)
def ONE_TO_ONE_MULTI_FIELD_INSTRUCTION():
    return f"""[OBJECTIVE]
Extract contract-level fields from full contract text.

[INSTRUCTIONS]
... (domain logic) ...

[OUTPUT FORMAT]
{JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS}

Each field value should be a JSON list, even if single value."""

Comprehensive List of Prompt Functions to Update:

CRITICAL: Format instructions are added to _INSTRUCTION() functions (cached), NOT to dynamic prompt functions.

All _INSTRUCTION() functions must be updated to include the appropriate JSON format instruction variable at the end. Here's the complete list organized by format type:

INSTRUCTION Functions to Update with JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS:

EXHIBIT_LEVEL_INSTRUCTION() ✓
DYNAMIC_ASSIGNMENT_INSTRUCTION()
ONE_TO_ONE_SINGLE_FIELD_INSTRUCTION()
ONE_TO_ONE_MULTI_FIELD_INSTRUCTION()
TIN_NPI_TEMPLATE_INSTRUCTION()
LESSER_OF_CHECK_INSTRUCTION()

INSTRUCTION Functions to Update with JSON_DICT_FORMAT_INSTRUCTIONS:

CARVEOUT_CHECK_INSTRUCTION() ✓
METHODOLOGY_BREAKOUT_INSTRUCTION()
FEE_SCHEDULE_BREAKOUT_INSTRUCTION()
GROUPER_BREAKOUT_INSTRUCTION()
FACILITY_ADJUSTMENT_BREAKOUT_INSTRUCTION()
LOB_RELATIONSHIP_INSTRUCTION()
EXHIBIT_LINKAGE_INSTRUCTION()
All special case breakout instructions (STOP_LOSS, OUTLIER, etc.)

INSTRUCTION Functions to Update with JSON_LIST_FORMAT_INSTRUCTIONS:

DYNAMIC_PRIMARY_TEXT_INSTRUCTION() ✓
LESSER_OF_DISTRIBUTION_INSTRUCTION()
REIMB_DATES_ASSIGNMENT_INSTRUCTION()
EXHIBIT_TITLE_MATCH_INSTRUCTION()
CODE_IMPLICIT_SPECIAL_INSTRUCTION()
CODE_PLACEHOLDER_INSTRUCTION()

INSTRUCTION Functions to Update with JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS:

REIMBURSEMENT_PRIMARY_INSTRUCTION() ✓
VALIDATE_REIMBURSEMENTS_INSTRUCTION()

Pattern for Updates:

# OLD (no format instructions in INSTRUCTION function):
def SOME_PROMPT_INSTRUCTION():
    return """[OBJECTIVE]
    Do something...
    
    [INSTRUCTIONS]
    Follow these rules..."""

# NEW (format instructions added to INSTRUCTION function):
def SOME_PROMPT_INSTRUCTION():
    return f"""[OBJECTIVE]
    Do something...
    
    [INSTRUCTIONS]
    Follow these rules...
    
    [OUTPUT FORMAT]
    {JSON_DICT_FORMAT_INSTRUCTIONS}"""

# Dynamic prompt remains clean (NO format instructions)
def SOME_PROMPT(context):
    return f"""[CONTEXT]
    {context}"""

Instruction Functions (cached - ADD format instruction variables):

CRITICAL: These are the functions that need updates. Format instructions must be ADDED to these cached instruction functions, NOT to the dynamic prompt functions.

Complete list of _INSTRUCTION() functions to update:

EXHIBIT_LEVEL_INSTRUCTION() ✓ - Add JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
DYNAMIC_PRIMARY_TEXT_INSTRUCTION() ✓ - Add JSON_LIST_FORMAT_INSTRUCTIONS
CARVEOUT_CHECK_INSTRUCTION() ✓ - Add JSON_DICT_FORMAT_INSTRUCTIONS
REIMBURSEMENT_PRIMARY_INSTRUCTION() ✓ - Add JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS
METHODOLOGY_BREAKOUT_INSTRUCTION() - Add JSON_DICT_FORMAT_INSTRUCTIONS
FEE_SCHEDULE_BREAKOUT_INSTRUCTION() - Add JSON_DICT_FORMAT_INSTRUCTIONS
GROUPER_BREAKOUT_INSTRUCTION() - Add JSON_DICT_FORMAT_INSTRUCTIONS
DYNAMIC_ASSIGNMENT_INSTRUCTION() - Add JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
REIMB_DATES_ASSIGNMENT_INSTRUCTION() - Add JSON_LIST_FORMAT_INSTRUCTIONS
LESSER_OF_DISTRIBUTION_INSTRUCTION() - Add JSON_LIST_FORMAT_INSTRUCTIONS
LESSER_OF_CHECK_INSTRUCTION() - Add JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
LOB_RELATIONSHIP_INSTRUCTION() - Add JSON_DICT_FORMAT_INSTRUCTIONS
ONE_TO_ONE_SINGLE_FIELD_INSTRUCTION() - Add JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
ONE_TO_ONE_MULTI_FIELD_INSTRUCTION() - Add JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
TIN_NPI_TEMPLATE_INSTRUCTION() - Add JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
EXHIBIT_LINKAGE_INSTRUCTION() - Add JSON_DICT_FORMAT_INSTRUCTIONS
VALIDATE_REIMBURSEMENTS_INSTRUCTION() - Add JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS
EXHIBIT_TITLE_MATCH_INSTRUCTION() - Add JSON_LIST_FORMAT_INSTRUCTIONS
All special case breakout instructions - Add JSON_DICT_FORMAT_INSTRUCTIONS

Dynamic Prompt Functions (do NOT add format instructions to these - they should remain clean)

5.3 Prompt Calling Layer

5.3.1 Modify: `src/pipelines/saas/prompts/prompt_calls.py`

Change Type: MAJOR REFACTOR

Changes:

Update Imports:

from src.utils import json_parsers  # Add this
# Remove references to extract_text_from_delimiters for new code

Update prompt_exhibit_level():

def prompt_exhibit_level(
    exhibit_text: str,
    exhibit_level_fields: FieldSet,
    constants: Constants,
    filename: str,
):
    logging.debug(f"Running exhibit Level Fields for {filename}:")
    logging.debug(exhibit_level_fields.print_prompt_dict(constants))
    if not exhibit_level_fields.contains_fields():
        return {}
    
    prompt = prompt_templates.EXHIBIT_LEVEL(
        exhibit_text, exhibit_level_fields.print_prompt_dict(constants)
    )
    
    llm_answer_raw = llm_utils.invoke_claude(
        prompt,
        "sonnet_latest",
        filename,
        max_tokens=8192,
        cache=True,
        instruction=prompt_templates.EXHIBIT_LEVEL_INSTRUCTION(),
        usage_label="EXHIBIT_LEVEL",
    )
    logging.debug(f"LLM raw output for {filename}: {llm_answer_raw}")

    # OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
    # NEW:
    llm_answer_final = json_parsers.parse_json_dict(llm_answer_raw)

    return llm_answer_final

Update prompt_exhibit_level_breakout():

def prompt_exhibit_level_breakout(
    exhibit_level_answers: dict[str, str], filename: str
) -> dict[str, str]:
    if not string_utils.is_empty(
        exhibit_level_answers.get("FACILITY_ADJUSTMENT_TERM", "")
    ):
        prompt = prompt_templates.FACILITY_ADJUSTMENT_BREAKOUT(
            exhibit_level_answers["FACILITY_ADJUSTMENT_TERM"]
        )
        llm_answer_raw = llm_utils.invoke_claude(
            prompt=prompt, model_id="sonnet_latest", filename=filename
        )
        
        # OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
        # NEW:
        llm_answer_final = json_parsers.parse_json_dict(llm_answer_raw)
        
        exhibit_level_answers.update(llm_answer_final)

    return exhibit_level_answers

Update prompt_dynamic_primary():

CRITICAL CHANGE: This function currently uses extract_text_from_delimiters() with PIPE delimiter. It must transition to JSON list output.

def prompt_dynamic_primary(
    exhibit_text: str, field: Field, constants: Constants, filename: str, TEMPLATE
):
    prompt = TEMPLATE(exhibit_text, field.field_name, field.get_prompt(constants))
    logging.debug(f"Dynamic primary prompt for {filename}; {field}: {prompt}")
    
    llm_answer_raw = llm_utils.invoke_claude(
        prompt,
        "sonnet_latest",
        filename,
        cache=True,
        instruction=prompt_templates.DYNAMIC_PRIMARY_TEXT_INSTRUCTION(),  # Updated instruction
    )
    logging.debug(f"Claude answer for {filename}; {field}: {llm_answer_raw}")
    
    # OLD:
    # llm_answer_final = string_utils.extract_text_from_delimiters(
    #     llm_answer_raw, string_utils.Delimiter.PIPE
    # )
    # if "," in llm_answer_final:
    #     llm_answer_final = "|".join(
    #         [item.strip() for item in llm_answer_final.split(",")]
    #     )
    
    # NEW: Parse as JSON list, then join with pipes for backward compatibility
    # (downstream code still expects pipe-delimited for now)
    try:
        answer_list = json_parsers.parse_json_list(llm_answer_raw)
        llm_answer_final = "|".join(str(item).strip() for item in answer_list)
    except json_parsers.JSONParseError as e:
        logging.error(f"Failed to parse dynamic primary answer: {e}")
        llm_answer_final = "N/A"
    
    return llm_answer_final

NOTE: This function converts JSON list back to pipe-delimited for backward compatibility during Phase 3. In Phase 4, once downstream consumers (particularly dynamic_funcs.py) are updated to handle JSON lists natively, this conversion will be removed.

Phase 3 (Temporary):

LLM outputs: ["Medicare", "Medicaid"]
Parser extracts: ["Medicare", "Medicaid"]
Function returns: "Medicare|Medicaid" (converted for compatibility)
Downstream code sees: pipe-delimited string (existing behavior)

Phase 4 (Final):

LLM outputs: ["Medicare", "Medicaid"]
Parser extracts: ["Medicare", "Medicaid"]
Function returns: ["Medicare", "Medicaid"] (no conversion)
Downstream code handles: JSON list natively

See Section 5.4.3 for downstream updates to dynamic_funcs.py.

Update prompt_reimbursement_primary():

def prompt_reimbursement_primary(
    page_text: str,
    filename: str,
) -> list[dict[str, str]]:

    prompt = prompt_templates.REIMBURSEMENT_PRIMARY(page_text)

    logging.debug(
        f"""Running reimbursement primary prompt for {filename} with prompt: {prompt}"""
    )
    llm_answer_raw = llm_utils.invoke_claude(
        prompt,
        "sonnet_latest",
        filename,
        max_tokens=25000,
        cache=True,
        instruction=prompt_templates.REIMBURSEMENT_PRIMARY_INSTRUCTION(),
        usage_label="REIMBURSEMENT_PRIMARY",
    )
    logging.debug(f"""LLM raw output for {filename}: {llm_answer_raw}""")

    # Check for the special "no results" case (could be empty array or special marker)
    try:
        # OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
        # NEW:
        llm_answer_final = json_parsers.parse_json_list(llm_answer_raw)
        
        # Handle empty list or special no-results marker
        if not llm_answer_final:
            return []
        
        # Check for special marker dict
        if len(llm_answer_final) == 1 and isinstance(llm_answer_final[0], dict):
            if llm_answer_final[0].get("NO_RESULTS") or "NO_REIMBURSEMENT_TERMS_FOUND" in str(llm_answer_final[0]):
                return []
        
        return llm_answer_final
        
    except json_parsers.JSONParseError as e:
        logging.error(f"Error parsing LLM response: {e}")
        logging.error(f"Raw LLM output: {llm_answer_raw}")
        raise

Update prompt_methodology_breakout():

def prompt_methodology_breakout(
    service_term: str,
    reimb_term: str,
    filename: str,
):
    prompt = prompt_templates.METHODOLOGY_BREAKOUT(service_term, reimb_term)
    logging.debug(f"Methodology Breakout Prompt for {filename}: {prompt}")
    
    llm_response = llm_utils.invoke_claude(
        prompt,
        "sonnet_latest",
        filename,
        cache=True,
        instruction=prompt_templates.METHODOLOGY_BREAKOUT_INSTRUCTION(),
        usage_label="METHODOLOGY_BREAKOUT",
    )
    logging.debug(f"LLM Response for {filename}: {llm_response}")
    
    try:
        # OLD: methodology_breakout_answers = string_utils.universal_json_load(llm_response)
        # NEW:
        methodology_breakout_answers = json_parsers.parse_json_dict(llm_response)
    except json_parsers.JSONParseError as e:
        logging.error(f"Error parsing LLM response: {e}")
        methodology_breakout_answers = {}
    
    return methodology_breakout_answers

Update prompt_fee_schedule_breakout():

def prompt_fee_schedule_breakout(
    methodology_breakout_dict: dict,
    reimbursement_method: str,
    filename: str,
):
    prompt = prompt_templates.FEE_SCHEDULE_BREAKOUT(
        reimbursement_method,
        methodology_breakout_dict.get("FEE_SCHEDULE"),
    )
    logging.debug(f"Fee Schedule Breakout Prompt for {filename}: {prompt}")
    
    llm_answer_raw = llm_utils.invoke_claude(
        prompt,
        "sonnet_latest",
        filename,
        cache=True,
        instruction=prompt_templates.FEE_SCHEDULE_BREAKOUT_INSTRUCTION(),
        usage_label="FEE_SCHEDULE_BREAKOUT",
    )
    logging.debug(f"LLM Response for {filename}: {llm_answer_raw}")
    
    try:
        # OLD: fs_breakout_dict = string_utils.universal_json_load(llm_answer_raw)
        # NEW:
        fs_breakout_dict = json_parsers.parse_json_dict(llm_answer_raw)
    except json_parsers.JSONParseError as e:
        logging.error(f"Error parsing fee schedule breakout: {e}")
        fs_breakout_dict = {}
    
    return fs_breakout_dict

Update prompt_grouper_breakout():

def prompt_grouper_breakout(
    service: str,
    reimbursement_method: str,
    filename: str,
):
    prompt = prompt_templates.GROUPER_BREAKOUT(service, reimbursement_method)
    logging.debug(f"Grouper Breakout Prompt for {filename}: {prompt}")
    
    llm_answer_raw = llm_utils.invoke_claude(
        prompt,
        "sonnet_latest",
        filename,
        cache=True,
        instruction=prompt_templates.GROUPER_BREAKOUT_INSTRUCTION(),
        usage_label="GROUPER_BREAKOUT",
    )
    logging.debug(f"LLM Response for {filename}: {llm_answer_raw}")
    
    try:
        # OLD: grouper_breakout_dict = string_utils.universal_json_load(llm_answer_raw)
        # NEW:
        grouper_breakout_dict = json_parsers.parse_json_dict(llm_answer_raw)
    except json_parsers.JSONParseError as e:
        logging.error(f"Error parsing grouper breakout: {e}")
        grouper_breakout_dict = {}

    return grouper_breakout_dict

Update prompt_carveout_check():

CRITICAL CHANGE: Currently uses extract_text_from_delimiters(). Must parse JSON dict.

def prompt_carveout_check(
    service_term: str,
    reimb_term: str,
    filename: str,
):
    carveout_prompt = prompt_templates.CARVEOUT_CHECK(service_term, reimb_term)
    logging.debug(
        f"Running carveout check for {filename} with prompt:\n{carveout_prompt}"
    )
    
    llm_answer_raw = llm_utils.invoke_claude(
        carveout_prompt,
        "sonnet_latest",
        filename,
        cache=True,
        instruction=prompt_templates.CARVEOUT_CHECK_INSTRUCTION(),
        usage_label="CARVEOUT_CHECK",
    )

    # OLD:
    # carveout_answer = string_utils.extract_text_from_delimiters(
    #     llm_answer_raw, Delimiter.PIPE
    # )
    
    # NEW:
    try:
        answer_dict = json_parsers.parse_json_dict(llm_answer_raw)
        carveout_answer = answer_dict.get("CASE", "N/A")
    except json_parsers.JSONParseError as e:
        logging.error(f"Failed to parse carveout check answer: {e}")
        carveout_answer = "N/A"
    
    return carveout_answer

Update prompt_special_case_breakout():

def prompt_special_case_breakout(breakout_template, term_answer, filename):
    prompt = breakout_template(term_answer)

    if not isinstance(prompt, str):
        logging.warning(
            f"Expected prompt string, got {type(prompt)} in prompt_special_case_breakout; coercing to str."
        )
        prompt = str(prompt)

    template_name = getattr(breakout_template, "__name__", "SPECIAL_CASE_BREAKOUT")
    instruction_func_name = f"{template_name}_INSTRUCTION"
    instruction_func = getattr(prompt_templates, instruction_func_name, None)

    if instruction_func and callable(instruction_func):
        llm_answer_raw = llm_utils.invoke_claude(
            prompt,
            "sonnet_latest",
            filename,
            cache=True,
            instruction=instruction_func(),
            usage_label=template_name,
        )
    else:
        llm_answer_raw = llm_utils.invoke_claude(
            prompt, "sonnet_latest", filename, usage_label="SPECIAL_CASE_BREAKOUT"
        )
    
    # OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
    # NEW:
    llm_answer_final = json_parsers.parse_json_dict(llm_answer_raw)
    
    return llm_answer_final

Update prompt_full_context():

def prompt_full_context(
    contract_text: str,
    full_context_fields: FieldSet,
    constants: Constants,
    filename: str,
):
    # ... (setup code unchanged)
    
    try:
        claude_answer_raw = llm_utils.invoke_claude(
            full_context_prompt,
            "sonnet_latest",
            filename,
            8192,
            cache=True,
            instruction=prompt_templates.ONE_TO_ONE_MULTI_FIELD_INSTRUCTION(),
        )

        # OLD: full_context_answers_dict = string_utils.universal_json_load(claude_answer_raw)
        # NEW:
        full_context_answers_dict = json_parsers.parse_json_dict(claude_answer_raw)
        
    except Exception as e:
        logging.error(f"Error processing high accuracy fields: {str(e)}")
        full_context_answers_dict = {}
        for field in full_context_fields.list_fields():
            full_context_answers_dict[field] = f"Error: {str(e)}"

    return full_context_answers_dict

Update prompt_dynamic():

def prompt_dynamic(
    text: str, field_prompts: dict, filename: str
):
    # ... (setup code unchanged)
    
    llm_answer_raw = llm_utils.invoke_claude(
        prompt,
        "sonnet_latest",
        filename,
        cache=True,
        instruction=prompt_templates.EXHIBIT_LEVEL_INSTRUCTION(),
    )
    logging.debug(f"Dynamic answer for {filename}: {llm_answer_raw}")
    
    # OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
    # NEW:
    llm_answer_final = json_parsers.parse_json_dict(llm_answer_raw)
    
    return llm_answer_final

Update prompt_dynamic_field_safe():

def prompt_dynamic_field_safe(
    # ... parameters
):
    # ... (prompt construction unchanged)
    
    llm_answer_raw = llm_utils.invoke_claude(
        # ... parameters
    )

    try:
        # OLD: llm_answer_final = string_utils.universal_json_load(llm_answer_raw)
        # NEW:
        llm_answer_final = json_parsers.parse_json_dict(llm_answer_raw)
        return llm_answer_final
    except json_parsers.JSONParseError:
        logging.error(
            f"Failed to parse Dynamic Field: {field_name}. Response: {llm_answer_raw}"
        )
        return {}

Summary: 11 call sites updated in this file

5.3.2 Modify: `src/pipelines/clients/clover/prompts/prompt_calls.py`

Change Type: SAME AS SAAS

Changes: Apply identical updates as in src/pipelines/saas/prompts/prompt_calls.py

Functions to Update:

prompt_exhibit_level_breakout()
prompt_dynamic_primary()
prompt_reimbursement_primary()
All other functions that use universal_json_load or extract_text_from_delimiters

5.3.3 Modify: `src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py`

Change Type: SAME AS SAAS

Changes: Apply identical updates as in src/pipelines/saas/prompts/prompt_calls.py

5.4 Downstream Pipe Split Consumers

5.4.1 Modify: `src/pipelines/shared/postprocessing/aarete_derived.py`

Change Type: LOGIC UPDATE

Current Issue: Code expects pipe-delimited strings from upstream

Changes:

Location 1 (lines 42-49):

# OLD:
all_lob_values = set()
existing_lob = answer_dict.get("AARETE_DERIVED_LOB", "")
if not string_utils.is_empty(existing_lob):
    if "|" in existing_lob:
        all_lob_values.update(
            v.strip()
            for v in existing_lob.split("|")
            if v.strip() and v.strip() != "N/A"
        )
    elif existing_lob.strip() and existing_lob.strip() != "N/A":
        all_lob_values.add(existing_lob.strip())

# NEW (handles both list and pipe-delimited for transition):
all_lob_values = set()
existing_lob = answer_dict.get("AARETE_DERIVED_LOB", "")

if not string_utils.is_empty(existing_lob):
    # Handle JSON list format (new)
    if isinstance(existing_lob, list):
        all_lob_values.update(
            str(v).strip()
            for v in existing_lob
            if str(v).strip() and str(v).strip() != "N/A"
        )
    # Handle pipe-delimited format (legacy)
    elif isinstance(existing_lob, str):
        if "|" in existing_lob:
            all_lob_values.update(
                v.strip()
                for v in existing_lob.split("|")
                if v.strip() and v.strip() != "N/A"
            )
        elif existing_lob.strip() and existing_lob.strip() != "N/A":
            all_lob_values.add(existing_lob.strip())

Apply similar logic to:

Lines 64-74 (program_lob_values)
Lines 90-100 (product_lob_values)
Lines 154-159 (from_field_value_list parsing)

Helper Function (add to file):

def _parse_multi_value_field(value: Any) -> list[str]:
    """
    Parse a multi-value field that may be in JSON list or pipe-delimited format.
    
    Args:
        value: Field value (list, str, or other)
    
    Returns:
        List of parsed values
    """
    if string_utils.is_empty(value):
        return []
    
    # JSON list format (preferred)
    if isinstance(value, list):
        return [str(v).strip() for v in value if not string_utils.is_empty(v)]
    
    # Pipe-delimited format (legacy)
    if isinstance(value, str):
        if "|" in value:
            return [v.strip() for v in value.split("|") if v.strip()]
        elif "," in value:
            return [v.strip() for v in value.split(",") if v.strip()]
        else:
            return [value.strip()] if value.strip() else []
    
    # Fallback: convert to string
    return [str(value).strip()] if str(value).strip() else []

Then refactor all pipe-split logic:

# Instead of:
if "|" in existing_lob:
    all_lob_values.update(v.strip() for v in existing_lob.split("|") ...)

# Use:
values = _parse_multi_value_field(existing_lob)
all_lob_values.update(v for v in values if v != "N/A")

5.4.2 Modify: `src/utils/crosswalk_utils.py`

Change Type: LOGIC UPDATE

Location: Lines 33-42

Changes:

def apply_crosswalk(val: str, mapping: dict[str, str], default: str = "N/A") -> str:
    # ... (initial checks unchanged)
    
    try:
        if "[" in val and "]" in val:
            # Try to parse as literal, but handle any structure returned
            parsed_val = ast.literal_eval(val)
            values = string_utils.flatten_to_strings(parsed_val)
        # NEW: Check if it's already a list object (JSON-parsed)
        elif isinstance(val, list):
            values = string_utils.flatten_to_strings(val)
        # Legacy string parsing
        elif "," in val:
            values = [v.strip() for v in val.split(",")]
        elif "|" in val:
            values = [v.strip() for v in val.split("|")]
        else:
            values = [val.strip()]
    except (ValueError, SyntaxError):
        # If parsing fails, treat val as a regular string
        values = [val.strip()]

    # ... (rest unchanged)

5.4.3 Modify: `src/pipelines/shared/extraction/dynamic_funcs.py`

Change Type: LOGIC UPDATE

Location: Lines 57-63

Changes:

def dynamic_primary(
    # ... parameters
):
    # ... (existing code)
    
    if not string_utils.is_empty(exhibit_text_answer):
        # Special handling for LOB
        if field.field_name == "LOB":
            # OLD: Parse pipe-delimited or comma-separated
            # values_list = [
            #     val.strip()
            #     for val in exhibit_text_answer.replace(",", "|").split("|")
            #     if val.strip()
            # ]
            
            # NEW: Handle both JSON list and pipe-delimited
            if isinstance(exhibit_text_answer, list):
                values_list = [str(val).strip() for val in exhibit_text_answer if val]
            elif isinstance(exhibit_text_answer, str):
                # Legacy: pipe or comma-delimited
                values_list = [
                    val.strip()
                    for val in exhibit_text_answer.replace(",", "|").split("|")
                    if val.strip()
                ]
            else:
                values_list = [str(exhibit_text_answer)]
            
            values_lower = [val.lower() for val in values_list]

            # Check if both Medicare and Medicaid are present
            has_medicare = any("medicare" in val for val in values_lower)
            has_medicaid = any("medicaid" in val for val in values_lower)

            if has_medicare and has_medicaid:
                if "duals" not in values_lower:
                    # OLD: exhibit_text_answer = exhibit_text_answer + " | Duals"
                    # NEW: Add to list or append with pipe
                    if isinstance(exhibit_text_answer, list):
                        exhibit_text_answer.append("Duals")
                    else:
                        exhibit_text_answer = exhibit_text_answer + " | Duals"
                    
                    logging.debug(
                        f"Added 'Duals' to LOB valid values because both Medicare and Medicaid were detected"
                    )

        # Update valid values (convert list to pipe-delimited for Field.update_valid_values)
        if isinstance(exhibit_text_answer, list):
            valid_values_str = "|".join(str(v) for v in exhibit_text_answer) + " | N/A"
        else:
            valid_values_str = str(exhibit_text_answer) + " | N/A"
        
        field.update_valid_values(valid_values_str)
        dynamic_reimbursement_fields.add_field(field)
        dynamic_primary_fields.remove_field(field.field_name)
    
    # ... (rest unchanged)

5.4.4 Modify: `src/pipelines/shared/postprocessing/postprocessing_funcs.py`

Change Type: REVIEW & UPDATE

Action: Search for all .split("|") occurrences and apply similar dual-format handling

5.4.5 Modify: Other Files

Files to update (apply similar dual-format parsing):

src/testbed/dynamic_primary_metrics.py
src/testbed/testbed_utils.py
src/codes/code_funcs.py
src/qc_qa/pipeline.py
src/utils/qa_qc_utils.py
src/parent_child/qc.py
src/pipelines/shared/extraction/tin_npi_funcs.py
src/testbed/model_evaluation_utils.py

Pattern: Add helper function to each file or use shared utility:

def _parse_multi_value_field(value):
    """Parse field that may be list or pipe-delimited string."""
    if isinstance(value, list):
        return value
    if isinstance(value, str) and "|" in value:
        return value.split("|")
    return [value] if value else []

5.5 Other Files

5.5.1 Modify: `src/codes/code_funcs.py`

Change Type: UPDATE universal_json_load USAGE

Action: Replace string_utils.universal_json_load() with json_parsers.parse_json_dict() or parse_json_list()

5.5.2 Modify: `src/pipelines/shared/extraction/tin_npi_funcs.py`

Change Type: UPDATE universal_json_load USAGE

Action: Replace with appropriate parser

5.5.3 Modify: `src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py`

Change Type: UPDATE universal_json_load USAGE

Action: Replace with json_parsers.parse_json_dict()

5.6 Test Files

5.6.1 Create: `src/tests/test_json_parsers.py`

Change Type: NEW FILE

Content:

"""
Unit tests for JSON parsing utilities.
"""

import pytest
from src.utils.json_parsers import (
    parse_json_dict,
    parse_json_list,
    JSONParseError,
)


class TestParseJsonDict:
    """Tests for parse_json_dict function."""

    def test_simple_dict(self):
        """Test parsing simple dictionary."""
        raw = '{"field": "value", "count": 5}'
        result = parse_json_dict(raw)
        assert result == {"field": "value", "count": 5}

    def test_dict_with_reasoning(self):
        """Test parsing dictionary with preceding reasoning text."""
        raw = """
        My reasoning for this answer is based on the contract text...
        
        {"LOB": ["Medicare", "Medicaid"], "NETWORK": "N/A"}
        """
        result = parse_json_dict(raw)
        assert result == {"LOB": ["Medicare", "Medicaid"], "NETWORK": "N/A"}

    def test_nested_dict(self):
        """Test parsing nested dictionary."""
        raw = '{"outer": {"inner": "value"}, "list": [1, 2, 3]}'
        result = parse_json_dict(raw)
        assert result == {"outer": {"inner": "value"}, "list": [1, 2, 3]}

    def test_multiple_dicts_returns_last(self):
        """Test that when multiple dicts present, last one is returned."""
        raw = '{"first": 1} some text {"second": 2}'
        result = parse_json_dict(raw)
        assert result == {"second": 2}

    def test_no_dict_raises_error(self):
        """Test that missing dict raises JSONParseError."""
        raw = "No JSON here"
        with pytest.raises(JSONParseError):
            parse_json_dict(raw)

    def test_list_instead_of_dict_raises_error(self):
        """Test that list input raises JSONParseError."""
        raw = '["item1", "item2"]'
        with pytest.raises(JSONParseError):
            parse_json_dict(raw)

    def test_malformed_json_raises_error(self):
        """Test that malformed JSON raises JSONParseError."""
        raw = '{"incomplete": '
        with pytest.raises(JSONParseError):
            parse_json_dict(raw)

    def test_dict_with_trailing_text(self):
        """Test parsing dict with trailing text."""
        raw = '{"field": "value"} and some trailing text'
        result = parse_json_dict(raw)
        assert result == {"field": "value"}


class TestParseJsonList:
    """Tests for parse_json_list function."""

    def test_simple_list(self):
        """Test parsing simple list."""
        raw = '["item1", "item2", "item3"]'
        result = parse_json_list(raw)
        assert result == ["item1", "item2", "item3"]

    def test_list_with_reasoning(self):
        """Test parsing list with preceding reasoning text."""
        raw = """
        The values found are:
        
        ["Medicare", "Medicaid", "Duals"]
        """
        result = parse_json_list(raw)
        assert result == ["Medicare", "Medicaid", "Duals"]

    def test_empty_list(self):
        """Test parsing empty list."""
        raw = "[]"
        result = parse_json_list(raw)
        assert result == []

    def test_list_of_dicts(self):
        """Test parsing list of dictionaries."""
        raw = '[{"name": "A", "value": 1}, {"name": "B", "value": 2}]'
        result = parse_json_list(raw)
        assert result == [{"name": "A", "value": 1}, {"name": "B", "value": 2}]

    def test_multiple_lists_returns_last(self):
        """Test that when multiple lists present, last one is returned."""
        raw = '["first"] some text ["second", "list"]'
        result = parse_json_list(raw)
        assert result == ["second", "list"]

    def test_no_list_raises_error(self):
        """Test that missing list raises JSONParseError."""
        raw = "No JSON here"
        with pytest.raises(JSONParseError):
            parse_json_list(raw)

    def test_dict_instead_of_list_raises_error(self):
        """Test that dict input raises JSONParseError."""
        raw = '{"field": "value"}'
        with pytest.raises(JSONParseError):
            parse_json_list(raw)


class TestBackwardCompatibility:
    """Tests for deprecated universal_json_load."""

    def test_universal_json_load_still_works(self):
        """Test that deprecated function still works."""
        from src.utils.json_parsers import universal_json_load
        
        raw = '{"field": "value"}'
        with pytest.warns(DeprecationWarning):
            result = universal_json_load(raw)
        assert result == {"field": "value"}


class TestEdgeCases:
    """Tests for edge cases and special scenarios."""

    def test_json_with_escaped_quotes(self):
        """Test parsing JSON with escaped quotes in strings."""
        raw = '{"text": "He said \\"hello\\""}'
        result = parse_json_dict(raw)
        assert result == {"text": 'He said "hello"'}

    def test_json_with_newlines(self):
        """Test parsing JSON with newlines in strings."""
        raw = '{"multiline": "line1\\nline2\\nline3"}'
        result = parse_json_dict(raw)
        assert "line1\nline2\nline3" in str(result.values())

    def test_non_string_input_raises_error(self):
        """Test that non-string input raises TypeError."""
        with pytest.raises(TypeError):
            parse_json_dict(123)
        
        with pytest.raises(TypeError):
            parse_json_list(None)

    def test_unicode_handling(self):
        """Test parsing JSON with unicode characters."""
        raw = '{"text": "café"}'
        result = parse_json_dict(raw)
        assert result == {"text": "café"}

5.6.2 Update: `src/tests/test_string_utils.py`

Change Type: ADD DEPRECATION TESTS

Add:

def test_universal_json_load_deprecated():
    """Test that universal_json_load shows deprecation warning."""
    with pytest.warns(DeprecationWarning):
        result = string_utils.universal_json_load('{"test": "value"}')
    assert result == {"test": "value"}


def test_extract_text_from_delimiters_deprecated():
    """Test that extract_text_from_delimiters shows deprecation warning."""
    from src.constants.delimiters import Delimiter
    
    with pytest.warns(DeprecationWarning):
        result = string_utils.extract_text_from_delimiters(
            "text |value| more text", Delimiter.PIPE
        )
    assert result == "value"

5.6.3 Update: Existing Test Files

Files to update:

src/tests/test_one_to_n.py
src/tests/test_code_funcs.py
src/tests/test_dynamic.py

Action: Update any tests that check for pipe-delimited outputs to check for JSON list/dict outputs instead

6. Migration Plan

6.0 Lessons from Previous Attempt

The previous implementation (commit faf7072) was reverted. This PRD incorporates lessons learned:

Previous Issues → Current Mitigations:

Issue	Mitigation in This PRD
Big-bang deployment caused failures	Phased rollout (6 phases over 5-6 weeks)
No backward compatibility	Dual-format support in all downstream consumers
Unclear format specifications	Comprehensive prompt-output mapping table (4.2.2)
Insufficient testing	Extensive test plan including golden tests (Section 7)
LLM responses didn't match expectations	Clear format instructions with examples in prompts
Hard to rollback	Explicit rollback plan with safe revert points (6.3)
Ambiguous prompt instructions	Four standardized format instruction variables (4.1.1)

Key Improvements:

Format Specification: Four distinct format types (JSON_DICT, JSON_LIST, JSON_DICT_WITH_LISTS, JSON_LIST_OF_DICTS) with clear usage guidelines
Caching Optimization: Format instructions in cached _INSTRUCTION() functions, not in dynamic prompts (saves tokens)
Backward Compatibility: All pipe-split consumers support both formats during transition
Testing Strategy: Golden tests compare semantic equivalence (not byte-for-byte)
Rollback Safety: Can revert at any phase without breaking production
Documentation: Comprehensive prompt mapping table eliminates ambiguity

6.1 Migration Strategy

Approach: Phased rollout with backward compatibility

Phase 1: Foundation (Week 1)

Create src/utils/json_parsers.py with new parsers
Add deprecation warnings to universal_json_load and extract_text_from_delimiters
Write unit tests for new parsers
Document migration guide

Phase 2: Prompt Templates (Week 2)

Update all prompt instruction functions to use JSON output format
Update examples in prompts to show JSON structure
Test prompts with updated instructions (manual QA)

Phase 3: Prompt Calling Layer (Week 2-3)

Update src/pipelines/saas/prompts/prompt_calls.py
- Replace universal_json_load → parse_json_dict / parse_json_list
- Replace extract_text_from_delimiters → parse_json_dict / parse_json_list
- Keep pipe-to-JSON conversion in prompt_dynamic_primary() temporarily
Update client-specific prompt_calls files (clover, bcbs_promise)
Update other files using universal_json_load

Phase 4: Downstream Consumers (Week 3-4)

Update aarete_derived.py to handle both JSON lists and pipe-delimited
Update crosswalk_utils.py to handle both formats
Update dynamic_funcs.py to handle both formats
Add _parse_multi_value_field() helper to shared utilities
Update all other files with pipe-split operations

Phase 5: Testing & Validation (Week 4-5)

Run full integration tests
Run on sample contracts and compare outputs
Golden test validation (ensure outputs match expected format)
Performance testing

Phase 6: Cleanup (Week 5-6)

Remove backward compatibility code from Phase 4 (dual-format handling)
Finalize deprecation of universal_json_load (plan removal date)
Update documentation
Code review and final QA

6.2 Backward Compatibility

6.2.1 During Migration

Dual-Format Support: All downstream consumers will temporarily support both formats:

JSON lists: ["value1", "value2"]
Pipe-delimited strings: "value1|value2"

Example:

def _parse_multi_value_field(value):
    """Handles both JSON list and pipe-delimited during migration."""
    if isinstance(value, list):
        return value  # New format
    if isinstance(value, str) and "|" in value:
        return value.split("|")  # Legacy format
    return [value] if value else []

6.2.2 After Migration

Deprecation Timeline:

Immediate: Add deprecation warnings
+3 months: Remove dual-format support
+6 months: Remove universal_json_load and extract_text_from_delimiters

6.3 Rollback Plan

If issues arise:

Prompt level: Revert prompt instruction changes (restore pipe format instructions)
Parser level: Keep using universal_json_load (backward compatible)
Downstream level: Dual-format support allows partial rollback

Rollback is safe because:

No breaking changes to external APIs
No database schema changes
Dual-format support in downstream code

7. Testing Plan

7.1 Unit Tests

7.1.1 Parser Unit Tests

File: src/tests/test_json_parsers.py

Coverage:

✅ Parse simple dict
✅ Parse dict with reasoning text
✅ Parse nested dict
✅ Parse simple list
✅ Parse list with reasoning text
✅ Parse empty list
✅ Parse list of dicts
✅ Error handling: No JSON found
✅ Error handling: Wrong type (dict vs list)
✅ Error handling: Malformed JSON
✅ Edge cases: Escaped quotes, newlines, unicode

7.1.2 Downstream Consumer Tests

Files to add tests to:

src/tests/test_aarete_derived.py (new or update existing)
src/tests/test_crosswalk_utils.py (new or update existing)
src/tests/test_dynamic_funcs.py (update existing)

Test cases:

Parsing JSON list inputs
Parsing pipe-delimited inputs (backward compatibility)
Mixed scenarios
Empty values
Edge cases (N/A handling, etc.)

7.2 Integration Tests

7.2.1 Prompt-to-Parser Integration

Test: End-to-end prompt execution with new format

Files: src/tests/test_integration_prompts.py (new)

Test cases:

def test_exhibit_level_json_output():
    """Test that exhibit level prompt returns valid JSON dict."""
    # Mock LLM response with JSON dict
    # Call prompt_exhibit_level()
    # Assert result is dict with expected fields


def test_reimbursement_primary_json_output():
    """Test that reimbursement primary prompt returns valid JSON list."""
    # Mock LLM response with JSON list
    # Call prompt_reimbursement_primary()
    # Assert result is list of dicts


def test_dynamic_primary_json_output():
    """Test that dynamic primary prompt returns valid JSON list."""
    # Mock LLM response with JSON list
    # Call prompt_dynamic_primary()
    # Assert result is correctly parsed

7.2.2 Full Pipeline Test

Test: Run complete extraction pipeline on sample contract

File: src/tests/test_end_to_end.py

Test case:

def test_full_pipeline_with_json_parsers():
    """
    Test complete extraction pipeline with JSON parsers.
    Ensures all components work together.
    """
    # Load sample contract
    # Run full extraction (1:N and 1:1)
    # Validate output structure
    # Compare to golden output (allow for equivalent formats)

7.3 Golden Tests

7.3.1 Output Format Validation

Approach: Compare outputs before and after migration

Process:

Run extraction on test contracts with OLD system (pipe-delimited)
Store outputs as "golden" files
Run extraction on same contracts with NEW system (JSON)
Compare outputs semantically (not byte-for-byte)
- Allow ["A", "B"] to match "A|B"
- Allow different whitespace/formatting
- Ensure same fields and values extracted

Tool: Create comparison script scripts/compare_outputs.py

7.4 LLM Prompt Testing

7.4.1 Manual QA

Process:

Test new prompt instructions with real LLM
Verify JSON format compliance
Check for any formatting errors
Validate multi-value fields use arrays

Sample prompts to test:

EXHIBIT_LEVEL
DYNAMIC_PRIMARY_TEXT
CARVEOUT_CHECK
REIMBURSEMENT_PRIMARY
All breakout prompts

7.4.2 Automated Prompt Validation

Script: scripts/validate_prompts.py

Checks:

All instruction functions updated
No lingering pipe format instructions
All examples use JSON format
Multi-value instructions mention arrays

7.5 Performance Testing

7.5.1 Parsing Performance

Test: Compare parsing speed

def test_parsing_performance():
    """Compare universal_json_load vs new parsers."""
    sample_json = '{"field": "value", ...}'  # Large JSON
    
    # Time old parser
    # Time new parser
    # Assert new parser is comparable or faster

7.5.2 End-to-End Performance

Test: Measure total pipeline time with new parsers

Expectation: No significant performance degradation (<5% slower acceptable)

8. Risks and Mitigations

8.1 Risk: Malformed JSON from LLM

Description: LLM may sometimes return malformed JSON or JSON with trailing text

Likelihood: Medium
Impact: High (pipeline failure)

Mitigation:

Robust parser: _extract_json_objects() handles text before/after JSON
Error handling: All parser calls wrapped in try/except
Fallback values: Return empty dict/list on parse failure
Logging: Log all parse failures for analysis
Prompt clarity: Clear instructions and examples reduce formatting errors

Monitoring:

Track parse failure rate
Alert if >5% of prompts fail to parse
Review and improve prompts with high failure rates

8.2 Risk: Partial Outputs

Description: LLM may return incomplete JSON (e.g., context length limits)

Likelihood: Low
Impact: Medium (missing data)

Mitigation:

Parser handles incomplete JSON: Returns what's parseable
Validation: Check for required fields after parsing
Token limits: Ensure prompts stay well under limits
Retry logic: Retry with shorter context if parsing fails

8.3 Risk: Model Verbosity Around JSON

Description: LLM may add extra text or formatting around JSON

Likelihood: Medium
Impact: Low (parser handles it)

Mitigation:

Parser designed for this: _extract_json_objects() finds JSON anywhere in text
Clear instructions: Prompts emphasize clean JSON output
Examples: Show clean JSON without extra formatting

8.4 Risk: Parsing Edge Cases

Description: Special characters, nested structures, etc. may cause issues

Likelihood: Medium
Impact: Medium

Mitigation:

Comprehensive unit tests: Cover edge cases
String escaping: JSON parser handles escaping correctly
Validation: Post-parse validation catches issues
Error logging: Track and fix edge cases as discovered

8.5 Risk: Backward Compatibility Breakage

Description: Changes may break existing code or data pipelines

Likelihood: Low
Impact: High

Mitigation:

Dual-format support: Downstream code handles both formats during transition
Deprecation warnings: Clear warnings before removal
Phased rollout: Test each phase before proceeding
Rollback plan: Can revert to old system if needed

8.6 Risk: Inconsistent Multi-Value Formats

Description: Some fields may still use pipes internally after parsing

Likelihood: Medium
Impact: Medium (confusion, bugs)

Mitigation:

Clear convention: Document when to use lists vs pipe-delimited
Helper functions: Provide _parse_multi_value_field() utility
Code review: Ensure consistency across codebase
Future refactor: Plan to eliminate internal pipe usage (out of scope)

8.7 Risk: Performance Degradation

Description: New parsing may be slower than old regex-based approach

Likelihood: Low
Impact: Low

Mitigation:

Performance tests: Benchmark before and after
Optimize if needed: Profile and optimize hot paths
Expectation: JSON parsing is well-optimized in Python

9. Work Items Checklist

9.1 Implementation Tasks

Phase 1: Foundation ✅ COMPLETED

Create src/utils/json_parsers.py → Renamed to json_utils.py
- Implement parse_json_dict()
- Implement parse_json_list()
- Implement parse_json_dict_or_list() helper
- Add universal_json_load() deprecated alias (in string_utils.py)
Update src/utils/string_utils.py
- Add deprecation warning to universal_json_load()
- Add deprecation warning to extract_text_from_delimiters()
Create src/tests/test_json_utils.py (formerly test_json_parsers.py)
- Write 20+ unit tests for parsers
- Test edge cases
- Test error handling
ARCHITECTURE ENHANCEMENT: Update all prompt templates to return (prompt, parser) tuples
- Add _json_dict_parser() and _json_list_parser() helper functions
- Update 35 prompt template functions with tuple returns
- Add type hints: -> Tuple[str, Callable[[str], dict/list]]
- Update docstrings with Returns section

Phase 2: Prompt Templates ✅ COMPLETED

Update src/prompts/prompt_templates.py
- Add four JSON format instruction constants (after PIPE_FORMAT_INSTRUCTIONS)
  - Add JSON_DICT_FORMAT_INSTRUCTIONS
  - Add JSON_LIST_FORMAT_INSTRUCTIONS
  - Add JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
  - Add JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS
- Deprecate PIPE_FORMAT_INSTRUCTIONS (marked as legacy)
- Update all _INSTRUCTION() functions (cached) to include format instructions:
  - EXHIBIT_LEVEL_INSTRUCTION() + JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
  - DYNAMIC_PRIMARY_TEXT_INSTRUCTION() + JSON_LIST_FORMAT_INSTRUCTIONS
  - CARVEOUT_CHECK_INSTRUCTION() + JSON_LIST_FORMAT_INSTRUCTIONS
  - REIMBURSEMENT_PRIMARY_INSTRUCTION() (already JSON)
  - METHODOLOGY_BREAKOUT_INSTRUCTION() (already JSON)
  - FEE_SCHEDULE_BREAKOUT_INSTRUCTION() (already JSON)
  - GROUPER_BREAKOUT_INSTRUCTION() (already JSON)
  - DYNAMIC_ASSIGNMENT_INSTRUCTION() + JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
  - REIMB_DATES_ASSIGNMENT_INSTRUCTION() + JSON_DICT_FORMAT_INSTRUCTIONS
  - LESSER_OF_DISTRIBUTION_INSTRUCTION() + JSON_LIST_FORMAT_INSTRUCTIONS
  - LESSER_OF_CHECK_INSTRUCTION() + JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
  - LOB_RELATIONSHIP_INSTRUCTION() + JSON_LIST_FORMAT_INSTRUCTIONS
  - ONE_TO_ONE_SINGLE_FIELD_INSTRUCTION() (already uses dict)
  - ONE_TO_ONE_MULTI_FIELD_INSTRUCTION() + JSON_DICT_FORMAT_INSTRUCTIONS
  - TIN_NPI_TEMPLATE_INSTRUCTION() (already uses list)
  - EXHIBIT_LINKAGE_INSTRUCTION() + JSON_LIST_FORMAT_INSTRUCTIONS
  - VALIDATE_REIMBURSEMENTS_INSTRUCTION() + JSON_LIST_FORMAT_INSTRUCTIONS
  - EXHIBIT_TITLE_MATCH_INSTRUCTION() + JSON_LIST_FORMAT_INSTRUCTIONS
  - All special case breakout _INSTRUCTION() functions (already JSON)
- Verified dynamic prompt functions remain clean (NO format instructions added)
- All 35 prompt template functions now return (prompt, parser) tuples
Manual QA testing
- Test each updated prompt with real LLM
- Verify JSON format compliance
- Fix any formatting issues

Commits:

Phase 1: Multiple commits (json_utils.py, test files, deprecation warnings)
Phase 2 Templates: Commit 5d1024e (by user)
Phase 2 Architecture: Commit 98554afc (tuple returns)

Phase 3: Prompt Calling Layer

Update src/pipelines/saas/prompts/prompt_calls.py
- Update prompt_exhibit_level()
- Update prompt_exhibit_level_breakout()
- Update prompt_dynamic_primary() (critical - pipe parsing)
- Update prompt_reimbursement_primary()
- Update prompt_methodology_breakout()
- Update prompt_fee_schedule_breakout()
- Update prompt_grouper_breakout()
- Update prompt_carveout_check() (critical - pipe parsing)
- Update prompt_special_case_breakout()
- Update prompt_full_context()
- Update prompt_dynamic()
- Update prompt_dynamic_field_safe()
Update src/pipelines/clients/clover/prompts/prompt_calls.py
- Apply same updates as saas
Update src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py
- Apply same updates as saas
Update src/codes/code_funcs.py
- Replace universal_json_load calls
Update src/pipelines/shared/extraction/tin_npi_funcs.py
- Replace universal_json_load calls
Update src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py
- Replace universal_json_load calls

Phase 4: Downstream Consumers

Create shared utility function
- Add _parse_multi_value_field() to src/utils/string_utils.py or new module
Update src/pipelines/shared/postprocessing/aarete_derived.py
- Update line 45 (LOB parsing)
- Update line 67 (PROGRAM parsing)
- Update line 93 (PRODUCT parsing)
- Update line 155 (crosswalk parsing)
- Add dual-format support to all pipe-split operations
Update src/utils/crosswalk_utils.py
- Update apply_crosswalk() to handle JSON lists
Update src/pipelines/shared/extraction/dynamic_funcs.py
- Update dynamic_primary() LOB parsing (line 60)
- Handle both JSON lists and pipe-delimited
Update src/pipelines/shared/postprocessing/postprocessing_funcs.py
- Find and update all pipe-split operations
Update remaining files with pipe splits:
- src/testbed/dynamic_primary_metrics.py
- src/testbed/testbed_utils.py
- src/qc_qa/pipeline.py
- src/utils/qa_qc_utils.py
- src/parent_child/qc.py
- src/testbed/model_evaluation_utils.py

Phase 5: Testing & Validation

Create integration test file
- src/tests/test_integration_prompts.py
- Test exhibit level integration
- Test reimbursement primary integration
- Test dynamic primary integration
Update existing test files
- src/tests/test_one_to_n.py
- src/tests/test_code_funcs.py
- src/tests/test_dynamic.py
Create golden test comparison
- Script: scripts/compare_outputs.py
- Run OLD system on test contracts
- Run NEW system on test contracts
- Compare outputs semantically
Run full test suite
- Unit tests pass
- Integration tests pass
- Golden tests pass
Manual testing
- Test on sample contracts
- Verify output quality
- Check for any regressions

Phase 6: Cleanup & Documentation

Remove backward compatibility code
- Remove dual-format support from downstream consumers
- Direct JSON list handling only
Update documentation
- Add migration guide to docs
- Update developer guide with new parsing patterns
- Document JSON output schemas
Code review
- Review all changes
- Check for missed occurrences
- Verify consistency
Final QA
- Run on production-like data
- Performance validation
- Error rate monitoring

9.2 Documentation Tasks

Create migration guide (docs/MIGRATION_GUIDE_JSON_PARSING.md)
Update developer onboarding docs
Add JSON schema examples to prompt documentation
Document new parser usage patterns
Update architecture diagrams
Create deprecation announcement

9.3 Monitoring & Rollout Tasks

Set up monitoring for parse failures
Create dashboard for JSON parsing metrics
Define rollback procedures
Create rollout checklist
Schedule phased deployment
Plan deprecation timeline communication

Appendix A: JSON Schema Examples

A.1 Exhibit Level Response

{
  "LOB": ["Medicare", "Medicaid"],
  "FACILITY_TYPE": "Inpatient",
  "NETWORK": "N/A",
  "EXHIBIT_NOTE": "This exhibit applies to all inpatient services"
}

A.2 Reimbursement Primary Response

[
  {
    "SERVICE_TERM": "Inpatient Services",
    "REIMB_TERM": "110% of Medicare"
  },
  {
    "SERVICE_TERM": "Outpatient Services",
    "REIMB_TERM": "$500 per visit"
  }
]

A.3 Dynamic Primary Response (LOB)

["Medicare", "Medicare Advantage"]

A.4 Carveout Check Response

{
  "CASE": "OUTLIER_TERM"
}

A.5 Methodology Breakout Response

{
  "REIMBURSEMENT_METHOD": "Percent of Charges",
  "PERCENT": "110",
  "BASIS": "Medicare",
  "FEE_SCHEDULE": "N/A"
}

Appendix B: Migration Guide Summary

For Developers

When adding new prompts:

Create both functions - An _INSTRUCTION() function and a dynamic prompt function:

# INSTRUCTION function (cached) - includes format instructions
def NEW_PROMPT_INSTRUCTION() -> str:
    return f"""[OBJECTIVE]
    Do something...
    
    [INSTRUCTIONS]
    Domain-specific rules...
    
    [OUTPUT FORMAT]
    {JSON_DICT_FORMAT_INSTRUCTIONS}"""  # Choose appropriate format variable

# Dynamic prompt (NOT cached) - NO format instructions
def NEW_PROMPT(context, data):
    return f"""[CONTEXT]
    {context}
    
    [DATA]
    {data}"""

Choose correct format instruction variable:
- Single entity, single values → JSON_DICT_FORMAT_INSTRUCTIONS
- Single entity, multi-value fields → JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS
- Multiple entities → JSON_LIST_OF_DICTS_FORMAT_INSTRUCTIONS
- Simple list → JSON_LIST_FORMAT_INSTRUCTIONS
Import and use parser:

from src.utils.json_parsers import parse_json_dict, parse_json_list

result = parse_json_dict(llm_output)  # or parse_json_list

Handle errors: Wrap in try/except with JSONParseError

When consuming LLM outputs:

For single-entity extractions: Expect dict
For multi-entity extractions: Expect list[dict]
For multi-value fields in dicts: Expect each value as list
For simple multi-value: Expect list

CRITICAL - Prompt Caching Optimization:

✅ Format instructions in _INSTRUCTION() functions (cached)
❌ Format instructions in dynamic prompt functions (sent every time)

During transition period:

Use helper: _parse_multi_value_field() to handle both formats
Test with both JSON and pipe-delimited inputs
Add logging for format detection

After migration complete:

Assume all multi-value fields are lists
No pipe splitting needed
JSON parsing only

Appendix C: Glossary

Pipe-delimited: Format using | as separator, e.g., "value1|value2|value3"
JSON dict: JSON object/dictionary, e.g., {"key": "value"}
JSON list: JSON array, e.g., ["item1", "item2"]
Universal JSON load: Legacy parser that extracts JSON from mixed text
Parse JSON dict: New parser specifically for dictionary outputs
Parse JSON list: New parser specifically for list/array outputs
Multi-value field: Field that can contain multiple values (LOB, PROGRAM, etc.)
Downstream consumer: Code that processes parsed LLM outputs
Dual-format support: Temporary ability to handle both JSON and pipe-delimited

End of PRD

101 KiB Raw Blame History

PRD: Standardize LLM Output Formats - Eliminate Pipe-Delimited Parsing

Implementation Progress

Executive Summary

Table of Contents

1. Background and Motivation

1.1 Problem Statement

1.2 Benefits of Standardization

1.3 Why Now?

2. Non-Goals

2.1 Out of Scope

2.2 Future Work (Not This PRD)

3. Current State Analysis

3.0 Previous Implementation Attempt

3.0.1 Commit faf7072 (Reverted)

3.1 Pipe-Delimited Usage

3.1.1 Prompt Template Constant

3.1.2 Extraction Function

3.2 Universal JSON Load Usage

3.3 Downstream Pipe Splitting

3.3.1 aarete_derived.py

3.3.2 crosswalk_utils.py

3.3.3 Other Files with Pipe Splits

3.4 Prompt Templates Mentioning Pipes

3.4.1 Instructions with Pipe References

4. Technical Design

4.1 JSON Output Instruction Variables

4.1.1 Standardized Format Instructions

4.2 Prompt Output Format Mapping

4.2.1 Output Format Classification

4.2.2 Comprehensive Prompt Mapping

4.2.3 Format Selection Logic

4.3 New Parser Architecture

4.3.1 Parser Function Specifications

4.1.2 Prompt Template Parser Coupling (IMPLEMENTED)

4.2 JSON Schema Expectations

4.2.1 Dictionary Outputs

4.2.2 List Outputs

4.2.3 Multi-Value Fields

5. File-by-File Change List

5.1 Core Utilities

5.1.1 Create: src/utils/json_parsers.py

5.1.2 Modify: src/utils/string_utils.py

5.1.3 Modify: src/constants/delimiters.py

5.2 Prompt Templates

5.2.1 Modify: src/prompts/prompt_templates.py

5.3 Prompt Calling Layer

5.3.1 Modify: src/pipelines/saas/prompts/prompt_calls.py

5.3.2 Modify: src/pipelines/clients/clover/prompts/prompt_calls.py

5.3.3 Modify: src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py

5.4 Downstream Pipe Split Consumers

5.4.1 Modify: src/pipelines/shared/postprocessing/aarete_derived.py

5.4.2 Modify: src/utils/crosswalk_utils.py

5.4.3 Modify: src/pipelines/shared/extraction/dynamic_funcs.py

5.4.4 Modify: src/pipelines/shared/postprocessing/postprocessing_funcs.py

5.4.5 Modify: Other Files

5.5 Other Files

5.5.1 Modify: src/codes/code_funcs.py

5.5.2 Modify: src/pipelines/shared/extraction/tin_npi_funcs.py

5.5.3 Modify: src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py

5.6 Test Files

5.6.1 Create: src/tests/test_json_parsers.py

5.6.2 Update: src/tests/test_string_utils.py

5.6.3 Update: Existing Test Files

6. Migration Plan

6.0 Lessons from Previous Attempt

6.1 Migration Strategy

Phase 1: Foundation (Week 1)

Phase 2: Prompt Templates (Week 2)

Phase 3: Prompt Calling Layer (Week 2-3)

Phase 4: Downstream Consumers (Week 3-4)

Phase 5: Testing & Validation (Week 4-5)

Phase 6: Cleanup (Week 5-6)

6.2 Backward Compatibility

6.2.1 During Migration

6.2.2 After Migration

6.3 Rollback Plan

7. Testing Plan

7.1 Unit Tests

7.1.1 Parser Unit Tests

101 KiB

Raw Blame History

3.0.1 Commit `faf7072` (Reverted)

5.1.1 Create: `src/utils/json_parsers.py`

5.1.2 Modify: `src/utils/string_utils.py`

5.1.3 Modify: `src/constants/delimiters.py`

5.2.1 Modify: `src/prompts/prompt_templates.py`

5.3.1 Modify: `src/pipelines/saas/prompts/prompt_calls.py`

5.3.2 Modify: `src/pipelines/clients/clover/prompts/prompt_calls.py`

5.3.3 Modify: `src/pipelines/clients/bcbs_promise/prompts/prompt_calls.py`

5.4.1 Modify: `src/pipelines/shared/postprocessing/aarete_derived.py`

5.4.2 Modify: `src/utils/crosswalk_utils.py`

5.4.3 Modify: `src/pipelines/shared/extraction/dynamic_funcs.py`

5.4.4 Modify: `src/pipelines/shared/postprocessing/postprocessing_funcs.py`

5.5.1 Modify: `src/codes/code_funcs.py`

5.5.2 Modify: `src/pipelines/shared/extraction/tin_npi_funcs.py`

5.5.3 Modify: `src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py`

5.6.1 Create: `src/tests/test_json_parsers.py`

5.6.2 Update: `src/tests/test_string_utils.py`