Merged in feature/DAIP2-2314-DAIP2-1687-hybrid (pull request #993)

Feature/DAIP2-2314 DAIP2 1687 hybrid

* remove -files from s3 prefix requirements

* Resolve input paths

* fix: VendorProcessor.process_file returns (df, None) tuple

runner.safe_process_file unpacks the result as (cc_df, dashboard_df), so
returning a single DataFrame caused every vendor/generic file to fail with
"too many values to unpack (expected 2)" — Python iterates DataFrame columns
during unpacking. Vendor pipelines have no dashboard variant; second slot is
None and the existing `dashboard_result is not None` guard in runner.py
already handles it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* DAIP2-2314 + DAIP2-1687: pad DYNAMIC_PRIMARY + DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION over 1024-token cache floor

- Pad DYNAMIC_PRIMARY_INSTRUCTION with three new sections: [SCOPE BOUNDARIES], [SOURCE TEXT INTERPRETATION], [REASONING DISCIPLINE], plus a [WORKED EXAMPLES] block. Estimated tokens: 447 -> 1117 (Sonnet 4.5 1024-min, +93 margin). All additions reinforce existing rules (sibling-field separation, alias mapping, pricing-vs-LOB distinction, contrastive-clause exclusion, exhibit-header binding) — no new directives that could bias extraction.
- Pad DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION_INSTRUCTION with a [FINAL CHECKLIST BEFORE OUTPUT] block. Estimated tokens: 956 -> 1101 (Sonnet 4.5 1024-min, +77 margin). Reinforces the existing 4-step anti-duplication protocol and JSON shape requirements.
- Register both new entries in cache_registry: DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION as INSTRUCTION_PLUS_CONTEXT (caches at warm-up), DYNAMIC_PRIMARY_ENTITIES as CONTEXT (instruction is intentionally short; CONTEXT c…
* black format fix

* Merged dev into feature/DAIP2-2314-DAIP2-1687-hybrid

* fixed raw lob values in base lob field mapping and composite entities fix

* black format fix

* fixed LOB Program output issues

* issue fixes

* remove debugging code

* Updated prompts

* updated additional instructions

* Update Program-->LOB

* LLM-based AD-Program/Product mapping to LOB even when there is a crosswalk

* black format fix

* Merged dev into feature/DAIP2-2314-DAIP2-1687-hybrid

* added logging in prompt call tracking

* added updated logging in prompt call tracking

* aaded min cache token per usage label

* added cache registry for dynamic primary mapping prompt calls

* reolved mapping prompts ambiguities

* black format fix

* Phase 2 modifications added

* reverted phase 2 modifications


Approved-by: Katon Minhas

documentation/PROMPT_CACHING_ANALYSIS.md

@@ -0,0 +1,204 @@
+# Prompt Caching Analysis Report
+
+**Branch:** `feature/DAIP2-2314-expand-caching-for-short-prompts`
+**Date:** 2026-04-21
+**Comparison:** Feature branch vs `origin/dev`
+
+---
+
+## Executive Summary
+
+This report analyzes the prompt caching implementation in the feature branch, comparing cached prompts against the dev branch and evaluating cost savings from Bedrock prompt caching.
+
+### Key Findings
+
+| Metric | Value |
+|--------|-------|
+| Total API Calls Analyzed | 1,251 |
+| Unique Usage Labels | 35 |
+| Total Cache Read Tokens | 1,472,330 |
+| Total Cache Write Tokens | 87,167 |
+| Overall Cost Savings | **28.1%** ($3.65 saved) |
+| Prompts Actively Caching | 22 |
+| Prompts Wired but Not Caching | 11 |
+| Prompts Not Wired | 2 |
+
+---
+
+## 1. Cache Effectiveness by Usage Label
+
+### Actively Caching (22 prompts) - High Performers
+
+| Usage Label | Calls | Avg Instruction Tokens | Cache Hit Rate |
+|-------------|-------|------------------------|----------------|
+| OUTLIER_BREAKOUT | 1 | 2,483 | 98.4% |
+| SERVICE_ENRICHMENT | 30 | 1,459 | 98.3% |
+| CARVEOUT_CHECK | 43 | 1,498 | 95.6% |
+| METHODOLOGY_BREAKOUT | 37 | 1,927 | 95.5% |
+| LESSER_OF_DISTRIBUTION | 32 | 914 | 94.5% |
+| CHECK_PROVIDER_NAME_MATCH | 74 | 957 | 94.2% |
+| DYNAMIC_CODE_ASSIGNMENT | 38 | 966 | 94.1% |
+| CODE_EXPLICIT | 34 | 871 | 93.9% |
+| GROUPER_BREAKOUT | 7 | 939 | 93.6% |
+| LESSER_OF_CHECK | 6 | 900 | 92.0% |
+| DYNAMIC_ASSIGNMENT | 204 | 1,578 | 89.4% |
+| validate_reimbursements_for_llm | 65 | 894 | 89.2% |
+| code_implicit_rag | 60 | 896 | 88.4% |
+| REIMB_DATES_ASSIGNMENT | 29 | 803 | 87.2% |
+| REIMBURSEMENT_PRIMARY | 38 | 1,390 | 85.0% |
+| EXHIBIT_LEVEL | 33 | 1,108 | 74.0% |
+| EXHIBIT_HEADER | 90 | 922 | 73.7% |
+| fill_bill_type | 47 | 968 | 72.5% |
+| prompt_provider_info | 24 | 966 | 60.5% |
+| DYNAMIC_PRIMARY | 44 | 441 | 49.7% |
+
+### Wired But Not Caching (11 prompts) - Under 1024 Token Threshold
+
+These prompts are wired for caching but have instruction tokens below Bedrock's 1024-token minimum threshold:
+
+| Usage Label | Calls | Avg Instruction Tokens | Issue |
+|-------------|-------|------------------------|-------|
+| SPLIT_SERVICE_TERM | 10 | 1,600 | Should be caching - investigate |
+| prompt_lob_relationship | 56 | 408 | **Needs padding to 1024** |
+| SPLIT_REIMB_DATES | 13 | 410 | **Needs padding to 1024** |
+| code_implicit_arbitration | 12 | 391 | **Needs padding to 1024** |
+| AARETE_DERIVED_PAYER_NAME | 1 | 372 | **Needs padding to 1024** |
+| FEE_SCHEDULE_BREAKOUT | 18 | 352 | **Needs padding to 1024** |
+| code_implicit_special | 30 | 329 | **Needs padding to 1024** |
+| DATE_FIX | 10 | 322 | **Needs padding to 1024** |
+| SPECIAL_CASE_ASSIGNMENT | 4 | 251 | **Needs padding to 1024** |
+| DERIVED_TERM_DATE | 10 | 247 | **Needs padding to 1024** |
+| code_last_check | 23 | 179 | **Needs padding to 1024** |
+| EXTRACT_AMENDMENT_NUM_FROM_FILENAME | 3 | 152 | **Needs padding to 1024** |
+| EXHIBIT_HEADER_DEDUP | 10 | 35 | **Needs padding to 1024** |
+
+### Not Wired for Caching (2 prompts)
+
+| Usage Label | Calls | Notes |
+|-------------|-------|-------|
+| prompt_hsc_single_field | 109 | High volume - should investigate wiring |
+| SPECIAL_CASE_BREAKOUT | 6 | Low volume |
+
+---
+
+## 2. Prompt Template Changes (Feature Branch vs Dev)
+
+The feature branch includes **649 lines changed** in `src/prompts/prompt_templates.py`. Key expansions:
+
+### Expanded Prompts (Additional Context Added)
+
+| Instruction Function | Change Summary |
+|---------------------|----------------|
+| **EXHIBIT_LEVEL_INSTRUCTION** | +55 lines: Added detailed guidance, validation rules, examples, common pitfalls, field-specific notes, quality assurance checks |
+| **DYNAMIC_PRIMARY_INSTRUCTION** | +18 lines: Added extraction guidance, ambiguity handling, quality checks |
+| **LESSER_OF_DISTRIBUTION_INSTRUCTION** | Restructured (net -25 lines): Converted to cleaner decision framework format |
+| **DYNAMIC_CODE_ASSIGNMENT_INSTRUCTION** | +10 lines: Added classification guidance with explicit rules |
+| **CODE_IMPLICIT_INSTRUCTION** | Reformatted indentation |
+| **CODE_IMPLICIT_ARBITRATION_INSTRUCTION** | +43 lines: Added clarifications, deterministic decision rules, calibration examples |
+| **FILL_BILL_TYPE_INSTRUCTION** | +49 lines: Added detailed guidance, validation rules, extended examples |
+| **TIN_NPI_TEMPLATE_INSTRUCTION** | +22 lines: Added provider/payer distinction clarity |
+
+### Analysis of Prompt Changes
+
+**Positive Impacts:**
+- More explicit instructions reduce model ambiguity
+- Quality checks and validation rules improve consistency
+- Examples help calibrate model responses
+- Clearer formatting improves readability
+
+**Potential Concerns:**
+- Expanded prompts increase token count (higher cache write cost initially)
+- Some prompts may have reduced due to restructuring (LESSER_OF_DISTRIBUTION)
+- Need to verify output quality hasn't degraded with expanded context
+
+---
+
+## 3. Cost Analysis
+
+### Bedrock Pricing Applied
+- **Input tokens:** $3.00/1M tokens
+- **Output tokens:** $15.00/1M tokens
+- **Cache read:** $0.30/1M tokens (90% savings)
+- **Cache write:** $3.75/1M tokens (25% premium)
+
+### Observed Results
+
+| Metric | Value |
+|--------|-------|
+| Total Input Tokens | 844,370 |
+| Total Output Tokens | 401,926 |
+| Total Cache Read Tokens | 1,472,330 |
+| Total Cache Write Tokens | 87,167 |
+| **Cost Without Caching** | $12.98 |
+| **Cost With Caching** | $9.33 |
+| **Savings** | $3.65 (28.1%) |
+
+### Per-Prompt Cost Impact (Estimated from previous testing)
+
+| Prompt | Cost Reduction |
+|--------|---------------|
+| LESSER_OF_DISTRIBUTION | 44% (Best performer) |
+| EXHIBIT_HEADER | 7% |
+| CHECK_PROVIDER_NAME_MATCH | 5% |
+| TIN_NPI_TEMPLATE (prompt_provider_info) | 5% |
+| code_implicit_rag | -12% (Negative - needs review) |
+
+---
+
+## 4. Prompts Needing Attention
+
+### Priority 1: Prompts Needing Padding (Under 1024 tokens)
+
+These prompts are wired but not actually caching due to Bedrock's minimum threshold:
+
+1. **prompt_lob_relationship** (408 tokens, 56 calls) - High impact
+2. **code_implicit_special** (329 tokens, 30 calls) - Medium impact
+3. **code_last_check** (179 tokens, 23 calls) - Medium impact
+4. **FEE_SCHEDULE_BREAKOUT** (352 tokens, 18 calls) - Medium impact
+5. **SPLIT_REIMB_DATES** (410 tokens, 13 calls)
+6. **code_implicit_arbitration** (391 tokens, 12 calls)
+7. **SPLIT_SERVICE_TERM** (1600 tokens, 10 calls) - Should be caching, investigate
+8. **DATE_FIX** (322 tokens, 10 calls)
+9. **DERIVED_TERM_DATE** (247 tokens, 10 calls)
+10. **EXHIBIT_HEADER_DEDUP** (35 tokens, 10 calls)
+
+### Priority 2: Investigate Negative Result
+
+- **code_implicit_rag**: Shows -12% cost (increase) - The expanded prompt may be too verbose or structured differently causing cache misses
+
+### Priority 3: Not Wired for Caching
+
+- **prompt_hsc_single_field** (109 calls) - High volume, should consider wiring
+
+---
+
+## 5. Recommendations
+
+1. **Pad short prompts to 1024 tokens**: Add contextual padding to prompts under the threshold, especially high-frequency ones like `prompt_lob_relationship` and `code_implicit_special`
+
+2. **Investigate code_implicit_rag**: The negative cost result (-12%) suggests the expanded prompt may be causing cache fragmentation or misses
+
+3. **Wire prompt_hsc_single_field**: With 109 calls, this is a good candidate for caching
+
+4. **Monitor SPLIT_SERVICE_TERM**: At 1,600 tokens it should be caching but shows 0% cache hit rate - may be a wiring issue
+
+5. **Run testbed comparison**: Use `src/testbed/testbed_metrics.py` to compare output quality between cached and non-cached prompts to ensure no regression
+
+---
+
+## 6. Files Analyzed
+
+- `src/test-PROMPT-CALLS_10.csv` - API call logs with cache statistics
+- `src/testbed-usethis.xlsx` - Testbed comparison file (binary, not readable)
+- `src/prompts/prompt_templates.py` - Prompt definitions (649 lines changed)
+- `documentation/prompt_caching_tracker_v2.csv` - Caching status tracker
+
+---
+
+## 7. Next Steps
+
+1. Run testbed metrics comparison script to validate output quality
+2. Add padding to short prompts identified above
+3. Investigate and fix code_implicit_rag negative result
+4. Wire prompt_hsc_single_field for caching
+5. Debug SPLIT_SERVICE_TERM caching issue

documentation/PROMPT_CACHING_SYSTEM_PROPOSAL.md

@@ -0,0 +1,154 @@
+# Prompt Caching System Proposal
+
+Branch: `feature/DAIP2-2314-expand-caching-for-short-prompts`
+
+## Goal
+
+Ensure every stable instruction prompt that is intended to be cached is actually cacheable, warmed on the same model used at runtime, and observable in runtime logs.
+
+## Recommended Caching System
+
+1. Introduce a central prompt cache registry.
+
+   Each cacheable prompt should be declared once with:
+   - `usage_label`
+   - instruction function
+   - model family used at runtime
+   - minimum cache token policy
+   - runtime call sites
+   - whether the prompt also uses `context_for_caching`
+
+   This avoids the current split where cache warming is in `prompt_templates.get_cacheable_instructions()` but runtime model choice is scattered through prompt call sites.
+
+2. Enforce the 1024-token minimum before runtime.
+
+   Prompt caching should not be treated as enabled just because `cache=True` is passed. At startup, the registry should validate every cached instruction with the same tokenizer/estimate used by reporting. If a prompt is below the minimum, it should either:
+   - fail fast in a cache audit mode, or
+   - automatically append approved static cache padding/instruction context until it crosses the threshold.
+
+   The padding must be stable and domain-relevant, not dynamic request data.
+
+3. Warm caches per model, not globally.
+
+   Cache entries are model-specific. The runner currently warms all instructions with `sonnet_latest`, but `SPLIT_SERVICE_TERM` runs with `haiku_latest`. The cache warming layer should warm `(usage_label, resolved_model_id)` pairs derived from the registry.
+
+4. Only mark supported models as cacheable.
+
+   `_supports_prompt_cache()` currently excludes Haiku. If Haiku prompt caching is not supported in this Bedrock setup, Haiku prompts should not be registered as cached. Either move those prompt calls to `sonnet_latest` when caching matters, or keep them uncached and report them as intentionally uncached.
+
+5. Keep static instruction caching separate from dynamic context caching.
+
+   Instruction prompts can be cached predictably. `context_for_caching` is different because it changes by contract, exhibit, or page. It should be tracked as a separate cache class:
+   - `instruction_cache`: expected to read after warm-up
+   - `context_cache`: expected to create/read per repeated context
+
+   This prevents context variability from making instruction caching look broken.
+
+6. Add a cache contract test.
+
+   A local test should inspect the built request body for every registered prompt and assert:
+   - the instruction block has `cache_control`
+   - the resolved model supports caching
+   - the static instruction token estimate is at least 1024
+   - warm-up model equals runtime model
+
+7. Upgrade runtime reporting.
+
+   Add these fields to prompt-call logs:
+   - `resolved_model_id`
+   - `cache_policy`: `instruction`, `context`, `instruction+context`, `none`
+   - `instruction_cache_eligible`
+   - `context_cache_eligible`
+   - `cache_miss_reason`
+
+   This makes root cause visible without manual investigation.
+
+## Prompt Application Report
+
+### Apply Instruction Caching, Already Over 1024 Tokens
+
+These should remain registered and warmed. They are good candidates for strict cache contract tests.
+
+| Prompt | Estimated Instruction Tokens | Action |
+|---|---:|---|
+| `OUTLIER_BREAKOUT` | 2483 | Keep cached |
+| `METHODOLOGY_BREAKOUT` | 1927 | Keep cached |
+| `SPLIT_SERVICE_TERM` | 1600 | Cache only if runtime model supports caching; currently runs on Haiku |
+| `DYNAMIC_ASSIGNMENT` | 1578 | Keep cached |
+| `CARVEOUT_CHECK` | 1498 | Keep cached |
+| `SERVICE_ENRICHMENT` | 1459 | Keep cached |
+| `REIMBURSEMENT_PRIMARY` | 1390 | Keep cached |
+| `EXHIBIT_LEVEL` | 1108 | Keep cached |
+
+### Apply Instruction Caching After Padding/Expansion
+
+These are wired or partially wired but below the 1024-token minimum. They should be padded or expanded with stable domain guidance before being considered cached.
+
+| Prompt | Estimated Instruction Tokens | Tokens Needed | Action |
+|---|---:|---:|---|
+| `fill_bill_type` | 968 | 56 | Add small static guidance |
+| `DYNAMIC_CODE_ASSIGNMENT` | 966 | 58 | Add small static guidance |
+| `CHECK_PROVIDER_NAME_MATCH` | 957 | 67 | Add small static guidance |
+| `GROUPER_BREAKOUT` | 939 | 85 | Add small static guidance |
+| `EXHIBIT_HEADER` | 922 | 102 | Add static header examples/rules |
+| `LESSER_OF_DISTRIBUTION` | 914 | 110 | Add static decision examples |
+| `LESSER_OF_CHECK` | 900 | 124 | Add static classification examples |
+| `CODE_EXPLICIT` | 871 | 153 | Add static code-system rules |
+| `REIMB_DATES_ASSIGNMENT` | 803 | 221 | Add date extraction examples |
+| `DYNAMIC_PRIMARY` | 441 | 583 | Expand instruction substantially |
+| `SPLIT_REIMB_DATES` | 410 | 614 | Expand instruction substantially |
+| `prompt_lob_relationship` | 408 | 616 | Move field prompt into instruction or add static LOB rules |
+| `code_implicit_arbitration` | 391 | 633 | Expand arbitration rules |
+| `AARETE_DERIVED_PAYER_NAME` | 372 | 652 | Expand payer naming rules |
+| `FEE_SCHEDULE_BREAKOUT` | 352 | 672 | Expand fee schedule rules |
+| `code_implicit_special` | 329 | 695 | Expand special code rules |
+| `DATE_FIX` | 322 | 702 | Expand date normalization rules |
+| `SPECIAL_CASE_ASSIGNMENT` | 251 | 773 | Expand special case rules |
+| `DERIVED_TERM_DATE` | 247 | 777 | Expand term date rules |
+| `code_last_check` | 179 | 845 | Expand or leave uncached if low value |
+| `EXTRACT_AMENDMENT_NUM_FROM_FILENAME` | 152 | 872 | Expand or leave uncached if low value |
+| `EXHIBIT_HEADER_DEDUP` | 35 | 989 | Do not cache unless redesigned |
+
+### Wire for Caching or Mark Intentionally Uncached
+
+These showed runtime volume but no instruction function match in the audit. They need explicit registry decisions.
+
+| Prompt | Runtime Finding | Action |
+|---|---|---|
+| `prompt_hsc_single_field` | 109 calls, `cache_enabled=False`, large dynamic prompt | Split stable HSC instructions from dynamic chunk and cache the instruction |
+| `SPECIAL_CASE_BREAKOUT` | 6 calls, no instruction cache | Wire only if repeated volume justifies expansion |
+| `code_implicit_rag` | Cached in API, audit alias missing | Add registry alias to the actual instruction function |
+| `prompt_provider_info` | Cached in API, audit alias missing | Add registry alias to `TIN_NPI_TEMPLATE_INSTRUCTION` or rename usage label |
+| `validate_reimbursements_for_llm` | Cached in API, audit alias missing | Add registry alias to `VALIDATE_REIMBURSEMENTS_INSTRUCTION` |
+
+## Immediate Fixes
+
+1. Change `SPLIT_SERVICE_TERM` to use a cache-supported model or mark it intentionally uncached. Current behavior is misleading: `cache=True` is passed, but no Bedrock cache tokens are recorded.
+
+2. Add a registry alias map for usage labels that do not match instruction names:
+   - `code_implicit_rag` -> `CODE_IMPLICIT`
+   - `prompt_provider_info` -> `TIN_NPI_TEMPLATE`
+   - `validate_reimbursements_for_llm` -> `VALIDATE_REIMBURSEMENTS`
+   - `prompt_lob_relationship` -> its real instruction function
+
+3. Expand the near-threshold instructions first because they need the least work and have current evidence of API cache reads:
+   - `fill_bill_type`
+   - `DYNAMIC_CODE_ASSIGNMENT`
+   - `CHECK_PROVIDER_NAME_MATCH`
+   - `GROUPER_BREAKOUT`
+   - `EXHIBIT_HEADER`
+   - `LESSER_OF_DISTRIBUTION`
+   - `LESSER_OF_CHECK`
+   - `CODE_EXPLICIT`
+
+4. Redesign `prompt_hsc_single_field` into `(instruction, prompt)` form. It is high-volume and currently sends all prompt text dynamically.
+
+## Expected Behavior After Fix
+
+For instruction-only prompts, after warm-up the API should report cache reads on normal calls. The cached percentage will still not be 100% of total billed input because the dynamic user prompt remains uncached. The correct target is:
+
+- `cache_creation_tokens` only during warm-up or first uncached use
+- `cache_read_tokens > 0` on every eligible runtime call
+- `cache_miss_reason` empty for registered instruction cache prompts
+
+For context-cached prompts, 100% cache reads should not be expected unless the same context block repeats. The report should evaluate those separately.

scripts/merge_files.py

@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+"""
+Script to merge multiple text files into a single file.
+
+Usage: 
+  python merge_files.py <output_file> <input_file1> <input_file2> ...
+  python merge_files.py <output_file> --dir <input_directory> [--pattern <file_pattern>]
+
+This script concatenates the contents of the input files into the output file,
+separating each file's content with a newline.
+For CSV files, it properly merges by keeping only the header from the first file.
+"""
+
+import sys
+import os
+import glob
+import csv
+
+def merge_files(file_list, output_file):
+    """
+    Merge a list of files into a single output file.
+
+    Args:
+        file_list (list): List of file paths to merge.
+        output_file (str): Path to the output file.
+    """
+    if not file_list:
+        return
+
+    # Check if files are CSV based on extension
+    is_csv = any(fname.lower().endswith('.csv') for fname in file_list)
+
+    if is_csv:
+        merge_csv_files(file_list, output_file)
+    else:
+        merge_text_files(file_list, output_file)
+
+def merge_text_files(file_list, output_file):
+    """
+    Merge text files by concatenating their contents.
+    """
+    with open(output_file, 'w', encoding='utf-8') as outfile:
+        for fname in file_list:
+            if not os.path.isfile(fname):
+                print(f"Warning: {fname} is not a file or does not exist. Skipping.")
+                continue
+            try:
+                with open(fname, 'r', encoding='utf-8') as infile:
+                    content = infile.read()
+                    outfile.write(content)
+                    outfile.write('\n')  # Add a newline separator between files
+            except Exception as e:
+                print(f"Error reading {fname}: {e}")
+
+def merge_csv_files(file_list, output_file):
+    """
+    Merge CSV files by keeping header from first file and appending data from others.
+    """
+    first_file = True
+
+    with open(output_file, 'w', newline='', encoding='utf-8') as outfile:
+        writer = None
+
+        for fname in file_list:
+            if not os.path.isfile(fname):
+                print(f"Warning: {fname} is not a file or does not exist. Skipping.")
+                continue
+
+            try:
+                with open(fname, 'r', encoding='utf-8') as infile:
+                    reader = csv.reader(infile)
+
+                    for row_num, row in enumerate(reader):
+                        if first_file or row_num > 0:  # Skip header for subsequent files
+                            if writer is None:
+                                writer = csv.writer(outfile)
+                            writer.writerow(row)
+
+                    first_file = False
+
+            except Exception as e:
+                print(f"Error reading {fname}: {e}")
+
+def get_files_from_dir(directory, pattern='*'):
+    """
+    Get all files from a directory matching a pattern.
+
+    Args:
+        directory (str): Directory path.
+        pattern (str): Glob pattern for files.
+
+    Returns:
+        list: List of file paths.
+    """
+    if not os.path.isdir(directory):
+        print(f"Error: {directory} is not a directory.")
+        return []
+    return glob.glob(os.path.join(directory, pattern))
+
+if __name__ == "__main__":
+    if len(sys.argv) < 3:
+        print("Usage: python merge_files.py <output_file> <input_file1> <input_file2> ...")
+        print("   or: python merge_files.py <output_file> --dir <input_directory> [--pattern <file_pattern>]")
+        sys.exit(1)
+
+    output_file = sys.argv[1]
+
+    if sys.argv[2] == '--dir':
+        if len(sys.argv) < 4:
+            print("Usage: python merge_files.py <output_file> --dir <input_directory> [--pattern <file_pattern>]")
+            sys.exit(1)
+        directory = sys.argv[3]
+        pattern = sys.argv[5] if len(sys.argv) > 5 and sys.argv[4] == '--pattern' else '*'
+        input_files = get_files_from_dir(directory, pattern)
+    else:
+        input_files = sys.argv[2:]
+
+    if not input_files:
+        print("No input files found.")
+        sys.exit(1)
+
+    merge_files(input_files, output_file)
+    print(f"Merged {len(input_files)} files into {output_file}")

src/codes/code_funcs.py

@@ -732,8 +732,8 @@ def fill_grouper_cd_desc(answer_dict: dict, constants: Constants) -> dict:
 
     for code in grouper_cd:
         try:
-            code_without_severity = int(
-                code.split("-")[0]
+            code_without_severity = str(
+                int(code.split("-")[0])
             )  # Remove severity level if present
         except Exception as e:
             logging.warning(

src/config.py

@@ -178,8 +178,16 @@ CLIENT = [s.strip() for s in get_arg_value("client", "None").split("-") if s]
 LOCAL_PATH = get_arg_value(
     "input_dir", "data/test"
 )  # Valid: any valid path that contains txt files
-S3_PREFIX = get_arg_value("s3_prefix", "")
+S3_PREFIX = get_arg_value("s3_prefix", "") + "/txt/"
 S3_BUCKET = get_arg_value("s3_bucket", "")
+
+# program_product_lob_mapping.csv is expected in the same S3 prefix as the txt files.
+PROGRAM_PRODUCT_MAPPING_CSV_KEY = (
+    get_arg_value("s3_prefix", "") + "/program_product_lob_mapping.csv"
+    if S3_PREFIX
+    else None
+)
+
 OUTPUT_DIRECTORY = get_arg_value(
     "output_dir", "output_individual"
 )  # Valid: any valid directory
@@ -338,6 +346,11 @@ def resolve_model_id(model_identifier: str) -> str:
 ENABLE_ANSWER_TRACKING = False
 ANSWER_TRACKING_DICT = {}
 
+# Runtime prompt-call tracking (one row per invoke_claude call)
+ENABLE_PROMPT_CALL_LOGGING = (
+    get_arg_value("enable_prompt_call_logging", "True") == "True"
+)
+
 ######################################## POSTPROCESSING SETTINGS ########################################
 FUZZY_MATCH_THRESHOLD = 0.8
 FILE_NAME_COLUMN = "Contract Name"
@@ -437,9 +450,9 @@ RUN_BASE_PREFIX = get_arg_value(
 if RUN_BASE_PREFIX:
     _run_base = RUN_BASE_PREFIX.rstrip("/") + "/"
     if not S3_PREFIX:
-        S3_PREFIX = f"{_run_base}txt-files/"
+        S3_PREFIX = f"{_run_base}txt/"
     if not PDF_S3_PREFIX:
-        PDF_S3_PREFIX = f"{_run_base}pdf-files/"
+        PDF_S3_PREFIX = f"{_run_base}pdf/"
     if not PDF_S3_BUCKET:
         PDF_S3_BUCKET = S3_BUCKET
 

src/constants/constants.py

@@ -1,10 +1,17 @@
+from __future__ import annotations
+
 from pathlib import Path
+from typing import TYPE_CHECKING
+
 from sentence_transformers import SentenceTransformer
 from src.crosswalk.crosswalk_builder import CrosswalkBuilder
 from src.crosswalk.list_builder import ListBuilder
 
 from src.constants.embedding import CodeEmbedding
 
+if TYPE_CHECKING:
+    from src.utils.program_product_mapping import ProgramProductMapping
+
 # Absolute path to src/ directory
 _SRC_DIR = Path(__file__).parent.parent.resolve()
 
@@ -17,7 +24,7 @@ def _path(relative_path: str) -> str:
 class Constants:
     """Class to manage all constants and mappings for a Doczy.ai execution"""
 
-    def __init__(self):
+    def __init__(self, program_product_mapping: "ProgramProductMapping | None" = None):
 
         #################################### Dynamic and Exhibit-Level ####################################
 
@@ -27,20 +34,30 @@ class Constants:
         )
         self.VALID_LOBS = list(self.CROSSWALK_LOB.mapping.keys())
 
-        self.CROSSWALK_PROGRAM = CrosswalkBuilder().from_json(
-            path=_path("constants/mappings/crosswalk_program.json")
-        )
-        self.VALID_PROGRAMS = list(self.CROSSWALK_PROGRAM.mapping.keys())
-
         self.CROSSWALK_NETWORK = CrosswalkBuilder().from_json(
             path=_path("constants/mappings/crosswalk_network.json")
         )
         self.VALID_NETWORKS = list(self.CROSSWALK_NETWORK.mapping.keys())
 
-        self.CROSSWALK_PRODUCT = CrosswalkBuilder().from_json(
-            path=_path("constants/mappings/crosswalk_product_lob.json")
+        # AARETE_DERIVED_PROGRAM → AARETE_DERIVED_LOB and AARETE_DERIVED_PRODUCT → AARETE_DERIVED_LOB:
+        # Built from program_product_lob_mapping.csv loaded at startup.  Empty when no mapping.
+        mapping = program_product_mapping
+        self.CROSSWALK_PROGRAM_LOB = CrosswalkBuilder().from_dict(
+            mapping=mapping.program_to_lob if mapping else {},
+            from_col="AARETE_DERIVED_PROGRAM",
+            to_col="AARETE_DERIVED_LOB",
+        )
+        self.CROSSWALK_PRODUCT_LOB = CrosswalkBuilder().from_dict(
+            mapping=mapping.product_to_lob if mapping else {},
+            from_col="AARETE_DERIVED_PRODUCT",
+            to_col="AARETE_DERIVED_LOB",
+        )
+        self.VALID_AARETE_DERIVED_PROGRAMS = (
+            mapping.valid_aarete_derived_programs if mapping else []
+        )
+        self.VALID_AARETE_DERIVED_PRODUCTS = (
+            mapping.valid_aarete_derived_products if mapping else []
         )
-        self.VALID_PRODUCTS = list(self.CROSSWALK_PRODUCT.mapping.keys())
 
         # Dynamic Codes
         self.CROSSWALK_PROV_SPECIALTY_CD = CrosswalkBuilder().from_json(
@@ -269,7 +286,6 @@ class Constants:
         return [
             v.upper()
             for v in self.VALID_LOBS
-            + self.VALID_PRODUCTS
             + list(
                 set(
                     CrosswalkBuilder()
@@ -278,13 +294,6 @@ class Constants:
                 )
             )
             + self.VALID_CLAIM_TYPE
-            + list(
-                set(
-                    CrosswalkBuilder()
-                    .from_json(path=_path("constants/mappings/crosswalk_program.json"))
-                    .mapping.values()
-                )
-            )
             + ["INPATIENT", "OUTPATIENT", "IN-PATIENT", "OUT-PATIENT", "IP", "OP"]
             + [
                 "COVERED SERVICES",

src/constants/mappings/crosswalk_lob.json

@@ -6,6 +6,7 @@
     "mapping": {
         "Medicaid" : "Medicaid",
         "Medicare" : "Medicare",
+        "Medicare Advantage (MA)" : "Medicare",
         "Duals" : "Duals",
         "Medicare-Medicaid (MM)" : "Duals",
         "Commercial" : "Commercial",

src/constants/mappings/crosswalk_product_lob.json

@@ -1,96 +0,0 @@
-{
-    "metadata": {
-        "from_col": "AARETE_DERIVED_PRODUCT",
-        "to_col": "AARETE_DERIVED_LOB"
-    },
-    "mapping": {
-    },
-    "client_mapping" : {
-        "Healthfirst" : {
-            "Medicare Select" : "Medicare"
-        },
-        "Centene" : {
-            
-        },
-        "Aetna" : {
-
-        },
-        "CHC" : {
-
-        },
-        "Fidelis" : {
-            
-        },
-        "Parkland" : {
-            "HEALTHfirst" : "Medicaid",
-            "KIDSfirst" : "Medicaid"
-        },
-        "Molina" :{
-            "Molina Medicare Options" : "Medicare",
-            "Molina Medicare Options Plus" : "Duals",
-            "Molina Health Benefit Exchange" : "Marketplace"
-        },
-        "Humana" : {
-
-        },
-        "Moda" : {
-            "Synergy" : "Commercial",
-            "Connexus" : "Commercial",
-            "Beacon" : "Commercial",
-            "Community Care Network" : "Commercial",
-            "OHSU" : "Commercial",
-            "PEBB" : "Commercial",
-            "OEBB" : "Commercial",
-            "Affinity" : "Marketplace",
-            "Moda Select" : "Marketplace"
-        },
-        "Healthnet" : {
-            "Indemnity" : "Commercial",
-            "Ambetter" : "Commercial",
-            "Healthy Families Benefit Program" : "Medicaid",
-            "Salud con Health Net" : ""
-        },
-        "Trillium" : {
-            "Universal" : "Medicaid"
-        },
-        "Sentara" : {
-
-        },
-        "CareSource" : {
-
-        },
-        "BCBS" : {
-            "adultBasic" : "Medicaid",
-            "Affordablue" : "Marketplace",
-            "Keystone Health Plan West" : "Commercial",
-            "PremierBlue" : "Commercial",
-            "Choice Blue" : "Commercial",
-            "Special Care Program" : "Marketplace",
-            "Classic Blue" : "Commercial",
-            "Preferred Blue" : "Commercial",
-            "Direct Blue" : "Commercial",
-            "Select Blue" : "Commerical",
-            "Community Blue" : "Commercial",
-            "Connect Blue" : "Commercial",
-            "My Blue Access" : "Marketplace",
-            "Short Term Blue" : "Commercial",
-            "Advance Blue" : "Commercial",
-            "Simply Blue" : "Commercial",
-            "Essential" : "Medicaid|Commercial",
-            "Essential Plan - Aliessa" : "Medicaid",
-            "Essential Plan - QHP" : "Commercial",
-            "Exchange" : "Marketplace",
-            "First Priority Health" : "Commercial",
-            "Highmark Choice Company" : "Commercial",
-            "Protection Plus" : "Commercial",
-            "Super Blue" : "Commercial",
-            "Super Blue Plus" : "Commercial",
-            "Super Blue Select" : "Commercial",
-            "Quality Blue" : "Commercial|Medicare",
-            "Blue High Performance Network (BlueHPN)" : "Commercial",
-            "Together Blue" : "Marketplace",
-            "Traditional" : "Commercial",
-            "True Performance Program" : "Commercial|Medicare"
-        }
-    }
-}

src/constants/mappings/crosswalk_program.json

@@ -1,269 +0,0 @@
-{
-    "metadata": {
-        "from_col": "PROGRAM",
-        "to_col": "AARETE_DERIVED_PROGRAM"
-    },
-    "mapping": {
-        "Children's Health Insurance Program (CHIP)" : "CHIP",
-        "Children's Health Insurance Program Perinate (CHIP-P)" : "CHIPP",
-        "Healthy Kids" : "CHIP",
-        "Medicaid Managed Care" : "MMC",
-        "Affordable Care Act (ACA)" : "ACA",
-        "Aged, Blind, and Disabled (ABD)" : "ABD",
-        "Alternative Benefit Plan (ABP)" : "ABP",
-        "Community First Choice (CFC)" : "CFC",
-        "Foster Care (FC)" : "FC",
-        "Home and Community Based Services Waiver (HCBS)" : "HCBS",
-        "Intellectual and Developmental Disability (IDD)" : "IDD",
-        "Institution for Medical Disease (IMD)" : "IMD",
-        "Managed Long Term Care (MLTC)" : "MLTC",
-        "Self-Directed Personal Assistant Services (PAS)" : "PAS",
-        "Psych Under 21 (PU21)" : "PU21",
-        "Supplemental Nutrition Assistance Program (SNAP)" : "SNAP",
-        "Temporary Assistance Needy Families (TANF)" : "TANF",
-        "Long-Term Services and Supports (LTSS)" : "LTSS",
-        "Special Needs Plan (SNP)" : "SNP",
-        "Medicare Advantage Special Needs Plan (MA-SNP)" : "MASNP",
-        "Chronic Condition Special Needs Plan (CSNP)" : "CSNP",
-        "Dual Eligible Special Needs Plan (DSNP)" : "DSNP",
-        "Fully Integrated Dual Eligible Special Needs Plan (FIDE-SNP)" : "FIDE SNP",
-        "Highly Integrated Dual Eligible Special Needs Plan (HIDE-SNP)" : "HIDE SNP",
-        "Institutional Special Needs Plan (I-SNP)" : "ISNP",
-        "Medicare-Medicaid Alignment Initiative (MMAI)" : "MMAI",
-        "Medicare-Medicaid Coordinated Plan (MMCP)" : "MMCP",
-        "Medicare-Medicaid Plan (MMP)" : "MMP",
-        "Program of All-Inclusive Care for the Elderly (PACE)" : "PACE",
-        "Advantage" : "MA",
-        "Medicare Advantage" : "MA",
-        "Medicare Part C" : "MA",
-        "Supplemental" : "Supp",
-        "Medigap" : "Supp",
-        "Self-Funded" : "SF",
-        "Administrative Services Only (ASO)" : "SF",
-        "Third Party Administrator (TPA)" : "SF",
-        "Fully-Funded" : "FF",
-        "Platinum" : "PLAT",
-        "Gold" : "GOLD",
-        "Silver" : "SLVR",
-        "Bronze" : "BRNZ",
-        "Connector" : "CONN",
-        "Catastrophic" : "CATA"
-    },
-    "state_mapping" : {
-        "AL" : {
-            "ALL Kids" : "CHIP"
-        },
-        "AK" : {
-            "Denali KidCare" : "CHIP",
-            "DenaliCare" : "DENALI"
-        },
-        "AZ" : {
-            "Arizona Health Care Cost Containment System (AHCCCS)" : "AHCCCS",
-            "KidsCare" : "CHIP"
-        },
-        "AR" : {
-            "ARKids First" : "CHIP"
-        },
-        "CA" : {
-            "Medi-Cal" : "MEDI_CAL",
-            "Covered California" : "COVEREDCAL",
-            "Medi-Cal Access Program (MCAP)" : "CHIP",
-            "Healthy Families Benefit Program" : "CHIP"
-        },
-        "CO" : {
-            "Health First Colorado" : "HFCO",
-            "Connect for Health Colorado" : "CFHCO",
-            "Child Health Plan Plus (CHP+)" : "CHIP"
-        },
-        "CT" : {
-            "Husky Health" : "HUSKY",
-            "Access Health CT" : "AHCT",
-            "HUSKY A" : "CHIP",
-            "HUSKY B" : "CHIP",
-            "HUSKY C" : "CHIP",
-            "HUSKY D" : "CHIP"
-        },
-        "DE" : {
-            "Diamond State Health Plan" : "DSHP",
-            "Choose Health Delaware" : "CHDE",
-            "Delaware Healthy Children Program (DHCP)" : "CHIP"
-        },
-        "DC" : {
-            "DC Health Link" : "DCHL",
-            "DC Healthy Families" : "CHIP"
-        },
-        "FL" : {
-            "Florida Statewide Medicaid Managed Care Program (SMMC)" : "SMMC",
-            "Florida KidCare" : "CHIP"
-        },
-        "GA" : {
-            "Georgia Access" : "GAACCESS",
-            "PeachCare for Kids" : "CHIP"
-        },
-        "HI" : {
-            "Med-Quest" : "MQUEST",
-            "QUEST Integration" : "CHIP"
-        },
-        "ID" : {
-            "Your Health Idaho" : "YHID",
-            "Idaho Medicaid for CHIP" : "CHIP"
-        },
-        "IL" : {
-            "HealthChoice Illinois" : "HCIL",
-            "Get Covered Illinois" : "GCIL",
-            "All Kids" : "CHIP"
-        },
-        "IN" : {
-            "Hoosier Healthwise" : "CHIP"
-        },
-        "IA" : {
-            "Iowa Health Link" : "IAHL",
-            "Healthy and Well Kids in Iowa (HAWK-I)" : "CHIP"
-        },
-        "KS" : {
-            "KanCare" : "KANCARE"
-        },
-        "KY" : {
-            "Kynect" : "KYNECT",
-            "Kentucky Children's Health Insurance Program (KCHIP)" : "CHIP"
-        },
-        "LA" : {
-            "Healthy Louisiana" : "HEALTHYLA",
-            "Louisiana State Insurance Exchange" : "LASIE",
-            "LaCHIP" : "CHIP"
-        },
-        "ME" : {
-            "MaineCare" : "MECARE",
-            "Cover ME" : "COVERME",
-            "Cub Care" : "CHIP"
-        },
-        "MD" : {
-            "Maryland HealthChoice" : "MDHC",
-            "Maryland Children's Health Program (MCHP)" : "CHIP"
-        },
-        "MA" : {
-            "MassHealth" : "MASSHEALTH",
-            "MassHealth CHIP" : "CHIP"
-        },
-        "MI" : {
-            "Healthy Michigan Plan" : "HMIP",
-            "MI Child" : "CHIP"
-        },
-        "MN" : {
-            "Minnesota Medical Assistance" : "MNMA",
-            "MNSure" : "MNSURE",
-            "MinnesotaCare" : "CHIP"
-        },
-        "MS" : {
-            "MississippiCAN" : "MSCAN",
-            "Mississippi CHIP" : "CHIP"
-        },
-        "MO" : {
-            "MO HealthNet" : "MOHN",
-            "MO HealthNet for Kids" : "CHIP"
-        },
-        "MT" : {
-            "Healthy Montana Kids (HMK)" : "CHIP"
-        },
-        "NE" : {
-            "Heritage Health" : "HHNE",
-            "Kids Connection" : "CHIP"
-        },
-        "NV" : {
-            "Nevada Health Link" : "NVHL",
-            "Nevada Check Up" : "CHIP"
-        },
-        "NH" : {
-            "NH Health Families" : "NHHF",
-            "New Hampshire Healthy Families" : "CHIP"
-        },
-        "NJ" : {
-            "FamilyCare" : "NJFC",
-            "Get Covered NJ" : "GCNJ"
-        },
-        "NM" : {
-            "Centennial Care" : "CCNM",
-            "WellNM" : "WELLNM",
-            "New MexiKids" : "CHIP"
-        },
-        "NY" : {
-            "NY State of Health" : "NYSOH",
-            "Health and Recovery Plan (HARP)" : "HARP",
-            "Child Health Plus (CHP)" : "CHPLUS",
-            "Essential Plan" : "EPNY",
-            "New York State Health Insurance Exchange Qualified Health Plan" : "QHP",
-            "Basic Health Plan" : "BHP"
-        },
-        "NC" : {
-            "NC Health Choice" : "CHIP"
-        },
-        "ND" : {
-            "Healthy Steps" : "CHIP"
-        },
-        "OH" : {
-            "Healthy Start" : "CHIP",
-            "MyCare Ohio" : "MYCAREOH"
-        },
-        "OK" : {
-            "SoonerCare" : "SOONER",
-            "SoonerSelect" : "SOONER"
-        },
-        "OR" : {
-            "Oregon Health Plan" : "ORHP",
-            "OregonHealthCare" : "ORHC"
-        },
-        "PA" : {
-            "Pennsylvania Medical Assistance" : "PAMA",
-            "Pennie" : "PENNIE",
-            "Pennsylvania CHIP" : "CHIP"
-        },
-        "RI" : {
-            "HealthSource RI" : "HSRI",
-            "Rite Care" : "CHIP"
-        },
-        "SC" : {
-            "Partners for Healthy Children" : "CHIP"
-        },
-        "SD" : {
-            "South Dakota Medicaid for CHIP" : "CHIP"
-        },
-        "TN" : {
-            "TennCare" : "TENNCARE",
-            "CoverKids" : "CHIP"
-        },
-        "TX" : {
-            "STAR" : "STAR",
-            "STAR Health" : "STAR",
-            "STAR Kids" : "STAR Kids",
-            "STAR+PLUS" : "STAR+PLUS"
-        },
-        "UT" : {
-            "Healthy U" : "HEALTHYU",
-            "Healthy Kids Plus" : "CHIP"
-        },
-        "VT" : {
-            "Green Mountain Care" : "GMCARE",
-            "Dr. Dynasaur" : "CHIP"
-        },
-        "VA" : {
-            "Cardinal Care" : "CCVA",
-            "Cover Virginia" : "COVERVA",
-            "FAMIS" : "CHIP"
-        },
-        "WA" : {
-            "Apple Health" : "AHWA",
-            "Washington Healthplanfinder" : "WAHPF",
-            "Apple Health for Kids" : "CHIP"
-        },
-        "WV" : {
-            "WVCHIP" : "CHIP"
-        },
-        "WI" : {
-            "BadgerCare Plus" : "BADGER"
-        },
-        "WY" : {
-            "EqualityCare" : "EQCAREWY",
-            "Kid Care CHIP" : "CHIP"
-        }
-    }
-}

src/constants/mappings/crosswalk_program_lob.json

@@ -1,224 +0,0 @@
-{
-    "metadata": {
-        "from_col": "AARETE_DERIVED_PROGRAM",
-        "to_col": "AARETE_DERIVED_LOB"
-    },
-    "mapping": {
-        "CHIP" : "Medicaid",
-        "CHIPP" : "Medicaid",
-        "MMC" : "Medicaid",
-        "ACA" : "Marketplace",
-        "ABD" : "Medicaid",
-        "ABP" : "Medicaid",
-        "CFC" : "Medicaid",
-        "FC" : "Medicaid",
-        "HCBS" : "Medicaid",
-        "HHS" : "Medicaid",
-        "HOSPICE" : "Medicaid",
-        "IDD" : "Medicaid",
-        "IMD" : "Medicaid",
-        "MLTC" : "Medicaid",
-        "PAS" : "Medicaid",
-        "PU21" : "Medicaid",
-        "SNAP" : "Medicaid",
-        "TANF" : "Medicaid",
-        "LTSS" : "Medicaid",
-        "BHP" : "Medicaid",
-        "SNP" : "Duals",
-        "CSNP" : "Duals",
-        "DSNP" : "Duals",
-        "FIDE SNP" : "Duals",
-        "HIDE SNP" : "Duals",
-        "ISNP" : "Duals",
-        "MMAI" : "Duals",
-        "MMCP" : "Duals",
-        "MMP" : "Duals",
-        "PACE" : "Duals",
-        "MA" : "Medicare",
-        "MASNP" : "Medicare",
-        "Supp" : "Medicare",
-        "SF" : "Commercial",
-        "FF" : "Commercial",
-        "PLAT" : "Marketplace",
-        "GOLD" : "Marketplace",
-        "SLVR" : "Marketplace",
-        "BRNZ" : "Marketplace",
-        "CONN" : "Marketplace",
-        "CATA" : "Marketplace",
-        "QHP" : "Marketplace"
-    },
-    "state_mapping" : {
-        "AL" : {
-            
-        },
-        "AK" : {
-            "DENALI" : "Medicaid"
-        },
-        "AZ" : {
-            "AHCCCS" : "Medicaid"
-        },
-        "AR" : {
-            
-        },
-        "CA" : {
-            "MEDI_CAL" : "Medicaid",
-            "COVEREDCAL" : "Marketplace"
-        },
-        "CO" : {
-            "HFCO" : "Medicaid",
-            "CFHCO" : "Marketplace"
-        },
-        "CT" : {
-            "HUSKY" : "Medicaid",
-            "AHCT" : "Marketplace"
-        },
-        "DE" : {
-            "DSHP" : "Medicaid",
-            "CHDE" : "Marketplace"
-        },
-        "DC" : {
-            "DCHL" : "Marketplace"
-        },
-        "FL" : {
-            "SMMC" : "Medicaid"
-        },
-        "GA" : {
-            "GAACCESS" : "Marketplace"
-        },
-        "HI" : {
-            "MQUEST" : "Medicaid"
-        },
-        "ID" : {
-            "YHID" : "Marketplace"
-        },
-        "IL" : {
-            "HCIL" : "Marketplace",
-            "GCIL" : "Marketplace"
-        },
-        "IN" : {
-
-        },
-        "IA" : {
-            "IAHL" : "Medicaid"
-        },
-        "KS" : {
-            "KANCARE" : "Medicaid"
-        },
-        "KY" : {
-            "KYNECT" : "Marketplace"
-        },
-        "LA" : {
-            "HEALTHYLA" : "Medicaid",
-            "LASIE" : "Marketplace"
-        },
-        "ME" : {
-            "MECARE" : "Medicaid",
-            "COVERME" : "Marketplace"
-        },
-        "MD" : {
-            "MDHC" : "Medicaid"
-        },
-        "MA" : {
-            "MASSHEALTH" : "Medicaid"
-        },
-        "MI" : {
-            "HMIP" : "Medicaid"
-        },
-        "MN" : {
-            "MNMA" : "Medicaid",
-            "MNSURE" : "Marketplace"
-        },
-        "MS" : {
-            "MSCAN" : "Medicaid"
-        },
-        "MO" : {
-            "MOHN" : "Medicaid"
-        },
-        "MT" : {
-            
-        },
-        "NE" : {
-            "HHNE" : "Medicaid"
-        },
-        "NV" : {
-            "NVHL" : "Marketplace"
-        },
-        "NH" : {
-            "NHHF" : "Marketplace"
-        },
-        "NJ" : {
-            "NJFC" : "Medicaid",
-            "GCNJ" : "Marketplace"
-        },
-        "NM" : {
-            "CCNM" : "Medicaid",
-            "WELLNM" : "Marketplace"
-        },
-        "NY" : {
-            "NYSOH" : "Marketplace",
-            "HARP" : "Medicaid",
-            "CHPLUS" : "Medicaid",
-            "EPNY" : "Medicaid"
-        },
-        "NC" : {
-            
-        },
-        "ND" : {
-            
-        },
-        "OH" : {
-            "MYCAREOH" : "Duals"
-        },
-        "OK" : {
-            "SOONER" : "Medicaid"
-        },
-        "OR" : {
-            "ORHP" : "Medicaid",
-            "ORHC" : "Marketplace"
-        },
-        "PA" : {
-            "PENNIE" : "Marketplace",
-            "PAMA" : "Medicaid"
-        },
-        "RI" : {
-            "HSRI" : "Marketplace"
-        },
-        "SC" : {
-            
-        },
-        "SD" : {
-            
-        },
-        "TN" : {
-            "TENNCARE" : "Medicaid"
-        },
-        "TX" : {
-            "STAR" : "Medicaid",
-            "STAR+PLUS" : "Medicaid",
-            "STAR Kids" : "Medicaid"
-        },
-        "UT" : {
-            "HEALTHYU" : "Medicaid"
-        },
-        "VT" : {
-            "GMCARE" : "Medicaid"
-        },
-        "VA" : {
-            "CCVA" : "Medicaid",
-            "COVERVA" : "Marketplace"
-        },
-        "WA" : {
-            "AHWA" : "Medicaid",
-            "WAHPF" : "Marketplace"
-        },
-        "WV" : {
-            
-        },
-        "WI" : {
-            "BADGER" : "Medicaid"
-        },
-        "WY" : {
-            "EQCAREWY" : "Medicaid"
-        }
-    }
-}

src/crosswalk/crosswalk_builder.py

@@ -2,7 +2,6 @@ import csv
 import json
 
 import pandas as pd
-from src.config import CLIENT, STATE
 
 
 class CrosswalkBuilder:
@@ -108,36 +107,7 @@ class CrosswalkBuilder:
 
         self.metadata = data.get("metadata", {})
 
-        mapping = data.get("mapping", {})
-
-        if "state_mapping" in data:
-            for state in STATE:
-                mapping.update(data.get("state_mapping", {}).get(state, {}))
-        if "client_mapping" in data:
-            client_mapping = data.get("client_mapping", {})
-            clients = [
-                str(c).strip()
-                for c in CLIENT
-                if c is not None and str(c).strip().lower() != "none"
-            ]
-
-            if clients:
-                for client in clients:
-                    mapping.update(client_mapping.get(client, {}))
-            else:
-                # No explicit client provided: merge all client mappings with deduplication.
-                # Keep first value for duplicate keys.
-                merged_client_mapping: dict[str, str] = {}
-                for client_map in client_mapping.values():
-                    if not isinstance(client_map, dict):
-                        continue
-
-                    for key, value in client_map.items():
-                        merged_client_mapping.setdefault(key, value)
-
-                mapping.update(merged_client_mapping)
-
-        self.mapping = mapping
+        self.mapping = data.get("mapping", {})
 
         return self
 

src/document_classification/main.py

@@ -22,6 +22,7 @@ from src.pipelines.shared.preprocessing.preprocessing_funcs import (
 )
 from src.prompts.fieldset import Field
 from src.utils import io_utils, llm_utils, duplicate_detection
+import src.utils.program_product_mapping as ppm_utils
 from src import config
 
 # Load prompts from JSON using Field class (with thread-safe caching)
@@ -59,17 +60,14 @@ _MAPPINGS_DIR = os.path.join(os.path.dirname(__file__), "..", "constants", "mapp
 
 _LOB_JSON_FILES = [
     "crosswalk_lob.json",
-    "crosswalk_program_lob.json",
-    "crosswalk_product_lob.json",
 ]
 
 
 def _load_lob_keywords_from_mappings():
     """Load LOB keywords and their normalized LOB values from mapping JSON files.
 
-    Reads crosswalk_lob.json, crosswalk_program_lob.json, and crosswalk_product_lob.json
-    and extracts all keyword -> normalized_LOB pairs from their mapping, state_mapping,
-    and client_mapping sections.
+    Reads crosswalk_lob.json and extracts all keyword -> normalized_LOB pairs from their
+    mapping, state_mapping, and client_mapping sections.
 
     Returns:
         dict: Mapping of keyword (str) -> normalized LOB category (str).
@@ -986,6 +984,21 @@ def generate_pre_doczy_report(input_dict, run_timestamp):
             f"{master_dup_count} master filename duplicates"
         )
 
+    # ── Load program/product mapping for PROGRAM and PRODUCT summary fields ──
+    programs_list: list = []
+    products_list: list = []
+    if config.S3_BUCKET and config.PROGRAM_PRODUCT_MAPPING_CSV_KEY:
+        _mapping = ppm_utils.load_from_s3(
+            config.S3_BUCKET, config.PROGRAM_PRODUCT_MAPPING_CSV_KEY
+        )
+        if _mapping is not None:
+            programs_list = _mapping.valid_aarete_derived_programs
+            products_list = _mapping.valid_aarete_derived_products
+    df["PROGRAM"] = [programs_list] * len(df)
+    df["PRODUCT"] = [products_list] * len(df)
+    print("Program/Product mapping loaded for summary fields")
+    print(f"Sample PROGRAM values: {programs_list}")
+    print(f"Sample PRODUCT values: {products_list}")
     col_order = [
         "FILE_NAME",
         "STATUS",
@@ -1000,6 +1013,8 @@ def generate_pre_doczy_report(input_dict, run_timestamp):
         "LOB_PAGES",
         "LOB_KEYWORDS_MATCHED",
         "LOB_DERIVED",
+        "PROGRAM",
+        "PRODUCT",
     ]
     df = df[[c for c in col_order if c in df.columns]]
 

src/pipelines/runner.py

@@ -28,6 +28,7 @@ random.seed(42)
 warnings.filterwarnings("ignore", message=".*protected namespace.*")
 
 import src.prompts.prompt_templates as prompt_templates
+import src.prompts.cache_registry as cache_registry
 import src.utils.instrumentation as instrumentation
 import src.utils.io_utils as io_utils
 import src.utils.llm_utils as llm_utils
@@ -35,6 +36,7 @@ import src.utils.logging_utils as logging_utils
 import src.utils.usage_tracking as usage_tracking
 import src.utils.timing_utils as timing_utils
 import src.utils.duplicate_detection as duplicate_detection
+import src.utils.program_product_mapping as ppm_utils
 from src.pipelines.runtime_context import set_active_client
 from src.pipelines.shared.extraction import one_to_one_funcs
 from src.pipelines.shared.postprocessing import postprocessing_funcs
@@ -167,17 +169,33 @@ def main(client: str = "saas", testing=False, test_params={}):
 
     # Windows paths cannot contain ':'; use '-' in time component
     run_timestamp = datetime.now().strftime(f"run_%Y%m%d_%H-%M_{config.BATCH_ID}")
+    config.PROMPT_CALL_RUN_TIMESTAMP = run_timestamp
 
     instrumentation.configure_for_run(config.BATCH_ID, run_timestamp)
 
     # Track overall pipeline time
     pipeline_start = time.time()
 
+    # Load program_product_lob_mapping.csv once (if available) and inject into Constants
+    program_product_mapping = None
+    if config.S3_BUCKET and config.PROGRAM_PRODUCT_MAPPING_CSV_KEY:
+        with timing_utils.timed_block("load_program_product_mapping", log_level="INFO"):
+            program_product_mapping = ppm_utils.load_from_s3(
+                bucket=config.S3_BUCKET,
+                key=config.PROGRAM_PRODUCT_MAPPING_CSV_KEY,
+            )
+            if program_product_mapping is None:
+                logging.warning(
+                    f"program_product_lob_mapping.csv not found at "
+                    f"s3://{config.S3_BUCKET}/{config.PROGRAM_PRODUCT_MAPPING_CSV_KEY}; "
+                    "crosswalk/derived mappings will be empty."
+                )
+
     # Read Constants
     with timing_utils.timed_block("read_constants", log_level="INFO"):
-        constants = Constants()
+        constants = Constants(program_product_mapping=program_product_mapping)
 
-    # Read and process input
+    # Read and process input — when mapping is loaded, txt files live in the txt_files/ subfolder
     with timing_utils.timed_block("read_input", log_level="INFO"):
         if not testing:
             input_dict = io_utils.read_input()
@@ -217,19 +235,53 @@ def main(client: str = "saas", testing=False, test_params={}):
         return
 
     # ========== COMMON: Warm prompt caches ==========
+    # Drive warming from `cache_registry`, which knows the (usage_label,
+    # runtime_model) pairing. Cache entries are model-specific in Bedrock,
+    # so warming `SPLIT_SERVICE_TERM` on `sonnet_latest` while it runs on
+    # `haiku_latest` was previously paying create-tokens for a cache the
+    # runtime call could never read. The registry filters to instruction-
+    # cached entries on cache-supporting models only.
     logging.info("Warming prompt caches before parallel processing...")
     with timing_utils.timed_block("warm_prompt_caches", log_level="INFO"):
+        warmed = 0
+        skipped = 0
         try:
-            cacheable_instructions = prompt_templates.get_cacheable_instructions()
-            for cache_key, instruction in cacheable_instructions.items():
-                llm_utils.warm_prompt_cache(
-                    instruction=instruction,
-                    cache_key=cache_key,
-                    model_id="sonnet_latest",
+            for (
+                usage_label,
+                resolved_model_id,
+                instruction_text,
+            ) in cache_registry.iter_warming_targets():
+                ok = llm_utils.warm_prompt_cache(
+                    instruction=instruction_text,
+                    cache_key=usage_label,
+                    model_id=resolved_model_id,
                 )
+                if ok:
+                    warmed += 1
+                else:
+                    skipped += 1
             logging.info(
-                f"Successfully warmed {len(cacheable_instructions)} prompt caches"
+                "Warmed %d prompt caches (skipped %d already-warm or "
+                "unsupported pairs).",
+                warmed,
+                skipped,
             )
+
+            # Surface registry entries whose instruction is below the
+            # 1024-token Bedrock minimum so the gap is visible in pipeline
+            # logs, not just at runtime.
+            below = cache_registry.audit_below_threshold()
+            if below:
+                logging.warning(
+                    "Cache audit: %d instruction prompts are below the "
+                    "%d-token minimum: %s",
+                    len(below),
+                    cache_registry.MIN_CACHE_TOKENS,
+                    ", ".join(
+                        f"{label}({tokens}<{minimum})"
+                        for label, tokens, minimum in below
+                    ),
+                )
         except Exception as e:
             logging.warning(f"Error warming caches (will proceed anyway): {str(e)}")
 

src/pipelines/saas/file_processing.py

@@ -156,7 +156,7 @@ def process_file(file_object, constants: Constants, run_timestamp):
             "merge_one_to_one_into_one_to_n", context=filename
         ):
             if not one_to_n_results.empty:
-                # BOTH processed - merge into one_to_n
+                # BOTH processed - merge into one_to_n results for unified output
                 final_results = row_funcs.merge_one_to_one_into_one_to_n(
                     one_to_n_results, one_to_one_results, constants
                 )
@@ -184,6 +184,20 @@ def process_file(file_object, constants: Constants, run_timestamp):
             final_results = code_funcs.grouper_breakout(results_with_code)
         logging.debug(f"{datetime_str()} Codes Complete - {filename}")
 
+    # Derive AARETE_DERIVED_PROGRAM / AARETE_DERIVED_PRODUCT from PROGRAM / PRODUCT
+    if not final_results.empty:
+        with timing_utils.timed_block(
+            "add_aarete_derived_program_product", context=filename
+        ):
+            final_results = one_to_n_funcs.add_aarete_derived_program_product(
+                final_results, constants, filename
+            )
+
+    # Derive AARETE_DERIVED_LOB from AARETE_DERIVED_PROGRAM / AARETE_DERIVED_PRODUCT
+    if not final_results.empty:
+        with timing_utils.timed_block("fill_na_mapping", context=filename):
+            final_results = aarete_derived.fill_na_mapping(final_results, constants)
+
     # POSTPROCESS
     with (
         instrumentation_context.instr_scope(segment="postprocess"),
@@ -335,6 +349,23 @@ def run_one_to_one_prompts(
             one_to_one_results, contract_text
         )
 
+    ################## Split Dynamic Primary Entities ##################
+    # If the DYNAMIC_PRIMARY_ENTITIES smart_chunked fallback was triggered (all four
+    # dynamic primary fields were empty across every one_to_n row), the HSC step above
+    # will have populated DYNAMIC_PRIMARY_ENTITIES in one_to_one_results.
+    # Classify it into LOB/PROGRAM/PRODUCT/NETWORK before crosswalk runs so that
+    # AARETE_DERIVED_LOB and AARETE_DERIVED_NETWORK are derived correctly.
+    if not string_utils.is_empty(one_to_one_results.get("DYNAMIC_PRIMARY_ENTITIES")):
+        split_results = one_to_n_funcs.split_dynamic_primary_entities(
+            [one_to_one_results], filename
+        )
+        one_to_one_results = split_results[0]
+
+    ################## Derive AARETE_DERIVED_LOB/NETWORK ##################
+    one_to_n_funcs.map_lob_network_to_aarete_derived(
+        [one_to_one_results], constants, filename
+    )
+
     ################## Crosswalk Fields ##################
     # All field format normalization is handled at prompt_calls level via field-aware parsers
     # Derived fields (AARETE_DERIVED_*, NUM_*_SIGNATORY_*) are created as strings directly
@@ -343,9 +374,6 @@ def run_one_to_one_prompts(
         [one_to_one_results], constants
     )
 
-    ################## Fill NA Mapping ##################
-    one_to_one_results = aarete_derived.fill_na_mapping(one_to_one_results)
-
     return one_to_one_results[0]
 
 

src/pipelines/saas/prompts/prompt_calls.py

@@ -154,6 +154,116 @@ def prompt_dynamic_primary(
     return llm_answer_final
 
 
+def prompt_dynamic_primary_entities(
+    exhibit_text: str,
+    combined_field_prompt: str,
+    filename: str,
+) -> list[Any]:
+    """Extract a combined exhibit-level dynamic primary entity list."""
+    context_text, prompt, _parser = prompt_templates.DYNAMIC_PRIMARY_ENTITIES(
+        exhibit_text,
+    )
+
+    llm_answer_raw = llm_utils.invoke_claude(
+        prompt,
+        "sonnet_latest",
+        filename,
+        cache=True,
+        instruction=prompt_templates.DYNAMIC_PRIMARY_ENTITIES_INSTRUCTION(
+            combined_field_prompt=combined_field_prompt
+        ),
+        context_for_caching=context_text,
+        usage_label="DYNAMIC_PRIMARY_ENTITIES",
+    )
+    return _parser(llm_answer_raw)
+
+
+def prompt_classify_dynamic_primary_entities(
+    dynamic_primary_entities: list[str],
+    filename: str,
+) -> dict[str, Any]:
+    """Classify extracted dynamic primary entities into LOB/PROGRAM/PRODUCT/NETWORK."""
+    context_text, prompt, _parser = (
+        prompt_templates.DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION(
+            dynamic_primary_entities,
+        )
+    )
+
+    llm_answer_raw = llm_utils.invoke_claude(
+        prompt,
+        "sonnet_latest",
+        filename,
+        cache=True,
+        instruction=prompt_templates.DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION_INSTRUCTION(),
+        context_for_caching=context_text,
+        usage_label="DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION",
+    )
+
+    return _parser(llm_answer_raw)
+
+
+def prompt_map_lob_to_valid(
+    lob_values: list[str],
+    valid_lobs: list[str],
+    filename: str,
+) -> list[str]:
+    """Map raw classified LOB values to valid LOB values using LLM.
+
+    Args:
+        lob_values: List of raw LOB strings from the classification step.
+        valid_lobs: Allowed output values (from VALID_LOBS constant).
+        filename: Filename for LLM invocation and logging.
+
+    Returns:
+        List of matched valid LOB values. Returns an empty list if no matches.
+    """
+    context_text, prompt, _parser = prompt_templates.MAP_LOB_TO_VALID(
+        lob_values, valid_lobs
+    )
+    llm_answer_raw = llm_utils.invoke_claude(
+        prompt,
+        "sonnet_latest",
+        filename,
+        cache=True,
+        instruction=prompt_templates.MAP_LOB_TO_VALID_INSTRUCTION(),
+        context_for_caching=context_text,
+        usage_label="MAP_LOB_TO_VALID",
+    )
+    result = _parser(llm_answer_raw)
+    return result if isinstance(result, list) else []
+
+
+def prompt_map_network_to_valid(
+    network_values: list[str],
+    valid_networks: list[str],
+    filename: str,
+) -> list[str]:
+    """Map raw classified NETWORK values to valid NETWORK values using LLM.
+
+    Args:
+        network_values: List of raw NETWORK strings from the classification step.
+        valid_networks: Allowed output values (from VALID_NETWORKS constant).
+        filename: Filename for LLM invocation and logging.
+
+    Returns:
+        List of matched valid NETWORK values. Returns an empty list if no matches.
+    """
+    context_text, prompt, _parser = prompt_templates.MAP_NETWORK_TO_VALID(
+        network_values, valid_networks
+    )
+    llm_answer_raw = llm_utils.invoke_claude(
+        prompt,
+        "sonnet_latest",
+        filename,
+        cache=True,
+        instruction=prompt_templates.MAP_NETWORK_TO_VALID_INSTRUCTION(),
+        context_for_caching=context_text,
+        usage_label="MAP_NETWORK_TO_VALID",
+    )
+    result = _parser(llm_answer_raw)
+    return result if isinstance(result, list) else []
+
+
 def prompt_reimbursement_primary(
     page_text: str,
     filename: str,
@@ -190,6 +300,7 @@ def prompt_reimbursement_primary(
 
 
 def prompt_dynamic_code_assignment(service_term: str, filename: str):
+    """Extract dynamic code fields associated with a service term."""
     prompt, _parser = prompt_templates.DYNAMIC_CODE_ASSIGNMENT(service_term)
     logging.debug(f"Dynamic Code Assignment Prompt for {filename}: {prompt}")
     llm_response = llm_utils.invoke_claude(
@@ -204,7 +315,7 @@ def prompt_dynamic_code_assignment(service_term: str, filename: str):
 
     try:
         dynamic_code_assignment_answers = _parser(llm_response)
-    except ValueError as e:
+    except ValueError:
         dynamic_code_assignment_answers = {}
     return dynamic_code_assignment_answers
 
@@ -234,6 +345,12 @@ def prompt_methodology_breakout(
     logging.debug(f"LLM Response for {filename}: {llm_response}")
     try:
         methodology_breakout_answers = _parser(llm_response)
+        # Normalize output - methodology_breakout returns list[dict] with methodology fields
+        # Use FieldSet to get field names from single source of truth
+        methodology_breakout_fields = FieldSet(
+            config.FIELD_JSON_PATH, field_type="methodology_breakout"
+        )
+        methodology_field_names = methodology_breakout_fields.list_fields()
         # Handle both list and single dict cases
         if isinstance(methodology_breakout_answers, dict):
             methodology_breakout_answers = [methodology_breakout_answers]
@@ -412,6 +529,72 @@ def prompt_lob_relationship(
     return normalized if normalized else "Exclusive"
 
 
+def prompt_derive_aarete_program(
+    program_values: list[str],
+    valid_aarete_derived_programs: list[str],
+    filename: str,
+) -> list[str]:
+    """Map PROGRAM values to AARETE_DERIVED_PROGRAM valid values using LLM.
+
+    Args:
+        program_values: List of raw PROGRAM strings extracted from the contract.
+        valid_aarete_derived_programs: Allowed output values (from VALID_AARETE_DERIVED_PROGRAMS).
+        filename: Filename for LLM invocation and logging.
+
+    Returns:
+        List of matched AARETE_DERIVED_PROGRAM values from the valid values list.
+        Returns an empty list if no matches are found.
+    """
+    context_text, prompt, _parser = prompt_templates.AARETE_DERIVED_PROGRAM(
+        program_values, valid_aarete_derived_programs
+    )
+    llm_answer_raw = llm_utils.invoke_claude(
+        prompt,
+        "sonnet_latest",
+        filename,
+        cache=True,
+        instruction=prompt_templates.AARETE_DERIVED_PROGRAM_INSTRUCTION(),
+        context_for_caching=context_text,
+        usage_label="AARETE_DERIVED_PROGRAM",
+    )
+    result = _parser(llm_answer_raw)
+    return result if isinstance(result, list) else []
+
+
+def prompt_derive_aarete_product(
+    product_values: list[str],
+    valid_aarete_derived_products: list[str],
+    filename: str,
+) -> list[str]:
+    """Map PRODUCT values to AARETE_DERIVED_PRODUCT valid values using LLM.
+
+    Args:
+        product_values: List of raw PRODUCT strings extracted from the contract.
+        valid_aarete_derived_products: Allowed output values (from VALID_AARETE_DERIVED_PRODUCTS).
+        filename: Filename for LLM invocation and logging.
+
+    Returns:
+        List of matched AARETE_DERIVED_PRODUCT values from the valid values list.
+        Returns an empty list if no matches are found.
+    """
+    context_text, prompt, _parser = prompt_templates.AARETE_DERIVED_PRODUCT(
+        product_values, valid_aarete_derived_products
+    )
+    llm_answer_raw = llm_utils.invoke_claude(
+        prompt,
+        "sonnet_latest",
+        filename,
+        cache=True,
+        instruction=prompt_templates.AARETE_DERIVED_PRODUCT_INSTRUCTION(),
+        context_for_caching=context_text,
+        usage_label="AARETE_DERIVED_PRODUCT",
+    )
+
+    result = _parser(llm_answer_raw)
+
+    return result if isinstance(result, list) else []
+
+
 def prompt_special_case_assignment(
     exhibit_text: str,
     reimbursement_dict: dict,
@@ -741,7 +924,7 @@ def prompt_dynamic_assignment(
     field_name = dynamic_field.field_name
     field_prompt = dynamic_field.get_prompt(constants)
 
-    # Use specialized prompt for REIMB_DATES assignment with context caching
+    # Use specialized prompts for selected dynamic assignment fields with context caching
     if field_name == "REIMB_DATES":
         context_text, prompt, _parser = prompt_templates.REIMB_DATES_ASSIGNMENT(
             service_term,
@@ -752,6 +935,18 @@ def prompt_dynamic_assignment(
         )
         instruction = prompt_templates.REIMB_DATES_ASSIGNMENT_INSTRUCTION()
         usage_label = "REIMB_DATES_ASSIGNMENT"
+    elif field_name == "DYNAMIC_PRIMARY_ENTITIES":
+        context_text, prompt, _parser = (
+            prompt_templates.DYNAMIC_PRIMARY_ENTITIES_ASSIGNMENT(
+                service_term,
+                reimb_term,
+                field_prompt,
+                exhibit_text_simplified,
+                page_num,
+            )
+        )
+        instruction = prompt_templates.DYNAMIC_PRIMARY_ENTITIES_ASSIGNMENT_INSTRUCTION()
+        usage_label = "DYNAMIC_PRIMARY_ENTITIES_ASSIGNMENT"
     else:
         # Use generic DYNAMIC_ASSIGNMENT for other fields with context caching
         context_text, prompt, _parser = prompt_templates.DYNAMIC_ASSIGNMENT(
@@ -1307,9 +1502,15 @@ def prompt_split_service_term(service_term: str, filename: str) -> list[str]:
     """
     prompt, _parser = prompt_templates.SPLIT_SERVICE_TERM(service_term)
 
+    # Runs on Sonnet (not Haiku) because Haiku 4.5 requires 4096 tokens
+    # for cache_control to be honored; the SPLIT_SERVICE_TERM instruction
+    # is ~1600 tokens, so it would cache on Sonnet (1024-token minimum)
+    # but never on Haiku. Service-term inputs repeat heavily across
+    # contracts, so cache reads on Sonnet beat uncached Haiku on cost
+    # even though Haiku's per-token rate is lower.
     llm_answer_raw = llm_utils.invoke_claude(
         prompt,
-        model_id="haiku_latest",
+        model_id="sonnet_latest",
         filename=filename,
         cache=True,
         instruction=prompt_templates.SPLIT_SERVICE_TERM_INSTRUCTION(),
@@ -1328,3 +1529,47 @@ def prompt_split_service_term(service_term: str, filename: str) -> list[str]:
         logging.error(f"Error parsing LLM response for SPLIT_SERVICE_TERM: {str(e)}")
         logging.error(f"Raw response: {llm_answer_raw}")
         return []
+
+
+def prompt_program_to_lob(
+    program_values: list,
+    valid_lobs: list,
+    filename: str,
+    payer_name: str = "",
+    payer_state: str = "",
+) -> list:
+    """Use LLM to derive LOB values from program values (raw PROGRAM + AARETE_DERIVED_PROGRAM) against the valid LOB list."""
+    context, parser = prompt_templates.PROGRAM_TO_LOB(
+        program_values, valid_lobs, payer_name=payer_name, payer_state=payer_state
+    )
+    llm_answer_raw = llm_utils.invoke_claude(
+        context,
+        "sonnet_latest",
+        filename,
+        cache=True,
+        instruction=prompt_templates.PROGRAM_TO_LOB_INSTRUCTION(),
+        usage_label="PROGRAM_TO_LOB",
+    )
+    return parser(llm_answer_raw)
+
+
+def prompt_product_to_lob(
+    product_values: list,
+    valid_lobs: list,
+    filename: str,
+    payer_name: str = "",
+    payer_state: str = "",
+) -> list:
+    """Use LLM to derive LOB values from product values (raw PRODUCT + AARETE_DERIVED_PRODUCT) against the valid LOB list."""
+    context, parser = prompt_templates.PRODUCT_TO_LOB(
+        product_values, valid_lobs, payer_name=payer_name, payer_state=payer_state
+    )
+    llm_answer_raw = llm_utils.invoke_claude(
+        context,
+        "sonnet_latest",
+        filename,
+        cache=True,
+        instruction=prompt_templates.PRODUCT_TO_LOB_INSTRUCTION(),
+        usage_label="PRODUCT_TO_LOB",
+    )
+    return parser(llm_answer_raw)

src/pipelines/shared/extraction/dynamic_funcs.py

@@ -20,6 +20,16 @@ if TYPE_CHECKING:
     from src.pipelines.shared.extraction.page_funcs import Page
 
 
+def _build_dynamic_primary_combined_prompt(
+    dynamic_primary_fields: FieldSet, constants: Constants
+) -> str:
+    """Build a single combined prompt from the dynamic primary fields in the FieldSet."""
+    parts = []
+    for field in dynamic_primary_fields.fields:
+        parts.append(f"{field.field_name}: {field.get_prompt(constants)}")
+    return "\n\n".join(parts)
+
+
 def dynamic_primary(
     exhibit_text: str,
     exhibit_level_answers: dict,
@@ -47,45 +57,36 @@ def dynamic_primary(
     dynamic_reimbursement_fields = FieldSet()
     exhibit_level_answer_dict = {}
 
-    for field in list(dynamic_primary_fields.fields):
-        exhibit_text_answer = prompt_calls.prompt_dynamic_primary(
-            exhibit_text,
-            field,
-            constants,
-            filename,
-            prompt_templates.DYNAMIC_PRIMARY,
-        )
-        # If there is at least ONE Non-N/A answers in the Exhibit
-        if not string_utils.is_empty(exhibit_text_answer):
-            # Special handling for LOB: If both Medicare and Medicaid are detected, add Duals
-            if field.field_name == "LOB":
-                values_lower = [val.lower() for val in exhibit_text_answer]
+    combined_prompt = _build_dynamic_primary_combined_prompt(
+        dynamic_primary_fields, constants
+    )
+    dynamic_primary_entities = prompt_calls.prompt_dynamic_primary_entities(
+        exhibit_text,
+        combined_prompt,
+        filename,
+    )
 
-                # Check if both Medicare and Medicaid are present (case-insensitive)
-                has_medicare = any("medicare" in val for val in values_lower)
-                has_medicaid = any("medicaid" in val for val in values_lower)
+    exhibit_level_answer_dict["DYNAMIC_PRIMARY_ENTITIES"] = dynamic_primary_entities
 
-                # If both are present and Duals is not already in the list, add it
-                if has_medicare and has_medicaid:
-                    if "duals" not in values_lower:
-                        if exhibit_text_answer:
-                            exhibit_text_answer.append("Duals")
-                        else:
-                            exhibit_text_answer = ["Duals"]
-                        logging.debug(
-                            f"Added 'Duals' to LOB valid values because both Medicare and Medicaid were detected"
-                        )
+    dynamic_primary_entities_field = Field.from_values(
+        field_name="DYNAMIC_PRIMARY_ENTITIES",
+        relationship="one_to_n",
+        field_type="dynamic_primary",
+        prompt=(
+            "Candidate dynamic primary entities extracted at exhibit level. "
+            "Assign only the subset that applies to the current reimbursement term: {valid_values}"
+        ),
+        valid_values=dynamic_primary_entities,
+    )
 
-            field.update_valid_values(exhibit_text_answer + ["N/A"])
-            dynamic_reimbursement_fields.add_field(field)
-            dynamic_primary_fields.remove_field(field.field_name)
-        # If the field is not found, add to Exhibit-Level answers (the answer will be N/A or similar)
-        else:
-            exhibit_level_answer_dict[field.field_name] = exhibit_text_answer
+    # dynamic_primary_entities_field.update_valid_values(dynamic_primary_entities)
+    dynamic_reimbursement_fields.add_field(dynamic_primary_entities_field)
+
+    for field_name in dynamic_primary_fields.list_fields().copy():
+        dynamic_primary_fields.remove_field(field_name)
 
     # Update
     exhibit_level_answers.update(exhibit_level_answer_dict)
-
     return exhibit_level_answers, dynamic_reimbursement_fields
 
 
@@ -258,38 +259,14 @@ def add_one_to_one_field(
     field_to_add.field_type = "smart_chunked"
     field_to_add.relationship = "one_to_one"
 
-    # Special Handling for Program, Product, and Network:
-    # Skip when full LOB (all rows have LOB). When partial LOB, pass PROGRAM/PRODUCT;
-    # merge fills only empty cells, so 1:N values are preserved.
-
-    if field_to_add.field_name in ["PROGRAM", "PRODUCT", "NETWORK"]:
-        lob_values = []
-        for answer_dict in answer_dicts:
-            lob_value = answer_dict.get("LOB", "")
-            if isinstance(lob_value, list):
-                for item in lob_value:
-                    if not string_utils.is_empty(item):
-                        lob_values.append(str(item))
-            elif not string_utils.is_empty(lob_value):
-                lob_values.append(lob_value)
-        has_lob = len(set(lob_values)) > 0
-        lob_empty_count = sum(
-            1 for d in answer_dicts if string_utils.is_empty(d.get("LOB"))
-        )
-        partial_lob = has_lob and lob_empty_count > 0
-        # Skip only when full LOB; allow PROGRAM, PRODUCT when partial
-        if has_lob and not partial_lob:
-            return one_to_one_fields
-
     if field_to_add.field_name == "CLAIM_TYPE_CD":
         field_to_add.prompt = (
             field_to_add.prompt
             + prompt_templates.HSC_CLAIM_TYPES_ADDITIONAL_INSTRUCTION()
         )
 
-    # Special instructions for dynamic primary fields when passed to 1:1
-    # Add specific template-language guidance, then append general full context instructions
-    if field_to_add.field_name in ["LOB", "PRODUCT", "PROGRAM", "NETWORK"]:
+    # Special instructions for dynamic primary entities field when passed to 1:1
+    if field_to_add.field_name == "DYNAMIC_PRIMARY_ENTITIES":
         field_to_add.prompt = (
             field_to_add.prompt + prompt_templates.HSC_DYNAMIC_PRIMARY_INSTRUCTION()
         )
@@ -311,49 +288,83 @@ def get_dynamic_one_to_one_fields(
     answer_dicts: list[dict[str, str]], constants: Constants
 ):
     """
-    Returns a FieldSet object of dynamic fields that have are empty for at least one dict in answer_dicts
+    Returns a FieldSet object of dynamic fields that have are empty for at least one dict in answer_dicts.
+
+    Special case: if ALL dynamic_primary fields are empty across every row (i.e. for each
+    exhibit, all dynamic primary fields were empty), adds a DYNAMIC_PRIMARY_ENTITIES
+    smart_chunked field instead of individual LOB/PROGRAM/PRODUCT/NETWORK fields so that
+    the full entity extraction + split path runs in one_to_one. All other dynamic fields
+    (e.g. CLAIM_TYPE_CD) still go through the normal loop.
     """
     # These fields get passed to 1:1 if ALL of the answers are empty
+    if not answer_dicts:
+        return FieldSet()
+
     all_empty_fields = FieldSet(
         config.FIELD_JSON_PATH, relationship="one_to_n", base_field=True
     )
 
     one_to_one_fields = FieldSet()  # Empty FieldSet to populate if criteria are met
 
-    # Check if any LOB value has been found - if so, skip PROGRAM, PRODUCT, NETWORK
-    # Only search for PROGRAM, PRODUCT, NETWORK when NO LOB has been found
-    # NOTE: Check raw LOB field, not AARETE_DERIVED_LOB, because AARETE_DERIVED_LOB
-    # is populated later via crosswalk and may be derived from PROGRAM/PRODUCT
-    # LOB can be a string or a list (from JSON format)
-    # Extract all LOB values, handling both string and list formats
-    # All values must be hashable (strings) to create a set
-    lob_values = []
-    for answer_dict in answer_dicts:
-        lob_value = answer_dict.get("LOB", "")
-        if isinstance(lob_value, list):
-            # If it's a list, extract each item and ensure it's a string
-            for item in lob_value:
-                if not string_utils.is_empty(item):
-                    # Convert to string to ensure hashability (handles edge cases)
-                    lob_values.append(str(item))
-        elif not string_utils.is_empty(lob_value):
-            # If it's a string, add it directly (already hashable)
-            lob_values.append(lob_value)
-    unique_lobs = set(lob_values)
-    has_lob = len(unique_lobs) > 0
+    # Derive the dynamic_primary field names directly from the dynamic_primary FieldSet
+    # (not from all_empty_fields which only contains crosswalk/derived entries).
+    dynamic_primary_base_fields = [
+        field.field_name
+        for field in FieldSet(
+            config.FIELD_JSON_PATH,
+            relationship="one_to_n",
+            field_type="dynamic_primary",
+        ).fields
+    ]
 
-    # Partial LOB: some rows have LOB, some don't (e.g. stripped headers)
-    # When partial, pass LOB+PROGRAM+PRODUCT to 1:1; merge fills only empty cells
-    lob_empty_count = sum(
-        1 for d in answer_dicts if string_utils.is_empty(d.get("LOB"))
+    # Trigger DYNAMIC_PRIMARY_ENTITIES path only when ALL dynamic_primary fields are empty
+    # across every row — meaning for each exhibit, all dynamic primary fields were empty.
+    use_dynamic_primary_entities_path = bool(dynamic_primary_base_fields) and all(
+        string_utils.is_empty(answer_dict.get(field_name))
+        for answer_dict in answer_dicts
+        for field_name in dynamic_primary_base_fields
     )
-    partial_lob = has_lob and lob_empty_count > 0
+    if use_dynamic_primary_entities_path:
+        dynamic_primary_fields = FieldSet(
+            config.FIELD_JSON_PATH,
+            relationship="one_to_n",
+            field_type="dynamic_primary",
+        )
+        combined_prompt = _build_dynamic_primary_combined_prompt(
+            dynamic_primary_fields, constants
+        )
+        dynamic_primary_entities_field = Field.from_values(
+            field_name="DYNAMIC_PRIMARY_ENTITIES",
+            relationship="one_to_one",
+            field_type="smart_chunked",
+            prompt=prompt_templates.DYNAMIC_PRIMARY_ENTITIES_INSTRUCTION(
+                combined_prompt
+            ),
+            retrieval_question="Extract all Lines of Business, Networks, Programs, and Products this contract applies to.",
+        )
+        one_to_one_fields = add_one_to_one_field(
+            one_to_one_fields, dynamic_primary_entities_field, answer_dicts, constants
+        )
+        logging.debug(
+            "All dynamic primary fields empty across contract — using DYNAMIC_PRIMARY_ENTITIES smart_chunked path for 1:1"
+        )
 
     # Handle ALL empty fields
     for field in all_empty_fields.fields:
         field_name = field.field_name
         if "REIMB" in field_name:  # Don't do this for the REIMB_ fields
             continue
+        if field_name in [
+            "LOB",
+            "PROGRAM",
+            "PRODUCT",
+            "NETWORK",
+            "AARETE_DERIVED_LOB",
+            "AARETE_DERIVED_PROGRAM",
+            "AARETE_DERIVED_PRODUCT",
+            "AARETE_DERIVED_NETWORK",
+        ]:
+            continue
 
         empty_count = sum(
             1
@@ -363,24 +374,15 @@ def get_dynamic_one_to_one_fields(
         total_count = len(answer_dicts)
         base = field.base_field if field.base_field else field_name
 
-        # LOB: pass when empty in ANY row (partial detection due to stripped headers)
-        # PROGRAM, PRODUCT: also pass when partial_lob and empty in any row (LOB relationship)
-        # Others: pass only when empty in ALL rows
-        should_pass_to_1to1 = (
-            empty_count == total_count
-            or (base == "LOB" and empty_count > 0)
-            or (partial_lob and base in ["PROGRAM", "PRODUCT"] and empty_count > 0)
-        )
+        should_pass_to_1to1 = empty_count == total_count
 
         if should_pass_to_1to1:
-            # Skip PROGRAM, PRODUCT, NETWORK when full LOB (all rows have LOB)
-            # Do NOT skip when partial_lob; pass PROGRAM, PRODUCT for consistency
-            if (
-                has_lob
-                and not partial_lob
-                and base in ["PROGRAM", "PRODUCT", "NETWORK"]
-            ):
-                continue
+            logging.debug(
+                f"Field '{field_name}' is empty in all {total_count} rows — adding to one-to-one FieldSet"
+            )
+            # Dynamic primary fields (LOB, PROGRAM, PRODUCT, NETWORK) and their
+            # derived counterparts (AARETE_DERIVED_*) are NEVER extracted individually —
+            # they always go through the DYNAMIC_PRIMARY_ENTITIES path.
 
             field_to_add = Field.load_from_file(
                 file_path=config.FIELD_JSON_PATH,
@@ -389,5 +391,4 @@ def get_dynamic_one_to_one_fields(
             one_to_one_fields = add_one_to_one_field(
                 one_to_one_fields, field_to_add, answer_dicts, constants
             )
-
     return one_to_one_fields

src/pipelines/shared/extraction/one_to_n_funcs.py

@@ -873,6 +873,372 @@ def get_lob_relationship(answer_dicts, exhibit_text, filename):
     return updated_dicts
 
 
+def _normalize_dynamic_primary_entities(raw_value) -> list[str]:
+    """Normalize dynamic primary entities to a clean, distinct list of strings."""
+    if string_utils.is_empty(raw_value):
+        return []
+
+    if isinstance(raw_value, list):
+        values = string_utils.flatten_to_strings(raw_value)
+    else:
+        values = string_utils.extract_nested_values(raw_value)
+
+    cleaned = []
+    for value in values:
+        text = str(value).strip()
+        if string_utils.is_empty(text):
+            continue
+        if text.upper() in {"N/A", "UNKNOWN", "NONE", "NULL"}:
+            continue
+        if text not in cleaned:
+            cleaned.append(text)
+    return cleaned
+
+
+def _map_lob_network_to_aarete_derived(
+    answer_dict: dict,
+    valid_lobs: list[str],
+    valid_networks: list[str],
+    constants: Constants,
+    filename: str,
+) -> None:
+    """Derive AARETE_DERIVED_LOB and AARETE_DERIVED_NETWORK from base LOB/NETWORK fields.
+
+    Two-step process — the intermediate step is never stored in the output:
+    1. LLM maps raw base-field text (exact contract language) to valid crosswalk keys.
+    2. Crosswalk maps those valid keys to AARETE_DERIVED values, which are stored
+       in answer_dict["AARETE_DERIVED_LOB"] / answer_dict["AARETE_DERIVED_NETWORK"].
+
+    Downstream get_crosswalk_fields will not overwrite pre-populated AARETE_DERIVED fields.
+    Modifies answer_dict in-place.
+    """
+    # Derive AARETE_DERIVED_LOB
+    if valid_lobs:
+        raw_lobs = _normalize_dynamic_primary_entities(answer_dict.get("LOB"))
+        if raw_lobs:
+            try:
+                lob_keys = prompt_calls.prompt_map_lob_to_valid(
+                    lob_values=raw_lobs,
+                    valid_lobs=valid_lobs,
+                    filename=filename,
+                )
+                if lob_keys:
+                    derived_lobs = []
+                    for key in lob_keys:
+                        val = constants.CROSSWALK_LOB.mapping.get(key)
+                        if val is None:
+                            continue
+                        if isinstance(val, list):
+                            derived_lobs.extend(v for v in val if v and v != "N/A")
+                        elif val and val != "N/A":
+                            derived_lobs.append(val)
+                    if derived_lobs:
+                        answer_dict["AARETE_DERIVED_LOB"] = list(
+                            dict.fromkeys(derived_lobs)
+                        )
+            except Exception:
+                logging.exception(
+                    "Failed to derive AARETE_DERIVED_LOB from LOB; skipping"
+                )
+
+    # Derive AARETE_DERIVED_NETWORK
+    if valid_networks:
+        raw_networks = _normalize_dynamic_primary_entities(answer_dict.get("NETWORK"))
+        if raw_networks:
+            try:
+                network_keys = prompt_calls.prompt_map_network_to_valid(
+                    network_values=raw_networks,
+                    valid_networks=valid_networks,
+                    filename=filename,
+                )
+                if network_keys:
+                    derived_networks = []
+                    for key in network_keys:
+                        val = constants.CROSSWALK_NETWORK.mapping.get(key)
+                        if val is None:
+                            continue
+                        if isinstance(val, list):
+                            derived_networks.extend(v for v in val if v and v != "N/A")
+                        elif val and val != "N/A":
+                            derived_networks.append(val)
+                    if derived_networks:
+                        answer_dict["AARETE_DERIVED_NETWORK"] = list(
+                            dict.fromkeys(derived_networks)
+                        )
+            except Exception:
+                logging.exception(
+                    "Failed to derive AARETE_DERIVED_NETWORK from NETWORK; skipping"
+                )
+
+
+def map_lob_network_to_aarete_derived(
+    all_exhibit_rows: list[dict], constants: Constants, filename: str
+) -> list[dict]:
+    """Derive AARETE_DERIVED_LOB/NETWORK for every row from their base LOB/NETWORK fields.
+
+    Runs the two-step intermediate mapping (exact contract text → crosswalk keys → AARETE_DERIVED
+    values) for each row.  Called explicitly just before get_crosswalk_fields so that
+    split_dynamic_primary_entities remains focused on classification only.
+    """
+    valid_lobs = constants.get_constant("VALID_LOBS")
+    valid_networks = constants.get_constant("VALID_NETWORKS")
+    for answer_dict in all_exhibit_rows:
+        _map_lob_network_to_aarete_derived(
+            answer_dict, valid_lobs, valid_networks, constants, filename
+        )
+    return all_exhibit_rows
+
+
+def split_dynamic_primary_entities(
+    all_exhibit_rows: list[dict], filename: str
+) -> list[dict]:
+    """Split DYNAMIC_PRIMARY_ENTITIES into LOB/PROGRAM/PRODUCT/NETWORK and remove temp field."""
+    if not all_exhibit_rows:
+        return all_exhibit_rows
+
+    for answer_dict in all_exhibit_rows:
+        raw_entities = answer_dict.get("DYNAMIC_PRIMARY_ENTITIES")
+        entities = _normalize_dynamic_primary_entities(raw_entities)
+
+        if not entities:
+            answer_dict.pop("DYNAMIC_PRIMARY_ENTITIES", None)
+            _deduplicate_primary_fields(answer_dict, filename)
+            continue
+
+        # Step 1: Classify entities semantically — no valid-value constraints at this stage
+        try:
+            classified = prompt_calls.prompt_classify_dynamic_primary_entities(
+                dynamic_primary_entities=entities,
+                filename=filename,
+            )
+        except Exception:
+            logging.exception(
+                "Failed to classify DYNAMIC_PRIMARY_ENTITIES; skipping row"
+            )
+            answer_dict.pop("DYNAMIC_PRIMARY_ENTITIES", None)
+            continue
+
+        # Step 2: Build collision-guard sets from raw classified LOB/NETWORK values to
+        # filter PROGRAM/PRODUCT values that duplicate a confirmed LOB or NETWORK entity.
+        mapped_lobs_lower = {
+            v.lower()
+            for v in _normalize_dynamic_primary_entities(classified.get("LOB"))
+        }
+        mapped_networks_lower = {
+            v.lower()
+            for v in _normalize_dynamic_primary_entities(classified.get("NETWORK"))
+        }
+
+        # Step 3: Merge classified values into answer_dict fields
+        for field_name in ["LOB", "NETWORK", "PROGRAM", "PRODUCT"]:
+            new_values = _normalize_dynamic_primary_entities(classified.get(field_name))
+            if not new_values:
+                continue
+
+            # Guard PROGRAM/PRODUCT against confirmed LOB or NETWORK values
+            if field_name in ("PROGRAM", "PRODUCT"):
+                new_values = [
+                    value
+                    for value in new_values
+                    if value.lower() not in mapped_lobs_lower
+                    and value.lower() not in mapped_networks_lower
+                ]
+
+            if not new_values:
+                continue
+
+            existing_values = _normalize_dynamic_primary_entities(
+                answer_dict.get(field_name)
+            )
+            merged = list(dict.fromkeys(existing_values + new_values))
+            answer_dict[field_name] = merged
+
+        answer_dict.pop("DYNAMIC_PRIMARY_ENTITIES", None)
+
+        # Step 4: Cross-field deduplication — enforce LOB > NETWORK > PROGRAM > PRODUCT priority.
+        # A value that appears in multiple fields (e.g. from the raw extraction putting the same
+        # entity into both PROGRAM and PRODUCT) is retained only in the highest-priority field.
+        _deduplicate_primary_fields(answer_dict, filename)
+
+    return all_exhibit_rows
+
+
+def _deduplicate_primary_fields(answer_dict: dict, filename: str = "") -> None:
+    """Enforce LOB > NETWORK > PROGRAM > PRODUCT priority across primary classification fields.
+
+    If a value appears in multiple fields, it is retained only in the highest-priority field
+    and removed from all lower-priority ones.  This handles cases where the raw one-to-n
+    extraction puts the same entity into both PROGRAM and PRODUCT (bypassing the
+    classification step entirely).
+
+    Modifies answer_dict in-place.
+    """
+
+    def _vals(field: str) -> list[str]:
+        return _normalize_dynamic_primary_entities(answer_dict.get(field))
+
+    lob_lower = {v.lower() for v in _vals("LOB")}
+    network_lower = {v.lower() for v in _vals("NETWORK")}
+    removals: dict[str, list[str]] = {}
+
+    # Remove from NETWORK anything already claimed by LOB
+    network_vals = _vals("NETWORK")
+    if network_vals:
+        filtered = [v for v in network_vals if v.lower() not in lob_lower]
+        if len(filtered) != len(network_vals):
+            removals["NETWORK"] = [v for v in network_vals if v.lower() in lob_lower]
+            answer_dict["NETWORK"] = filtered
+        network_lower = {v.lower() for v in filtered}
+
+    # Remove from PROGRAM anything already claimed by LOB or NETWORK
+    program_vals = _vals("PROGRAM")
+    if program_vals:
+        filtered = [
+            v
+            for v in program_vals
+            if v.lower() not in lob_lower and v.lower() not in network_lower
+        ]
+        if len(filtered) != len(program_vals):
+            removals["PROGRAM"] = [
+                v
+                for v in program_vals
+                if v.lower() in lob_lower or v.lower() in network_lower
+            ]
+            answer_dict["PROGRAM"] = filtered
+        program_lower = {v.lower() for v in filtered}
+    else:
+        program_lower = set()
+
+    # Remove from PRODUCT anything already claimed by LOB, NETWORK, or PROGRAM
+    product_vals = _vals("PRODUCT")
+    if product_vals:
+        filtered = [
+            v
+            for v in product_vals
+            if v.lower() not in lob_lower
+            and v.lower() not in network_lower
+            and v.lower() not in program_lower
+        ]
+        if len(filtered) != len(product_vals):
+            removals["PRODUCT"] = [
+                v
+                for v in product_vals
+                if v.lower() in lob_lower
+                or v.lower() in network_lower
+                or v.lower() in program_lower
+            ]
+            answer_dict["PRODUCT"] = filtered
+
+
+def _normalize_to_uppercase_list(value) -> list[str]:
+    """Normalize a PROGRAM or PRODUCT value to a list of trimmed, uppercase strings."""
+    if string_utils.is_empty(value):
+        return []
+    if isinstance(value, list):
+        return [str(item).strip().upper() for item in value if str(item).strip()]
+    text = str(value).strip()
+    return [text.upper()] if text else []
+
+
+def add_aarete_derived_program_product(
+    df: pd.DataFrame, constants: Constants, filename: str
+) -> pd.DataFrame:
+    """Derive AARETE_DERIVED_PROGRAM and AARETE_DERIVED_PRODUCT from PROGRAM and PRODUCT.
+
+    For each field declared as ``field_type="derived_dynamic_primary"`` in the FieldSet:
+
+    - If the corresponding valid values list (resolved from Constants) is **non-empty**,
+      the LLM is used to map each row's base field values (PROGRAM / PRODUCT) to
+      standardized derived values constrained to the valid values list.
+    - If the valid values list is **empty or absent**, the function falls back to simple
+      trim-and-titlecase normalization (preserving the current behaviour for PRODUCT when
+      no client-specific valid values are configured).
+
+    Existing (non-empty) values for the derived field on a row are preserved and the new
+    values are merged in (deduplicated, order-stable).
+
+    Args:
+        df: pd.DataFrame containing rows with PROGRAM and/or PRODUCT fields.
+        constants: Constants object used to look up VALID_AARETE_DERIVED_PROGRAMS and
+            VALID_AARETE_DERIVED_PRODUCTS.
+        filename: Filename passed to LLM calls and logging.
+
+    Returns:
+        pd.DataFrame with AARETE_DERIVED_PROGRAM and AARETE_DERIVED_PRODUCT set.
+    """
+    if df.empty:
+        return df
+    all_exhibit_rows = df.to_dict(orient="records")
+
+    derived_fields = FieldSet(
+        config.FIELD_JSON_PATH, field_type="derived_dynamic_primary"
+    )
+
+    for field in derived_fields.fields:
+        derived_field_name = field.field_name  # e.g. AARETE_DERIVED_PROGRAM
+        base_field_name = field.base_field  # e.g. PROGRAM
+        valid_values_key = field.valid_values  # e.g. "VALID_AARETE_DERIVED_PROGRAMS"
+
+        # Resolve valid values from constants (None / empty → fallback mode)
+        valid_values: list[str] = constants.get_constant(valid_values_key) or []
+        use_llm = bool(valid_values)
+        # Build a lowercase → canonical lookup for post-LLM enforcement
+        valid_values_by_lower: dict[str, str] = {v.lower(): v for v in valid_values}
+        for answer_dict in all_exhibit_rows:
+            raw_value = answer_dict.get(base_field_name)
+            if string_utils.is_empty(raw_value):
+                continue
+
+            # Normalise base values to a flat list of non-empty strings,
+            # handling both real lists and serialized list strings (e.g. '["A", "B"]')
+            base_values = _normalize_dynamic_primary_entities(raw_value)
+
+            if not base_values:
+                continue
+
+            if use_llm:
+                try:
+                    if derived_field_name == "AARETE_DERIVED_PROGRAM":
+                        derived_values = prompt_calls.prompt_derive_aarete_program(
+                            base_values, valid_values, filename
+                        )
+                    elif derived_field_name == "AARETE_DERIVED_PRODUCT":
+                        derived_values = prompt_calls.prompt_derive_aarete_product(
+                            base_values, valid_values, filename
+                        )
+                except Exception:
+                    logging.exception(
+                        f"LLM derivation failed for {derived_field_name}; skipping"
+                    )
+                    derived_values = []
+                # Enforce valid-values constraint: filter out any LLM hallucinations,
+                # returning the canonical value exactly as it appears in valid_values
+                derived_values = [
+                    valid_values_by_lower[v.lower()]
+                    for v in derived_values
+                    if v.lower() in valid_values_by_lower
+                ]
+            else:
+                # Fallback: trim + title-case, preserving all-uppercase words as
+                # acronyms (e.g. CHIP, DSNP, STAR+PLUS stay unchanged).
+                def _title_preserve_acronyms(text: str) -> str:
+                    return " ".join(
+                        word if word.isupper() else word.title()
+                        for word in text.split()
+                    )
+
+                derived_values = [_title_preserve_acronyms(v) for v in base_values]
+
+            # Merge with any existing values already present on this row
+            existing_raw = answer_dict.get(derived_field_name)
+            existing_list = _normalize_dynamic_primary_entities(existing_raw)
+
+            merged = list(dict.fromkeys(existing_list + derived_values))
+            answer_dict[derived_field_name] = merged
+
+    return pd.DataFrame(all_exhibit_rows)
+
+
 def set_carveout_indicators(all_exhibit_rows: list[dict]) -> list[dict]:
     """
     Sets CARVEOUT_IND and DEFAULT_IND based on CARVEOUT_CD values.
@@ -901,6 +1267,20 @@ def set_carveout_indicators(all_exhibit_rows: list[dict]) -> list[dict]:
 def one_to_n_cleaning(
     all_exhibit_rows: list[dict], exhibit_text: str, constants: Constants, filename: str
 ):
+    ################################ Split Dynamic Primary Entities ################################
+    with timing_utils.timed_block(
+        "one_to_n_cleaning.split_dynamic_primary_entities", context=filename
+    ):
+        all_exhibit_rows = split_dynamic_primary_entities(all_exhibit_rows, filename)
+
+    ################################ Derive AARETE_DERIVED_LOB/NETWORK ################################
+    with timing_utils.timed_block(
+        "one_to_n_cleaning.derive_aarete_lob_network", context=filename
+    ):
+        all_exhibit_rows = map_lob_network_to_aarete_derived(
+            all_exhibit_rows, constants, filename
+        )
+
     ################################ Crosswalk Fields ################################
     with timing_utils.timed_block(
         "one_to_n_cleaning.crosswalk_fields", context=filename
@@ -917,12 +1297,6 @@ def one_to_n_cleaning(
             all_exhibit_rows, exhibit_text, filename
         )
 
-    ################################ Fill NA Mapping ################################
-    with timing_utils.timed_block(
-        "one_to_n_cleaning.fill_na_mapping", context=filename
-    ):
-        all_exhibit_rows = aarete_derived.fill_na_mapping(all_exhibit_rows)
-
     ################################ Split REIMB_DATES ################################
     with timing_utils.timed_block(
         "one_to_n_cleaning.split_reimb_dates", context=filename

src/pipelines/shared/extraction/row_funcs.py

@@ -3,7 +3,6 @@ from copy import deepcopy
 import pandas as pd
 import src.pipelines.shared.prompts.prompt_calls as prompt_calls
 from src.constants.constants import Constants
-from src.pipelines.shared.postprocessing import aarete_derived
 from src.pipelines.shared.extraction import one_to_one_funcs
 from src.utils import string_utils
 from src.prompts.fieldset import FieldSet
@@ -100,9 +99,6 @@ def merge_one_to_one_into_one_to_n(
                 )
             )
 
-    # one_to_one merge happens after one_to_n cleaning. Re-run fill mapping so
-    # propagated PROGRAM/PRODUCT values can derive LOB in rows where it is empty.
     merged_rows = one_to_n_results.to_dict(orient="records")
-    merged_rows = aarete_derived.fill_na_mapping(merged_rows)
 
     return pd.DataFrame(merged_rows)

src/pipelines/shared/postprocessing/aarete_derived.py

@@ -1,14 +1,22 @@
 import logging
+import pandas as pd
 import src.config as config
 import src.utils.string_utils as string_utils
 from src.constants.constants import Constants
 from src.constants.investment_columns import FIELD_FORMAT_MAPPING
 from src.crosswalk.crosswalk_builder import CrosswalkBuilder
+from src.pipelines.saas.prompts import prompt_calls
 from src.prompts.fieldset import FieldSet
 from src.utils.crosswalk_utils import apply_crosswalk
 from src.utils.formatting_utils import normalize_field_value
 
 
+def _canonical_lob(val: str, valid_lobs_lower: dict[str, str]) -> str:
+    """Return the canonical-cased LOB from valid_lobs (case-insensitive match).
+    Falls back to title case if no match found."""
+    return valid_lobs_lower.get(val.strip().lower(), val.strip().title())
+
+
 def fill_na_from_field(results_dict, to_field, from_field, crosswalk_path):
     from_str = results_dict.get(from_field)
     if string_utils.is_empty(from_str):
@@ -18,156 +26,197 @@ def fill_na_from_field(results_dict, to_field, from_field, crosswalk_path):
     return to_value if to_value else results_dict.get(to_field)
 
 
-def fill_na_mapping(answer_dicts):
+def fill_na_mapping(df: pd.DataFrame, constants: Constants) -> pd.DataFrame:
     """
-    Fill in missing values in fields using mappings from other fields.
-    Always checks all crosswalks (PROGRAM and PRODUCT) and merges unique values.
+    Derive AARETE_DERIVED_LOB from AARETE_DERIVED_PROGRAM and AARETE_DERIVED_PRODUCT.
+    Uses crosswalk mapping when available; falls back to LLM when the mapping is empty.
+    Merges unique LOB values from both sources.
 
     Args:
-        answer_dicts (list[dict]): List of answer dictionaries
+        df (pd.DataFrame): DataFrame with extracted fields.
+        constants (Constants): Shared Constants instance (loaded once at startup).
 
     Returns:
-        list[dict]: List of answer dictionaries with filled/merged values
+        pd.DataFrame: DataFrame with filled/merged AARETE_DERIVED_LOB values.
     """
+    if df.shape[0] == 0:
+        return df
+
+    filename = df["FILE_NAME"].iloc[0] if "FILE_NAME" in df.columns else "unknown"
+    payer_name_raw = df["PAYER_NAME"].iloc[0] if "PAYER_NAME" in df.columns else ""
+    payer_state_raw = df["PAYER_STATE"].iloc[0] if "PAYER_STATE" in df.columns else ""
+    payer_name = (
+        str(payer_name_raw) if not string_utils.is_empty(payer_name_raw) else ""
+    )
+    payer_state = (
+        str(payer_state_raw) if not string_utils.is_empty(payer_state_raw) else ""
+    )
+
+    answer_dicts = df.to_dict(orient="records")
+    # Crosswalk values are the AARETE_DERIVED_LOB values - pass these to the LLM
+    lob_mapping_values = constants.CROSSWALK_LOB.mapping.values()
+    valid_lobs = sorted(
+        {
+            item
+            for v in lob_mapping_values
+            for item in (v if isinstance(v, list) else [v])
+            if item and item != "N/A"
+        }
+    )
+    # Case-insensitive lookup → canonical form (e.g. "MEDICAID" → "Medicaid")
+    valid_lobs_lower: dict[str, str] = {v.lower(): v for v in valid_lobs}
+
+    def _to_str_list(val) -> list:
+        """Normalize a field value to a deduplicated list of non-empty strings."""
+        if string_utils.is_empty(val):
+            return []
+        if isinstance(val, list):
+            return [str(v).strip() for v in val if str(v).strip()]
+        if isinstance(val, str):
+            return [val.strip()] if val.strip() else []
+        return []
+
+    # --- Pre-compute LOB caches keyed by unique input tuples ---
+    # Collect unique keys across all rows so each distinct value is resolved once.
+    unique_aarete_program_keys: set[tuple] = set()
+    unique_program_keys: set[tuple] = set()
+    unique_aarete_product_keys: set[tuple] = set()
+    unique_product_keys: set[tuple] = set()
+
+    for answer_dict in answer_dicts:
+        adp = _to_str_list(answer_dict.get("AARETE_DERIVED_PROGRAM"))
+        if adp:
+            unique_aarete_program_keys.add(tuple(adp))
+        p = _to_str_list(answer_dict.get("PROGRAM"))
+        if p:
+            unique_program_keys.add(tuple(p))
+        adprod = _to_str_list(answer_dict.get("AARETE_DERIVED_PRODUCT"))
+        if adprod:
+            unique_aarete_product_keys.add(tuple(adprod))
+        prod = _to_str_list(answer_dict.get("PRODUCT"))
+        if prod:
+            unique_product_keys.add(tuple(prod))
+
+    # Crosswalk cache: AARETE_DERIVED_PROGRAM → LOB list
+    program_crosswalk_cache: dict[tuple, list] = {}
+    if constants.CROSSWALK_PROGRAM_LOB.mapping:
+        for key in unique_aarete_program_keys:
+            program_crosswalk_cache[key] = [
+                constants.CROSSWALK_PROGRAM_LOB.mapping[v]
+                for v in key
+                if v in constants.CROSSWALK_PROGRAM_LOB.mapping
+            ]
+
+    # LLM cache: PROGRAM → LOB list (one call per unique program list)
+    program_llm_cache: dict[tuple, list] = {}
+    for key in unique_program_keys:
+        program_llm_cache[key] = (
+            prompt_calls.prompt_program_to_lob(
+                list(key),
+                valid_lobs,
+                filename,
+                payer_name=payer_name,
+                payer_state=payer_state,
+            )
+            or []
+        )
+
+    # Crosswalk cache: AARETE_DERIVED_PRODUCT → LOB list
+    product_crosswalk_cache: dict[tuple, list] = {}
+    if constants.CROSSWALK_PRODUCT_LOB.mapping:
+        for key in unique_aarete_product_keys:
+            product_crosswalk_cache[key] = [
+                constants.CROSSWALK_PRODUCT_LOB.mapping[v]
+                for v in key
+                if v in constants.CROSSWALK_PRODUCT_LOB.mapping
+            ]
+
+    # LLM cache: PRODUCT → LOB list (one call per unique product list)
+    product_llm_cache: dict[tuple, list] = {}
+    for key in unique_product_keys:
+        product_llm_cache[key] = (
+            prompt_calls.prompt_product_to_lob(
+                list(key),
+                valid_lobs,
+                filename,
+                payer_name=payer_name,
+                payer_state=payer_state,
+            )
+            or []
+        )
+
+    # --- Apply cached results to each row ---
     for answer_dict in answer_dicts:
-        # Collect all AARETE_DERIVED_LOB values from different sources
         all_lob_values = set()
 
         # Get existing AARETE_DERIVED_LOB values (if any)
         # Handle both list and string formats (JSON lists or pipe-delimited strings)
         existing_lob_list = answer_dict.get("AARETE_DERIVED_LOB", "")
-
         if not string_utils.is_empty(existing_lob_list):
-            # Normalize to list format
             if isinstance(existing_lob_list, list):
                 lob_items = existing_lob_list
             elif isinstance(existing_lob_list, str):
-                # Handle pipe-delimited format (backward compatibility)
-                if "|" in existing_lob_list:
-                    lob_items = [v.strip() for v in existing_lob_list.split("|")]
-                else:
-                    lob_items = [existing_lob_list]
+                lob_items = [existing_lob_list]
             else:
                 lob_items = [str(existing_lob_list)]
-
-            # Process each LOB value
             for lob_val in lob_items:
                 if (
                     isinstance(lob_val, str)
                     and lob_val.strip()
                     and lob_val.strip() != "N/A"
                 ):
-                    all_lob_values.add(lob_val.strip())
+                    all_lob_values.add(_canonical_lob(lob_val, valid_lobs_lower))
 
-        # Get AARETE_DERIVED_LOB from AARETE_DERIVED_PROGRAM crosswalk (always check if PROGRAM exists)
+        # Get AARETE_DERIVED_LOB from AARETE_DERIVED_PROGRAM + PROGRAM (via caches)
         program_lob_values = set()
-        if not string_utils.is_empty(answer_dict.get("AARETE_DERIVED_PROGRAM")):
-            program_filled_value_list = fill_na_from_field(
-                answer_dict,
-                "AARETE_DERIVED_LOB",
-                "AARETE_DERIVED_PROGRAM",
-                "src/constants/mappings/crosswalk_program_lob.json",
-            )
-            if (
-                not string_utils.is_empty(program_filled_value_list)
-                and "N/A" not in program_filled_value_list
-            ):
-                # Normalize to list format (handle both list and string)
-                if isinstance(program_filled_value_list, list):
-                    program_lob_items = program_filled_value_list
-                elif isinstance(program_filled_value_list, str):
-                    # Handle pipe-delimited format (backward compatibility)
-                    if "|" in program_filled_value_list:
-                        program_lob_items = [
-                            v.strip() for v in program_filled_value_list.split("|")
-                        ]
-                    else:
-                        program_lob_items = [program_filled_value_list]
-                else:
-                    program_lob_items = [str(program_filled_value_list)]
+        adp_key = tuple(_to_str_list(answer_dict.get("AARETE_DERIVED_PROGRAM")))
+        p_key = tuple(_to_str_list(answer_dict.get("PROGRAM")))
+        program_lob_list = program_crosswalk_cache.get(
+            adp_key, []
+        ) + program_llm_cache.get(p_key, [])
+        filtered = [
+            _canonical_lob(v, valid_lobs_lower)
+            for v in program_lob_list
+            if isinstance(v, str) and v.strip() and v.strip() != "N/A"
+        ]
+        if filtered:
+            program_lob_values.update(filtered)
+            all_lob_values.update(filtered)
 
-                # Process each program LOB value
-                for program_lob_val in program_lob_items:
-                    if (
-                        isinstance(program_lob_val, str)
-                        and program_lob_val.strip()
-                        and program_lob_val.strip() != "N/A"
-                    ):
-                        program_lob_values.add(program_lob_val.strip())
-                # Update all_lob_values after processing all program LOB values
-                all_lob_values.update(program_lob_values)
-
-        # Get AARETE_DERIVED_LOB from product crosswalk.
-        # Prefer AARETE_DERIVED_PRODUCT, fallback to PRODUCT when derived is empty.
+        # Get AARETE_DERIVED_LOB from AARETE_DERIVED_PRODUCT + PRODUCT (via caches)
         product_lob_values = set()
-        product_source_field = None
-        if not string_utils.is_empty(answer_dict.get("AARETE_DERIVED_PRODUCT")):
-            product_source_field = "AARETE_DERIVED_PRODUCT"
-        elif not string_utils.is_empty(answer_dict.get("PRODUCT")):
-            product_source_field = "PRODUCT"
-
-        if product_source_field is not None:
-            product_filled_value_list = fill_na_from_field(
-                answer_dict,
-                "AARETE_DERIVED_LOB",
-                product_source_field,
-                "src/constants/mappings/crosswalk_product_lob.json",
-            )
-            if (
-                not string_utils.is_empty(product_filled_value_list)
-                and "N/A" not in product_filled_value_list
-            ):
-                # Normalize to list format (handle both list and string)
-                if isinstance(product_filled_value_list, list):
-                    product_lob_items = product_filled_value_list
-                elif isinstance(product_filled_value_list, str):
-                    # Handle pipe-delimited format (backward compatibility)
-                    if "|" in product_filled_value_list:
-                        product_lob_items = [
-                            v.strip() for v in product_filled_value_list.split("|")
-                        ]
-                    else:
-                        product_lob_items = [product_filled_value_list]
-                else:
-                    product_lob_items = [str(product_filled_value_list)]
-
-                # Process each product LOB value
-                for product_lob_val in product_lob_items:
-                    if (
-                        isinstance(product_lob_val, str)
-                        and product_lob_val.strip()
-                        and product_lob_val.strip() != "N/A"
-                    ):
-                        product_lob_values.add(product_lob_val.strip())
-                # Update all_lob_values after processing all product LOB values
-                all_lob_values.update(product_lob_values)
+        adprod_key = tuple(_to_str_list(answer_dict.get("AARETE_DERIVED_PRODUCT")))
+        prod_key = tuple(_to_str_list(answer_dict.get("PRODUCT")))
+        product_lob_list = product_crosswalk_cache.get(
+            adprod_key, []
+        ) + product_llm_cache.get(prod_key, [])
+        filtered = [
+            _canonical_lob(v, valid_lobs_lower)
+            for v in product_lob_list
+            if isinstance(v, str) and v.strip() and v.strip() != "N/A"
+        ]
+        if filtered:
+            product_lob_values.update(filtered)
+            all_lob_values.update(filtered)
 
         # Set the merged, deduplicated value
         if all_lob_values:
             answer_dict["AARETE_DERIVED_LOB"] = sorted(all_lob_values)
-            if string_utils.is_empty(answer_dict.get("LOB")):
-                answer_dict["LOB"] = sorted(all_lob_values)
-            # Set relationship flags if values came from PROGRAM or PRODUCT
             # Only set if field is empty or "N/A" (preserve LLM-determined Inclusive/Exclusive)
             if program_lob_values:
                 existing_relationship = answer_dict.get("LOB_PROGRAM_RELATIONSHIP", "")
-                # Original code (CHANGED - now preserves LLM values):
-                # answer_dict["LOB_PROGRAM_RELATIONSHIP"] = "Exclusive"
                 if string_utils.is_empty(existing_relationship):
                     answer_dict["LOB_PROGRAM_RELATIONSHIP"] = "Exclusive"
             if product_lob_values:
                 existing_relationship = answer_dict.get("LOB_PRODUCT_RELATIONSHIP", "")
-                # Original code (CHANGED - now preserves LLM values):
-                # answer_dict["LOB_PRODUCT_RELATIONSHIP"] = "Exclusive"
                 if string_utils.is_empty(existing_relationship):
                     answer_dict["LOB_PRODUCT_RELATIONSHIP"] = "Exclusive"
         elif string_utils.is_empty(answer_dict.get("AARETE_DERIVED_LOB")):
             # If still empty after all attempts, keep it as is (will be N/A or empty)
             pass
 
-        # Handle cases where PROGRAM/PRODUCT is NA/blank (LLM was bypassed)
-        # Set relationship to "Exclusive" as default when PROGRAM/PRODUCT is missing and relationship is not set
-        # This covers both cases: LOB exists but PROGRAM/PRODUCT is NA, or both LOB and PROGRAM/PRODUCT are NA
+        # Handle cases where PROGRAM/PRODUCT is NA/blank
+        # Set relationship to "Exclusive" as default when missing and relationship is not set
         if string_utils.is_empty(answer_dict.get("AARETE_DERIVED_PROGRAM")):
             if string_utils.is_empty(answer_dict.get("LOB_PROGRAM_RELATIONSHIP")):
                 answer_dict["LOB_PROGRAM_RELATIONSHIP"] = "Exclusive"
@@ -176,7 +225,7 @@ def fill_na_mapping(answer_dicts):
             if string_utils.is_empty(answer_dict.get("LOB_PRODUCT_RELATIONSHIP")):
                 answer_dict["LOB_PRODUCT_RELATIONSHIP"] = "Exclusive"
 
-    return answer_dicts
+    return pd.DataFrame(answer_dicts)
 
 
 def get_crosswalk_fields(answer_dicts: list, constants: Constants):
@@ -248,5 +297,4 @@ def get_crosswalk_fields(answer_dicts: list, constants: Constants):
                     # The crosswalk should only create/update the target field (to_field_name),
                     # not modify the source field (from_field_name)
                     # If from_field_name needs to be a list, that should be handled elsewhere
-
     return answer_dicts

src/pipelines/shared/postprocessing/postprocess.py

@@ -4,7 +4,7 @@ import logging
 import src.config as config
 from src.constants.constants import Constants
 from src.constants.investment_columns import FIELD_FORMAT_MAPPING
-from src.pipelines.shared.postprocessing import postprocessing_funcs
+from src.pipelines.shared.postprocessing import postprocessing_funcs, aarete_derived
 from src.utils import timing_utils, string_utils
 
 
@@ -130,9 +130,6 @@ def standard_postprocess(df, constants: Constants):
     # Process PATIENT_AGE_RANGE into PATIENT_AGE_MIN and PATIENT_AGE_MAX
     df = postprocessing_funcs.process_patient_age_range(df)
 
-    # Add AARETE_DERIVED_PRODUCT
-    df = postprocessing_funcs.add_aarete_derived_product(df)
-
     df = postprocessing_funcs.update_grouper_base_rate_and_grouper_pct_rate(df)
 
     # Fill empty claim type from CONTRACT_TITLE as a safety net

src/pipelines/shared/postprocessing/postprocessing_funcs.py

@@ -733,14 +733,11 @@ def add_aarete_derived_amendment_num(df: pd.DataFrame) -> pd.DataFrame:
     return df
 
 
-def add_aarete_derived_product(df):
+def add_aarete_derived_program_product(df):
+    """No-op pass-through: AARETE_DERIVED_PROGRAM/PRODUCT are now derived upstream
+    in one_to_n_cleaning (one_to_n_funcs.add_aarete_derived_program_product) before
+    the DataFrame is assembled.  This stub is retained for backward compatibility.
     """
-    Add AARETE_DERIVED_PRODUCT column to df, based on PRODUCT column.
-
-    The new column should be Title Case of the PRODUCT column value.
-    """
-    if "PRODUCT" in df.columns:
-        df["AARETE_DERIVED_PRODUCT"] = df["PRODUCT"]
     return df
 
 

src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py

@@ -470,14 +470,25 @@ def prompt_hsc_single_field(
             f"context with len {len(context)}: {context[:500]}........{context[-500:]}"
         )  # log only the beginning and end of the context to avoid clutter
 
-        # LLM call
+        # LLM call. The static extraction rules live in
+        # ONE_TO_ONE_SINGLE_FIELD_INSTRUCTION so they cache once across all
+        # ~30 fields per document instead of being re-billed every call.
+        # The dynamic prompt still carries the per-field question and the
+        # retrieved chunk; only the rule block is cached.
         with timing_utils.timed_block(
             f"hsc.field.{field.field_name}.llm_call", context=filename
         ):
             llm_answer_raw = llm_utils.invoke_claude(
-                prompt, "sonnet_latest", filename, max_tokens=8192
+                prompt,
+                "sonnet_latest",
+                filename,
+                max_tokens=8192,
+                cache=True,
+                instruction=prompt_templates.ONE_TO_ONE_SINGLE_FIELD_INSTRUCTION(),
+                usage_label="ONE_TO_ONE_SINGLE_FIELD",
             )
         logging.debug(f"Raw LLM answer for {field.field_name}: {llm_answer_raw}")
+
         try:
             llm_answer_final = _parser(llm_answer_raw)  # returns dict
             field_value = llm_answer_final.get(field.field_name)

src/pipelines/vendors/shared/generic_processor.py

@@ -127,7 +127,7 @@ class VendorProcessor:
     Generic vendor processor driven by config.yaml + vendor_fields.json.
 
     Matches the interface expected by runner.py:
-      - process_file(file_object, constants, run_timestamp) -> pd.DataFrame
+      - process_file(file_object, constants, run_timestamp) -> tuple[pd.DataFrame, None]
       - SHEET_COLUMN_SCHEMAS -> dict[str, list[str]]
     """
 
@@ -222,7 +222,9 @@ class VendorProcessor:
     # Public entry point
     # ------------------------------------------------------------------
 
-    def process_file(self, file_object, constants, run_timestamp) -> pd.DataFrame:
+    def process_file(
+        self, file_object, constants, run_timestamp
+    ) -> tuple[pd.DataFrame, None]:
         """Process a single vendor contract file.
 
         Args:
@@ -231,8 +233,10 @@ class VendorProcessor:
             run_timestamp: Timestamp string for output naming.
 
         Returns:
-            Flat DataFrame combining 1-to-1 and SLA/KPI rows (broadcast style).
-            Each SLA/KPI row carries all 1-to-1 field values, matching saas behaviour.
+            (flat_df, None) — matches the (cc_df, dashboard_df) contract used by
+            the saas pipeline. Vendors do not produce a dashboard variant, so the
+            second slot is always None. flat_df combines 1-to-1 and SLA/KPI rows
+            (broadcast style); each SLA/KPI row carries all 1-to-1 field values.
         """
         filename, contract_text = file_object
         logging_utils.set_current_file(filename)
@@ -307,7 +311,7 @@ class VendorProcessor:
         logging.info(
             f"{datetime_str()} [{self.vendor_name}] Writing complete — {filename}"
         )
-        return flat_df
+        return flat_df, None
 
     # ------------------------------------------------------------------
     # Extraction helpers

src/prompts/cache_registry.py

@@ -0,0 +1,622 @@
+"""Central prompt cache registry.
+
+Single source of truth for which prompts are eligible for prompt caching,
+which model they actually run on at runtime, and which cache class they
+use. This is consumed by the cache-warming layer in
+`src.pipelines.runner` and by reporting / contract tests.
+
+Why this exists
+---------------
+Before the registry, cache warming and runtime call sites were split:
+- `prompt_templates.get_cacheable_instructions()` produced the warmed list
+- runtime code in `pipelines/.../prompt_calls.py` chose its own model id
+The two could (and did) drift. For example `SPLIT_SERVICE_TERM` was warmed
+on `sonnet_latest` but invoked on `haiku_latest`, so warm-up tokens were
+spent against a cache key the runtime never read.
+
+Cache classes
+-------------
+- INSTRUCTION:        the system instruction block is cached; cache reads
+                      are expected on every call after warm-up.
+- CONTEXT:            a dynamic `context_for_caching` block is cached;
+                      reads are only expected when the same context
+                      repeats (e.g. the same exhibit text across fields).
+- INSTRUCTION_PLUS_CONTEXT: both blocks are cached.
+- NONE:               explicitly opted out; either the runtime model
+                      does not support caching in this Bedrock setup,
+                      or the static instruction is too small to qualify
+                      and is not yet padded.
+
+Reporting should evaluate INSTRUCTION and CONTEXT cache classes
+separately so per-context variability does not make instruction caching
+look broken.
+"""
+
+from __future__ import annotations
+
+import logging
+import re
+from dataclasses import dataclass
+from typing import Callable, Iterable, Iterator
+
+import src.config as config
+import src.prompts.prompt_templates as prompt_templates
+
+
+# Default minimum tokens for a cache_control block to be honored.
+# Per-model overrides live in `llm_utils._MODEL_MIN_CACHE_TOKENS`. This
+# constant is the conservative fallback used when an entry has no
+# explicit min_instruction_tokens set and the resolved model is unknown.
+MIN_CACHE_TOKENS = 1024
+
+# Same heuristic used by `prompt_call_tracking._estimate_tokens` so warm-up
+# audits agree with runtime token reporting.
+_WORDS_TO_TOKENS_MULTIPLIER = 1.3
+
+
+# ---------------------------------------------------------------------------
+# Cache classes
+# ---------------------------------------------------------------------------
+
+
+class CacheClass:
+    INSTRUCTION = "instruction"
+    CONTEXT = "context"
+    INSTRUCTION_PLUS_CONTEXT = "instruction+context"
+    NONE = "none"
+
+
+_VALID_CACHE_CLASSES = {
+    CacheClass.INSTRUCTION,
+    CacheClass.CONTEXT,
+    CacheClass.INSTRUCTION_PLUS_CONTEXT,
+    CacheClass.NONE,
+}
+
+
+@dataclass(frozen=True)
+class CachePromptEntry:
+    """One cacheable prompt declaration.
+
+    Attributes:
+        usage_label: Stable label passed to `invoke_claude(usage_label=...)`
+            and emitted in runtime logs. Must match what runtime call sites
+            pass.
+        instruction_func: Zero-arg callable returning the static
+            instruction text. May be None if the prompt only uses context
+            caching (cache_class == CONTEXT) and has no static instruction
+            we want to warm.
+        runtime_model: Model alias or full id this prompt is invoked with
+            at runtime (e.g. "sonnet_latest"). The warming layer resolves
+            this through `config.resolve_model_id`.
+        cache_class: One of the CacheClass values.
+        min_instruction_tokens: Optional explicit minimum size for the
+            instruction block. When None (the default) the warming layer
+            uses the per-model threshold from
+            `llm_utils._min_cache_tokens(resolved_model_id)` — 1024 for
+            Sonnet 4.5, 4096 for Haiku 4.5, etc. Set this only when an
+            entry needs a stricter floor than the model defaults.
+        notes: Free text — usually the reason this entry deviates from the
+            default (e.g. why something is NONE).
+    """
+
+    usage_label: str
+    instruction_func: Callable[[], str] | None
+    runtime_model: str
+    cache_class: str
+    min_instruction_tokens: int | None = None
+    notes: str = ""
+
+    def __post_init__(self) -> None:
+        if self.cache_class not in _VALID_CACHE_CLASSES:
+            raise ValueError(
+                f"Unknown cache_class={self.cache_class!r} on {self.usage_label}. "
+                f"Expected one of {sorted(_VALID_CACHE_CLASSES)}."
+            )
+        if (
+            self.cache_class
+            in (
+                CacheClass.INSTRUCTION,
+                CacheClass.INSTRUCTION_PLUS_CONTEXT,
+            )
+            and self.instruction_func is None
+        ):
+            raise ValueError(
+                f"{self.usage_label}: instruction-cached entries require "
+                f"an instruction_func."
+            )
+
+
+# ---------------------------------------------------------------------------
+# Token estimation
+# ---------------------------------------------------------------------------
+
+
+def estimate_instruction_tokens(text: str | None) -> int:
+    """Estimate token count for an instruction string.
+
+    Uses the same words*1.3 heuristic as runtime prompt-call tracking so
+    warm-up audits and runtime reports stay aligned.
+    """
+    if not text:
+        return 0
+    words = len(re.findall(r"\S+", text))
+    return int(round(words * _WORDS_TO_TOKENS_MULTIPLIER))
+
+
+# ---------------------------------------------------------------------------
+# Registry
+# ---------------------------------------------------------------------------
+# Conventions:
+#   - usage_label MUST match what call sites pass to `invoke_claude(...)`.
+#   - runtime_model is the alias the call site passes, not the resolved id.
+#   - Instruction-only entries that are below 1024 tokens after the latest
+#     pad expansion are still listed as INSTRUCTION; the warming layer logs
+#     a below-threshold warning so the gap is visible without silently
+#     hiding the prompt.
+
+_REGISTRY: tuple[CachePromptEntry, ...] = (
+    # ---------- one_to_n primary / breakout family ----------
+    CachePromptEntry(
+        usage_label="REIMBURSEMENT_PRIMARY",
+        instruction_func=prompt_templates.REIMBURSEMENT_PRIMARY_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="METHODOLOGY_BREAKOUT",
+        instruction_func=prompt_templates.METHODOLOGY_BREAKOUT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="DYNAMIC_CODE_ASSIGNMENT",
+        instruction_func=prompt_templates.DYNAMIC_CODE_ASSIGNMENT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="FEE_SCHEDULE_BREAKOUT",
+        instruction_func=prompt_templates.FEE_SCHEDULE_BREAKOUT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="GROUPER_BREAKOUT",
+        instruction_func=prompt_templates.GROUPER_BREAKOUT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="OUTLIER_BREAKOUT",
+        instruction_func=prompt_templates.OUTLIER_BREAKOUT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="CARVEOUT_CHECK",
+        instruction_func=prompt_templates.CARVEOUT_CHECK_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="DYNAMIC_ASSIGNMENT",
+        instruction_func=prompt_templates.DYNAMIC_ASSIGNMENT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="REIMB_DATES_ASSIGNMENT",
+        instruction_func=prompt_templates.REIMB_DATES_ASSIGNMENT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="LESSER_OF_DISTRIBUTION",
+        instruction_func=prompt_templates.LESSER_OF_DISTRIBUTION_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="LESSER_OF_CHECK",
+        instruction_func=prompt_templates.LESSER_OF_CHECK_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="VALIDATE_REIMBURSEMENTS",
+        instruction_func=prompt_templates.VALIDATE_REIMBURSEMENTS_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="LOB_RELATIONSHIP",
+        instruction_func=prompt_templates.LOB_RELATIONSHIP_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+        notes=(
+            "Runtime label is `prompt_lob_relationship` (see alias). "
+            "Currently below 1024-token threshold; warming layer logs a "
+            "below-threshold warning until the instruction is expanded."
+        ),
+    ),
+    # ---------- exhibit-level (instruction + context cache) ----------
+    CachePromptEntry(
+        usage_label="EXHIBIT_LEVEL",
+        instruction_func=prompt_templates.EXHIBIT_LEVEL_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
+        notes=(
+            "Calls also pass context_for_caching for the exhibit text; "
+            "instruction reads are stable, context reads only repeat per "
+            "exhibit."
+        ),
+    ),
+    CachePromptEntry(
+        usage_label="DYNAMIC_PRIMARY",
+        instruction_func=prompt_templates.DYNAMIC_PRIMARY_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
+        notes="Exhibit text is sent as context_for_caching across fields.",
+    ),
+    CachePromptEntry(
+        usage_label="DYNAMIC_PRIMARY_ENTITIES",
+        instruction_func=prompt_templates.DYNAMIC_PRIMARY_ENTITIES_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.CONTEXT,
+        notes=(
+            "Static instruction is intentionally short (~180 tokens) and "
+            "below the 1024-token Sonnet 4.5 cache minimum; padding it "
+            "would dwarf the actual prompt. Caching here relies on the "
+            "context_for_caching block (the exhibit text) repeating "
+            "within a run. CONTEXT class skips instruction warm-up to "
+            "avoid spending warm-tokens on a block that cannot cache."
+        ),
+    ),
+    CachePromptEntry(
+        usage_label="DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION",
+        instruction_func=prompt_templates.DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
+        notes=(
+            "Classification step that runs after entity extraction. "
+            "Instruction expanded over the 1024-token Sonnet 4.5 cache "
+            "minimum so the static block caches; the dynamic entity list "
+            "in context_for_caching is too small to cache reliably and "
+            "is expected to miss on the context block."
+        ),
+    ),
+    CachePromptEntry(
+        usage_label="DYNAMIC_PRIMARY_ENTITIES_ASSIGNMENT",
+        instruction_func=prompt_templates.DYNAMIC_PRIMARY_ENTITIES_ASSIGNMENT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
+        notes="Field-level assignment step for DYNAMIC_PRIMARY_ENTITIES; exhibit text passed as context_for_caching.",
+    ),
+    # ---------- LOB / program / product mapping ----------
+    CachePromptEntry(
+        usage_label="MAP_LOB_TO_VALID",
+        instruction_func=prompt_templates.MAP_LOB_TO_VALID_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
+        notes="Maps raw LOB values to valid crosswalk keys; valid-LOB list passed as context_for_caching.",
+    ),
+    CachePromptEntry(
+        usage_label="MAP_NETWORK_TO_VALID",
+        instruction_func=prompt_templates.MAP_NETWORK_TO_VALID_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
+        notes="Maps raw NETWORK values to valid crosswalk keys; valid-network list passed as context_for_caching.",
+    ),
+    CachePromptEntry(
+        usage_label="AARETE_DERIVED_PROGRAM",
+        instruction_func=prompt_templates.AARETE_DERIVED_PROGRAM_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
+        notes="Maps extracted PROGRAM names to AARETE_DERIVED_PROGRAM values; valid-values list passed as context_for_caching.",
+    ),
+    CachePromptEntry(
+        usage_label="AARETE_DERIVED_PRODUCT",
+        instruction_func=prompt_templates.AARETE_DERIVED_PRODUCT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
+        notes="Maps extracted PRODUCT names to AARETE_DERIVED_PRODUCT values; valid-values list passed as context_for_caching.",
+    ),
+    CachePromptEntry(
+        usage_label="PROGRAM_TO_LOB",
+        instruction_func=prompt_templates.PROGRAM_TO_LOB_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+        notes="Derives LOB values from raw PROGRAM + AARETE_DERIVED_PROGRAM; no context_for_caching at runtime.",
+    ),
+    CachePromptEntry(
+        usage_label="PRODUCT_TO_LOB",
+        instruction_func=prompt_templates.PRODUCT_TO_LOB_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+        notes="Derives LOB values from raw PRODUCT + AARETE_DERIVED_PRODUCT; no context_for_caching at runtime.",
+    ),
+    # ---------- service term split (Sonnet 4.5 — Haiku 4.5 won't cache) ----------
+    CachePromptEntry(
+        usage_label="SPLIT_SERVICE_TERM",
+        instruction_func=prompt_templates.SPLIT_SERVICE_TERM_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+        notes=(
+            "Moved off haiku_latest because Haiku 4.5's 4096-token "
+            "minimum cache size means the ~1600-token SPLIT_SERVICE_TERM "
+            "instruction was never cached. Sonnet 4.5's 1024-token "
+            "minimum makes it cache cleanly. With heavy repetition of "
+            "service terms across exhibits, cache reads on Sonnet beat "
+            "uncached Haiku on cost despite the higher per-token rate."
+        ),
+    ),
+    # ---------- preprocessing ----------
+    CachePromptEntry(
+        usage_label="EXHIBIT_HEADER",
+        instruction_func=prompt_templates.EXHIBIT_HEADER_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="EXHIBIT_LINKAGE",
+        instruction_func=prompt_templates.EXHIBIT_LINKAGE_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="EXHIBIT_TITLE_MATCH",
+        instruction_func=prompt_templates.EXHIBIT_TITLE_MATCH_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    # ---------- utility ----------
+    CachePromptEntry(
+        usage_label="DATE_FIX",
+        instruction_func=prompt_templates.DATE_FIX_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="DERIVED_TERM_DATE",
+        instruction_func=prompt_templates.DERIVED_TERM_DATE_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="CHECK_PROVIDER_NAME_MATCH",
+        instruction_func=prompt_templates.CHECK_PROVIDER_NAME_MATCH_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="SPECIAL_CASE_ASSIGNMENT",
+        instruction_func=prompt_templates.SPECIAL_CASE_ASSIGNMENT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="SPLIT_REIMB_DATES",
+        instruction_func=prompt_templates.SPLIT_REIMB_DATES_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    # ---------- code extraction family ----------
+    CachePromptEntry(
+        usage_label="CODE_EXPLICIT",
+        instruction_func=prompt_templates.CODE_EXPLICIT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="CODE_CATEGORY",
+        instruction_func=prompt_templates.CODE_CATEGORY_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="CODE_IMPLICIT_SPECIAL",
+        instruction_func=prompt_templates.CODE_IMPLICIT_SPECIAL_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="CODE_IMPLICIT",
+        instruction_func=prompt_templates.CODE_IMPLICIT_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="CODE_IMPLICIT_ARBITRATION",
+        instruction_func=prompt_templates.CODE_IMPLICIT_ARBITRATION_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="CODE_LAST_CHECK",
+        instruction_func=prompt_templates.CODE_LAST_CHECK_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="FILL_BILL_TYPE",
+        instruction_func=prompt_templates.FILL_BILL_TYPE_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    # ---------- one_to_one ----------
+    CachePromptEntry(
+        usage_label="TIN_NPI_TEMPLATE",
+        instruction_func=prompt_templates.TIN_NPI_TEMPLATE_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+    CachePromptEntry(
+        usage_label="ONE_TO_ONE_SINGLE_FIELD",
+        instruction_func=prompt_templates.ONE_TO_ONE_SINGLE_FIELD_INSTRUCTION,
+        runtime_model="sonnet_latest",
+        cache_class=CacheClass.INSTRUCTION,
+    ),
+)
+
+
+# Aliases let runtime usage_labels that don't match the canonical
+# instruction-function name still resolve to one registry entry.
+# Callers should prefer canonical labels; this is a compatibility shim.
+_USAGE_LABEL_ALIASES: dict[str, str] = {
+    "code_implicit_rag": "CODE_IMPLICIT",
+    "prompt_provider_info": "TIN_NPI_TEMPLATE",
+    "validate_reimbursements_for_llm": "VALIDATE_REIMBURSEMENTS",
+    "fill_bill_type": "FILL_BILL_TYPE",
+    "prompt_lob_relationship": "LOB_RELATIONSHIP",
+}
+
+
+def _build_index() -> dict[str, CachePromptEntry]:
+    index: dict[str, CachePromptEntry] = {}
+    for entry in _REGISTRY:
+        if entry.usage_label in index:
+            raise ValueError(
+                f"Duplicate registry entry for usage_label={entry.usage_label}"
+            )
+        index[entry.usage_label] = entry
+    return index
+
+
+_INDEX: dict[str, CachePromptEntry] = _build_index()
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+
+def all_entries() -> tuple[CachePromptEntry, ...]:
+    """Return every registered entry (including NONE / opted-out ones)."""
+    return _REGISTRY
+
+
+def get_entry(usage_label: str) -> CachePromptEntry | None:
+    """Look up an entry by usage_label. Resolves aliases."""
+    if usage_label in _INDEX:
+        return _INDEX[usage_label]
+    canonical = _USAGE_LABEL_ALIASES.get(usage_label)
+    if canonical is not None:
+        return _INDEX.get(canonical)
+    return None
+
+
+def _resolve_min_tokens(entry: CachePromptEntry, resolved_model_id: str) -> int:
+    """Pick the threshold to enforce: explicit on the entry > per-model > default."""
+    if entry.min_instruction_tokens is not None:
+        return entry.min_instruction_tokens
+    # Imported lazily so this module stays importable without llm_utils.
+    from src.utils.llm_utils import _min_cache_tokens
+
+    return _min_cache_tokens(resolved_model_id)
+
+
+def iter_warming_targets() -> Iterator[tuple[str, str, str]]:
+    """Yield (usage_label, resolved_model_id, instruction_text) tuples for
+    every entry whose instruction block is supposed to be cached.
+
+    Skips:
+      - cache_class == NONE
+      - cache_class == CONTEXT (no static instruction to warm)
+      - entries whose resolved model does not support prompt caching
+      - entries whose instruction function fails to render
+    """
+    # Imported here to avoid a circular import: llm_utils → cache_registry
+    # is allowed, but cache_registry → llm_utils only at call time.
+    from src.utils.llm_utils import _supports_prompt_cache
+
+    for entry in _REGISTRY:
+        if entry.cache_class in (CacheClass.NONE, CacheClass.CONTEXT):
+            continue
+        if entry.instruction_func is None:
+            continue
+
+        resolved_model_id = config.resolve_model_id(entry.runtime_model)
+        if not _supports_prompt_cache(resolved_model_id):
+            logging.info(
+                "cache_registry: skipping warm-up for %s — model %s "
+                "(resolved %s) does not support prompt caching.",
+                entry.usage_label,
+                entry.runtime_model,
+                resolved_model_id,
+            )
+            continue
+
+        try:
+            instruction_text = entry.instruction_func()
+        except Exception as exc:  # field bundles can be unavailable in CLI/tests
+            logging.debug(
+                "cache_registry: skipping %s — instruction function raised: %s",
+                entry.usage_label,
+                exc,
+            )
+            continue
+
+        token_estimate = estimate_instruction_tokens(instruction_text)
+        threshold = _resolve_min_tokens(entry, resolved_model_id)
+        if token_estimate < threshold:
+            # Warn but still warm — the call still tracks
+            # cache_creation_input_tokens in the response, and the runtime
+            # cache_miss_reason makes the gap visible.
+            logging.warning(
+                "cache_registry: %s instruction is ~%d tokens, below the "
+                "%d-token minimum for %s; cache_control block may not be "
+                "honored.",
+                entry.usage_label,
+                token_estimate,
+                threshold,
+                resolved_model_id,
+            )
+
+        yield entry.usage_label, resolved_model_id, instruction_text
+
+
+def audit_below_threshold() -> list[tuple[str, int, int]]:
+    """Return (usage_label, estimated_tokens, min_required) for every
+    INSTRUCTION-class entry whose instruction is below threshold. The
+    threshold is resolved per-model: Sonnet 4.5 = 1024, Haiku 4.5 = 4096,
+    etc. Useful for a startup audit / contract test.
+    """
+    findings: list[tuple[str, int, int]] = []
+    for entry in _REGISTRY:
+        if entry.cache_class not in (
+            CacheClass.INSTRUCTION,
+            CacheClass.INSTRUCTION_PLUS_CONTEXT,
+        ):
+            continue
+        if entry.instruction_func is None:
+            continue
+        try:
+            text = entry.instruction_func()
+        except Exception:
+            continue
+        resolved_model_id = config.resolve_model_id(entry.runtime_model)
+        threshold = _resolve_min_tokens(entry, resolved_model_id)
+        tokens = estimate_instruction_tokens(text)
+        if tokens < threshold:
+            findings.append((entry.usage_label, tokens, threshold))
+    return findings
+
+
+def instruction_cache_entries() -> Iterable[CachePromptEntry]:
+    """Entries where the instruction block is expected to be cached."""
+    return tuple(
+        e
+        for e in _REGISTRY
+        if e.cache_class
+        in (CacheClass.INSTRUCTION, CacheClass.INSTRUCTION_PLUS_CONTEXT)
+    )
+
+
+def context_cache_entries() -> Iterable[CachePromptEntry]:
+    """Entries where a dynamic context block is cached separately."""
+    return tuple(
+        e
+        for e in _REGISTRY
+        if e.cache_class in (CacheClass.CONTEXT, CacheClass.INSTRUCTION_PLUS_CONTEXT)
+    )

src/prompts/fieldset.py

@@ -166,7 +166,7 @@ class Field:
         """Resolves placeholders in the prompt using the valid_values property."""
 
         # If no valid_values to resolve
-        if string_utils.is_empty(self.valid_values):  #
+        if string_utils.is_empty(self.valid_values):
             return self.prompt
 
         if not string_utils.is_empty(self.valid_values) and not constants:
@@ -177,12 +177,13 @@ class Field:
         resolved_prompt = self.prompt
         if "{valid_values}" in resolved_prompt:
             resolved_value = self.valid_values
-            if hasattr(constants, resolved_value):
+            # Only call hasattr if resolved_value is a string
+            if isinstance(resolved_value, str) and hasattr(constants, resolved_value):
                 try:
                     resolved_prompt = resolved_prompt.replace(
                         "{valid_values}", str(constants.get_constant(resolved_value))
                     )
-                except:
+                except Exception:
                     resolved_prompt = resolved_prompt.replace(
                         "{valid_values}", str(resolved_value)
                     )

src/prompts/investment_prompts.json

@@ -99,8 +99,7 @@
         "field_name": "LOB",
         "relationship": "one_to_n",
         "field_type": "dynamic_primary",
-        "prompt": "A distinct, high-level category of health insurance coverage. Choose LOBs ONLY from the following values: {valid_values}. Do NOT extract Programs, Products, or other entities listed. Choose ONLY from the valid values presented. Only return values that are explicitly written - do NOT fill this value out by deducing the answer from sub-Programs that may fall under one of the valid answers. If only the sub-Program is seen, return 'N/A'. Do NOT extract an LOB that appears only incidentally as part of a rate name, reimbursement methodology, or pricing reference (e.g., 'Medicaid Base DRG rate'). Only extract an LOB if the text explicitly declares or restricts the scope of the exhibit or agreement to that LOB.",
-        "valid_values": "VALID_LOBS",
+        "prompt": "A distinct, high-level category of health insurance coverage. Only return values that are explicitly written in the contract. Do NOT extract an LOB that appears only incidentally as part of a rate name, reimbursement methodology, or pricing reference (e.g., 'Medicaid Base DRG rate'). Only extract an LOB if the text explicitly declares or restricts the scope of the exhibit or agreement to that LOB.",
         "retrieval_question": "What Line of Business does this contract apply to?"
     },
     {
@@ -115,8 +114,7 @@
         "field_name": "NETWORK",
         "relationship": "one_to_n",
         "field_type": "dynamic_primary",
-        "prompt": "A type of managed care structures that defines what providers a patient can use. Choose Networks ONLY from the following values: {valid_values}. Only return values that are explicitly written.",
-        "valid_values": "VALID_NETWORKS",
+        "prompt": "A type of managed care structures that defines what providers a patient can use. Only return values that are explicitly written in the contract.",
         "retrieval_question": "What Network (HMO, PPO, etc) does this contract apply to?"
     },
     {
@@ -131,32 +129,29 @@
         "field_name": "PRODUCT",
         "relationship": "one_to_n",
         "field_type": "dynamic_primary",
-        "prompt": "The brand name of a health plan or product offered by the Payer. Choose Products ONLY from the following values: {valid_values}. Choose ONLY from the valid values presented; if no value from the list is explicitly mentioned, return 'N/A'. Do NOT include the name of the Payer itself (such as 'Molina', 'Centene' etc) - rather, look for the Product that these healthplans offer. Do NOT extract LOBs (such as 'Medicaid', 'Medicare', 'Marketplace', 'Duals') or Programs (such as 'Medicaid Managed Care', 'Medicare Advantage', 'ACA') as products - those belong in the LOB and PROGRAM fields respectively. Return values exactly as written in the valid values list.",
-        "valid_values": "VALID_PRODUCTS",
+        "prompt": "The brand name of a health plan or product offered by the Payer. Do NOT include the name of the Payer itself (such as 'Molina', 'Centene' etc) - rather, look for the Product that these health plans offer. List any products explicitly stated in the text.",
         "retrieval_question": "What Product does this contract apply to?"
     },
     {
         "field_name": "AARETE_DERIVED_PRODUCT",
         "relationship": "one_to_n",
-        "field_type": "TBD",
-        "prompt": "TBD",
-        "base_field": "PRODUCT"
+        "field_type": "derived_dynamic_primary",
+        "base_field": "PRODUCT",
+        "valid_values": "VALID_AARETE_DERIVED_PRODUCTS"
     },
     {
         "field_name": "PROGRAM",
         "relationship": "one_to_n",
         "field_type": "dynamic_primary",
-        "prompt": "The brand name of a health plan or product offered by the State or Federal government. List any Programs explicitly stated in the text. Choose ONLY from the following: {valid_values}. Note that 'Medicare' on it's own does NOT mean 'Medicare Advantage'. If only 'Medicare' is seen, do not write 'Medicare Advantage'. IMPORTANT - CHIP vs CHIPP distinction: 'CHIP' (Children's Health Insurance Program) is a valid standalone value. If you see 'CHIP HMO' or just 'CHIP' (without 'Perinate'), extract 'CHIP'. However, if you see 'CHIP Perinate' or 'CHIP-P' (Children's Health Insurance Program Perinate), extract 'CHIPP' (NOT 'CHIP'). These are two distinct programs - do not confuse them. Choose the answer from the list that most closely matches what is found in the text. Use reasonable judgment for obvious synonyms and equivalent terms (e.g., 'Medicare-Medicaid Program' matches 'Medicare-Medicaid Plan'). Do NOT extract a program value when it appears only as a regulatory citation, legal reference, or compliance note (e.g., 'section 2713 of ACA', 'pursuant to 42 CFR', 'under ERISA', 'as defined by ACA'). Only extract a program if the text explicitly identifies it as the type of coverage or program governing the scope of the agreement or exhibit.",
-        "valid_values": "VALID_PROGRAMS",
+        "prompt": "The brand name of a health plan or product offered by the State or Federal government. List any Programs explicitly stated in the text. Do NOT extract a program value when it appears only as a regulatory citation, legal reference, or compliance note (e.g., 'section 2713 of ACA', 'pursuant to 42 CFR', 'under ERISA', 'as defined by ACA'). Only extract a program if the text explicitly identifies it as the type of coverage or program governing the scope of the agreement or exhibit.",
         "retrieval_question": "What Program(s) does this contract apply to?"
     },
     {
         "field_name": "AARETE_DERIVED_PROGRAM",
         "relationship": "one_to_n",
-        "field_type": "crosswalk",
-        "prompt": "TBD",
-        "crosswalk": "CROSSWALK_PROGRAM",
-        "base_field": "PROGRAM"
+        "field_type": "derived_dynamic_primary",
+        "base_field": "PROGRAM",
+        "valid_values": "VALID_AARETE_DERIVED_PROGRAMS"
     },
     {
         "field_name": "CONTRACT_TITLE",

src/scripts/prompt_call_log_report.py

@@ -0,0 +1,254 @@
+import importlib.util
+import inspect
+import re
+from pathlib import Path
+
+import pandas as pd
+
+ANTHROPIC_CACHE_MIN_TOKENS = 1024
+WORDS_TO_TOKENS_MULTIPLIER = 1.3
+
+ALIASES = {
+    "date_fix": "DATE_FIX",
+    "derive_term_date": "DERIVED_TERM_DATE",
+}
+
+ROUND_COLS = [
+    "avg_prompt_tokens_est",
+    "avg_instruction_tokens_est",
+    "avg_context_tokens_est",
+    "avg_input_tokens",
+    "avg_api_cached_input_tokens",
+    "avg_api_total_billed_input_tokens",
+]
+
+TOKENS_PER_CALL_COLS = [
+    "timestamp",
+    "contract_filename",
+    "usage_label",
+    "model_id",
+    "resolved_model_id",
+    "status",
+    "from_local_memory_cache",
+    "prompt_tokens_est",
+    "instruction_tokens_est",
+    "context_tokens_est",
+    "total_input_tokens_est",
+    "api_cached_input_tokens",
+    "api_total_billed_input_tokens",
+    "est_minus_api_total_tokens",
+    "longest_component",
+    "input_tokens",
+    "output_tokens",
+    "cache_creation_tokens",
+    "cache_read_tokens",
+]
+
+INSTRUCTION_AUDIT_COLS = [
+    "requested_name",
+    "normalized_prompt",
+    "instruction_function_found",
+    "instruction_chars",
+    "instruction_tokens_est",
+    "cache_min_tokens",
+    "cacheable_at_threshold",
+    "tokens_to_reach_threshold",
+    "priority",
+    "instruction_preview",
+]
+
+
+def _min_cache_tokens_for_label(
+    df: pd.DataFrame,
+    label: str,
+) -> int:
+    """Return the most restrictive cache-minimum across all model IDs seen for
+    *label* in the prompt-call log.  Falls back to ANTHROPIC_CACHE_MIN_TOKENS
+    when the column is absent or empty."""
+    from src.utils.llm_utils import _min_cache_tokens
+
+    col = "resolved_model_id" if "resolved_model_id" in df.columns else "model_id"
+    mask = df.get("usage_label", pd.Series(dtype=str)).astype(str).str.strip() == label
+    model_ids = df.loc[mask, col].dropna().unique() if col in df.columns else []
+    if len(model_ids) == 0:
+        return ANTHROPIC_CACHE_MIN_TOKENS
+    return max(_min_cache_tokens(str(mid)) for mid in model_ids)
+
+
+def _build_instruction_audit(
+    df: pd.DataFrame,
+    prompt_templates_file: Path,
+) -> pd.DataFrame:
+    instruction_texts: dict[str, str] = {}
+    if not prompt_templates_file.exists():
+        raise FileNotFoundError(
+            f"Prompt templates file not found: {prompt_templates_file}"
+        )
+
+    spec = importlib.util.spec_from_file_location(
+        "prompt_templates_for_audit", prompt_templates_file
+    )
+    if spec is None or spec.loader is None:
+        raise ImportError(f"Unable to load module from {prompt_templates_file}")
+
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+
+    for name, fn in inspect.getmembers(module, inspect.isfunction):
+        if not name.endswith("_INSTRUCTION"):
+            continue
+        label = name.removesuffix("_INSTRUCTION")
+        try:
+            text = fn()
+        except TypeError:
+            text = ""
+        instruction_texts[label] = text if isinstance(text, str) else str(text)
+
+    prompt_names = sorted(
+        {str(v) for v in df.get("usage_label", pd.Series(dtype=str)).dropna().unique()}
+    )
+
+    rows: list[dict] = []
+    for raw_name in prompt_names:
+        normalized = ALIASES.get(raw_name.strip(), raw_name.strip().upper())
+        text = instruction_texts.get(normalized, "")
+        tokens = int(round(len(re.findall(r"\S+", text)) * WORDS_TO_TOKENS_MULTIPLIER))
+        threshold = _min_cache_tokens_for_label(df, raw_name.strip())
+        gap = max(0, threshold - tokens)
+        rows.append(
+            {
+                "requested_name": raw_name,
+                "normalized_prompt": normalized,
+                "instruction_function_found": normalized in instruction_texts,
+                "instruction_chars": len(text),
+                "instruction_tokens_est": tokens,
+                "cache_min_tokens": threshold,
+                "cacheable_at_threshold": tokens >= threshold,
+                "tokens_to_reach_threshold": gap,
+                "priority": "OK" if gap == 0 else ("HIGH" if gap >= 300 else "MEDIUM"),
+                "instruction_preview": (
+                    (text[:240].replace("\n", " ") + "...") if text else ""
+                ),
+            }
+        )
+
+    if not rows:
+        return pd.DataFrame(columns=INSTRUCTION_AUDIT_COLS)
+
+    return pd.DataFrame(rows).sort_values(
+        by=["cacheable_at_threshold", "tokens_to_reach_threshold", "normalized_prompt"],
+        ascending=[True, False, True],
+    )
+
+
+def run_reports(
+    prompt_call_log: Path,
+    output_dir: Path,
+    prompt_templates_file: Path,
+) -> dict[str, Path]:
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    if not prompt_call_log.exists():
+        raise FileNotFoundError(f"Prompt call log not found: {prompt_call_log}")
+    df = pd.read_csv(prompt_call_log)
+
+    tokens_per_call = df[[c for c in TOKENS_PER_CALL_COLS if c in df.columns]].copy()
+    tokens_per_call = tokens_per_call.sort_values(
+        by=["contract_filename", "usage_label", "timestamp"]
+    )
+
+    success_df = df[df["status"] == "success"]
+    calls_per_contract = (
+        success_df.groupby(["contract_filename", "usage_label"], dropna=False)
+        .agg(
+            call_count=("usage_label", "size"),
+            avg_prompt_tokens_est=("prompt_tokens_est", "mean"),
+            avg_instruction_tokens_est=("instruction_tokens_est", "mean"),
+            avg_context_tokens_est=("context_tokens_est", "mean"),
+            avg_input_tokens=("input_tokens", "mean"),
+            avg_api_cached_input_tokens=("api_cached_input_tokens", "mean"),
+            avg_api_total_billed_input_tokens=("api_total_billed_input_tokens", "mean"),
+        )
+        .reset_index()
+        .sort_values(by=["contract_filename", "call_count"], ascending=[True, False])
+    )
+    if not calls_per_contract.empty:
+        calls_per_contract[ROUND_COLS] = calls_per_contract[ROUND_COLS].round(2)
+
+    longest_component = (
+        success_df.groupby(["usage_label", "longest_component"], dropna=False)
+        .size()
+        .reset_index(name="occurrences")
+        .merge(
+            success_df.groupby(["usage_label"], dropna=False)
+            .size()
+            .reset_index(name="total_calls"),
+            on=["usage_label"],
+            how="left",
+        )
+    )
+    if longest_component.empty:
+        longest_component = pd.DataFrame(
+            columns=[
+                "usage_label",
+                "longest_component",
+                "occurrences",
+                "percent_of_calls",
+            ]
+        )
+    else:
+        longest_component["percent_of_calls"] = (
+            (longest_component["occurrences"] / longest_component["total_calls"]) * 100
+        ).round(2)
+        longest_component = longest_component.drop(columns=["total_calls"]).sort_values(
+            by=["usage_label", "occurrences"], ascending=[True, False]
+        )
+
+    instruction_audit = _build_instruction_audit(
+        df, prompt_templates_file=prompt_templates_file
+    )
+
+    tokens_path = output_dir / "runtime_prompt_tokens_per_call.csv"
+    calls_path = output_dir / "runtime_prompt_calls_per_contract.csv"
+    longest_path = output_dir / "runtime_longest_component_summary.csv"
+    audit_path = output_dir / "prompt_instruction_audit.csv"
+
+    tokens_per_call.to_csv(tokens_path, index=False)
+    calls_per_contract.to_csv(calls_path, index=False)
+    longest_component.to_csv(longest_path, index=False)
+    instruction_audit.to_csv(audit_path, index=False)
+
+    return {
+        "tokens_per_call": tokens_path,
+        "calls_per_contract": calls_path,
+        "longest_component": longest_path,
+        "instruction_audit": audit_path,
+    }
+
+
+PROMPT_TEMPLATES_FILE = Path("src/prompts/prompt_templates.py")
+
+
+def main() -> None:
+    prompt_call_log = max(
+        Path("outputs").glob("**/tracking/*-PROMPT-CALLS.csv"),
+        key=lambda p: p.stat().st_mtime,
+        default=None,
+    )
+    if prompt_call_log is None:
+        raise FileNotFoundError("No prompt call logs found under outputs/**/tracking")
+
+    outputs = run_reports(
+        prompt_call_log=prompt_call_log,
+        output_dir=prompt_call_log.parent,
+        prompt_templates_file=PROMPT_TEMPLATES_FILE,
+    )
+
+    print(f"Source log: {prompt_call_log}")
+    print("Generated report files:")
+    for key, path in outputs.items():
+        print(f"- {key}: {path}")
+
+
+if __name__ == "__main__":
+    main()

src/tests/test_crosswalk_utils.py

@@ -128,28 +128,6 @@ def test_to_csv_no_metadata(tmp_path):
     assert result_dict == {"A": "one", "B": "two"}
 
 
-def test_from_json_client_none_merges_all_client_mappings(tmp_path, monkeypatch):
-    data = {
-        "metadata": {"from_col": "from_col", "to_col": "to_col"},
-        "mapping": {},
-        "client_mapping": {
-            "ClientA": {"Connexus": "Commercial", "Synergy": "Commercial"},
-            "ClientB": {"HEALTHfirst": "Medicaid"},
-        },
-    }
-    file_path = tmp_path / "client_merge.json"
-    with open(file_path, "w") as f:
-        json.dump(data, f)
-
-    monkeypatch.setattr("src.crosswalk.crosswalk_builder.CLIENT", ["None"])
-    monkeypatch.setattr("src.crosswalk.crosswalk_builder.STATE", ["None"])
-
-    builder = CrosswalkBuilder().from_json(file_path)
-    assert builder.mapping["Connexus"] == "Commercial"
-    assert builder.mapping["Synergy"] == "Commercial"
-    assert builder.mapping["HEALTHfirst"] == "Medicaid"
-
-
 def test_apply_crosswalk():
     mapping = {"A": "1", "B": "2", "C": "3"}
     assert apply_crosswalk("A", mapping) == ["1"]

src/tests/test_one_to_n.py

@@ -648,14 +648,10 @@ class TestOneToNFuncs(unittest.TestCase):
         "src.pipelines.shared.extraction.one_to_n_funcs.aarete_derived.get_crosswalk_fields"
     )
     @patch("src.pipelines.shared.extraction.one_to_n_funcs.get_lob_relationship")
-    @patch(
-        "src.pipelines.shared.extraction.one_to_n_funcs.aarete_derived.fill_na_mapping"
-    )
     @patch("src.pipelines.shared.extraction.one_to_n_funcs.split_reimb_dates")
     def test_one_to_n_cleaning_full_pipeline(
         self,
         mock_split_dates,
-        mock_fill_na,
         mock_get_lob,
         mock_crosswalk,
     ):
@@ -664,7 +660,6 @@ class TestOneToNFuncs(unittest.TestCase):
 
         mock_crosswalk.return_value = input_data
         mock_get_lob.return_value = input_data
-        mock_fill_na.return_value = input_data
         mock_split_dates.return_value = input_data
 
         result = one_to_n_funcs.one_to_n_cleaning(
@@ -674,7 +669,6 @@ class TestOneToNFuncs(unittest.TestCase):
         # Verify all cleaning steps were called
         mock_crosswalk.assert_called_once()
         mock_get_lob.assert_called_once()
-        mock_fill_na.assert_called_once()
         mock_split_dates.assert_called_once()
 
         self.assertEqual(result, input_data)

src/tests/test_postprocess.py

@@ -943,24 +943,37 @@ class TestPostprocessFunctions(unittest.TestCase):
         self.assertEqual(result_df.loc[0, "CONTRACT_TITLE"], "")
         self.assertEqual(result_df.loc[1, "CONTRACT_TITLE"], "Another Valid")
 
-    def test_fill_na_mapping_derives_lob_from_aarete_derived_product(self):
-        """fill_na_mapping should derive LOB from AARETE_DERIVED_PRODUCT."""
-        rows = [
-            {
-                "PRODUCT": "",
-                "AARETE_DERIVED_PRODUCT": '["Connexus", "Synergy"]',
-                "AARETE_DERIVED_LOB": "",
-                "LOB": "",
-                "AARETE_DERIVED_PROGRAM": "",
-                "LOB_PROGRAM_RELATIONSHIP": "",
-                "LOB_PRODUCT_RELATIONSHIP": "",
-            }
-        ]
+    @patch(
+        "src.pipelines.shared.postprocessing.aarete_derived.prompt_calls.prompt_product_to_lob",
+        return_value=["Commercial"],
+    )
+    def test_fill_na_mapping_derives_lob_from_aarete_derived_product(
+        self, mock_product_lob
+    ):
+        """fill_na_mapping should derive LOB from AARETE_DERIVED_PRODUCT via LLM."""
+        rows = pd.DataFrame(
+            [
+                {
+                    "FILE_NAME": "test_file",
+                    "PRODUCT": ["Connexus", "Synergy"],
+                    "AARETE_DERIVED_PRODUCT": ["Connexus", "Synergy"],
+                    "AARETE_DERIVED_LOB": "",
+                    "LOB": "",
+                    "AARETE_DERIVED_PROGRAM": "",
+                    "LOB_PROGRAM_RELATIONSHIP": "",
+                    "LOB_PRODUCT_RELATIONSHIP": "",
+                }
+            ]
+        )
 
-        result = aarete_derived.fill_na_mapping(rows)
+        constants = MagicMock()
+        constants.CROSSWALK_PROGRAM_LOB.mapping = {}
+        constants.CROSSWALK_PRODUCT_LOB.mapping = {}
+        constants.CROSSWALK_LOB.mapping = {}
 
-        self.assertEqual(result[0]["AARETE_DERIVED_LOB"], ["Commercial"])
-        self.assertEqual(result[0]["LOB"], ["Commercial"])
+        result = aarete_derived.fill_na_mapping(rows, constants)
+
+        self.assertEqual(result.iloc[0]["AARETE_DERIVED_LOB"], ["Commercial"])
 
 
 class TestStandardizeReimbMethodAndFeeScheduleUOM(unittest.TestCase):

src/utils/llm_utils.py

@@ -11,6 +11,7 @@ from collections import OrderedDict
 
 import anthropic
 import src.utils.string_utils as string_utils
+import src.utils.prompt_call_tracking as prompt_call_tracking
 import src.utils.usage_tracking as usage_tracking
 from botocore.exceptions import ClientError, ReadTimeoutError
 from src import config
@@ -103,22 +104,56 @@ def _validate_model_support(
 def _supports_prompt_cache(model_id: str) -> bool:
     """Return True if the given model supports prompt caching via system cache_control.
 
-    Enable only for models that are known to support prompt caching:
+    Enable for models that are known to support prompt caching:
     - Claude 3.5 Sonnet v2+ (20241022-v2:0)
     - Claude 3.7 Sonnet
     - Claude 4 Sonnet
     - Claude 4.5 Sonnet
+    - Claude 4.5 Haiku
     """
     supported_ids = {
         getattr(config, "MODEL_ID_CLAUDE35_SONNET_V2", None),
         getattr(config, "MODEL_ID_CLAUDE37_SONNET", None),
         getattr(config, "MODEL_ID_CLAUDE4_SONNET", None),
         getattr(config, "MODEL_ID_CLAUDE45_SONNET", None),
+        getattr(config, "MODEL_ID_CLAUDE45_HAIKU", None),
     }
     # Filter out None values in case any model IDs are not configured
     return model_id in {m for m in supported_ids if m is not None}
 
 
+# Per Anthropic docs (https://docs.anthropic.com/en/docs/prompt-caching),
+# the minimum cacheable prompt length differs by model family:
+#   1024 tokens  — Sonnet 3.7 / 4 / 4.5, Opus 4 / 4.1
+#   2048 tokens  — Sonnet 4.6, Haiku 3.5 (deprecated)
+#   4096 tokens  — Sonnet/Opus Mythos, Opus 4.5/4.6/4.7, Haiku 4.5
+# Caches against blocks below the threshold are silently dropped — the
+# request still succeeds and returns content, but `cache_creation_tokens`
+# and `cache_read_tokens` will both be 0 forever. That is the
+# `no_cache_event` signature surfaced in runtime tracking.
+_MODEL_MIN_CACHE_TOKENS_DEFAULT = 1024
+_MODEL_MIN_CACHE_TOKENS: dict[str, int] = {
+    # Sonnet 4.5 — 1024
+    getattr(config, "MODEL_ID_CLAUDE45_SONNET", "_unset_45_sonnet"): 1024,
+    # Sonnet 4 / 3.7 / 3.5 v2 — 1024
+    getattr(config, "MODEL_ID_CLAUDE4_SONNET", "_unset_4_sonnet"): 1024,
+    getattr(config, "MODEL_ID_CLAUDE37_SONNET", "_unset_37_sonnet"): 1024,
+    getattr(config, "MODEL_ID_CLAUDE35_SONNET_V2", "_unset_35_sonnet_v2"): 1024,
+    # Haiku 4.5 — 4096
+    getattr(config, "MODEL_ID_CLAUDE45_HAIKU", "_unset_45_haiku"): 4096,
+}
+
+
+def _min_cache_tokens(model_id: str) -> int:
+    """Return the minimum prompt-cache block size honored by the model.
+
+    Falls back to 1024 (the historical Sonnet minimum) when the model is
+    unknown — that is the most permissive choice; callers only get a
+    surprise `no_cache_event` if the actual minimum is higher.
+    """
+    return _MODEL_MIN_CACHE_TOKENS.get(model_id, _MODEL_MIN_CACHE_TOKENS_DEFAULT)
+
+
 def _classify_cache_outcome(
     *,
     has_context_cache: bool,
@@ -410,11 +445,12 @@ def _build_claude_3_request_body(
     if context_for_caching:
         if cache and model_id and _supports_prompt_cache(model_id):
             token_estimate = len(context_for_caching.split()) * 1.3
-            if token_estimate < 1024:
+            min_tokens = _min_cache_tokens(model_id)
+            if token_estimate < min_tokens:
                 logging.warning(
                     f"context_for_caching is ~{token_estimate:.0f} tokens, below the "
-                    "Anthropic 1024-token minimum for prompt caching; context will be "
-                    "sent but the cache_control block may not actually be cached."
+                    f"Anthropic {min_tokens}-token minimum for prompt caching on {model_id}; "
+                    "context will be sent but the cache_control block may not actually be cached."
                 )
             user_content.append(
                 {
@@ -557,10 +593,30 @@ def invoke_claude(
         context_for_caching=context_for_caching,
     )
 
+    # Resolve alias up-front so cache-policy classification in tracking
+    # logs uses the durable id, not whatever alias the caller happened to
+    # pass.
+    resolved_model_id = config.resolve_model_id(model_id)
+    model_supports_cache = _supports_prompt_cache(resolved_model_id)
+
     # Check cache first - thread-safe access
     with _cache_lock:
         if cache_key in claude_cache:
             claude_cache.move_to_end(cache_key)  # Mark as recently used
+            prompt_call_tracking.log_prompt_call(
+                contract_filename=filename,
+                usage_label=usage_label,
+                model_id=model_id,
+                cache_enabled=cache,
+                max_tokens=max_tokens,
+                prompt=prompt,
+                instruction=instruction,
+                context_for_caching=context_for_caching,
+                status="success",
+                from_local_memory_cache=True,
+                resolved_model_id=resolved_model_id,
+                model_supports_cache=model_supports_cache,
+            )
             # One llm_call_inmem_hit event per in-memory cache return.
             from src.utils import instrumentation, instrumentation_context
 
@@ -594,49 +650,63 @@ def invoke_claude(
                 )
             return claude_cache[cache_key]
 
-    # Resolve alias to actual model ID
-    resolved_model_id = config.resolve_model_id(model_id)
-
     # Validate model support (checks for deprecated and unsupported models)
     _validate_model_support(model_id, resolved_model_id)
 
     # Push call metadata into context so _track_usage_from_response can read it.
     from src.utils.instrumentation_context import instr_scope
 
-    with instr_scope(
-        _usage_label=usage_label,
-        _has_context_cache=bool(context_for_caching),
-        _context_chars=len(context_for_caching) if context_for_caching else 0,
-        _prompt_chars=len(prompt) if prompt else 0,
-        _cache_key=cache_key,
-    ):
-        if config.RUN_MODE == "local":
-            response = local_claude_3_and_up(
-                prompt,
-                resolved_model_id,
-                filename,
-                max_tokens,
-                cache=cache,
-                instruction=instruction,
-                usage_label=usage_label,
-                context_for_caching=context_for_caching,
-            )
-        elif config.RUN_MODE == "ec2":
-            response = ec2_claude_3_and_up(
-                prompt,
-                resolved_model_id,
-                filename,
-                max_tokens,
-                cache=cache,
-                instruction=instruction,
-                usage_label=usage_label,
-                context_for_caching=context_for_caching,
-            )
-        else:
-            logging.error(
-                "Usage: python local_main.py <-local> OR python main.py <ec2>"
-            )
-            raise
+    try:
+        with instr_scope(
+            _usage_label=usage_label,
+            _has_context_cache=bool(context_for_caching),
+            _context_chars=len(context_for_caching) if context_for_caching else 0,
+            _prompt_chars=len(prompt) if prompt else 0,
+            _cache_key=cache_key,
+        ):
+            if config.RUN_MODE == "local":
+                response = local_claude_3_and_up(
+                    prompt,
+                    resolved_model_id,
+                    filename,
+                    max_tokens,
+                    cache=cache,
+                    instruction=instruction,
+                    usage_label=usage_label,
+                    context_for_caching=context_for_caching,
+                )
+            elif config.RUN_MODE == "ec2":
+                response = ec2_claude_3_and_up(
+                    prompt,
+                    resolved_model_id,
+                    filename,
+                    max_tokens,
+                    cache=cache,
+                    instruction=instruction,
+                    usage_label=usage_label,
+                    context_for_caching=context_for_caching,
+                )
+            else:
+                logging.error(
+                    "Usage: python local_main.py <-local> OR python main.py <ec2>"
+                )
+                raise
+    except Exception as e:
+        prompt_call_tracking.log_prompt_call(
+            contract_filename=filename,
+            usage_label=usage_label,
+            model_id=resolved_model_id,
+            cache_enabled=cache,
+            max_tokens=max_tokens,
+            prompt=prompt,
+            instruction=instruction,
+            context_for_caching=context_for_caching,
+            status="error",
+            error=str(e),
+            resolved_model_id=resolved_model_id,
+            model_supports_cache=model_supports_cache,
+        )
+        raise
 
     # Store response in cache with LRU eviction - thread-safe
     _store_in_cache(cache_key, response)
@@ -806,9 +876,34 @@ def local_claude_3_and_up(
             response_body = json.loads(response.get("body").read())
             response_text = _extract_text_from_claude_response(response_body)
 
+            (
+                input_tokens,
+                output_tokens,
+                cache_creation_tokens,
+                cache_read_tokens,
+            ) = usage_tracking.extract_usage_from_response(response_body)
+
             # Track usage and costs from API response (including cache tokens)
             _track_usage_from_response(response_body, filename, model_id)
 
+            prompt_call_tracking.log_prompt_call(
+                contract_filename=filename,
+                usage_label=usage_label or "UNLABELED",
+                model_id=model_id,
+                cache_enabled=cache,
+                max_tokens=max_tokens,
+                prompt=prompt,
+                instruction=instruction,
+                context_for_caching=context_for_caching,
+                status="success",
+                input_tokens=input_tokens,
+                output_tokens=output_tokens,
+                cache_creation_tokens=cache_creation_tokens,
+                cache_read_tokens=cache_read_tokens,
+                resolved_model_id=model_id,
+                model_supports_cache=_supports_prompt_cache(model_id),
+            )
+
             if config.ENABLE_ANSWER_TRACKING:
                 config.ANSWER_TRACKING_DICT[prompt] = response_text
 
@@ -1050,9 +1145,34 @@ def ec2_claude_3_and_up(
                     response_body = json.loads(response.get("body").read())
                     response_text = _extract_text_from_claude_response(response_body)
 
+                    (
+                        input_tokens,
+                        output_tokens,
+                        cache_creation_tokens,
+                        cache_read_tokens,
+                    ) = usage_tracking.extract_usage_from_response(response_body)
+
                     # Track usage and costs from API response (including cache tokens)
                     _track_usage_from_response(response_body, filename, model_id)
 
+                    prompt_call_tracking.log_prompt_call(
+                        contract_filename=filename,
+                        usage_label=usage_label or "UNLABELED",
+                        model_id=model_id,
+                        cache_enabled=cache,
+                        max_tokens=max_tokens,
+                        prompt=prompt,
+                        instruction=instruction,
+                        context_for_caching=context_for_caching,
+                        status="success",
+                        input_tokens=input_tokens,
+                        output_tokens=output_tokens,
+                        cache_creation_tokens=cache_creation_tokens,
+                        cache_read_tokens=cache_read_tokens,
+                        resolved_model_id=model_id,
+                        model_supports_cache=_supports_prompt_cache(model_id),
+                    )
+
                     return response_text
 
                 except ValueError as e:
@@ -1142,9 +1262,34 @@ def ec2_claude_3_and_up(
                 response_body = json.loads(response.get("body").read())
                 response_text = _extract_text_from_claude_response(response_body)
 
+                (
+                    input_tokens,
+                    output_tokens,
+                    cache_creation_tokens,
+                    cache_read_tokens,
+                ) = usage_tracking.extract_usage_from_response(response_body)
+
                 # Track usage and costs from API response (including cache tokens)
                 _track_usage_from_response(response_body, filename, model_id)
 
+                prompt_call_tracking.log_prompt_call(
+                    contract_filename=filename,
+                    usage_label=usage_label or "UNLABELED",
+                    model_id=model_id,
+                    cache_enabled=cache,
+                    max_tokens=max_tokens,
+                    prompt=prompt,
+                    instruction=instruction,
+                    context_for_caching=context_for_caching,
+                    status="success",
+                    input_tokens=input_tokens,
+                    output_tokens=output_tokens,
+                    cache_creation_tokens=cache_creation_tokens,
+                    cache_read_tokens=cache_read_tokens,
+                    resolved_model_id=model_id,
+                    model_supports_cache=_supports_prompt_cache(model_id),
+                )
+
                 return response_text
 
             except ValueError as e:
@@ -1241,32 +1386,58 @@ def warm_prompt_cache(
 ):
     """
     Warms the prompt cache by sending a minimal request with the instruction.
-    This should be called once before parallel processing to ensure the cache is registered.
+    This should be called once before parallel processing to ensure the cache
+    is registered.
+
+    Cache entries are model-specific in Bedrock — a cache created against
+    `sonnet_latest` is NOT readable from a runtime call invoked against
+    `haiku_latest`. The warmed-state map is therefore keyed by
+    (cache_key, resolved_model_id) so repeated calls with different models
+    each get warmed exactly once.
 
     Args:
         instruction (str): The instruction text to cache
         cache_key (str): A unique identifier for this cache (e.g., "REIMBURSEMENT_PRIMARY")
-        model_id (str): The model ID to use. Defaults to "sonnet_latest"
+        model_id (str): The model ID or alias to use. Defaults to "sonnet_latest"
         max_tokens (int): Minimal tokens for cache warming. Defaults to 10
 
     Returns:
-        bool: True if cache was warmed, False if it was already warm
+        bool: True if cache was warmed, False if it was already warm or
+              the resolved model does not support prompt caching.
     """
     global _cache_warmed
 
+    # Resolve model ID up-front so the warmed-state key matches what the
+    # runtime call will use after its own resolve_model_id().
+    resolved_model_id = config.resolve_model_id(model_id)
+
+    # Hard skip on models that do not support prompt caching in this
+    # Bedrock setup. Without this, warming spends create-tokens against a
+    # cache the runtime call can never read.
+    if not _supports_prompt_cache(resolved_model_id):
+        logging.info(
+            "Skipping cache warming for %s on %s (resolved %s): model does "
+            "not support prompt caching.",
+            cache_key,
+            model_id,
+            resolved_model_id,
+        )
+        return False
+
+    warmed_key = (cache_key, resolved_model_id)
+
     # Check if already warmed (thread-safe)
     with _cache_warming_lock:
-        if _cache_warmed.get(cache_key, False):
-            logging.debug(f"Cache already warmed for {cache_key}")
+        if _cache_warmed.get(warmed_key, False):
+            logging.debug(
+                f"Cache already warmed for {cache_key} on {resolved_model_id}"
+            )
             return False
 
         # Mark as being warmed
-        _cache_warmed[cache_key] = True
+        _cache_warmed[warmed_key] = True
 
-    logging.info(f"Warming prompt cache for {cache_key}...")
-
-    # Resolve model ID
-    resolved_model_id = config.resolve_model_id(model_id)
+    logging.info(f"Warming prompt cache for {cache_key} on {resolved_model_id}...")
 
     # Create minimal prompt with the instruction to register the cache
     minimal_prompt = "Respond with OK."
@@ -1323,5 +1494,5 @@ def warm_prompt_cache(
         logging.error(f"Error warming cache for {cache_key}: {str(e)}")
         # Mark as not warmed on error so it can be retried
         with _cache_warming_lock:
-            _cache_warmed[cache_key] = False
+            _cache_warmed[warmed_key] = False
         return False

src/utils/program_product_mapping.py

@@ -0,0 +1,156 @@
+"""Loader for program_product_lob_mapping.csv.
+
+The CSV lives in the same S3 prefix as the txt files:
+    s3://<bucket>/<s3_prefix>/program_product_lob_mapping.csv
+    s3://<bucket>/<s3_prefix>/<txt files>
+
+Key columns:
+    MBR_PROGRAM_DESC             -> AARETE_DERIVED_PROGRAM valid values
+    MBR_PRODUCT_DESC             -> AARETE_DERIVED_PRODUCT valid values
+    MBR_PRODUCT_LINE_OF_BUSINESS -> AARETE_DERIVED_LOB value for that row
+
+The mapping is loaded once at pipeline startup and injected into Constants so every
+file processed in the same batch shares the same read-only data.
+If the CSV is absent, the pipeline continues normally with empty mapping lists.
+"""
+
+import logging
+from dataclasses import dataclass, field
+from io import StringIO
+
+import pandas as pd
+from botocore.exceptions import ClientError
+
+import src.config as config
+
+
+# ---------------------------------------------------------------------------
+# Data class
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class ProgramProductMapping:
+    """Holds all derived data extracted from program_product_lob_mapping.csv."""
+
+    # AARETE_DERIVED_PROGRAM -> AARETE_DERIVED_LOB  (non-empty when CSV has program rows)
+    program_to_lob: dict[str, str] = field(default_factory=dict)
+
+    # AARETE_DERIVED_PRODUCT -> AARETE_DERIVED_LOB  (non-empty when CSV has product rows)
+    product_to_lob: dict[str, str] = field(default_factory=dict)
+
+    # Sorted list of unique AARETE_DERIVED_PROGRAM values from the CSV
+    valid_aarete_derived_programs: list[str] = field(default_factory=list)
+
+    # Sorted list of unique AARETE_DERIVED_PRODUCT values from the CSV
+    valid_aarete_derived_products: list[str] = field(default_factory=list)
+
+    @property
+    def use_crosswalk_for_program(self) -> bool:
+        """True when the CSV has program->LOB entries (use crosswalk, skip LLM)."""
+        return bool(self.program_to_lob)
+
+    @property
+    def use_crosswalk_for_product(self) -> bool:
+        """True when the CSV has product->LOB entries (use crosswalk, skip LLM)."""
+        return bool(self.product_to_lob)
+
+
+# ---------------------------------------------------------------------------
+# Internal builder
+# ---------------------------------------------------------------------------
+
+
+def _build_from_dataframe(df: pd.DataFrame) -> ProgramProductMapping:
+    """Parse the loaded DataFrame into a ProgramProductMapping."""
+    program_to_lob: dict[str, str] = {}
+    product_to_lob: dict[str, str] = {}
+    programs: set[str] = set()
+    products: set[str] = set()
+
+    for _, row in df.iterrows():
+        lob_raw = row.get("MBR_PRODUCT_LINE_OF_BUSINESS", "")
+        lob = str(lob_raw).strip() if pd.notna(lob_raw) else ""
+        if not lob or lob.lower() in ("nan", "n/a", "none", ""):
+            continue
+
+        prog_raw = row.get("MBR_PROGRAM_DESC", "")
+        prog = str(prog_raw).strip() if pd.notna(prog_raw) else ""
+        if prog and prog.lower() not in ("nan", "n/a", "none", ""):
+            program_to_lob[prog] = lob
+            programs.add(prog)
+
+        prod_raw = row.get("MBR_PRODUCT_DESC", "")
+        prod = str(prod_raw).strip() if pd.notna(prod_raw) else ""
+        if prod and prod.lower() not in ("nan", "n/a", "none", ""):
+            product_to_lob[prod] = lob
+            products.add(prod)
+
+    mapping = ProgramProductMapping(
+        program_to_lob=program_to_lob,
+        product_to_lob=product_to_lob,
+        valid_aarete_derived_programs=sorted(programs),
+        valid_aarete_derived_products=sorted(products),
+    )
+    logging.info(
+        f"program_product_mapping loaded: "
+        f"{len(programs)} programs, {len(products)} products, "
+        f"program_crosswalk={'enabled' if mapping.use_crosswalk_for_program else 'disabled (LLM fallback)'}, "
+        f"product_crosswalk={'enabled' if mapping.use_crosswalk_for_product else 'disabled (LLM fallback)'}"
+    )
+    return mapping
+
+
+# ---------------------------------------------------------------------------
+# Public loaders
+# ---------------------------------------------------------------------------
+
+
+def load_from_s3(bucket: str, key: str) -> ProgramProductMapping | None:
+    """Load program_product_lob_mapping.csv from S3.
+
+    Args:
+        bucket: S3 bucket name.
+        key: S3 object key (e.g. "run-folder/program_product_lob_mapping.csv").
+
+    Returns:
+        ProgramProductMapping on success, None if the file does not exist or on failure.
+    """
+    try:
+        response = config.S3_CLIENT.get_object(Bucket=bucket, Key=key)
+        content = response["Body"].read().decode("latin-1")
+        df = pd.read_csv(StringIO(content), sep=",", dtype=str)
+        return _build_from_dataframe(df)
+    except ClientError as e:
+        if e.response["Error"]["Code"] in ("NoSuchKey", "404"):
+            logging.warning(
+                f"program_product_lob_mapping.csv not found at "
+                f"s3://{bucket}/{key}; crosswalk/derived mappings will be empty."
+            )
+        else:
+            logging.error(
+                f"Failed to load program_product_mapping from s3://{bucket}/{key}: {e}"
+            )
+        return None
+    except Exception as e:
+        logging.error(
+            f"Failed to load program_product_mapping from s3://{bucket}/{key}: {e}"
+        )
+        return None
+
+
+def load_from_local(path: str) -> ProgramProductMapping | None:
+    """Load program_product_lob_mapping.csv from a local file path.
+
+    Args:
+        path: Absolute or relative path to the CSV file.
+
+    Returns:
+        ProgramProductMapping on success, None on failure.
+    """
+    try:
+        df = pd.read_csv(path, encoding="latin-1", sep=",", dtype=str)
+        return _build_from_dataframe(df)
+    except Exception as e:
+        logging.error(f"Failed to load program_product_mapping from {path}: {e}")
+        return None

src/utils/prompt_call_tracking.py

@@ -0,0 +1,354 @@
+"""Runtime prompt-call tracking for prompt caching analysis.
+
+Writes one CSV row per invoke_claude call so we can answer:
+- token sizes per prompt component
+- number of calls per contract and prompt label
+- longest prompt component (context vs instruction vs field prompt)
+- which cache class each call exercised and why a cache read was missed
+"""
+
+import csv
+import logging
+import os
+import re
+import threading
+from datetime import datetime
+from pathlib import Path
+
+from src import config
+
+WORDS_TO_TOKENS_MULTIPLIER = 1.3
+
+# Default minimum tokens for a `cache_control` block to be honored.
+# Per-model overrides are pulled lazily from `llm_utils._min_cache_tokens`
+# at call time. Falling back to 1024 here matches the historical Sonnet
+# minimum and is the most permissive default.
+_DEFAULT_MIN_CACHE_TOKENS = 1024
+
+_PROMPT_CALL_LOCK = threading.Lock()
+
+
+# Cache policy values. Mirrors the CacheClass enum in cache_registry.
+class CachePolicy:
+    INSTRUCTION = "instruction"
+    CONTEXT = "context"
+    INSTRUCTION_PLUS_CONTEXT = "instruction+context"
+    NONE = "none"
+
+
+# Cache miss reason values. Empty string means "no miss" — either the
+# call had a cache read or it was served from local memory cache.
+class CacheMissReason:
+    NONE = ""  # cache read happened, or call was served locally
+    CACHE_DISABLED = "cache_disabled"
+    MODEL_UNSUPPORTED = "model_unsupported"
+    NO_CACHEABLE_BLOCK = "no_cacheable_block"
+    BELOW_1024_THRESHOLD = "below_1024_threshold"
+    WARMUP_OR_EVICTION = "warmup_or_eviction"
+    NO_CACHE_EVENT = "no_cache_event"
+
+
+PROMPT_CALL_FIELDS = [
+    "timestamp",
+    "contract_filename",
+    "usage_label",
+    "model_id",
+    # New: full resolved model id (e.g. "us.anthropic.claude-sonnet-4-5-...")
+    # so logs are durable when MODEL_ALIASES change. `model_id` may still be
+    # an alias like "sonnet_latest" for back-compat with older log readers.
+    "resolved_model_id",
+    "cache_enabled",
+    # New: separate cache classes — `instruction` and `context` are
+    # tracked independently because they have different expected
+    # behaviours. instruction caches read every call; context caches only
+    # read when the same context block repeats. Reporting must evaluate
+    # them separately so context variability does not make instruction
+    # caching look broken.
+    "cache_policy",
+    "instruction_cache_eligible",
+    "context_cache_eligible",
+    "cache_miss_reason",
+    "max_tokens",
+    "status",
+    "error",
+    "from_local_memory_cache",
+    "prompt_tokens_est",
+    "instruction_tokens_est",
+    "context_tokens_est",
+    "total_input_tokens_est",
+    "api_cached_input_tokens",
+    "api_total_billed_input_tokens",
+    "est_minus_api_total_tokens",
+    "longest_component",
+    "input_tokens",
+    "output_tokens",
+    "cache_creation_tokens",
+    "cache_read_tokens",
+]
+
+
+def _classify_cache_policy(
+    cache_enabled: bool,
+    instruction_tokens: int,
+    context_tokens: int,
+    model_supports_cache: bool,
+) -> str:
+    """Return the actual cache_policy this call exercised on the wire.
+
+    This is the runtime view: what `_build_claude_3_request_body` actually
+    placed `cache_control` on, given what the caller passed. It is NOT the
+    registry's intended class — comparing the two reveals wiring bugs.
+    """
+    if not cache_enabled or not model_supports_cache:
+        return CachePolicy.NONE
+    has_instruction = instruction_tokens > 0
+    has_context = context_tokens > 0
+    if has_instruction and has_context:
+        return CachePolicy.INSTRUCTION_PLUS_CONTEXT
+    if has_instruction:
+        return CachePolicy.INSTRUCTION
+    if has_context:
+        return CachePolicy.CONTEXT
+    return CachePolicy.NONE
+
+
+def _derive_cache_miss_reason(
+    cache_enabled: bool,
+    model_supports_cache: bool,
+    instruction_tokens: int,
+    context_tokens: int,
+    cache_read_tokens: int,
+    cache_creation_tokens: int,
+    from_local_memory_cache: bool,
+    min_cache_tokens: int = _DEFAULT_MIN_CACHE_TOKENS,
+) -> str:
+    """Explain why a call did not record a cache read.
+
+    Returns "" when a read happened (or the call was served from local
+    memory and thus did not hit the API at all). Otherwise returns a
+    short stable code from `CacheMissReason`. Reporting groups by this
+    column to surface root causes without manual investigation.
+
+    `min_cache_tokens` is the model-specific Bedrock threshold below
+    which `cache_control` blocks are silently dropped. Sonnet 4.5 = 1024,
+    Haiku 4.5 = 4096, etc.
+    """
+    if from_local_memory_cache:
+        # No API call happened; cache reads are not applicable.
+        return CacheMissReason.NONE
+    if cache_read_tokens > 0:
+        return CacheMissReason.NONE
+    if not cache_enabled:
+        return CacheMissReason.CACHE_DISABLED
+    if not model_supports_cache:
+        return CacheMissReason.MODEL_UNSUPPORTED
+    if instruction_tokens == 0 and context_tokens == 0:
+        return CacheMissReason.NO_CACHEABLE_BLOCK
+
+    # Eligible blocks are those at or above the per-model Bedrock minimum.
+    instruction_eligible = instruction_tokens >= min_cache_tokens
+    context_eligible = context_tokens >= min_cache_tokens
+    if not (instruction_eligible or context_eligible):
+        return CacheMissReason.BELOW_1024_THRESHOLD
+
+    if cache_creation_tokens > 0:
+        # Bedrock created (or refreshed) the cache on this call; a future
+        # call within TTL is what will actually read.
+        return CacheMissReason.WARMUP_OR_EVICTION
+
+    return CacheMissReason.NO_CACHE_EVENT
+
+
+def _estimate_tokens(text: str | None) -> int:
+    if not text:
+        return 0
+    words = len(re.findall(r"\S+", text))
+    return int(round(words * WORDS_TO_TOKENS_MULTIPLIER))
+
+
+def _get_prompt_call_log_path() -> str:
+    # Preferred path: same tracking directory as USAGE files.
+    batch_id = getattr(config, "BATCH_ID", None) or "adhoc"
+    file_name = f"{batch_id}-PROMPT-CALLS.csv"
+    run_timestamp = getattr(config, "PROMPT_CALL_RUN_TIMESTAMP", "")
+    if run_timestamp:
+        return str(
+            Path(config.CONSOLIDATED_OUTPUT_DIRECTORY)
+            / run_timestamp
+            / "tracking"
+            / file_name
+        )
+
+    # Fallback for ad-hoc invocations outside the main pipeline.
+    return str(Path(config.CONSOLIDATED_OUTPUT_DIRECTORY) / "tracking" / file_name)
+
+
+def _ensure_prompt_call_log_file() -> str:
+    path = _get_prompt_call_log_path()
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+
+    if not os.path.exists(path):
+        with open(path, "w", newline="", encoding="utf-8") as f:
+            writer = csv.DictWriter(f, fieldnames=PROMPT_CALL_FIELDS)
+            writer.writeheader()
+
+    return path
+
+
+def _longest_component(
+    prompt_tokens: int,
+    instruction_tokens: int,
+    context_tokens: int,
+) -> str:
+    components = {
+        "field_instructions": prompt_tokens,
+        "prompt_instructions": instruction_tokens,
+        "context": context_tokens,
+    }
+    if all(v == 0 for v in components.values()):
+        return "unknown"
+    return max(components, key=components.get)
+
+
+def log_prompt_call(
+    contract_filename: str,
+    usage_label: str,
+    model_id: str,
+    cache_enabled: bool,
+    max_tokens: int,
+    prompt: str | None,
+    instruction: str | None,
+    context_for_caching: str | None,
+    status: str,
+    error: str = "",
+    from_local_memory_cache: bool = False,
+    input_tokens: int = 0,
+    output_tokens: int = 0,
+    cache_creation_tokens: int = 0,
+    cache_read_tokens: int = 0,
+    resolved_model_id: str | None = None,
+    model_supports_cache: bool | None = None,
+) -> None:
+    """Write one prompt-call tracking row.
+
+    This function never raises to avoid impacting the extraction pipeline.
+
+    Args:
+        resolved_model_id: Full model id after `config.resolve_model_id`.
+            When None, falls back to `model_id` so older callers still log
+            something durable.
+        model_supports_cache: Whether the resolved model supports prompt
+            caching at all. When None, the function imports
+            `_supports_prompt_cache` lazily and computes it. Pass it
+            explicitly from the call site when available to avoid the
+            import.
+    """
+    if not getattr(config, "ENABLE_PROMPT_CALL_LOGGING", True):
+        return
+
+    try:
+        prompt = prompt or ""
+        instruction = instruction or ""
+        context_for_caching = context_for_caching or ""
+
+        prompt_tokens_est = _estimate_tokens(prompt)
+        instruction_tokens_est = _estimate_tokens(instruction)
+        context_tokens_est = _estimate_tokens(context_for_caching)
+
+        total_input_tokens_est = (
+            prompt_tokens_est + instruction_tokens_est + context_tokens_est
+        )
+        api_cached_input_tokens = cache_creation_tokens + cache_read_tokens
+        api_total_billed_input_tokens = input_tokens + api_cached_input_tokens
+
+        # Resolve model id, cache support, and per-model min threshold if
+        # the caller didn't pass them.
+        if resolved_model_id is None:
+            resolved_model_id = config.resolve_model_id(model_id)
+
+        min_cache_tokens = _DEFAULT_MIN_CACHE_TOKENS
+        try:
+            from src.utils.llm_utils import (
+                _min_cache_tokens,
+                _supports_prompt_cache,
+            )
+
+            if model_supports_cache is None:
+                model_supports_cache = _supports_prompt_cache(resolved_model_id)
+            min_cache_tokens = _min_cache_tokens(resolved_model_id)
+        except Exception:
+            if model_supports_cache is None:
+                model_supports_cache = False
+
+        cache_policy = _classify_cache_policy(
+            cache_enabled,
+            instruction_tokens_est,
+            context_tokens_est,
+            model_supports_cache,
+        )
+        instruction_cache_eligible = (
+            cache_enabled
+            and model_supports_cache
+            and instruction_tokens_est >= min_cache_tokens
+        )
+        context_cache_eligible = (
+            cache_enabled
+            and model_supports_cache
+            and context_tokens_est >= min_cache_tokens
+        )
+        cache_miss_reason = _derive_cache_miss_reason(
+            cache_enabled=cache_enabled,
+            model_supports_cache=model_supports_cache,
+            instruction_tokens=instruction_tokens_est,
+            context_tokens=context_tokens_est,
+            cache_read_tokens=cache_read_tokens,
+            cache_creation_tokens=cache_creation_tokens,
+            from_local_memory_cache=from_local_memory_cache,
+            min_cache_tokens=min_cache_tokens,
+        )
+
+        row = {
+            "timestamp": datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S"),
+            "contract_filename": contract_filename,
+            "usage_label": usage_label,
+            "model_id": model_id,
+            "resolved_model_id": resolved_model_id,
+            "cache_enabled": cache_enabled,
+            "cache_policy": cache_policy,
+            "instruction_cache_eligible": instruction_cache_eligible,
+            "context_cache_eligible": context_cache_eligible,
+            "cache_miss_reason": cache_miss_reason,
+            "max_tokens": max_tokens,
+            "status": status,
+            "error": error,
+            "from_local_memory_cache": from_local_memory_cache,
+            "prompt_tokens_est": prompt_tokens_est,
+            "instruction_tokens_est": instruction_tokens_est,
+            "context_tokens_est": context_tokens_est,
+            "total_input_tokens_est": total_input_tokens_est,
+            "api_cached_input_tokens": api_cached_input_tokens,
+            "api_total_billed_input_tokens": api_total_billed_input_tokens,
+            "est_minus_api_total_tokens": total_input_tokens_est
+            - api_total_billed_input_tokens,
+            "longest_component": _longest_component(
+                prompt_tokens_est,
+                instruction_tokens_est,
+                context_tokens_est,
+            ),
+            "input_tokens": input_tokens,
+            "output_tokens": output_tokens,
+            "cache_creation_tokens": cache_creation_tokens,
+            "cache_read_tokens": cache_read_tokens,
+        }
+
+        with _PROMPT_CALL_LOCK:
+            file_path = _ensure_prompt_call_log_file()
+            with open(file_path, "a", newline="", encoding="utf-8") as f:
+                writer = csv.DictWriter(f, fieldnames=PROMPT_CALL_FIELDS)
+                writer.writerow(row)
+    except Exception as e:
+        # Tracking must not break the pipeline.
+        logging.error(
+            f"Failed to log prompt call for {contract_filename} / {usage_label}: {e}"
+        )
+        return