doczyai-pipelines/documentation/PROMPT_CACHING_ANALYSIS.md

# Prompt Caching Analysis Report

**Branch:** `feature/DAIP2-2314-expand-caching-for-short-prompts`
**Date:** 2026-04-21
**Comparison:** Feature branch vs `origin/dev`

---

## Executive Summary

This report analyzes the prompt caching implementation in the feature branch, comparing cached prompts against the dev branch and evaluating cost savings from Bedrock prompt caching.

### Key Findings

| Metric | Value |
|--------|-------|
| Total API Calls Analyzed | 1,251 |
| Unique Usage Labels | 35 |
| Total Cache Read Tokens | 1,472,330 |
| Total Cache Write Tokens | 87,167 |
| Overall Cost Savings | **28.1%** ($3.65 saved) |
| Prompts Actively Caching | 22 |
| Prompts Wired but Not Caching | 11 |
| Prompts Not Wired | 2 |

---

## 1. Cache Effectiveness by Usage Label

### Actively Caching (22 prompts) - High Performers

| Usage Label | Calls | Avg Instruction Tokens | Cache Hit Rate |
|-------------|-------|------------------------|----------------|
| OUTLIER_BREAKOUT | 1 | 2,483 | 98.4% |
| SERVICE_ENRICHMENT | 30 | 1,459 | 98.3% |
| CARVEOUT_CHECK | 43 | 1,498 | 95.6% |
| METHODOLOGY_BREAKOUT | 37 | 1,927 | 95.5% |
| LESSER_OF_DISTRIBUTION | 32 | 914 | 94.5% |
| CHECK_PROVIDER_NAME_MATCH | 74 | 957 | 94.2% |
| DYNAMIC_CODE_ASSIGNMENT | 38 | 966 | 94.1% |
| CODE_EXPLICIT | 34 | 871 | 93.9% |
| GROUPER_BREAKOUT | 7 | 939 | 93.6% |
| LESSER_OF_CHECK | 6 | 900 | 92.0% |
| DYNAMIC_ASSIGNMENT | 204 | 1,578 | 89.4% |
| validate_reimbursements_for_llm | 65 | 894 | 89.2% |
| code_implicit_rag | 60 | 896 | 88.4% |
| REIMB_DATES_ASSIGNMENT | 29 | 803 | 87.2% |
| REIMBURSEMENT_PRIMARY | 38 | 1,390 | 85.0% |
| EXHIBIT_LEVEL | 33 | 1,108 | 74.0% |
| EXHIBIT_HEADER | 90 | 922 | 73.7% |
| fill_bill_type | 47 | 968 | 72.5% |
| prompt_provider_info | 24 | 966 | 60.5% |
| DYNAMIC_PRIMARY | 44 | 441 | 49.7% |

### Wired But Not Caching (11 prompts) - Under 1024 Token Threshold

These prompts are wired for caching but have instruction tokens below Bedrock's 1024-token minimum threshold:

| Usage Label | Calls | Avg Instruction Tokens | Issue |
|-------------|-------|------------------------|-------|
| SPLIT_SERVICE_TERM | 10 | 1,600 | Should be caching - investigate |
| prompt_lob_relationship | 56 | 408 | **Needs padding to 1024** |
| SPLIT_REIMB_DATES | 13 | 410 | **Needs padding to 1024** |
| code_implicit_arbitration | 12 | 391 | **Needs padding to 1024** |
| AARETE_DERIVED_PAYER_NAME | 1 | 372 | **Needs padding to 1024** |
| FEE_SCHEDULE_BREAKOUT | 18 | 352 | **Needs padding to 1024** |
| code_implicit_special | 30 | 329 | **Needs padding to 1024** |
| DATE_FIX | 10 | 322 | **Needs padding to 1024** |
| SPECIAL_CASE_ASSIGNMENT | 4 | 251 | **Needs padding to 1024** |
| DERIVED_TERM_DATE | 10 | 247 | **Needs padding to 1024** |
| code_last_check | 23 | 179 | **Needs padding to 1024** |
| EXTRACT_AMENDMENT_NUM_FROM_FILENAME | 3 | 152 | **Needs padding to 1024** |
| EXHIBIT_HEADER_DEDUP | 10 | 35 | **Needs padding to 1024** |

### Not Wired for Caching (2 prompts)

| Usage Label | Calls | Notes |
|-------------|-------|-------|
| prompt_hsc_single_field | 109 | High volume - should investigate wiring |
| SPECIAL_CASE_BREAKOUT | 6 | Low volume |

---

## 2. Prompt Template Changes (Feature Branch vs Dev)

The feature branch includes **649 lines changed** in `src/prompts/prompt_templates.py`. Key expansions:

### Expanded Prompts (Additional Context Added)

| Instruction Function | Change Summary |
|---------------------|----------------|
| **EXHIBIT_LEVEL_INSTRUCTION** | +55 lines: Added detailed guidance, validation rules, examples, common pitfalls, field-specific notes, quality assurance checks |
| **DYNAMIC_PRIMARY_INSTRUCTION** | +18 lines: Added extraction guidance, ambiguity handling, quality checks |
| **LESSER_OF_DISTRIBUTION_INSTRUCTION** | Restructured (net -25 lines): Converted to cleaner decision framework format |
| **DYNAMIC_CODE_ASSIGNMENT_INSTRUCTION** | +10 lines: Added classification guidance with explicit rules |
| **CODE_IMPLICIT_INSTRUCTION** | Reformatted indentation |
| **CODE_IMPLICIT_ARBITRATION_INSTRUCTION** | +43 lines: Added clarifications, deterministic decision rules, calibration examples |
| **FILL_BILL_TYPE_INSTRUCTION** | +49 lines: Added detailed guidance, validation rules, extended examples |
| **TIN_NPI_TEMPLATE_INSTRUCTION** | +22 lines: Added provider/payer distinction clarity |

### Analysis of Prompt Changes

**Positive Impacts:**
- More explicit instructions reduce model ambiguity
- Quality checks and validation rules improve consistency
- Examples help calibrate model responses
- Clearer formatting improves readability

**Potential Concerns:**
- Expanded prompts increase token count (higher cache write cost initially)
- Some prompts may have reduced due to restructuring (LESSER_OF_DISTRIBUTION)
- Need to verify output quality hasn't degraded with expanded context

---

## 3. Cost Analysis

### Bedrock Pricing Applied
- **Input tokens:** $3.00/1M tokens
- **Output tokens:** $15.00/1M tokens
- **Cache read:** $0.30/1M tokens (90% savings)
- **Cache write:** $3.75/1M tokens (25% premium)

### Observed Results

| Metric | Value |
|--------|-------|
| Total Input Tokens | 844,370 |
| Total Output Tokens | 401,926 |
| Total Cache Read Tokens | 1,472,330 |
| Total Cache Write Tokens | 87,167 |
| **Cost Without Caching** | $12.98 |
| **Cost With Caching** | $9.33 |
| **Savings** | $3.65 (28.1%) |

### Per-Prompt Cost Impact (Estimated from previous testing)

| Prompt | Cost Reduction |
|--------|---------------|
| LESSER_OF_DISTRIBUTION | 44% (Best performer) |
| EXHIBIT_HEADER | 7% |
| CHECK_PROVIDER_NAME_MATCH | 5% |
| TIN_NPI_TEMPLATE (prompt_provider_info) | 5% |
| code_implicit_rag | -12% (Negative - needs review) |

---

## 4. Prompts Needing Attention

### Priority 1: Prompts Needing Padding (Under 1024 tokens)

These prompts are wired but not actually caching due to Bedrock's minimum threshold:

1. **prompt_lob_relationship** (408 tokens, 56 calls) - High impact
2. **code_implicit_special** (329 tokens, 30 calls) - Medium impact
3. **code_last_check** (179 tokens, 23 calls) - Medium impact
4. **FEE_SCHEDULE_BREAKOUT** (352 tokens, 18 calls) - Medium impact
5. **SPLIT_REIMB_DATES** (410 tokens, 13 calls)
6. **code_implicit_arbitration** (391 tokens, 12 calls)
7. **SPLIT_SERVICE_TERM** (1600 tokens, 10 calls) - Should be caching, investigate
8. **DATE_FIX** (322 tokens, 10 calls)
9. **DERIVED_TERM_DATE** (247 tokens, 10 calls)
10. **EXHIBIT_HEADER_DEDUP** (35 tokens, 10 calls)

### Priority 2: Investigate Negative Result

- **code_implicit_rag**: Shows -12% cost (increase) - The expanded prompt may be too verbose or structured differently causing cache misses

### Priority 3: Not Wired for Caching

- **prompt_hsc_single_field** (109 calls) - High volume, should consider wiring

---

## 5. Recommendations

1. **Pad short prompts to 1024 tokens**: Add contextual padding to prompts under the threshold, especially high-frequency ones like `prompt_lob_relationship` and `code_implicit_special`

2. **Investigate code_implicit_rag**: The negative cost result (-12%) suggests the expanded prompt may be causing cache fragmentation or misses

3. **Wire prompt_hsc_single_field**: With 109 calls, this is a good candidate for caching

4. **Monitor SPLIT_SERVICE_TERM**: At 1,600 tokens it should be caching but shows 0% cache hit rate - may be a wiring issue

5. **Run testbed comparison**: Use `src/testbed/testbed_metrics.py` to compare output quality between cached and non-cached prompts to ensure no regression

---

## 6. Files Analyzed

- `src/test-PROMPT-CALLS_10.csv` - API call logs with cache statistics
- `src/testbed-usethis.xlsx` - Testbed comparison file (binary, not readable)
- `src/prompts/prompt_templates.py` - Prompt definitions (649 lines changed)
- `documentation/prompt_caching_tracker_v2.csv` - Caching status tracker

---

## 7. Next Steps

1. Run testbed metrics comparison script to validate output quality
2. Add padding to short prompts identified above
3. Investigate and fix code_implicit_rag negative result
4. Wire prompt_hsc_single_field for caching
5. Debug SPLIT_SERVICE_TERM caching issue