doczyai-pipelines/documentation/PROMPT_CACHING_SYSTEM_PROPOSAL.md

# Prompt Caching System Proposal

Branch: `feature/DAIP2-2314-expand-caching-for-short-prompts`

## Goal

Ensure every stable instruction prompt that is intended to be cached is actually cacheable, warmed on the same model used at runtime, and observable in runtime logs.

## Recommended Caching System

1. Introduce a central prompt cache registry.

   Each cacheable prompt should be declared once with:
   - `usage_label`
   - instruction function
   - model family used at runtime
   - minimum cache token policy
   - runtime call sites
   - whether the prompt also uses `context_for_caching`

   This avoids the current split where cache warming is in `prompt_templates.get_cacheable_instructions()` but runtime model choice is scattered through prompt call sites.

2. Enforce the 1024-token minimum before runtime.

   Prompt caching should not be treated as enabled just because `cache=True` is passed. At startup, the registry should validate every cached instruction with the same tokenizer/estimate used by reporting. If a prompt is below the minimum, it should either:
   - fail fast in a cache audit mode, or
   - automatically append approved static cache padding/instruction context until it crosses the threshold.

   The padding must be stable and domain-relevant, not dynamic request data.

3. Warm caches per model, not globally.

   Cache entries are model-specific. The runner currently warms all instructions with `sonnet_latest`, but `SPLIT_SERVICE_TERM` runs with `haiku_latest`. The cache warming layer should warm `(usage_label, resolved_model_id)` pairs derived from the registry.

4. Only mark supported models as cacheable.

   `_supports_prompt_cache()` currently excludes Haiku. If Haiku prompt caching is not supported in this Bedrock setup, Haiku prompts should not be registered as cached. Either move those prompt calls to `sonnet_latest` when caching matters, or keep them uncached and report them as intentionally uncached.

5. Keep static instruction caching separate from dynamic context caching.

   Instruction prompts can be cached predictably. `context_for_caching` is different because it changes by contract, exhibit, or page. It should be tracked as a separate cache class:
   - `instruction_cache`: expected to read after warm-up
   - `context_cache`: expected to create/read per repeated context

   This prevents context variability from making instruction caching look broken.

6. Add a cache contract test.

   A local test should inspect the built request body for every registered prompt and assert:
   - the instruction block has `cache_control`
   - the resolved model supports caching
   - the static instruction token estimate is at least 1024
   - warm-up model equals runtime model

7. Upgrade runtime reporting.

   Add these fields to prompt-call logs:
   - `resolved_model_id`
   - `cache_policy`: `instruction`, `context`, `instruction+context`, `none`
   - `instruction_cache_eligible`
   - `context_cache_eligible`
   - `cache_miss_reason`

   This makes root cause visible without manual investigation.

## Prompt Application Report

### Apply Instruction Caching, Already Over 1024 Tokens

These should remain registered and warmed. They are good candidates for strict cache contract tests.

| Prompt | Estimated Instruction Tokens | Action |
|---|---:|---|
| `OUTLIER_BREAKOUT` | 2483 | Keep cached |
| `METHODOLOGY_BREAKOUT` | 1927 | Keep cached |
| `SPLIT_SERVICE_TERM` | 1600 | Cache only if runtime model supports caching; currently runs on Haiku |
| `DYNAMIC_ASSIGNMENT` | 1578 | Keep cached |
| `CARVEOUT_CHECK` | 1498 | Keep cached |
| `SERVICE_ENRICHMENT` | 1459 | Keep cached |
| `REIMBURSEMENT_PRIMARY` | 1390 | Keep cached |
| `EXHIBIT_LEVEL` | 1108 | Keep cached |

### Apply Instruction Caching After Padding/Expansion

These are wired or partially wired but below the 1024-token minimum. They should be padded or expanded with stable domain guidance before being considered cached.

| Prompt | Estimated Instruction Tokens | Tokens Needed | Action |
|---|---:|---:|---|
| `fill_bill_type` | 968 | 56 | Add small static guidance |
| `DYNAMIC_CODE_ASSIGNMENT` | 966 | 58 | Add small static guidance |
| `CHECK_PROVIDER_NAME_MATCH` | 957 | 67 | Add small static guidance |
| `GROUPER_BREAKOUT` | 939 | 85 | Add small static guidance |
| `EXHIBIT_HEADER` | 922 | 102 | Add static header examples/rules |
| `LESSER_OF_DISTRIBUTION` | 914 | 110 | Add static decision examples |
| `LESSER_OF_CHECK` | 900 | 124 | Add static classification examples |
| `CODE_EXPLICIT` | 871 | 153 | Add static code-system rules |
| `REIMB_DATES_ASSIGNMENT` | 803 | 221 | Add date extraction examples |
| `DYNAMIC_PRIMARY` | 441 | 583 | Expand instruction substantially |
| `SPLIT_REIMB_DATES` | 410 | 614 | Expand instruction substantially |
| `prompt_lob_relationship` | 408 | 616 | Move field prompt into instruction or add static LOB rules |
| `code_implicit_arbitration` | 391 | 633 | Expand arbitration rules |
| `AARETE_DERIVED_PAYER_NAME` | 372 | 652 | Expand payer naming rules |
| `FEE_SCHEDULE_BREAKOUT` | 352 | 672 | Expand fee schedule rules |
| `code_implicit_special` | 329 | 695 | Expand special code rules |
| `DATE_FIX` | 322 | 702 | Expand date normalization rules |
| `SPECIAL_CASE_ASSIGNMENT` | 251 | 773 | Expand special case rules |
| `DERIVED_TERM_DATE` | 247 | 777 | Expand term date rules |
| `code_last_check` | 179 | 845 | Expand or leave uncached if low value |
| `EXTRACT_AMENDMENT_NUM_FROM_FILENAME` | 152 | 872 | Expand or leave uncached if low value |
| `EXHIBIT_HEADER_DEDUP` | 35 | 989 | Do not cache unless redesigned |

### Wire for Caching or Mark Intentionally Uncached

These showed runtime volume but no instruction function match in the audit. They need explicit registry decisions.

| Prompt | Runtime Finding | Action |
|---|---|---|
| `prompt_hsc_single_field` | 109 calls, `cache_enabled=False`, large dynamic prompt | Split stable HSC instructions from dynamic chunk and cache the instruction |
| `SPECIAL_CASE_BREAKOUT` | 6 calls, no instruction cache | Wire only if repeated volume justifies expansion |
| `code_implicit_rag` | Cached in API, audit alias missing | Add registry alias to the actual instruction function |
| `prompt_provider_info` | Cached in API, audit alias missing | Add registry alias to `TIN_NPI_TEMPLATE_INSTRUCTION` or rename usage label |
| `validate_reimbursements_for_llm` | Cached in API, audit alias missing | Add registry alias to `VALIDATE_REIMBURSEMENTS_INSTRUCTION` |

## Immediate Fixes

1. Change `SPLIT_SERVICE_TERM` to use a cache-supported model or mark it intentionally uncached. Current behavior is misleading: `cache=True` is passed, but no Bedrock cache tokens are recorded.

2. Add a registry alias map for usage labels that do not match instruction names:
   - `code_implicit_rag` -> `CODE_IMPLICIT`
   - `prompt_provider_info` -> `TIN_NPI_TEMPLATE`
   - `validate_reimbursements_for_llm` -> `VALIDATE_REIMBURSEMENTS`
   - `prompt_lob_relationship` -> its real instruction function

3. Expand the near-threshold instructions first because they need the least work and have current evidence of API cache reads:
   - `fill_bill_type`
   - `DYNAMIC_CODE_ASSIGNMENT`
   - `CHECK_PROVIDER_NAME_MATCH`
   - `GROUPER_BREAKOUT`
   - `EXHIBIT_HEADER`
   - `LESSER_OF_DISTRIBUTION`
   - `LESSER_OF_CHECK`
   - `CODE_EXPLICIT`

4. Redesign `prompt_hsc_single_field` into `(instruction, prompt)` form. It is high-volume and currently sends all prompt text dynamically.

## Expected Behavior After Fix

For instruction-only prompts, after warm-up the API should report cache reads on normal calls. The cached percentage will still not be 100% of total billed input because the dynamic user prompt remains uncached. The correct target is:

- `cache_creation_tokens` only during warm-up or first uncached use
- `cache_read_tokens > 0` on every eligible runtime call
- `cache_miss_reason` empty for registered instruction cache prompts

For context-cached prompts, 100% cache reads should not be expected unless the same context block repeats. The report should evaluate those separately.