53a0c76d8e
Bugfix/stg to main prep * Merged in dev (pull request #1001) Dev * Revert premature merge of bugfix/retire_stale_client_file_processing PR #959 was merged into dev without approval. This reverts commits5e143c10,63f32c41,849aa626, and927abcaeto restore dev to its pre-merge state. The changes will be re-submitted via a new PR after proper review. * Merged in bugfix/retire_stale_client_file_processing (pull request #961) Return None for dashboard output when dashboard postprocessing is off * Return None for dashboard output when dashboard postprocessing is off FINAL_RESULT_DF_DASHBOARD was initialized as an empty DataFrame even when RUN_DASHBOARD_POSTPROCESSING was False, causing downstream code to needlessly process it (reorder_columns, etc). Now returns None when dashboard is not requested, matching the postprocess() contract. * Merged dev into bugfix/retire_stale_client_file_processing * Merge dev (with revert) into feature branch * Re-apply retire stale client file_processing changes Revert of the revert (4f53b528) to… * fix grouping_key casing in service term standardization qc.py uppercases all PC_Details columns to GROUPING_KEY, but the service-term reader looked up lowercase 'grouping_key'. The lookup silently failed and degraded per-group standardization to global mode. * enable prompt caching on service_term_standardization LLM calls cache=True was missing; the 2.5k-token instruction was being re-billed on every chunk and mapping-batch call instead of reading from the shared instruction cache. * register SERVICE_ENRICHMENT_INSTRUCTION in cache registry The runtime call at code_funcs.py:75 already passes cache=True with usage_label="SERVICE_ENRICHMENT", but the registry entry was missing, so the prompt was never warmed and every first call paid create-tokens. * remove stray debug prints from document_classification.generate_pre_doczy_report Three print() calls were dumping the Program/Product mapping to stdout on every run. * demote code_explicit raw-answer log from INFO to DEBUG Full raw LLM responses were spamming INFO-level logs on every call. * flag vendor prompt_templates convention drift for follow-up Vendor pipeline still uses legacy hand-rolled JSON wording and has one stale 'Return result in pipes' reference. Adopting the JSON_*_FORMAT_INSTRUCTIONS macros from the main pipeline is deferred to a dedicated follow-up; header comment points to the review tracker. * skip healthcare finalizer for vendor pipelines runner.main applied add_aarete_derived_payer_name/provider_name and the FIELD_FORMAT_MAPPING reorder unconditionally to every client. The mapping is healthcare-only (src/constants/investment_columns.py), so reorder_columns dropped every vendor-specific column (TITLE, VENDOR_NAME, AMOUNT, SLA/KPI, ...) from the combined cc_results_full output. Per-file individual outputs survived because they are written inside generic_processor before the finalizer runs. Guard the entire healthcare finalizer block with registry.is_vendor(client) so vendor results keep their own flat schema. * move DTC short-circuit above duplicate_detection in runner In DTC mode, dedup ran twice: once in runner before the short-circuit (immediately discarded at return), and again inside dtc_main.main on the same input_dict. Reorder so DTC exits before the runner-scope dedup is computed. dtc_main keeps its own internal dedup pass. Inline comment notes the cleaner follow-up — have dtc_main.main take (original_files_to_process, file_status_map) so both call sites share a single computation. * remove dead code from saas/main.py This module is a thin delegator to runner.main; the actual safe_process_file lives in runner.py:101. The local copy at saas/main.py referenced logging, traceback, pd, and file_processing — none of which were imported, so any caller would have raised NameError immediately. Drops the function and the four unused module-level imports (parent_child_main, dtc_main, standardize_service_terms, qc_qa.pipeline.*). File shrinks from 102 to 28 lines. * Merge origin/dev into bugfix/stg-to-main-prep Reconciles the 9 stg-to-main prep fixes with 197 commits dev gained since 2026-03-16. Resolved 14 conflicts: Group A (preserve our 9 fixes): - saas/main.py: dead-code cleanup preserved - code_funcs.py: code_explicit log demoted to DEBUG preserved - document_classification/main.py: 3 stray prints removed preserved - vendors/prompt_templates.py: NOTE flagging legacy JSON wording preserved - cache_registry.py: SERVICE_ENRICHMENT registry entry preserved - service_term_standardization.py: GROUPING_KEY casing + cache=True preserved - runner.py: vendor finalizer guard + DTC short-circuit reorder preserved Group B (dev prevails; our fixes don't touch these): - saas/file_processing.py: dev's per-field confidence scoring (DAIP2-2692) - postprocessing_funcs.py: dev's add_amendment_intent + _CONF carve-out - preprocess.py: dev's removal of stale fallback comment - preprocessing_funcs.py: dev's _find_header_in_text import (needed by body) - prompts/prompt_templates.py: dev… Approved-by: Siddhant Medar
629 lines
25 KiB
Python
629 lines
25 KiB
Python
"""Central prompt cache registry.
|
|
|
|
Single source of truth for which prompts are eligible for prompt caching,
|
|
which model they actually run on at runtime, and which cache class they
|
|
use. This is consumed by the cache-warming layer in
|
|
`src.pipelines.runner` and by reporting / contract tests.
|
|
|
|
Why this exists
|
|
---------------
|
|
Before the registry, cache warming and runtime call sites were split:
|
|
- `prompt_templates.get_cacheable_instructions()` produced the warmed list
|
|
- runtime code in `pipelines/.../prompt_calls.py` chose its own model id
|
|
The two could (and did) drift. For example `SPLIT_SERVICE_TERM` was warmed
|
|
on `sonnet_latest` but invoked on `haiku_latest`, so warm-up tokens were
|
|
spent against a cache key the runtime never read.
|
|
|
|
Cache classes
|
|
-------------
|
|
- INSTRUCTION: the system instruction block is cached; cache reads
|
|
are expected on every call after warm-up.
|
|
- CONTEXT: a dynamic `context_for_caching` block is cached;
|
|
reads are only expected when the same context
|
|
repeats (e.g. the same exhibit text across fields).
|
|
- INSTRUCTION_PLUS_CONTEXT: both blocks are cached.
|
|
- NONE: explicitly opted out; either the runtime model
|
|
does not support caching in this Bedrock setup,
|
|
or the static instruction is too small to qualify
|
|
and is not yet padded.
|
|
|
|
Reporting should evaluate INSTRUCTION and CONTEXT cache classes
|
|
separately so per-context variability does not make instruction caching
|
|
look broken.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import re
|
|
from dataclasses import dataclass
|
|
from typing import Callable, Iterable, Iterator
|
|
|
|
import src.config as config
|
|
import src.prompts.prompt_templates as prompt_templates
|
|
|
|
|
|
# Default minimum tokens for a cache_control block to be honored.
|
|
# Per-model overrides live in `llm_utils._MODEL_MIN_CACHE_TOKENS`. This
|
|
# constant is the conservative fallback used when an entry has no
|
|
# explicit min_instruction_tokens set and the resolved model is unknown.
|
|
MIN_CACHE_TOKENS = 1024
|
|
|
|
# Same heuristic used by `prompt_call_tracking._estimate_tokens` so warm-up
|
|
# audits agree with runtime token reporting.
|
|
_WORDS_TO_TOKENS_MULTIPLIER = 1.3
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Cache classes
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class CacheClass:
|
|
INSTRUCTION = "instruction"
|
|
CONTEXT = "context"
|
|
INSTRUCTION_PLUS_CONTEXT = "instruction+context"
|
|
NONE = "none"
|
|
|
|
|
|
_VALID_CACHE_CLASSES = {
|
|
CacheClass.INSTRUCTION,
|
|
CacheClass.CONTEXT,
|
|
CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
CacheClass.NONE,
|
|
}
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class CachePromptEntry:
|
|
"""One cacheable prompt declaration.
|
|
|
|
Attributes:
|
|
usage_label: Stable label passed to `invoke_claude(usage_label=...)`
|
|
and emitted in runtime logs. Must match what runtime call sites
|
|
pass.
|
|
instruction_func: Zero-arg callable returning the static
|
|
instruction text. May be None if the prompt only uses context
|
|
caching (cache_class == CONTEXT) and has no static instruction
|
|
we want to warm.
|
|
runtime_model: Model alias or full id this prompt is invoked with
|
|
at runtime (e.g. "sonnet_latest"). The warming layer resolves
|
|
this through `config.resolve_model_id`.
|
|
cache_class: One of the CacheClass values.
|
|
min_instruction_tokens: Optional explicit minimum size for the
|
|
instruction block. When None (the default) the warming layer
|
|
uses the per-model threshold from
|
|
`llm_utils._min_cache_tokens(resolved_model_id)` — 1024 for
|
|
Sonnet 4.5, 4096 for Haiku 4.5, etc. Set this only when an
|
|
entry needs a stricter floor than the model defaults.
|
|
notes: Free text — usually the reason this entry deviates from the
|
|
default (e.g. why something is NONE).
|
|
"""
|
|
|
|
usage_label: str
|
|
instruction_func: Callable[[], str] | None
|
|
runtime_model: str
|
|
cache_class: str
|
|
min_instruction_tokens: int | None = None
|
|
notes: str = ""
|
|
|
|
def __post_init__(self) -> None:
|
|
if self.cache_class not in _VALID_CACHE_CLASSES:
|
|
raise ValueError(
|
|
f"Unknown cache_class={self.cache_class!r} on {self.usage_label}. "
|
|
f"Expected one of {sorted(_VALID_CACHE_CLASSES)}."
|
|
)
|
|
if (
|
|
self.cache_class
|
|
in (
|
|
CacheClass.INSTRUCTION,
|
|
CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
)
|
|
and self.instruction_func is None
|
|
):
|
|
raise ValueError(
|
|
f"{self.usage_label}: instruction-cached entries require "
|
|
f"an instruction_func."
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Token estimation
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def estimate_instruction_tokens(text: str | None) -> int:
|
|
"""Estimate token count for an instruction string.
|
|
|
|
Uses the same words*1.3 heuristic as runtime prompt-call tracking so
|
|
warm-up audits and runtime reports stay aligned.
|
|
"""
|
|
if not text:
|
|
return 0
|
|
words = len(re.findall(r"\S+", text))
|
|
return int(round(words * _WORDS_TO_TOKENS_MULTIPLIER))
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Registry
|
|
# ---------------------------------------------------------------------------
|
|
# Conventions:
|
|
# - usage_label MUST match what call sites pass to `invoke_claude(...)`.
|
|
# - runtime_model is the alias the call site passes, not the resolved id.
|
|
# - Instruction-only entries that are below 1024 tokens after the latest
|
|
# pad expansion are still listed as INSTRUCTION; the warming layer logs
|
|
# a below-threshold warning so the gap is visible without silently
|
|
# hiding the prompt.
|
|
|
|
_REGISTRY: tuple[CachePromptEntry, ...] = (
|
|
# ---------- one_to_n primary / breakout family ----------
|
|
CachePromptEntry(
|
|
usage_label="REIMBURSEMENT_PRIMARY",
|
|
instruction_func=prompt_templates.REIMBURSEMENT_PRIMARY_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="METHODOLOGY_BREAKOUT",
|
|
instruction_func=prompt_templates.METHODOLOGY_BREAKOUT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="DYNAMIC_CODE_ASSIGNMENT",
|
|
instruction_func=prompt_templates.DYNAMIC_CODE_ASSIGNMENT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="FEE_SCHEDULE_BREAKOUT",
|
|
instruction_func=prompt_templates.FEE_SCHEDULE_BREAKOUT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="GROUPER_BREAKOUT",
|
|
instruction_func=prompt_templates.GROUPER_BREAKOUT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="OUTLIER_BREAKOUT",
|
|
instruction_func=prompt_templates.OUTLIER_BREAKOUT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="CARVEOUT_CHECK",
|
|
instruction_func=prompt_templates.CARVEOUT_CHECK_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="DYNAMIC_ASSIGNMENT",
|
|
instruction_func=prompt_templates.DYNAMIC_ASSIGNMENT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="REIMB_DATES_ASSIGNMENT",
|
|
instruction_func=prompt_templates.REIMB_DATES_ASSIGNMENT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="LESSER_OF_DISTRIBUTION",
|
|
instruction_func=prompt_templates.LESSER_OF_DISTRIBUTION_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="LESSER_OF_CHECK",
|
|
instruction_func=prompt_templates.LESSER_OF_CHECK_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="VALIDATE_REIMBURSEMENTS",
|
|
instruction_func=prompt_templates.VALIDATE_REIMBURSEMENTS_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="LOB_RELATIONSHIP",
|
|
instruction_func=prompt_templates.LOB_RELATIONSHIP_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
notes=(
|
|
"Runtime label is `prompt_lob_relationship` (see alias). "
|
|
"Currently below 1024-token threshold; warming layer logs a "
|
|
"below-threshold warning until the instruction is expanded."
|
|
),
|
|
),
|
|
# ---------- exhibit-level (instruction + context cache) ----------
|
|
CachePromptEntry(
|
|
usage_label="EXHIBIT_LEVEL",
|
|
instruction_func=prompt_templates.EXHIBIT_LEVEL_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
notes=(
|
|
"Calls also pass context_for_caching for the exhibit text; "
|
|
"instruction reads are stable, context reads only repeat per "
|
|
"exhibit."
|
|
),
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="DYNAMIC_PRIMARY",
|
|
instruction_func=prompt_templates.DYNAMIC_PRIMARY_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
notes="Exhibit text is sent as context_for_caching across fields.",
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="DYNAMIC_PRIMARY_ENTITIES",
|
|
instruction_func=prompt_templates.DYNAMIC_PRIMARY_ENTITIES_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.CONTEXT,
|
|
notes=(
|
|
"Static instruction is intentionally short (~180 tokens) and "
|
|
"below the 1024-token Sonnet 4.5 cache minimum; padding it "
|
|
"would dwarf the actual prompt. Caching here relies on the "
|
|
"context_for_caching block (the exhibit text) repeating "
|
|
"within a run. CONTEXT class skips instruction warm-up to "
|
|
"avoid spending warm-tokens on a block that cannot cache."
|
|
),
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION",
|
|
instruction_func=prompt_templates.DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
notes=(
|
|
"Classification step that runs after entity extraction. "
|
|
"Instruction expanded over the 1024-token Sonnet 4.5 cache "
|
|
"minimum so the static block caches; the dynamic entity list "
|
|
"in context_for_caching is too small to cache reliably and "
|
|
"is expected to miss on the context block."
|
|
),
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="DYNAMIC_PRIMARY_ENTITIES_ASSIGNMENT",
|
|
instruction_func=prompt_templates.DYNAMIC_PRIMARY_ENTITIES_ASSIGNMENT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
notes="Field-level assignment step for DYNAMIC_PRIMARY_ENTITIES; exhibit text passed as context_for_caching.",
|
|
),
|
|
# ---------- LOB / program / product mapping ----------
|
|
CachePromptEntry(
|
|
usage_label="MAP_LOB_TO_VALID",
|
|
instruction_func=prompt_templates.MAP_LOB_TO_VALID_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
notes="Maps raw LOB values to valid crosswalk keys; valid-LOB list passed as context_for_caching.",
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="MAP_NETWORK_TO_VALID",
|
|
instruction_func=prompt_templates.MAP_NETWORK_TO_VALID_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
notes="Maps raw NETWORK values to valid crosswalk keys; valid-network list passed as context_for_caching.",
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="AARETE_DERIVED_PROGRAM",
|
|
instruction_func=prompt_templates.AARETE_DERIVED_PROGRAM_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
notes="Maps extracted PROGRAM names to AARETE_DERIVED_PROGRAM values; valid-values list passed as context_for_caching.",
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="AARETE_DERIVED_PRODUCT",
|
|
instruction_func=prompt_templates.AARETE_DERIVED_PRODUCT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
notes="Maps extracted PRODUCT names to AARETE_DERIVED_PRODUCT values; valid-values list passed as context_for_caching.",
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="PROGRAM_TO_LOB",
|
|
instruction_func=prompt_templates.PROGRAM_TO_LOB_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
notes="Derives LOB values from raw PROGRAM + AARETE_DERIVED_PROGRAM; no context_for_caching at runtime.",
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="PRODUCT_TO_LOB",
|
|
instruction_func=prompt_templates.PRODUCT_TO_LOB_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
notes="Derives LOB values from raw PRODUCT + AARETE_DERIVED_PRODUCT; no context_for_caching at runtime.",
|
|
),
|
|
# ---------- service term split (Sonnet 4.5 — Haiku 4.5 won't cache) ----------
|
|
CachePromptEntry(
|
|
usage_label="SPLIT_SERVICE_TERM",
|
|
instruction_func=prompt_templates.SPLIT_SERVICE_TERM_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
notes=(
|
|
"Moved off haiku_latest because Haiku 4.5's 4096-token "
|
|
"minimum cache size means the ~1600-token SPLIT_SERVICE_TERM "
|
|
"instruction was never cached. Sonnet 4.5's 1024-token "
|
|
"minimum makes it cache cleanly. With heavy repetition of "
|
|
"service terms across exhibits, cache reads on Sonnet beat "
|
|
"uncached Haiku on cost despite the higher per-token rate."
|
|
),
|
|
),
|
|
# ---------- preprocessing ----------
|
|
CachePromptEntry(
|
|
usage_label="EXHIBIT_HEADER",
|
|
instruction_func=prompt_templates.EXHIBIT_HEADER_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="EXHIBIT_LINKAGE",
|
|
instruction_func=prompt_templates.EXHIBIT_LINKAGE_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="EXHIBIT_TITLE_MATCH",
|
|
instruction_func=prompt_templates.EXHIBIT_TITLE_MATCH_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
# ---------- utility ----------
|
|
CachePromptEntry(
|
|
usage_label="DATE_FIX",
|
|
instruction_func=prompt_templates.DATE_FIX_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="DERIVED_TERM_DATE",
|
|
instruction_func=prompt_templates.DERIVED_TERM_DATE_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="CHECK_PROVIDER_NAME_MATCH",
|
|
instruction_func=prompt_templates.CHECK_PROVIDER_NAME_MATCH_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="SPECIAL_CASE_ASSIGNMENT",
|
|
instruction_func=prompt_templates.SPECIAL_CASE_ASSIGNMENT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="SPLIT_REIMB_DATES",
|
|
instruction_func=prompt_templates.SPLIT_REIMB_DATES_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
# ---------- code extraction family ----------
|
|
CachePromptEntry(
|
|
usage_label="SERVICE_ENRICHMENT",
|
|
instruction_func=prompt_templates.SERVICE_ENRICHMENT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="CODE_EXPLICIT",
|
|
instruction_func=prompt_templates.CODE_EXPLICIT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="CODE_CATEGORY",
|
|
instruction_func=prompt_templates.CODE_CATEGORY_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="CODE_IMPLICIT_SPECIAL",
|
|
instruction_func=prompt_templates.CODE_IMPLICIT_SPECIAL_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="CODE_IMPLICIT",
|
|
instruction_func=prompt_templates.CODE_IMPLICIT_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="CODE_IMPLICIT_ARBITRATION",
|
|
instruction_func=prompt_templates.CODE_IMPLICIT_ARBITRATION_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="CODE_LAST_CHECK",
|
|
instruction_func=prompt_templates.CODE_LAST_CHECK_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="FILL_BILL_TYPE",
|
|
instruction_func=prompt_templates.FILL_BILL_TYPE_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
# ---------- one_to_one ----------
|
|
CachePromptEntry(
|
|
usage_label="TIN_NPI_TEMPLATE",
|
|
instruction_func=prompt_templates.TIN_NPI_TEMPLATE_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
CachePromptEntry(
|
|
usage_label="ONE_TO_ONE_SINGLE_FIELD",
|
|
instruction_func=prompt_templates.ONE_TO_ONE_SINGLE_FIELD_INSTRUCTION,
|
|
runtime_model="sonnet_latest",
|
|
cache_class=CacheClass.INSTRUCTION,
|
|
),
|
|
)
|
|
|
|
|
|
# Aliases let runtime usage_labels that don't match the canonical
|
|
# instruction-function name still resolve to one registry entry.
|
|
# Callers should prefer canonical labels; this is a compatibility shim.
|
|
_USAGE_LABEL_ALIASES: dict[str, str] = {
|
|
"code_implicit_rag": "CODE_IMPLICIT",
|
|
"prompt_provider_info": "TIN_NPI_TEMPLATE",
|
|
"validate_reimbursements_for_llm": "VALIDATE_REIMBURSEMENTS",
|
|
"fill_bill_type": "FILL_BILL_TYPE",
|
|
"prompt_lob_relationship": "LOB_RELATIONSHIP",
|
|
}
|
|
|
|
|
|
def _build_index() -> dict[str, CachePromptEntry]:
|
|
index: dict[str, CachePromptEntry] = {}
|
|
for entry in _REGISTRY:
|
|
if entry.usage_label in index:
|
|
raise ValueError(
|
|
f"Duplicate registry entry for usage_label={entry.usage_label}"
|
|
)
|
|
index[entry.usage_label] = entry
|
|
return index
|
|
|
|
|
|
_INDEX: dict[str, CachePromptEntry] = _build_index()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Public API
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def all_entries() -> tuple[CachePromptEntry, ...]:
|
|
"""Return every registered entry (including NONE / opted-out ones)."""
|
|
return _REGISTRY
|
|
|
|
|
|
def get_entry(usage_label: str) -> CachePromptEntry | None:
|
|
"""Look up an entry by usage_label. Resolves aliases."""
|
|
if usage_label in _INDEX:
|
|
return _INDEX[usage_label]
|
|
canonical = _USAGE_LABEL_ALIASES.get(usage_label)
|
|
if canonical is not None:
|
|
return _INDEX.get(canonical)
|
|
return None
|
|
|
|
|
|
def _resolve_min_tokens(entry: CachePromptEntry, resolved_model_id: str) -> int:
|
|
"""Pick the threshold to enforce: explicit on the entry > per-model > default."""
|
|
if entry.min_instruction_tokens is not None:
|
|
return entry.min_instruction_tokens
|
|
# Imported lazily so this module stays importable without llm_utils.
|
|
from src.utils.llm_utils import _min_cache_tokens
|
|
|
|
return _min_cache_tokens(resolved_model_id)
|
|
|
|
|
|
def iter_warming_targets() -> Iterator[tuple[str, str, str]]:
|
|
"""Yield (usage_label, resolved_model_id, instruction_text) tuples for
|
|
every entry whose instruction block is supposed to be cached.
|
|
|
|
Skips:
|
|
- cache_class == NONE
|
|
- cache_class == CONTEXT (no static instruction to warm)
|
|
- entries whose resolved model does not support prompt caching
|
|
- entries whose instruction function fails to render
|
|
"""
|
|
# Imported here to avoid a circular import: llm_utils → cache_registry
|
|
# is allowed, but cache_registry → llm_utils only at call time.
|
|
from src.utils.llm_utils import _supports_prompt_cache
|
|
|
|
for entry in _REGISTRY:
|
|
if entry.cache_class in (CacheClass.NONE, CacheClass.CONTEXT):
|
|
continue
|
|
if entry.instruction_func is None:
|
|
continue
|
|
|
|
resolved_model_id = config.resolve_model_id(entry.runtime_model)
|
|
if not _supports_prompt_cache(resolved_model_id):
|
|
logging.info(
|
|
"cache_registry: skipping warm-up for %s — model %s "
|
|
"(resolved %s) does not support prompt caching.",
|
|
entry.usage_label,
|
|
entry.runtime_model,
|
|
resolved_model_id,
|
|
)
|
|
continue
|
|
|
|
try:
|
|
instruction_text = entry.instruction_func()
|
|
except Exception as exc: # field bundles can be unavailable in CLI/tests
|
|
logging.debug(
|
|
"cache_registry: skipping %s — instruction function raised: %s",
|
|
entry.usage_label,
|
|
exc,
|
|
)
|
|
continue
|
|
|
|
token_estimate = estimate_instruction_tokens(instruction_text)
|
|
threshold = _resolve_min_tokens(entry, resolved_model_id)
|
|
if token_estimate < threshold:
|
|
# Warn but still warm — the call still tracks
|
|
# cache_creation_input_tokens in the response, and the runtime
|
|
# cache_miss_reason makes the gap visible.
|
|
logging.warning(
|
|
"cache_registry: %s instruction is ~%d tokens, below the "
|
|
"%d-token minimum for %s; cache_control block may not be "
|
|
"honored.",
|
|
entry.usage_label,
|
|
token_estimate,
|
|
threshold,
|
|
resolved_model_id,
|
|
)
|
|
|
|
yield entry.usage_label, resolved_model_id, instruction_text
|
|
|
|
|
|
def audit_below_threshold() -> list[tuple[str, int, int]]:
|
|
"""Return (usage_label, estimated_tokens, min_required) for every
|
|
INSTRUCTION-class entry whose instruction is below threshold. The
|
|
threshold is resolved per-model: Sonnet 4.5 = 1024, Haiku 4.5 = 4096,
|
|
etc. Useful for a startup audit / contract test.
|
|
"""
|
|
findings: list[tuple[str, int, int]] = []
|
|
for entry in _REGISTRY:
|
|
if entry.cache_class not in (
|
|
CacheClass.INSTRUCTION,
|
|
CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
|
):
|
|
continue
|
|
if entry.instruction_func is None:
|
|
continue
|
|
try:
|
|
text = entry.instruction_func()
|
|
except Exception:
|
|
continue
|
|
resolved_model_id = config.resolve_model_id(entry.runtime_model)
|
|
threshold = _resolve_min_tokens(entry, resolved_model_id)
|
|
tokens = estimate_instruction_tokens(text)
|
|
if tokens < threshold:
|
|
findings.append((entry.usage_label, tokens, threshold))
|
|
return findings
|
|
|
|
|
|
def instruction_cache_entries() -> Iterable[CachePromptEntry]:
|
|
"""Entries where the instruction block is expected to be cached."""
|
|
return tuple(
|
|
e
|
|
for e in _REGISTRY
|
|
if e.cache_class
|
|
in (CacheClass.INSTRUCTION, CacheClass.INSTRUCTION_PLUS_CONTEXT)
|
|
)
|
|
|
|
|
|
def context_cache_entries() -> Iterable[CachePromptEntry]:
|
|
"""Entries where a dynamic context block is cached separately."""
|
|
return tuple(
|
|
e
|
|
for e in _REGISTRY
|
|
if e.cache_class in (CacheClass.CONTEXT, CacheClass.INSTRUCTION_PLUS_CONTEXT)
|
|
)
|