doczyai-pipelines/documentation/PROGRAM_PRODUCT_CANONICAL_DERIVATION.md

AARETE Derived Program & Product Canonical Derivation

---

Objective

Raw contracts reference the same healthcare program or health plan product using many different names. This feature normalizes them all to a single canonical name per entity, stored in two Dashboard columns:

• AARETE_DERIVED_PROGRAM — canonical program name (e.g., CHIP, MMC, STAR)
• AARETE_DERIVED_PRODUCT — canonical product/plan name (e.g., HA+, HMO)

The canonical is always one of the actual values seen in the data — nothing is invented.

---

Pipeline (4 Steps)

Raw PROGRAM / PRODUCT column  (comma-separated strings in Dashboard DF)
  │
  ├─ Step 1  Collect all unique non-empty values across the full DataFrame
  │
  ├─ Step 2  Extract payer context from the DataFrame
  │           (PAYER_NAME, PAYER_STATE, FILE_NAME from the first row)
  │
  ├─ Step 3  ONE global LLM batch call with all unique values + payer context
  │           + valid-values soft anchor → returns {raw: canonical} dict
  │           The LLM groups semantically equivalent variants and assigns one
  │           canonical per group, resolving brand-prefix noise vs. meaningful
  │           qualifiers in a single pass
  │
  └─ Step 4  Apply {raw → canonical} map to every row
              Output is comma-separated string, e.g. "CHIP, MMC"

Inputs & Outputs

	PROGRAM	PRODUCT
Input column	PROGRAM	PRODUCT
Output column	AARETE_DERIVED_PROGRAM	AARETE_DERIVED_PRODUCT
Valid-values list	constants.VALID_AARETE_DERIVED_PROGRAMS	constants.VALID_AARETE_DERIVED_PRODUCTS
Output format	str, comma-separated	str, comma-separated

Skip condition: If the valid-values list is non-empty and the derived column is already fully populated, the function returns the DataFrame unchanged.

---

Exceptions & Edge Cases

Scenario	Behaviour
PROGRAM / PRODUCT column missing	Warning logged, DataFrame returned unchanged
All values empty / null	Info logged, no LLM call made
Sentinel values (N/A, UNKNOWN, NONE, NULL)	Filtered out before LLM call
LLM omits a raw value from its response	Falls back to raw.upper() for that key
LLM response parse failure	Error logged; entire map falls back to raw.upper() for all values
Multi-value cell e.g. "CHIP, MMC"	Each value mapped independently, re-joined
Duplicate canonicals in one cell	Deduplicated before joining

---

LLM Prompt Design

Both batch prompts follow the same structure: a static cached instruction (registered in src/prompts/cache_registry.py) plus a dynamic template that injects unique values and payer context at runtime.

The instruction defines 7 rules evaluated in order:

#	Rule	Description
1	PAYER PREFIX	Strip the payer brand name when it appears as a leading prefix (e.g., "Molina Medicaid" → "Medicaid"). For PRODUCT, the base name after stripping must be identical for two values to merge — "Options" ≠ "Options Plus".
2	STATE PREFIX NOT NOISE	A state abbreviation prefix is meaningful if the payer contracts span multiple states; keep it unless context confirms it is redundant.
3	DISTINGUISHING QUALIFIERS	Qualifiers that change meaning (e.g., "Perinatal", "Enhanced") must keep values as separate canonicals. "Plus" is usually distinguishing unless Rule 5 applies.
4	ABBREVIATION / FULL-NAME	Prefer the recognized abbreviation as the canonical (e.g., "CHIP" over "Children's Health Insurance Program").
5	ABBREVIATIONS ABSORB BRAND-PREFIX AND PLUS EXPANSIONS	A widely-used abbreviation can absorb brand-prefix or "Plus" expansions if it already encodes them (e.g., MMOP absorbs "Molina Medicaid Options Plus").
6	DO NOT STRIP GENERIC SUFFIXES	Generic trailing words like "Plan" or "Program" are part of the canonical and must be preserved — unless another group in the same batch already uses the suffix-stripped form as its canonical (collision avoidance).
7	TIEBREAKER — PREFER SPLINTER	When uncertain, keep two values as separate canonicals rather than merging them. Over-splitting is safer than over-merging.

The valid-values list is passed as a soft anchor (canonical selection priority (a)). When a raw value clearly matches a valid value, that exact string is used. Otherwise the LLM selects from the raw input values.

---

Code Changes

File	What changed
src/pipelines/shared/postprocessing/aarete_derived.py	Removed clustering helpers (_merge_acronym_clusters, _split_subset_clusters); rewrote add_aarete_derived_program_canonical and add_aarete_derived_product_canonical to use a single batch LLM call
src/prompts/prompt_templates.py	Added 4 functions: 2 cached batch instructions + 2 dynamic batch templates
src/pipelines/saas/prompts/prompt_calls.py	Added 2 batch wrapper functions: prompt_aarete_derived_program_canonical_batch and prompt_aarete_derived_product_canonical_batch
src/prompts/cache_registry.py	Added 2 cache entries for the batch prompt instructions
test_program_product_canonical.py	Updated for batch architecture; removed clustering imports and per-cluster validation checks

---

Function Reference

aarete_derived.py

Function	Visibility	Purpose
_split_multi_value_separators(text)	private	Splits a raw cell value on commas, semicolons, and similar delimiters.
_collect_unique_values_from_list_column(column)	private	Deduplicated list of non-empty, non-sentinel values from a Series. Handles both comma-string and list formats.
_all_empty_in_list_column(column)	private	Returns True if every cell is empty — used as the skip-derivation guard.
_map_row_list_to_canonicals(raw_val, canonical_map)	private	Maps one cell to its canonical(s). Splits on ,, looks up each part, deduplicates, returns comma-separated string.
add_aarete_derived_program_canonical(df, valid_programs)	public	Runs the 4-step batch pipeline for PROGRAM → AARETE_DERIVED_PROGRAM.
add_aarete_derived_product_canonical(df, valid_products)	public	Same pipeline for PRODUCT → AARETE_DERIVED_PRODUCT.

prompt_templates.py

Function	Purpose
AARETE_DERIVED_PROGRAM_CANONICAL_BATCH_INSTRUCTION()	Static cached system instruction for batch PROGRAM canonicalization (7 rules). Registered in cache_registry.py under label AARETE_DERIVED_PROGRAM_CANONICAL_BATCH.
AARETE_DERIVED_PROGRAM_CANONICAL_BATCH(unique_values, valid_programs, payer_name, payer_state)	Dynamic template. Returns (context_text, prompt_text, parser) tuple.
AARETE_DERIVED_PRODUCT_CANONICAL_BATCH_INSTRUCTION()	Static cached system instruction for batch PRODUCT canonicalization (7 rules). Registered under label AARETE_DERIVED_PRODUCT_CANONICAL_BATCH.
AARETE_DERIVED_PRODUCT_CANONICAL_BATCH(unique_values, valid_products, payer_name, payer_state)	Dynamic template. Returns (context_text, prompt_text, parser) tuple.

prompt_calls.py

Function	Purpose
prompt_aarete_derived_program_canonical_batch(unique_values, valid_programs, filename, payer_name, payer_state)	Single LLM call for all PROGRAM values. Returns dict[str, str] mapping each raw value to its canonical. Falls back to raw.upper() for any key the LLM omits; falls back to {v: v.upper()} for all values on parse failure.
prompt_aarete_derived_product_canonical_batch(unique_values, valid_products, filename, payer_name, payer_state)	Same for PRODUCT.

---

Example

Raw PROGRAM values (all unique values across the DataFrame): CHIP, Children's Health Insurance Program, Children's Health Insurance Program (CHIP), MMC, Medicaid Managed Care, Medicaid

Batch LLM call (Step 3):

All 6 values are sent in one call with payer context. The LLM determines:

• CHIP, Children's Health Insurance Program, Children's Health Insurance Program (CHIP) → same program (Rule 4: prefer abbreviation) → canonical: CHIP
• MMC, Medicaid Managed Care → same program (Rule 4: prefer abbreviation) → canonical: MMC
• Medicaid → distinct broader program; no other group uses MEDICAID → canonical: MEDICAID

Resulting {raw → canonical} map:

Raw value	Canonical
CHIP	CHIP
Children's Health Insurance Program	CHIP
Children's Health Insurance Program (CHIP)	CHIP
Medicaid Managed Care	MMC
MMC	MMC
Medicaid	MEDICAID

Final output applied to rows:

PROGRAM (raw)	AARETE_DERIVED_PROGRAM
CHIP	CHIP
Children's Health Insurance Program	CHIP
Children's Health Insurance Program (CHIP)	CHIP
Medicaid Managed Care	MMC
Medicaid	MEDICAID
CHIP, Medicaid Managed Care	CHIP, MMC