doczyai-pipelines/documentation/PROGRAM_PRODUCT_CANONICAL_DERIVATION.md

# AARETE Derived Program & Product Canonical Derivation

---

## Objective

Raw contracts reference the same healthcare program or health plan product using many different names. This feature normalizes them all to a single canonical name per entity, stored in two Dashboard columns:

- `AARETE_DERIVED_PROGRAM` — canonical program name (e.g., CHIP, MMC, STAR)
- `AARETE_DERIVED_PRODUCT` — canonical product/plan name (e.g., HA+, HMO)

The canonical is always **one of the actual values seen in the data** — nothing is invented.

---

## Pipeline (4 Steps)

```
Raw PROGRAM / PRODUCT column  (comma-separated strings in Dashboard DF)
  │
  ├─ Step 1  Collect all unique non-empty values across the full DataFrame
  │
  ├─ Step 2  Extract payer context from the DataFrame
  │           (PAYER_NAME, PAYER_STATE, FILE_NAME from the first row)
  │
  ├─ Step 3  ONE global LLM batch call with all unique values + payer context
  │           + valid-values soft anchor → returns {raw: canonical} dict
  │           The LLM groups semantically equivalent variants and assigns one
  │           canonical per group, resolving brand-prefix noise vs. meaningful
  │           qualifiers in a single pass
  │
  └─ Step 4  Apply {raw → canonical} map to every row
              Output is comma-separated string, e.g. "CHIP, MMC"
```

## Inputs & Outputs

| | PROGRAM | PRODUCT |
|---|---|---|
| **Input column** | `PROGRAM` | `PRODUCT` |
| **Output column** | `AARETE_DERIVED_PROGRAM` | `AARETE_DERIVED_PRODUCT` |
| **Valid-values list** | `constants.VALID_AARETE_DERIVED_PROGRAMS` | `constants.VALID_AARETE_DERIVED_PRODUCTS` |
| **Output format** | `str`, comma-separated | `str`, comma-separated |

**Skip condition:** If the valid-values list is non-empty **and** the derived column is already fully populated, the function returns the DataFrame unchanged.

---

## Exceptions & Edge Cases

| Scenario | Behaviour |
|---|---|
| `PROGRAM` / `PRODUCT` column missing | Warning logged, DataFrame returned unchanged |
| All values empty / null | Info logged, no LLM call made |
| Sentinel values (`N/A`, `UNKNOWN`, `NONE`, `NULL`) | Filtered out before LLM call |
| LLM omits a raw value from its response | Falls back to `raw.upper()` for that key |
| LLM response parse failure | Error logged; entire map falls back to `raw.upper()` for all values |
| Multi-value cell e.g. `"CHIP, MMC"` | Each value mapped independently, re-joined |
| Duplicate canonicals in one cell | Deduplicated before joining |

---

## LLM Prompt Design

Both batch prompts follow the same structure: a **static cached instruction** (registered in `src/prompts/cache_registry.py`) plus a **dynamic template** that injects unique values and payer context at runtime.

The instruction defines 7 rules evaluated in order:

| # | Rule | Description |
|---|---|---|
| 1 | **PAYER PREFIX** | Strip the payer brand name when it appears as a leading prefix (e.g., "Molina Medicaid" → "Medicaid"). For PRODUCT, the base name after stripping must be identical for two values to merge — "Options" ≠ "Options Plus". |
| 2 | **STATE PREFIX NOT NOISE** | A state abbreviation prefix is meaningful if the payer contracts span multiple states; keep it unless context confirms it is redundant. |
| 3 | **DISTINGUISHING QUALIFIERS** | Qualifiers that change meaning (e.g., "Perinatal", "Enhanced") must keep values as separate canonicals. "Plus" is usually distinguishing unless Rule 5 applies. |
| 4 | **ABBREVIATION / FULL-NAME** | Prefer the recognized abbreviation as the canonical (e.g., "CHIP" over "Children's Health Insurance Program"). |
| 5 | **ABBREVIATIONS ABSORB BRAND-PREFIX AND PLUS EXPANSIONS** | A widely-used abbreviation can absorb brand-prefix or "Plus" expansions if it already encodes them (e.g., MMOP absorbs "Molina Medicaid Options Plus"). |
| 6 | **DO NOT STRIP GENERIC SUFFIXES** | Generic trailing words like "Plan" or "Program" are part of the canonical and must be preserved — unless another group in the same batch already uses the suffix-stripped form as its canonical (collision avoidance). |
| 7 | **TIEBREAKER — PREFER SPLINTER** | When uncertain, keep two values as separate canonicals rather than merging them. Over-splitting is safer than over-merging. |

The valid-values list is passed as a soft anchor (canonical selection priority (a)). When a raw value clearly matches a valid value, that exact string is used. Otherwise the LLM selects from the raw input values.

---

## Code Changes

| File | What changed |
|---|---|
| [src/pipelines/shared/postprocessing/aarete_derived.py](../src/pipelines/shared/postprocessing/aarete_derived.py) | Removed clustering helpers (`_merge_acronym_clusters`, `_split_subset_clusters`); rewrote `add_aarete_derived_program_canonical` and `add_aarete_derived_product_canonical` to use a single batch LLM call |
| [src/prompts/prompt_templates.py](../src/prompts/prompt_templates.py) | Added 4 functions: 2 cached batch instructions + 2 dynamic batch templates |
| [src/pipelines/saas/prompts/prompt_calls.py](../src/pipelines/saas/prompts/prompt_calls.py) | Added 2 batch wrapper functions: `prompt_aarete_derived_program_canonical_batch` and `prompt_aarete_derived_product_canonical_batch` |
| [src/prompts/cache_registry.py](../src/prompts/cache_registry.py) | Added 2 cache entries for the batch prompt instructions |
| [test_program_product_canonical.py](../test_program_product_canonical.py) | Updated for batch architecture; removed clustering imports and per-cluster validation checks |

---

## Function Reference

### `aarete_derived.py`

| Function | Visibility | Purpose |
|---|---|---|
| `_split_multi_value_separators(text)` | private | Splits a raw cell value on commas, semicolons, and similar delimiters. |
| `_collect_unique_values_from_list_column(column)` | private | Deduplicated list of non-empty, non-sentinel values from a Series. Handles both comma-string and list formats. |
| `_all_empty_in_list_column(column)` | private | Returns `True` if every cell is empty — used as the skip-derivation guard. |
| `_map_row_list_to_canonicals(raw_val, canonical_map)` | private | Maps one cell to its canonical(s). Splits on `,`, looks up each part, deduplicates, returns comma-separated string. |
| `add_aarete_derived_program_canonical(df, valid_programs)` | **public** | Runs the 4-step batch pipeline for `PROGRAM` → `AARETE_DERIVED_PROGRAM`. |
| `add_aarete_derived_product_canonical(df, valid_products)` | **public** | Same pipeline for `PRODUCT` → `AARETE_DERIVED_PRODUCT`. |

### `prompt_templates.py`

| Function | Purpose |
|---|---|
| `AARETE_DERIVED_PROGRAM_CANONICAL_BATCH_INSTRUCTION()` | Static cached system instruction for batch PROGRAM canonicalization (7 rules). Registered in `cache_registry.py` under label `AARETE_DERIVED_PROGRAM_CANONICAL_BATCH`. |
| `AARETE_DERIVED_PROGRAM_CANONICAL_BATCH(unique_values, valid_programs, payer_name, payer_state)` | Dynamic template. Returns `(context_text, prompt_text, parser)` tuple. |
| `AARETE_DERIVED_PRODUCT_CANONICAL_BATCH_INSTRUCTION()` | Static cached system instruction for batch PRODUCT canonicalization (7 rules). Registered under label `AARETE_DERIVED_PRODUCT_CANONICAL_BATCH`. |
| `AARETE_DERIVED_PRODUCT_CANONICAL_BATCH(unique_values, valid_products, payer_name, payer_state)` | Dynamic template. Returns `(context_text, prompt_text, parser)` tuple. |

### `prompt_calls.py`

| Function | Purpose |
|---|---|
| `prompt_aarete_derived_program_canonical_batch(unique_values, valid_programs, filename, payer_name, payer_state)` | Single LLM call for all PROGRAM values. Returns `dict[str, str]` mapping each raw value to its canonical. Falls back to `raw.upper()` for any key the LLM omits; falls back to `{v: v.upper()}` for all values on parse failure. |
| `prompt_aarete_derived_product_canonical_batch(unique_values, valid_products, filename, payer_name, payer_state)` | Same for PRODUCT. |

---

## Example

**Raw PROGRAM values (all unique values across the DataFrame):**
`CHIP`, `Children's Health Insurance Program`, `Children's Health Insurance Program (CHIP)`, `MMC`, `Medicaid Managed Care`, `Medicaid`

**Batch LLM call (Step 3):**

All 6 values are sent in one call with payer context. The LLM determines:
- `CHIP`, `Children's Health Insurance Program`, `Children's Health Insurance Program (CHIP)` → same program (Rule 4: prefer abbreviation) → canonical: `CHIP`
- `MMC`, `Medicaid Managed Care` → same program (Rule 4: prefer abbreviation) → canonical: `MMC`
- `Medicaid` → distinct broader program; no other group uses `MEDICAID` → canonical: `MEDICAID`

**Resulting `{raw → canonical}` map:**

| Raw value | Canonical |
|---|---|
| CHIP | CHIP |
| Children's Health Insurance Program | CHIP |
| Children's Health Insurance Program (CHIP) | CHIP |
| Medicaid Managed Care | MMC |
| MMC | MMC |
| Medicaid | MEDICAID |

**Final output applied to rows:**

| PROGRAM (raw) | AARETE_DERIVED_PROGRAM |
|---|---|
| CHIP | CHIP |
| Children's Health Insurance Program | CHIP |
| Children's Health Insurance Program (CHIP) | CHIP |
| Medicaid Managed Care | MMC |
| Medicaid | MEDICAID |
| CHIP, Medicaid Managed Care | CHIP, MMC |