Merged in dev (pull request #1016)
Dev * Merged in bugfix/DAIP2-2823-generic-issue-fixes-one-to-one (pull request #1011) Bugfix/DAIP2-2823 generic issue fixes one to one * updated auto renewal ind prompt * updated CONTRACT_AMENDMENT_NUM prompt * Merged dev into bugfix/DAIP2-2823-generic-issue-fixes-one-to-one Approved-by: Praneel Panchigar Approved-by: Siddhant Medar * Merged in feature/DAIP2-2698-phase-3-program-product-lob-mapping (pull request #1000) Feature/DAIP2-2698 phase 3 program product lob mapping * added missing phase 2 modifications * added phase 3 modifications * Fixed acronym issues * black format fix * Merged dev into feature/DAIP2-2698-phase-3-program-product-lob-mapping * standardization fixes * Merged dev into feature/DAIP2-2698-phase-3-program-product-lob-mapping * added hyphenated suffix names fix * black format fix * standardization fix * dynamic primary fix * Merge Dev into feature/DAIP2-2698-phase-3-program-product-lob-mapping * updated Program Product Standardization * updated prompt instrcutions * added documenttaion markdown file * add Rule 5 counter-example for Plus-less sibling in PRODUCT canonical prompt Clarify that Rule 5's abbreviation absorption does not extend to the bare-form sibling when the anchor's expansion itself contains 'Plus'. Addresses a row in the may18 canonical run where 'Molina Medicare Options' (no Plus) was… Approved-by: Siddhant Medar
This commit is contained in:
Praneel Panchigar
committed by
Siddhant Medar
Siddhant Medar
parent
4d153aa090
commit
a082b1864c
@@ -0,0 +1,158 @@
|
||||
# AARETE Derived Program & Product Canonical Derivation
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
Raw contracts reference the same healthcare program or health plan product using many different names. This feature normalizes them all to a single canonical name per entity, stored in two Dashboard columns:
|
||||
|
||||
- `AARETE_DERIVED_PROGRAM` — canonical program name (e.g., CHIP, MMC, STAR)
|
||||
- `AARETE_DERIVED_PRODUCT` — canonical product/plan name (e.g., HA+, HMO)
|
||||
|
||||
The canonical is always **one of the actual values seen in the data** — nothing is invented.
|
||||
|
||||
---
|
||||
|
||||
## Pipeline (4 Steps)
|
||||
|
||||
```
|
||||
Raw PROGRAM / PRODUCT column (comma-separated strings in Dashboard DF)
|
||||
│
|
||||
├─ Step 1 Collect all unique non-empty values across the full DataFrame
|
||||
│
|
||||
├─ Step 2 Extract payer context from the DataFrame
|
||||
│ (PAYER_NAME, PAYER_STATE, FILE_NAME from the first row)
|
||||
│
|
||||
├─ Step 3 ONE global LLM batch call with all unique values + payer context
|
||||
│ + valid-values soft anchor → returns {raw: canonical} dict
|
||||
│ The LLM groups semantically equivalent variants and assigns one
|
||||
│ canonical per group, resolving brand-prefix noise vs. meaningful
|
||||
│ qualifiers in a single pass
|
||||
│
|
||||
└─ Step 4 Apply {raw → canonical} map to every row
|
||||
Output is comma-separated string, e.g. "CHIP, MMC"
|
||||
```
|
||||
|
||||
## Inputs & Outputs
|
||||
|
||||
| | PROGRAM | PRODUCT |
|
||||
|---|---|---|
|
||||
| **Input column** | `PROGRAM` | `PRODUCT` |
|
||||
| **Output column** | `AARETE_DERIVED_PROGRAM` | `AARETE_DERIVED_PRODUCT` |
|
||||
| **Valid-values list** | `constants.VALID_AARETE_DERIVED_PROGRAMS` | `constants.VALID_AARETE_DERIVED_PRODUCTS` |
|
||||
| **Output format** | `str`, comma-separated | `str`, comma-separated |
|
||||
|
||||
**Skip condition:** If the valid-values list is non-empty **and** the derived column is already fully populated, the function returns the DataFrame unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Exceptions & Edge Cases
|
||||
|
||||
| Scenario | Behaviour |
|
||||
|---|---|
|
||||
| `PROGRAM` / `PRODUCT` column missing | Warning logged, DataFrame returned unchanged |
|
||||
| All values empty / null | Info logged, no LLM call made |
|
||||
| Sentinel values (`N/A`, `UNKNOWN`, `NONE`, `NULL`) | Filtered out before LLM call |
|
||||
| LLM omits a raw value from its response | Falls back to `raw.upper()` for that key |
|
||||
| LLM response parse failure | Error logged; entire map falls back to `raw.upper()` for all values |
|
||||
| Multi-value cell e.g. `"CHIP, MMC"` | Each value mapped independently, re-joined |
|
||||
| Duplicate canonicals in one cell | Deduplicated before joining |
|
||||
|
||||
---
|
||||
|
||||
## LLM Prompt Design
|
||||
|
||||
Both batch prompts follow the same structure: a **static cached instruction** (registered in `src/prompts/cache_registry.py`) plus a **dynamic template** that injects unique values and payer context at runtime.
|
||||
|
||||
The instruction defines 7 rules evaluated in order:
|
||||
|
||||
| # | Rule | Description |
|
||||
|---|---|---|
|
||||
| 1 | **PAYER PREFIX** | Strip the payer brand name when it appears as a leading prefix (e.g., "Molina Medicaid" → "Medicaid"). For PRODUCT, the base name after stripping must be identical for two values to merge — "Options" ≠ "Options Plus". |
|
||||
| 2 | **STATE PREFIX NOT NOISE** | A state abbreviation prefix is meaningful if the payer contracts span multiple states; keep it unless context confirms it is redundant. |
|
||||
| 3 | **DISTINGUISHING QUALIFIERS** | Qualifiers that change meaning (e.g., "Perinatal", "Enhanced") must keep values as separate canonicals. "Plus" is usually distinguishing unless Rule 5 applies. |
|
||||
| 4 | **ABBREVIATION / FULL-NAME** | Prefer the recognized abbreviation as the canonical (e.g., "CHIP" over "Children's Health Insurance Program"). |
|
||||
| 5 | **ABBREVIATIONS ABSORB BRAND-PREFIX AND PLUS EXPANSIONS** | A widely-used abbreviation can absorb brand-prefix or "Plus" expansions if it already encodes them (e.g., MMOP absorbs "Molina Medicaid Options Plus"). |
|
||||
| 6 | **DO NOT STRIP GENERIC SUFFIXES** | Generic trailing words like "Plan" or "Program" are part of the canonical and must be preserved — unless another group in the same batch already uses the suffix-stripped form as its canonical (collision avoidance). |
|
||||
| 7 | **TIEBREAKER — PREFER SPLINTER** | When uncertain, keep two values as separate canonicals rather than merging them. Over-splitting is safer than over-merging. |
|
||||
|
||||
The valid-values list is passed as a soft anchor (canonical selection priority (a)). When a raw value clearly matches a valid value, that exact string is used. Otherwise the LLM selects from the raw input values.
|
||||
|
||||
---
|
||||
|
||||
## Code Changes
|
||||
|
||||
| File | What changed |
|
||||
|---|---|
|
||||
| [src/pipelines/shared/postprocessing/aarete_derived.py](../src/pipelines/shared/postprocessing/aarete_derived.py) | Removed clustering helpers (`_merge_acronym_clusters`, `_split_subset_clusters`); rewrote `add_aarete_derived_program_canonical` and `add_aarete_derived_product_canonical` to use a single batch LLM call |
|
||||
| [src/prompts/prompt_templates.py](../src/prompts/prompt_templates.py) | Added 4 functions: 2 cached batch instructions + 2 dynamic batch templates |
|
||||
| [src/pipelines/saas/prompts/prompt_calls.py](../src/pipelines/saas/prompts/prompt_calls.py) | Added 2 batch wrapper functions: `prompt_aarete_derived_program_canonical_batch` and `prompt_aarete_derived_product_canonical_batch` |
|
||||
| [src/prompts/cache_registry.py](../src/prompts/cache_registry.py) | Added 2 cache entries for the batch prompt instructions |
|
||||
| [test_program_product_canonical.py](../test_program_product_canonical.py) | Updated for batch architecture; removed clustering imports and per-cluster validation checks |
|
||||
|
||||
---
|
||||
|
||||
## Function Reference
|
||||
|
||||
### `aarete_derived.py`
|
||||
|
||||
| Function | Visibility | Purpose |
|
||||
|---|---|---|
|
||||
| `_split_multi_value_separators(text)` | private | Splits a raw cell value on commas, semicolons, and similar delimiters. |
|
||||
| `_collect_unique_values_from_list_column(column)` | private | Deduplicated list of non-empty, non-sentinel values from a Series. Handles both comma-string and list formats. |
|
||||
| `_all_empty_in_list_column(column)` | private | Returns `True` if every cell is empty — used as the skip-derivation guard. |
|
||||
| `_map_row_list_to_canonicals(raw_val, canonical_map)` | private | Maps one cell to its canonical(s). Splits on `,`, looks up each part, deduplicates, returns comma-separated string. |
|
||||
| `add_aarete_derived_program_canonical(df, valid_programs)` | **public** | Runs the 4-step batch pipeline for `PROGRAM` → `AARETE_DERIVED_PROGRAM`. |
|
||||
| `add_aarete_derived_product_canonical(df, valid_products)` | **public** | Same pipeline for `PRODUCT` → `AARETE_DERIVED_PRODUCT`. |
|
||||
|
||||
### `prompt_templates.py`
|
||||
|
||||
| Function | Purpose |
|
||||
|---|---|
|
||||
| `AARETE_DERIVED_PROGRAM_CANONICAL_BATCH_INSTRUCTION()` | Static cached system instruction for batch PROGRAM canonicalization (7 rules). Registered in `cache_registry.py` under label `AARETE_DERIVED_PROGRAM_CANONICAL_BATCH`. |
|
||||
| `AARETE_DERIVED_PROGRAM_CANONICAL_BATCH(unique_values, valid_programs, payer_name, payer_state)` | Dynamic template. Returns `(context_text, prompt_text, parser)` tuple. |
|
||||
| `AARETE_DERIVED_PRODUCT_CANONICAL_BATCH_INSTRUCTION()` | Static cached system instruction for batch PRODUCT canonicalization (7 rules). Registered under label `AARETE_DERIVED_PRODUCT_CANONICAL_BATCH`. |
|
||||
| `AARETE_DERIVED_PRODUCT_CANONICAL_BATCH(unique_values, valid_products, payer_name, payer_state)` | Dynamic template. Returns `(context_text, prompt_text, parser)` tuple. |
|
||||
|
||||
### `prompt_calls.py`
|
||||
|
||||
| Function | Purpose |
|
||||
|---|---|
|
||||
| `prompt_aarete_derived_program_canonical_batch(unique_values, valid_programs, filename, payer_name, payer_state)` | Single LLM call for all PROGRAM values. Returns `dict[str, str]` mapping each raw value to its canonical. Falls back to `raw.upper()` for any key the LLM omits; falls back to `{v: v.upper()}` for all values on parse failure. |
|
||||
| `prompt_aarete_derived_product_canonical_batch(unique_values, valid_products, filename, payer_name, payer_state)` | Same for PRODUCT. |
|
||||
|
||||
---
|
||||
|
||||
## Example
|
||||
|
||||
**Raw PROGRAM values (all unique values across the DataFrame):**
|
||||
`CHIP`, `Children's Health Insurance Program`, `Children's Health Insurance Program (CHIP)`, `MMC`, `Medicaid Managed Care`, `Medicaid`
|
||||
|
||||
**Batch LLM call (Step 3):**
|
||||
|
||||
All 6 values are sent in one call with payer context. The LLM determines:
|
||||
- `CHIP`, `Children's Health Insurance Program`, `Children's Health Insurance Program (CHIP)` → same program (Rule 4: prefer abbreviation) → canonical: `CHIP`
|
||||
- `MMC`, `Medicaid Managed Care` → same program (Rule 4: prefer abbreviation) → canonical: `MMC`
|
||||
- `Medicaid` → distinct broader program; no other group uses `MEDICAID` → canonical: `MEDICAID`
|
||||
|
||||
**Resulting `{raw → canonical}` map:**
|
||||
|
||||
| Raw value | Canonical |
|
||||
|---|---|
|
||||
| CHIP | CHIP |
|
||||
| Children's Health Insurance Program | CHIP |
|
||||
| Children's Health Insurance Program (CHIP) | CHIP |
|
||||
| Medicaid Managed Care | MMC |
|
||||
| MMC | MMC |
|
||||
| Medicaid | MEDICAID |
|
||||
|
||||
**Final output applied to rows:**
|
||||
|
||||
| PROGRAM (raw) | AARETE_DERIVED_PROGRAM |
|
||||
|---|---|
|
||||
| CHIP | CHIP |
|
||||
| Children's Health Insurance Program | CHIP |
|
||||
| Children's Health Insurance Program (CHIP) | CHIP |
|
||||
| Medicaid Managed Care | MMC |
|
||||
| Medicaid | MEDICAID |
|
||||
| CHIP, Medicaid Managed Care | CHIP, MMC |
|
||||
+13
-1
@@ -39,7 +39,7 @@ import src.utils.duplicate_detection as duplicate_detection
|
||||
import src.utils.program_product_mapping as ppm_utils
|
||||
from src.pipelines.runtime_context import set_active_client
|
||||
from src.pipelines.shared.extraction import one_to_one_funcs
|
||||
from src.pipelines.shared.postprocessing import postprocessing_funcs
|
||||
from src.pipelines.shared.postprocessing import aarete_derived, postprocessing_funcs
|
||||
from src.constants.investment_columns import FIELD_FORMAT_MAPPING
|
||||
from src.constants.constants import Constants
|
||||
from src import config
|
||||
@@ -383,6 +383,18 @@ def main(client: str = "saas", testing=False, test_params={}):
|
||||
FINAL_RESULT_DF_DASHBOARD, config.STATE_FLAG
|
||||
)
|
||||
)
|
||||
FINAL_RESULT_DF_DASHBOARD = (
|
||||
aarete_derived.add_aarete_derived_program_canonical(
|
||||
FINAL_RESULT_DF_DASHBOARD,
|
||||
constants.VALID_AARETE_DERIVED_PROGRAMS,
|
||||
)
|
||||
)
|
||||
FINAL_RESULT_DF_DASHBOARD = (
|
||||
aarete_derived.add_aarete_derived_product_canonical(
|
||||
FINAL_RESULT_DF_DASHBOARD,
|
||||
constants.VALID_AARETE_DERIVED_PRODUCTS,
|
||||
)
|
||||
)
|
||||
|
||||
FINAL_RESULT_DF_CC = postprocessing_funcs.reorder_columns(
|
||||
FINAL_RESULT_DF_CC, FIELD_FORMAT_MAPPING
|
||||
|
||||
@@ -1580,6 +1580,122 @@ def prompt_aarete_derived_provider_name(
|
||||
return str(fallback).upper()
|
||||
|
||||
|
||||
def prompt_aarete_derived_program_canonical_batch(
|
||||
unique_combinations: list[tuple[str, str, str]],
|
||||
valid_programs: list[str],
|
||||
) -> dict[str, str]:
|
||||
"""Map all unique raw PROGRAM values to canonical names in a single LLM call.
|
||||
|
||||
Args:
|
||||
unique_combinations: Unique (payer_name, payer_state, raw_value) triples
|
||||
collected across the full dashboard DataFrame. Each raw value is paired
|
||||
with the payer context of the row it came from, giving the LLM correct
|
||||
per-entry context for brand-prefix stripping across multiple payers.
|
||||
valid_programs: Client-supplied valid AARETE_DERIVED_PROGRAM values (soft anchor).
|
||||
|
||||
Returns:
|
||||
Dict mapping each unique raw value to its canonical string. Falls back to
|
||||
``raw.upper()`` for any key the LLM omits.
|
||||
"""
|
||||
if not unique_combinations:
|
||||
return {}
|
||||
|
||||
context_text, prompt_text, _parser = (
|
||||
prompt_templates.AARETE_DERIVED_PROGRAM_CANONICAL_BATCH(
|
||||
unique_combinations, valid_programs
|
||||
)
|
||||
)
|
||||
logging.debug(f"AARETE_DERIVED_PROGRAM_CANONICAL_BATCH: {prompt_text}")
|
||||
llm_answer_raw = llm_utils.invoke_claude(
|
||||
f"{context_text}\n\n{prompt_text}",
|
||||
"sonnet_latest",
|
||||
"all_files",
|
||||
cache=True,
|
||||
instruction=prompt_templates.AARETE_DERIVED_PROGRAM_CANONICAL_BATCH_INSTRUCTION(),
|
||||
usage_label="AARETE_DERIVED_PROGRAM_CANONICAL_BATCH",
|
||||
)
|
||||
logging.debug(
|
||||
f"AARETE_DERIVED_PROGRAM_CANONICAL_BATCH raw response: {llm_answer_raw}"
|
||||
)
|
||||
# Unique raw values, preserving first-seen order
|
||||
unique_raw_values = list(dict.fromkeys(combo[2] for combo in unique_combinations))
|
||||
try:
|
||||
result = _parser(llm_answer_raw)
|
||||
if isinstance(result, dict):
|
||||
canonical_map: dict[str, str] = {}
|
||||
for raw in unique_raw_values:
|
||||
canon = result.get(raw)
|
||||
if canon and isinstance(canon, str) and canon.strip():
|
||||
canonical_map[raw] = canon.strip().upper()
|
||||
else:
|
||||
canonical_map[raw] = raw.upper()
|
||||
return canonical_map
|
||||
except Exception as e:
|
||||
logging.error(
|
||||
f"Error parsing AARETE_DERIVED_PROGRAM_CANONICAL_BATCH response: {e}"
|
||||
)
|
||||
# Fallback: uppercase each unique raw value
|
||||
return {combo[2]: combo[2].upper() for combo in unique_combinations}
|
||||
|
||||
|
||||
def prompt_aarete_derived_product_canonical_batch(
|
||||
unique_combinations: list[tuple[str, str, str]],
|
||||
valid_products: list[str],
|
||||
) -> dict[str, str]:
|
||||
"""Map all unique raw PRODUCT values to canonical names in a single LLM call.
|
||||
|
||||
Args:
|
||||
unique_combinations: Unique (payer_name, payer_state, raw_value) triples
|
||||
collected across the full dashboard DataFrame. Each raw value is paired
|
||||
with the payer context of the row it came from, giving the LLM correct
|
||||
per-entry context for brand-prefix stripping across multiple payers.
|
||||
valid_products: Client-supplied valid AARETE_DERIVED_PRODUCT values (soft anchor).
|
||||
|
||||
Returns:
|
||||
Dict mapping each unique raw value to its canonical string. Falls back to
|
||||
``raw.upper()`` for any key the LLM omits.
|
||||
"""
|
||||
if not unique_combinations:
|
||||
return {}
|
||||
|
||||
context_text, prompt_text, _parser = (
|
||||
prompt_templates.AARETE_DERIVED_PRODUCT_CANONICAL_BATCH(
|
||||
unique_combinations, valid_products
|
||||
)
|
||||
)
|
||||
logging.debug(f"AARETE_DERIVED_PRODUCT_CANONICAL_BATCH: {prompt_text}")
|
||||
llm_answer_raw = llm_utils.invoke_claude(
|
||||
f"{context_text}\n\n{prompt_text}",
|
||||
"sonnet_latest",
|
||||
"all_files",
|
||||
cache=True,
|
||||
instruction=prompt_templates.AARETE_DERIVED_PRODUCT_CANONICAL_BATCH_INSTRUCTION(),
|
||||
usage_label="AARETE_DERIVED_PRODUCT_CANONICAL_BATCH",
|
||||
)
|
||||
logging.debug(
|
||||
f"AARETE_DERIVED_PRODUCT_CANONICAL_BATCH raw response: {llm_answer_raw}"
|
||||
)
|
||||
# Unique raw values, preserving first-seen order
|
||||
unique_raw_values = list(dict.fromkeys(combo[2] for combo in unique_combinations))
|
||||
try:
|
||||
result = _parser(llm_answer_raw)
|
||||
if isinstance(result, dict):
|
||||
canonical_map: dict[str, str] = {}
|
||||
for raw in unique_raw_values:
|
||||
canon = result.get(raw)
|
||||
if canon and isinstance(canon, str) and canon.strip():
|
||||
canonical_map[raw] = canon.strip().upper()
|
||||
else:
|
||||
canonical_map[raw] = raw.upper()
|
||||
return canonical_map
|
||||
except Exception as e:
|
||||
logging.error(
|
||||
f"Error parsing AARETE_DERIVED_PRODUCT_CANONICAL_BATCH response: {e}"
|
||||
)
|
||||
# Fallback: uppercase each unique raw value
|
||||
return {combo[2]: combo[2].upper() for combo in unique_combinations}
|
||||
|
||||
|
||||
def prompt_split_service_term(service_term: str, filename: str) -> list[str]:
|
||||
"""
|
||||
Split a service term into its components (e.g., service description and modifiers).
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import logging
|
||||
import re
|
||||
import concurrent.futures
|
||||
from typing import TYPE_CHECKING, Optional
|
||||
|
||||
@@ -66,6 +67,20 @@ def dynamic_primary(
|
||||
filename,
|
||||
)
|
||||
|
||||
# Normalise multi-value separators immediately after extraction so all
|
||||
# downstream steps (reimbursement-level assignment and classification into
|
||||
# LOB/PROGRAM/PRODUCT/NETWORK) receive clean, individual entity strings.
|
||||
# Healthcare contracts use '/' and ' & ' between entity names to express
|
||||
# multiple values, e.g. "HA/HA+" means two separate entities: "HA" and "HA+".
|
||||
if isinstance(dynamic_primary_entities, list):
|
||||
_split: list[str] = []
|
||||
for _entity in dynamic_primary_entities:
|
||||
_parts = re.split(r"/|\s+&\s+", str(_entity))
|
||||
_split.extend(_p.strip() for _p in _parts if _p.strip())
|
||||
dynamic_primary_entities = list(
|
||||
dict.fromkeys(_split)
|
||||
) # deduplicate, preserve order
|
||||
|
||||
exhibit_level_answer_dict["DYNAMIC_PRIMARY_ENTITIES"] = dynamic_primary_entities
|
||||
|
||||
dynamic_primary_entities_field = Field.from_values(
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import concurrent.futures
|
||||
import logging
|
||||
import re
|
||||
import pandas as pd
|
||||
from src.pipelines.shared.extraction import dynamic_funcs
|
||||
from src.pipelines.shared.prompts import prompt_calls
|
||||
@@ -989,6 +990,33 @@ def map_lob_network_to_aarete_derived(
|
||||
return all_exhibit_rows
|
||||
|
||||
|
||||
def _normalize_program_product_separators(
|
||||
all_exhibit_rows: list[dict],
|
||||
) -> list[dict]:
|
||||
"""Split PROGRAM and PRODUCT values on '/' and ' & ' separators.
|
||||
|
||||
Healthcare contracts routinely use these delimiters to express multiple
|
||||
distinct entities in a single cell, e.g. ``"HA/HA+"`` means two separate
|
||||
products: ``"HA"`` and ``"HA+"``.
|
||||
|
||||
Normalising here (before ``dashboard_postprocess`` joins list elements with
|
||||
``", "``) ensures the Dashboard CSV always uses commas as the sole separator,
|
||||
so downstream code only needs to split on commas.
|
||||
"""
|
||||
for row in all_exhibit_rows:
|
||||
for field in ("PROGRAM", "PRODUCT"):
|
||||
raw = row.get(field)
|
||||
if string_utils.is_empty(raw):
|
||||
continue
|
||||
items = _normalize_dynamic_primary_entities(raw)
|
||||
split_items: list[str] = []
|
||||
for item in items:
|
||||
parts = re.split(r"/|\s+&\s+", item)
|
||||
split_items.extend(p.strip() for p in parts if p.strip())
|
||||
row[field] = list(dict.fromkeys(split_items)) # deduplicate, preserve order
|
||||
return all_exhibit_rows
|
||||
|
||||
|
||||
def split_dynamic_primary_entities(
|
||||
all_exhibit_rows: list[dict], filename: str
|
||||
) -> list[dict]:
|
||||
@@ -1219,15 +1247,8 @@ def add_aarete_derived_program_product(
|
||||
if v.lower() in valid_values_by_lower
|
||||
]
|
||||
else:
|
||||
# Fallback: trim + title-case, preserving all-uppercase words as
|
||||
# acronyms (e.g. CHIP, DSNP, STAR+PLUS stay unchanged).
|
||||
def _title_preserve_acronyms(text: str) -> str:
|
||||
return " ".join(
|
||||
word if word.isupper() else word.title()
|
||||
for word in text.split()
|
||||
)
|
||||
|
||||
derived_values = [_title_preserve_acronyms(v) for v in base_values]
|
||||
# if no valid values present, derived values will be an empty list.
|
||||
derived_values = []
|
||||
|
||||
# Merge with any existing values already present on this row
|
||||
existing_raw = answer_dict.get(derived_field_name)
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import logging
|
||||
import re
|
||||
import pandas as pd
|
||||
import src.config as config
|
||||
import src.utils.string_utils as string_utils
|
||||
@@ -171,9 +172,14 @@ def fill_na_mapping(df: pd.DataFrame, constants: Constants) -> pd.DataFrame:
|
||||
program_lob_values = set()
|
||||
adp_key = tuple(_to_str_list(answer_dict.get("AARETE_DERIVED_PROGRAM")))
|
||||
p_key = tuple(_to_str_list(answer_dict.get("PROGRAM")))
|
||||
program_lob_list = program_crosswalk_cache.get(
|
||||
adp_key, []
|
||||
) + program_llm_cache.get(p_key, [])
|
||||
if program_crosswalk_cache:
|
||||
logging.info(
|
||||
f"Using crosswalk cache for AARETE_DERIVED_PROGRAM key: {adp_key}"
|
||||
)
|
||||
program_lob_list = program_crosswalk_cache.get(adp_key, [])
|
||||
else:
|
||||
logging.info(f"Using LLM cache for PROGRAM key: {p_key}")
|
||||
program_lob_list = program_llm_cache.get(p_key, [])
|
||||
filtered = [
|
||||
_canonical_lob(v, valid_lobs_lower)
|
||||
for v in program_lob_list
|
||||
@@ -187,9 +193,14 @@ def fill_na_mapping(df: pd.DataFrame, constants: Constants) -> pd.DataFrame:
|
||||
product_lob_values = set()
|
||||
adprod_key = tuple(_to_str_list(answer_dict.get("AARETE_DERIVED_PRODUCT")))
|
||||
prod_key = tuple(_to_str_list(answer_dict.get("PRODUCT")))
|
||||
product_lob_list = product_crosswalk_cache.get(
|
||||
adprod_key, []
|
||||
) + product_llm_cache.get(prod_key, [])
|
||||
if product_crosswalk_cache:
|
||||
logging.info(
|
||||
f"Using crosswalk cache for AARETE_DERIVED_PRODUCT key: {adprod_key}"
|
||||
)
|
||||
product_lob_list = product_crosswalk_cache.get(adprod_key, [])
|
||||
else:
|
||||
logging.info(f"Using LLM cache for PRODUCT key: {prod_key}")
|
||||
product_lob_list = product_llm_cache.get(prod_key, [])
|
||||
filtered = [
|
||||
_canonical_lob(v, valid_lobs_lower)
|
||||
for v in product_lob_list
|
||||
@@ -298,3 +309,289 @@ def get_crosswalk_fields(answer_dicts: list, constants: Constants):
|
||||
# not modify the source field (from_field_name)
|
||||
# If from_field_name needs to be a list, that should be handled elsewhere
|
||||
return answer_dicts
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers shared by PROGRAM and PRODUCT canonical derivation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _split_multi_value_separators(text: str) -> list[str]:
|
||||
"""Split a PROGRAM/PRODUCT cell on all recognised list separators.
|
||||
|
||||
In practice these columns use three separator conventions:
|
||||
|
||||
* comma ``","`` — primary Dashboard separator
|
||||
* slash ``"/"`` — e.g. ``"Healthy Advantage/Healthy Advantage Plus"``
|
||||
* ampersand ``" & "`` — e.g. ``"HA & HA+"``
|
||||
(requires surrounding whitespace to avoid splitting abbreviated compound
|
||||
names such as ``"B&G"`` or ``"A&M"``)
|
||||
|
||||
Returns a list of stripped, non-empty parts.
|
||||
"""
|
||||
parts = re.split(r",|/|\s+&\s+", text)
|
||||
return [p.strip() for p in parts if p.strip()]
|
||||
|
||||
|
||||
def _collect_unique_values_from_list_column(
|
||||
column: pd.Series,
|
||||
) -> list[str]:
|
||||
"""Collect all unique, non-empty, non-sentinel values from a PROGRAM/PRODUCT column.
|
||||
|
||||
In the Dashboard DataFrame (the only place this is called), ``dashboard_postprocess``
|
||||
has already run and converted every list column to a comma-separated string, e.g.
|
||||
``"Medicaid Managed Care, Commercial PPO"``.
|
||||
This helper handles both that string form AND the original list form so it is safe
|
||||
regardless of call order.
|
||||
"""
|
||||
_SENTINELS = {"N/A", "UNKNOWN", "NONE", "NULL"}
|
||||
seen: set[str] = set()
|
||||
unique: list[str] = []
|
||||
for val in column:
|
||||
if string_utils.is_empty(val):
|
||||
continue
|
||||
if isinstance(val, list):
|
||||
# Pre-dashboard-postprocess form: Python list — each element may itself
|
||||
# contain secondary separators (/ or &), so split each item further.
|
||||
raw_items = string_utils.flatten_to_strings(val)
|
||||
items = []
|
||||
for raw in raw_items:
|
||||
items.extend(_split_multi_value_separators(raw))
|
||||
else:
|
||||
# Post-dashboard-postprocess form: comma-separated string (may also
|
||||
# contain / or & separators inside individual cells).
|
||||
items = _split_multi_value_separators(str(val))
|
||||
for item in items:
|
||||
text = item.strip()
|
||||
if not string_utils.is_empty(text) and text.upper() not in _SENTINELS:
|
||||
if text not in seen:
|
||||
seen.add(text)
|
||||
unique.append(text)
|
||||
return unique
|
||||
|
||||
|
||||
def _collect_unique_payer_value_combos(
|
||||
df: pd.DataFrame, value_col: str
|
||||
) -> list[tuple[str, str, str]]:
|
||||
"""Collect unique (payer_name, payer_state, raw_value) triples from the DataFrame.
|
||||
|
||||
Iterates every row, splits the value_col cell on all recognised separators,
|
||||
and pairs each resulting item with the row's PAYER_NAME / PAYER_STATE.
|
||||
Combinations are deduplicated while preserving insertion order.
|
||||
|
||||
Used by add_aarete_derived_program_canonical and add_aarete_derived_product_canonical
|
||||
to build the per-entry payer-context input for the batch LLM call.
|
||||
"""
|
||||
_SENTINELS = {"N/A", "UNKNOWN", "NONE", "NULL"}
|
||||
seen: set[tuple[str, str, str]] = set()
|
||||
result: list[tuple[str, str, str]] = []
|
||||
has_payer_name = "PAYER_NAME" in df.columns
|
||||
has_payer_state = "PAYER_STATE" in df.columns
|
||||
for _, row in df.iterrows():
|
||||
payer_name = (
|
||||
str(row["PAYER_NAME"]).strip()
|
||||
if has_payer_name and not string_utils.is_empty(row["PAYER_NAME"])
|
||||
else ""
|
||||
)
|
||||
payer_state = (
|
||||
str(row["PAYER_STATE"]).strip()
|
||||
if has_payer_state and not string_utils.is_empty(row["PAYER_STATE"])
|
||||
else ""
|
||||
)
|
||||
val = row[value_col]
|
||||
if string_utils.is_empty(val):
|
||||
continue
|
||||
if isinstance(val, list):
|
||||
raw_items = string_utils.flatten_to_strings(val)
|
||||
items: list[str] = []
|
||||
for raw in raw_items:
|
||||
items.extend(_split_multi_value_separators(raw))
|
||||
else:
|
||||
items = _split_multi_value_separators(str(val))
|
||||
for item in items:
|
||||
text = item.strip()
|
||||
if not string_utils.is_empty(text) and text.upper() not in _SENTINELS:
|
||||
combo = (payer_name, payer_state, text)
|
||||
if combo not in seen:
|
||||
seen.add(combo)
|
||||
result.append(combo)
|
||||
return result
|
||||
|
||||
|
||||
def _all_empty_in_list_column(column: pd.Series) -> bool:
|
||||
"""Return True when every cell in a list[str] column is empty / null."""
|
||||
return all(string_utils.is_empty(val) for val in column)
|
||||
|
||||
|
||||
def _map_row_list_to_canonicals(raw_val, canonical_map: dict[str, str]) -> str:
|
||||
"""Map one PROGRAM/PRODUCT cell to a comma-separated string of canonical names.
|
||||
|
||||
Input can be a Python list (pre-dashboard-postprocess) or a comma-separated
|
||||
string (post-dashboard-postprocess, which is the normal case for the Dashboard DF).
|
||||
Output is always a comma-separated string to match the Dashboard DF column format.
|
||||
"""
|
||||
_SENTINELS = {"N/A", "UNKNOWN", "NONE", "NULL"}
|
||||
if string_utils.is_empty(raw_val):
|
||||
return ""
|
||||
if isinstance(raw_val, list):
|
||||
raw_items = string_utils.flatten_to_strings(raw_val)
|
||||
items = []
|
||||
for raw in raw_items:
|
||||
items.extend(_split_multi_value_separators(raw))
|
||||
else:
|
||||
# Post-dashboard-postprocess: comma-separated string (may also contain
|
||||
# / or & separators within individual cells, e.g. "HA/HA+").
|
||||
items = _split_multi_value_separators(str(raw_val))
|
||||
result: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for v in items:
|
||||
text = v.strip()
|
||||
if string_utils.is_empty(text) or text.upper() in _SENTINELS:
|
||||
continue
|
||||
canonical = canonical_map.get(text, text.upper())
|
||||
if canonical and canonical not in seen:
|
||||
seen.add(canonical)
|
||||
result.append(canonical)
|
||||
elif canonical in seen:
|
||||
logging.debug(
|
||||
"_map_row_list_to_canonicals: dropped duplicate canonical '%s' "
|
||||
"for raw='%s' (already mapped from an earlier value in this cell)",
|
||||
canonical,
|
||||
text,
|
||||
)
|
||||
return ", ".join(result)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API — called from runner.py (Dashboard only)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def add_aarete_derived_program_canonical(
|
||||
df: pd.DataFrame, valid_programs: list[str]
|
||||
) -> pd.DataFrame:
|
||||
"""Derive canonical program names from the raw PROGRAM column.
|
||||
|
||||
Architecture (single global LLM call):
|
||||
1. Collect all unique raw PROGRAM values across the entire Dashboard DataFrame.
|
||||
2. Make ONE LLM call with all unique values + payer context + valid_programs anchor.
|
||||
The LLM groups semantically equivalent variants and assigns a single canonical
|
||||
per group, resolving brand-prefix noise vs. meaningful qualifiers correctly.
|
||||
3. Build a {original_name -> canonical_name} lookup and apply it to every row.
|
||||
|
||||
Trigger conditions (either is sufficient):
|
||||
- ``valid_programs`` is empty (no client-supplied valid-values list), OR
|
||||
- the ``AARETE_DERIVED_PROGRAM`` column is entirely empty/null for all rows.
|
||||
|
||||
Args:
|
||||
df: Dashboard DataFrame containing at minimum a ``PROGRAM`` column.
|
||||
valid_programs: ``Constants.VALID_AARETE_DERIVED_PROGRAMS`` list.
|
||||
|
||||
Returns:
|
||||
DataFrame with ``AARETE_DERIVED_PROGRAM`` populated/overwritten with
|
||||
canonical names derived from the raw PROGRAM values.
|
||||
"""
|
||||
df_copy = df.copy()
|
||||
|
||||
if "AARETE_DERIVED_PROGRAM" not in df_copy.columns:
|
||||
df_copy["AARETE_DERIVED_PROGRAM"] = ""
|
||||
|
||||
has_valid_programs = bool(valid_programs)
|
||||
adp_col_empty = _all_empty_in_list_column(df_copy["AARETE_DERIVED_PROGRAM"])
|
||||
|
||||
if has_valid_programs and not adp_col_empty:
|
||||
logging.info(
|
||||
"Skipping canonical PROGRAM derivation: valid mapping exists and "
|
||||
"AARETE_DERIVED_PROGRAM is already populated."
|
||||
)
|
||||
return df_copy
|
||||
|
||||
if "PROGRAM" not in df_copy.columns:
|
||||
logging.warning(
|
||||
"PROGRAM column not found in DataFrame; skipping canonical program derivation."
|
||||
)
|
||||
return df_copy
|
||||
|
||||
# Step 1 — collect unique (payer_name, payer_state, raw_value) combinations
|
||||
unique_combos = _collect_unique_payer_value_combos(df_copy, "PROGRAM")
|
||||
if not unique_combos:
|
||||
logging.info(
|
||||
"No non-empty PROGRAM values found; skipping canonical program derivation."
|
||||
)
|
||||
return df_copy
|
||||
|
||||
# Step 2 — single global LLM call returns {raw: canonical} dict
|
||||
canonical_map = prompt_calls.prompt_aarete_derived_program_canonical_batch(
|
||||
unique_combos, valid_programs
|
||||
)
|
||||
|
||||
# Step 4 — apply mapping to each row
|
||||
df_copy["AARETE_DERIVED_PROGRAM"] = df_copy["PROGRAM"].apply(
|
||||
lambda val: _map_row_list_to_canonicals(val, canonical_map)
|
||||
)
|
||||
return df_copy
|
||||
|
||||
|
||||
def add_aarete_derived_product_canonical(
|
||||
df: pd.DataFrame, valid_products: list[str]
|
||||
) -> pd.DataFrame:
|
||||
"""Derive canonical product names from the raw PRODUCT column.
|
||||
|
||||
Architecture (single global LLM call):
|
||||
1. Collect all unique raw PRODUCT values across the entire Dashboard DataFrame.
|
||||
2. Make ONE LLM call with all unique values + payer context + valid_products anchor.
|
||||
The LLM groups semantically equivalent variants and assigns a single canonical
|
||||
per group, resolving brand-prefix noise vs. meaningful qualifiers correctly.
|
||||
3. Build a {original_name -> canonical_name} lookup and apply it to every row.
|
||||
|
||||
Trigger conditions (either is sufficient):
|
||||
- ``valid_products`` is empty (no client-supplied valid-values list), OR
|
||||
- the ``AARETE_DERIVED_PRODUCT`` column is entirely empty/null for all rows.
|
||||
|
||||
Args:
|
||||
df: Dashboard DataFrame containing at minimum a ``PRODUCT`` column.
|
||||
valid_products: ``Constants.VALID_AARETE_DERIVED_PRODUCTS`` list.
|
||||
|
||||
Returns:
|
||||
DataFrame with ``AARETE_DERIVED_PRODUCT`` populated/overwritten with
|
||||
canonical names derived from the raw PRODUCT values.
|
||||
"""
|
||||
df_copy = df.copy()
|
||||
|
||||
if "AARETE_DERIVED_PRODUCT" not in df_copy.columns:
|
||||
df_copy["AARETE_DERIVED_PRODUCT"] = ""
|
||||
|
||||
has_valid_products = bool(valid_products)
|
||||
adp_col_empty = _all_empty_in_list_column(df_copy["AARETE_DERIVED_PRODUCT"])
|
||||
|
||||
if has_valid_products and not adp_col_empty:
|
||||
logging.info(
|
||||
"Skipping canonical PRODUCT derivation: valid mapping exists and "
|
||||
"AARETE_DERIVED_PRODUCT is already populated."
|
||||
)
|
||||
return df_copy
|
||||
|
||||
if "PRODUCT" not in df_copy.columns:
|
||||
logging.warning(
|
||||
"PRODUCT column not found in DataFrame; skipping canonical product derivation."
|
||||
)
|
||||
return df_copy
|
||||
|
||||
# Step 1 — collect unique (payer_name, payer_state, raw_value) combinations
|
||||
unique_combos = _collect_unique_payer_value_combos(df_copy, "PRODUCT")
|
||||
if not unique_combos:
|
||||
logging.info(
|
||||
"No non-empty PRODUCT values found; skipping canonical product derivation."
|
||||
)
|
||||
return df_copy
|
||||
|
||||
# Step 2 — single global LLM call returns {raw: canonical} dict
|
||||
canonical_map = prompt_calls.prompt_aarete_derived_product_canonical_batch(
|
||||
unique_combos, valid_products
|
||||
)
|
||||
|
||||
# Step 4 — apply mapping to each row
|
||||
df_copy["AARETE_DERIVED_PRODUCT"] = df_copy["PRODUCT"].apply(
|
||||
lambda val: _map_row_list_to_canonicals(val, canonical_map)
|
||||
)
|
||||
return df_copy
|
||||
|
||||
@@ -322,6 +322,20 @@ _REGISTRY: tuple[CachePromptEntry, ...] = (
|
||||
cache_class=CacheClass.INSTRUCTION_PLUS_CONTEXT,
|
||||
notes="Maps extracted PRODUCT names to AARETE_DERIVED_PRODUCT values; valid-values list passed as context_for_caching.",
|
||||
),
|
||||
CachePromptEntry(
|
||||
usage_label="AARETE_DERIVED_PROGRAM_CANONICAL_BATCH",
|
||||
instruction_func=prompt_templates.AARETE_DERIVED_PROGRAM_CANONICAL_BATCH_INSTRUCTION,
|
||||
runtime_model="sonnet_latest",
|
||||
cache_class=CacheClass.INSTRUCTION,
|
||||
notes="Global batch canonical derivation for PROGRAM names; all unique values + payer context passed dynamically.",
|
||||
),
|
||||
CachePromptEntry(
|
||||
usage_label="AARETE_DERIVED_PRODUCT_CANONICAL_BATCH",
|
||||
instruction_func=prompt_templates.AARETE_DERIVED_PRODUCT_CANONICAL_BATCH_INSTRUCTION,
|
||||
runtime_model="sonnet_latest",
|
||||
cache_class=CacheClass.INSTRUCTION,
|
||||
notes="Global batch canonical derivation for PRODUCT names; all unique values + payer context passed dynamically.",
|
||||
),
|
||||
CachePromptEntry(
|
||||
usage_label="PROGRAM_TO_LOB",
|
||||
instruction_func=prompt_templates.PROGRAM_TO_LOB_INSTRUCTION,
|
||||
|
||||
@@ -177,7 +177,7 @@
|
||||
"field_name": "CONTRACT_AMENDMENT_NUM",
|
||||
"relationship": "one_to_one",
|
||||
"field_type": "smart_chunked",
|
||||
"prompt": "Extract the amendment number or identifier if this document is an amendment to a contract. The amendment identifier can be numeric, letters only, or alphanumeric. Follow these guidelines:\n1. Look for phrases like 'Amendment No. X', 'X Amendment', 'Amendment X to the Agreement', 'Xth Amendment' in the document title, preamble, or header sections.\n2. Return the amendment number/identifier exactly as it appears, only if it consists of numbers, letters, or alphanumeric characters. (e.g., for 'Amendment No. 3' return '3', for 'Amendment A' return 'A', for 'Amendment 3A' return '3A').\n3. If the amendment number is written as a word (e.g., 'First Amendment'), convert it to a number (e.g., '1').\n4. If the amendment identifier is alphanumeric, return the whole value (e.g., for '2A' return '2A').\n5. If the amendment number uses Roman numerals (e.g., 'Amendment IV'), convert it to a number (e.g., '4').\n6. If the amendment number includes decimal points (e.g., 'Amendment 2.1'), return the full numeric value (e.g., '2.1').\n7. If the amendment identifier is letters only (e.g., 'Amendment AB'), return the letters exactly as they appear (e.g., 'AB',for 'Amendment XYZ' return 'XYZ').\n8. If multiple amendment numbers appear, return the one most clearly associated with the current document.\n9. If the document is an amendment but the number/identifier cannot be determined, return 'UNKNOWN'.\n10. If the document is not an amendment, return 'N/A'.\n11. If the amendment number contains any leading zeroes (e.g., '001'), drop all leading zeros (e.g., '1').",
|
||||
"prompt": "Extract the amendment number or identifier if this document is an amendment to a contract. The amendment identifier can be numeric, letters only, or alphanumeric. Follow these guidelines:\n1. Look for phrases like 'Amendment No. X', 'X Amendment', 'Amendment X to the Agreement', 'Xth Amendment' in the document title, preamble, or header sections.\n2. Return the amendment number/identifier exactly as it appears, only if it consists of numbers, letters, or alphanumeric characters. (e.g., for 'Amendment No. 3' return '3', for 'Amendment A' return 'A', for 'Amendment 3A' return '3A').\n3. If the amendment number is written as a word (e.g., 'First Amendment'), convert it to a number (e.g., '1').\n4. If the amendment identifier is alphanumeric, return the whole value (e.g., for '2A' return '2A').\n5. If the amendment number uses Roman numerals (e.g., 'Amendment IV'), convert it to a number (e.g., '4').\n6. If the amendment number includes decimal points (e.g., 'Amendment 2.1'), return the full numeric value (e.g., '2.1').\n7. If the amendment identifier is letters only (e.g., 'Amendment AB'), return the letters exactly as they appear (e.g., 'AB',for 'Amendment XYZ' return 'XYZ').\n8. If multiple amendment numbers appear, return the one most clearly associated with the current document.\n9. NEVER return a date or date-like string as the amendment number. Date patterns such as 'MM-DD-YYYY', 'YYYY-MM-DD', 'M-D-YYYY', or any string that resembles a calendar date (e.g., '1-1-2014', '01-01-2014', '2014-01-01') are effective dates, NOT amendment identifiers. If the only candidate identifier is a date, discard it and treat the amendment number as undetermined.\n10. If the document is an amendment but the number/identifier cannot be determined, return 'UNKNOWN'.\n11. If the document is not an amendment, return 'N/A'.\n12. If the amendment number contains any leading zeroes (e.g., '001'), drop all leading zeros (e.g., '1').",
|
||||
"retrieval_question": "What is the amendment number, letter, or alphanumeric identifier of this document? Amendment No., Amendment Number, Amendment Letter, Amendment Code, ordinal amendment, numbered amendment, lettered amendment, coded amendment? Amendment to agreement, amendment to contract."
|
||||
},
|
||||
{
|
||||
@@ -328,7 +328,7 @@
|
||||
"field_name": "AUTO_RENEWAL_IND",
|
||||
"relationship": "one_to_one",
|
||||
"field_type": "smart_chunked",
|
||||
"prompt": "This pertains to the term and termination section of the contract. Determine whether the contract includes an automatic renewal. Answer Y only if the contract renews automatically without any action required by either party (for example, it renews unless a party gives notice to terminate). Answer N if renewal requires any action, such as written agreement, consent, negotiation, or signing a renewal. If you cannot determine, answer N. Output only Y or N.",
|
||||
"prompt": "This pertains to the term and termination section of the contract. Determine whether the contract includes an automatic renewal. Answer Y only if the contract itself explicitly states that it renews automatically without any action required by either party (for example, it renews unless a party gives notice to terminate). Answer N in all of the following cases: (1) renewal requires any action such as written agreement, consent, negotiation, or signing a renewal; (2) the document is an amendment, addendum, or exhibit that merely states it renews 'with', 'under', or 'pursuant to' the terms of a parent or base Agreement — because the auto-renewal determination depends on the unseen parent Agreement, not this document; (3) the renewal language only defers to or incorporates the renewal provisions of another document without specifying automatic renewal in this document. If you cannot determine, answer N. Output only Y or N.",
|
||||
"keywords": [
|
||||
"Renew",
|
||||
"renew",
|
||||
|
||||
@@ -4079,6 +4079,297 @@ STATE_FLAG:
|
||||
return (prompt, _json_dict_parser)
|
||||
|
||||
|
||||
def AARETE_DERIVED_PROGRAM_CANONICAL_BATCH_INSTRUCTION() -> str:
|
||||
return f"""[OBJECTIVE]
|
||||
Map every raw PROGRAM name from a healthcare contract to a single canonical string.
|
||||
Goal: every appearance of the same raw value lands on the same canonical within
|
||||
this batch. Cross-batch standardization is NOT the goal here — batch consistency is.
|
||||
When in doubt, keep variants as separate canonicals rather than collapsing them.
|
||||
|
||||
[GROUPING RULES — apply before choosing a canonical form]
|
||||
|
||||
1. PAYER-NAME PREFIX NOISE — When two raw values differ only by the presence of the
|
||||
payer name shown in their [payer: ...] annotation, treat them as the same program.
|
||||
The payer's brand name is scoping noise and should be stripped.
|
||||
- Entry "Molina MMC" with [payer: Molina] and entry "MMC" → same program.
|
||||
- Entry "BCBS Premium" with [payer: BCBS] and entry "Premium" → same program.
|
||||
- Entry "Aetna Better Health" with [payer: Aetna] and entry "Better Health" → same program.
|
||||
Do NOT apply this rule when removing the payer name leaves an ambiguous or
|
||||
non-standard term that on its own would not be recognizable as a program.
|
||||
|
||||
2. STATE/REGION PREFIXES ARE NOT NOISE BY DEFAULT — A leading U.S. state name is
|
||||
treated as part of the program identity. State-prefixed and non-prefixed
|
||||
variants are kept as separate canonicals unless context strongly suggests they
|
||||
are the same program. Batch consistency makes this safe — splintered canonicals
|
||||
still join correctly downstream.
|
||||
|
||||
3. SEMANTICALLY DISTINGUISHING QUALIFIERS — A qualifier that changes the clinical
|
||||
meaning or eligibility population creates a DISTINCT program; keep them as
|
||||
separate canonicals. Common distinguishing qualifiers:
|
||||
Perinatal, Coordinated, Kids, Duals, SNP, Special Needs, Behavioral, Dental, Plus.
|
||||
- "CHIP" and "CHIP Perinatal" → two separate canonicals.
|
||||
- "MMP" and "MMCP" (Medicare-Medicaid Coordinated Plan) → two separate canonicals.
|
||||
- "DSNP" and "DSNP Dental" → two separate canonicals.
|
||||
|
||||
4. ABBREVIATION / FULL-NAME VARIANTS — When an abbreviated form and its full-name
|
||||
equivalent both appear, they are the same program; prefer the abbreviation.
|
||||
- "CHIP" and "Children's Health Insurance Program" → same program; prefer "CHIP".
|
||||
- "MMC" and "Medicaid Managed Care" → same program; prefer "MMC".
|
||||
- "DSNP" and "Dual Special Needs Plan" → same program; prefer "DSNP".
|
||||
|
||||
5. PLUS-VARIANT ABBREVIATIONS — A short form ending with "+" and its spelled-out
|
||||
"Plus" counterpart are the same program (notational variants, not distinct).
|
||||
- "HA+" and "Healthy Advantage Plus" → same program; prefer "HA+".
|
||||
|
||||
6. DO NOT STRIP GENERIC SUFFIXES — Generic trailing words like "Program" or "Plan"
|
||||
are part of the canonical name and must be preserved, UNLESS another group in
|
||||
this batch already maps to the suffix-stripped form as its canonical. In that
|
||||
case, keep them separate to avoid a collision.
|
||||
- "PCP P4Q Program" → "PCP P4Q PROGRAM", not "PCP P4Q" (no other group uses "PCP P4Q").
|
||||
- "Medicaid Managed Care Program" → "MEDICAID MANAGED CARE PROGRAM" (no other group uses "Medicaid Managed Care").
|
||||
|
||||
7. TIEBREAKER — PREFER SPLINTER OVER COLLAPSE — When you cannot decide between
|
||||
collapsing and splintering two raw values, keep them as separate canonicals.
|
||||
Over-merging produces silently wrong canonicals; over-splintering produces
|
||||
extra canonicals that still join correctly within this batch.
|
||||
- If [VALID PROGRAM VALUES] contains exactly one match for both raw values,
|
||||
collapse to that valid value.
|
||||
- If the per-entry payer context makes one interpretation clearly more likely AND
|
||||
that interpretation is supported by Rules 1–5, follow it.
|
||||
- Otherwise, keep them separate.
|
||||
|
||||
[CANONICAL FORM SELECTION — for each group of equivalent raw values]
|
||||
|
||||
Priority order:
|
||||
a. If a value from [VALID PROGRAM VALUES] is provided and a raw value clearly
|
||||
matches it, use that exact string.
|
||||
b. If a recognized abbreviation is present among the group's raw values, use it
|
||||
(uppercased).
|
||||
c. Otherwise, uppercase the most standard or concise raw value in the group.
|
||||
|
||||
[OUTPUT CONTRACT]
|
||||
|
||||
- Every unique raw input value MUST appear as a key in the output JSON dictionary.
|
||||
- Each key maps to a single canonical string (never a list, null, or empty string).
|
||||
- Two raw values that represent the same program MUST map to the SAME canonical.
|
||||
- Two raw values that represent DISTINCT programs MUST map to DIFFERENT canonicals.
|
||||
- Do NOT invent canonical strings not derivable from the inputs or
|
||||
[VALID PROGRAM VALUES].
|
||||
|
||||
[OUTPUT FORMAT]
|
||||
Return a JSON dictionary where every key is a raw PROGRAM value from the input and
|
||||
every value is its canonical form. Include all input keys.
|
||||
{JSON_DICT_FORMAT_INSTRUCTIONS}"""
|
||||
|
||||
|
||||
def AARETE_DERIVED_PROGRAM_CANONICAL_BATCH(
|
||||
unique_combinations: list[tuple[str, str, str]],
|
||||
valid_programs: list[str],
|
||||
) -> Tuple[str, str, Callable[[str], dict]]:
|
||||
"""Build prompt payload for global-batch canonical PROGRAM name derivation.
|
||||
|
||||
Args:
|
||||
unique_combinations: Unique (payer_name, payer_state, raw_value) triples
|
||||
collected across the full dashboard DataFrame. Each raw value is paired
|
||||
with the payer context of the row it came from, giving the LLM correct
|
||||
per-entry context for brand-prefix stripping across multiple payers.
|
||||
valid_programs: Client-supplied valid AARETE_DERIVED_PROGRAM values (soft anchor).
|
||||
|
||||
Returns:
|
||||
Tuple of (context_text, prompt_text, parser_function).
|
||||
"""
|
||||
|
||||
def _combo_line(payer_name: str, payer_state: str, raw_value: str) -> str:
|
||||
parts = []
|
||||
if payer_name:
|
||||
parts.append(f"payer: {payer_name}")
|
||||
if payer_state:
|
||||
parts.append(f"state: {payer_state}")
|
||||
ctx = f" [{', '.join(parts)}]" if parts else ""
|
||||
return f' "{raw_value}"{ctx}'
|
||||
|
||||
values_text = "\n".join(_combo_line(*combo) for combo in unique_combinations)
|
||||
valid_text = (
|
||||
", ".join(f'"{v}"' for v in valid_programs)
|
||||
if valid_programs
|
||||
else "none provided"
|
||||
)
|
||||
|
||||
context_text = f"""[RAW PROGRAM VALUES TO MAP]
|
||||
Each entry shows the raw value and its source payer context (used for payer-prefix
|
||||
identification only). If the same raw value appears for multiple payers, provide one
|
||||
canonical for that raw value in the output JSON.
|
||||
{values_text}
|
||||
|
||||
[VALID PROGRAM VALUES (soft anchor — prefer these exact strings when a raw value clearly matches)]
|
||||
{valid_text}"""
|
||||
|
||||
prompt_text = """[TASK]
|
||||
Apply the grouping rules and canonical form selection rules from the instruction to build
|
||||
a JSON dictionary that maps EVERY raw program value above to its canonical string.
|
||||
Briefly explain your grouping decisions, then output the JSON dictionary."""
|
||||
|
||||
return (context_text, prompt_text, _json_dict_parser)
|
||||
|
||||
|
||||
def AARETE_DERIVED_PRODUCT_CANONICAL_BATCH_INSTRUCTION() -> str:
|
||||
return f"""[OBJECTIVE]
|
||||
Map every raw PRODUCT name from a healthcare contract to a single canonical string.
|
||||
Goal: every appearance of the same raw value lands on the same canonical within
|
||||
this batch. Cross-batch standardization is NOT the goal — batch consistency is.
|
||||
When in doubt, keep variants as separate canonicals rather than collapsing them.
|
||||
|
||||
[GROUPING RULES — apply before choosing a canonical form]
|
||||
|
||||
1. PAYER-NAME PREFIX NOISE — When two raw values differ only by the presence of the
|
||||
payer name shown in their [payer: ...] annotation AND the remaining base name is
|
||||
identical, treat them as the same product.
|
||||
- Entry "Molina Medicare Options Plus" with [payer: Molina] and entry "Medicare Options Plus"
|
||||
→ same product (base "Medicare Options Plus" is identical).
|
||||
- Entry "Aetna Better Health Premier" with [payer: Aetna] and entry "Better Health Premier"
|
||||
→ same product.
|
||||
- Entry "UHC Dual Complete" with [payer: UnitedHealthcare] and entry "Dual Complete"
|
||||
→ same product.
|
||||
Do NOT apply this rule when removing the prefix yields a DIFFERENT base name:
|
||||
- "Molina Options" and "Molina Options Plus" → NOT same product
|
||||
(after stripping payer: "Options" ≠ "Options Plus").
|
||||
|
||||
2. STATE/REGION PREFIXES ARE NOT NOISE BY DEFAULT — A leading U.S. state name is
|
||||
treated as part of the product identity. State-prefixed and non-prefixed variants
|
||||
are kept as separate canonicals unless context strongly suggests they are the
|
||||
same product. Batch consistency makes this safe.
|
||||
|
||||
3. SEMANTICALLY DISTINGUISHING QUALIFIERS — A qualifier that changes the clinical
|
||||
meaning or target population creates a DISTINCT product; keep them separate.
|
||||
Common distinguishing qualifiers: Coordinated, Kids, SNP, Special Needs, Dental,
|
||||
Vision, Behavioral, Perinatal.
|
||||
|
||||
The word "Plus" (and its abbreviation "+") is USUALLY a distinguishing qualifier —
|
||||
it typically marks an enhanced-coverage variant. Default to treating Plus as
|
||||
distinguishing UNLESS Rule 5 applies.
|
||||
- "Options" and "Options Plus" → two separate canonicals (no anchor present).
|
||||
- "Medicare Advantage" and "Medicare Advantage Plus" → two separate canonicals.
|
||||
- "Healthy Advantage" and "Healthy Advantage Plus" → two separate canonicals
|
||||
UNLESS "HA+" also appears in the input set (see Rule 5).
|
||||
|
||||
4. ABBREVIATION / FULL-NAME VARIANTS — When an abbreviated form and its full-name
|
||||
equivalent both appear, they are the same product; prefer the abbreviation.
|
||||
- "PCHP" and "Parkland Community Health Plan" → same product; prefer "PCHP".
|
||||
- "HMO Premier" and "Health Maintenance Organization Premier" → same product;
|
||||
prefer "HMO Premier".
|
||||
|
||||
5. ABBREVIATIONS ABSORB THEIR BRAND-PREFIX AND PLUS EXPANSIONS — When an
|
||||
abbreviation is present in the input set or in [VALID PRODUCT VALUES], ALL
|
||||
longer renditions of the same product (with or without payer prefix, with or
|
||||
without "Plus", with or without LOB tokens like "Medicare") collapse into the
|
||||
abbreviation. This rule OVERRIDES Rule 3's default Plus behavior, and is the
|
||||
ONLY exception to Rule 6's prefer-splinter default.
|
||||
- When "MMOP" is anchored: "MMOP", "Molina Medicare Options Plus", "Medicare
|
||||
Options Plus", and "Options Plus" → all map to "MMOP".
|
||||
- When "HA+" is anchored: "HA+", "Healthy Advantage Plus", and "Molina Healthy
|
||||
Advantage Plus" → all map to "HA+".
|
||||
- When "DSNP" is anchored: "DSNP", "Dual Special Needs Plan", and "Molina DSNP"
|
||||
→ all map to "DSNP" (but "DSNP Dental" stays separate, per Rule 3).
|
||||
COUNTER-EXAMPLE — when the abbreviation's expansion itself contains "Plus" (or
|
||||
"+"), Rule 5 does NOT extend to the bare-form sibling (the same expansion MINUS
|
||||
"Plus"). That sibling is a distinct product per Rule 3 and stays separate.
|
||||
- Suppose "DCP" (UHC Dual Complete Plus) is anchored: "DCP", "Dual Complete
|
||||
Plus", and "UHC Dual Complete Plus" → all map to "DCP". But "Dual Complete"
|
||||
and "UHC Dual Complete" (no "Plus") do NOT map to "DCP" — they are the
|
||||
Plus-less sibling, a different product, with canonical "DUAL COMPLETE".
|
||||
This rule applies WHEN AND ONLY WHEN the abbreviated form appears as an anchor.
|
||||
Without an abbreviation anchor, fall back to Rule 3's default.
|
||||
|
||||
6. DO NOT STRIP GENERIC SUFFIXES — Generic trailing words like "Plan" or "Program"
|
||||
are part of the canonical name and must be preserved, UNLESS another group in
|
||||
this batch already maps to the suffix-stripped form as its canonical. In that
|
||||
case, keep them separate to avoid a collision.
|
||||
- "Premier Health Plan" → "PREMIER HEALTH PLAN", not "PREMIER HEALTH" (no other group uses "PREMIER HEALTH").
|
||||
- "Dual Special Needs Plan" → "DUAL SPECIAL NEEDS PLAN".
|
||||
|
||||
7. TIEBREAKER — PREFER SPLINTER OVER COLLAPSE — When you cannot decide between
|
||||
collapsing and splintering two raw values, keep them as separate canonicals.
|
||||
Over-merging produces silently wrong canonicals; over-splintering produces
|
||||
extra canonicals that still join correctly within this batch.
|
||||
- If [VALID PRODUCT VALUES] contains exactly one match for both raw values,
|
||||
collapse to that valid value.
|
||||
- If Rule 5 applies (abbreviation anchored), follow Rule 5.
|
||||
- Otherwise, keep them separate.
|
||||
|
||||
[CANONICAL FORM SELECTION — for each group of equivalent raw values]
|
||||
|
||||
Priority order:
|
||||
a. If a value from [VALID PRODUCT VALUES] is provided and a raw value clearly
|
||||
matches it, use that exact string.
|
||||
b. If a recognized abbreviation is present among the group's raw values, use it
|
||||
(uppercased).
|
||||
c. Otherwise, uppercase the most standard or concise raw value in the group.
|
||||
|
||||
[OUTPUT CONTRACT]
|
||||
|
||||
- Every raw input value MUST appear as a key in the output JSON dictionary.
|
||||
- Each key maps to a single canonical string.
|
||||
- Two raw values that represent the same product MUST map to the SAME canonical.
|
||||
- Two raw values that represent DISTINCT products MUST map to DIFFERENT canonicals.
|
||||
- Do NOT invent canonical strings not derivable from the inputs or
|
||||
[VALID PRODUCT VALUES].
|
||||
|
||||
[OUTPUT FORMAT]
|
||||
Return a JSON dictionary where every key is a raw PRODUCT value from the input and
|
||||
every value is its canonical form. Include all input keys.
|
||||
{JSON_DICT_FORMAT_INSTRUCTIONS}"""
|
||||
|
||||
|
||||
def AARETE_DERIVED_PRODUCT_CANONICAL_BATCH(
|
||||
unique_combinations: list[tuple[str, str, str]],
|
||||
valid_products: list[str],
|
||||
) -> Tuple[str, str, Callable[[str], dict]]:
|
||||
"""Build prompt payload for global-batch canonical PRODUCT name derivation.
|
||||
|
||||
Args:
|
||||
unique_combinations: Unique (payer_name, payer_state, raw_value) triples
|
||||
collected across the full dashboard DataFrame. Each raw value is paired
|
||||
with the payer context of the row it came from, giving the LLM correct
|
||||
per-entry context for brand-prefix stripping across multiple payers.
|
||||
valid_products: Client-supplied valid AARETE_DERIVED_PRODUCT values (soft anchor).
|
||||
|
||||
Returns:
|
||||
Tuple of (context_text, prompt_text, parser_function).
|
||||
"""
|
||||
|
||||
def _combo_line(payer_name: str, payer_state: str, raw_value: str) -> str:
|
||||
parts = []
|
||||
if payer_name:
|
||||
parts.append(f"payer: {payer_name}")
|
||||
if payer_state:
|
||||
parts.append(f"state: {payer_state}")
|
||||
ctx = f" [{', '.join(parts)}]" if parts else ""
|
||||
return f' "{raw_value}"{ctx}'
|
||||
|
||||
values_text = "\n".join(_combo_line(*combo) for combo in unique_combinations)
|
||||
valid_text = (
|
||||
", ".join(f'"{v}"' for v in valid_products)
|
||||
if valid_products
|
||||
else "none provided"
|
||||
)
|
||||
|
||||
context_text = f"""[RAW PRODUCT VALUES TO MAP]
|
||||
Each entry shows the raw value and its source payer context (used for payer-prefix
|
||||
identification only). If the same raw value appears for multiple payers, provide one
|
||||
canonical for that raw value in the output JSON.
|
||||
{values_text}
|
||||
|
||||
[VALID PRODUCT VALUES (soft anchor — prefer these exact strings when a raw value clearly matches)]
|
||||
{valid_text}"""
|
||||
|
||||
prompt_text = """[TASK]
|
||||
Apply the grouping rules and canonical form selection rules from the instruction to build
|
||||
a JSON dictionary that maps EVERY raw product value above to its canonical string.
|
||||
Briefly explain your grouping decisions, then output the JSON dictionary."""
|
||||
|
||||
return (context_text, prompt_text, _json_dict_parser)
|
||||
|
||||
|
||||
def SPLIT_SERVICE_TERM(
|
||||
service_term: str,
|
||||
) -> Tuple[str, Callable[[str], list]]:
|
||||
|
||||
@@ -112,6 +112,8 @@ USAGE_LABEL_TO_SEGMENT: dict[str, str] = {
|
||||
# --- postprocess (runs after all per-contract extraction completes) ---
|
||||
"AARETE_DERIVED_PAYER_NAME": "postprocess", # runner.py:290 (post-run clustering)
|
||||
"AARETE_DERIVED_PROVIDER_NAME": "postprocess", # runner.py:300 (post-run clustering)
|
||||
"AARETE_DERIVED_PROGRAM_CANONICAL": "postprocess", # runner.py (Dashboard-only canonical derivation)
|
||||
"AARETE_DERIVED_PRODUCT_CANONICAL": "postprocess", # runner.py (Dashboard-only canonical derivation)
|
||||
# --- active_rates (batch-level, runs after parent-child mapping) ---
|
||||
# exhibit_title_standardization: 2-pass LLM taxonomy derivation + mapping.
|
||||
# Fires only from active_rates._standardize_and_derive_intent (exhibit_title_standardization.py:68).
|
||||
|
||||
Reference in New Issue
Block a user