doczyai-pipelines/documentation/SERVICE_TERM_STANDARDIZATION.md

# Service Term Standardization

## Overview

The service term standardization feature ensures that raw, inconsistently-named `SERVICE_TERM` values ingested from payer PC pipeline outputs are normalized into a canonical form stored in a new column: `AARETE_DERIVED_SERVICE_TERM`. This improves downstream reporting accuracy, cross-payer comparability, and analytics consistency.

---

## Problem Statement

Payer PC output files contain a `SERVICE_TERM` column populated by source systems with no enforced vocabulary. The same concept (e.g., "Medical/Surgical") may appear as dozens of variants: mixed case, abbreviations, typos, or legacy codes. Without standardization, grouping and aggregating by service term produces fragmented, unreliable results.

---

## Solution

A post-processing step (`standardize_service_terms`) is applied after the PC pipeline runs. It:

1. Reads the raw `SERVICE_TERM` column from the combined pipeline output.
2. Optionally reads a `PC_Details` sheet from a PC Excel file to enable **per-group** mapping (different payers or file groups may use different term vocabularies).
3. Maps each raw term to a canonical `AARETE_DERIVED_SERVICE_TERM` value.
4. Appends the new column to the DataFrame without modifying the original `SERVICE_TERM`.

The logic lives in:

```
src/pipelines/shared/postprocessing/service_term_standardization.py
```

---

## Standalone Script

For manual inspection and retroactive application, a standalone script is provided:

```
src/scripts/retroactively_standardize_service_term.py
```

### Usage

```bash
python src/scripts/retroactively_standardize_service_term.py <input_file> [pc_excel_path]
```

| Argument | Required | Description |
|---|---|---|
| `input_file` | Yes | CSV or Excel file with a `SERVICE_TERM` column |
| `pc_excel_path` | No | Separate PC Excel file containing a `PC_Details` sheet for per-group mode |

**Auto-detection:** If `input_file` is an Excel file that already contains a `PC_Details` sheet, per-group mode is enabled automatically — no second argument needed.

### Output

Both output files are written to the **same folder as the input file**:

| File | Description |
|---|---|
| `<input>_standardized_services.csv/.xlsx` | Full dataset with the new `AARETE_DERIVED_SERVICE_TERM` column appended |
| `<input>_standardized_services_mapping.csv` | Unique `SERVICE_TERM → AARETE_DERIVED_SERVICE_TERM` mapping table for audit |

The output format (CSV vs. Excel) matches the input file format.

### Console Summary

The script prints a summary to stdout:

```
======================================================================
STANDARDIZATION SUMMARY
======================================================================
  Total unique mappings : 142
  Changed               : 89
  Unchanged (pass-thru) : 53

CHANGED MAPPINGS (first 50):
  RAW SERVICE_TERM                                         →  AARETE_DERIVED_SERVICE_TERM
  ...

UNCHANGED TERMS (first 20):
  ...
```

---

## Modes of Operation

| Mode | When it applies | Behavior |
|---|---|---|
| **Global** | No `PC_Details` sheet found | Single mapping applied to all rows |
| **Per-group** | `PC_Details` sheet present (auto-detected or explicitly provided) | Mapping scoped per file group / payer using `FILE_NAME` and `grouping_key` from `PC_Details` |

---

## Column Reference

| Column | Source | Description |
|---|---|---|
| `SERVICE_TERM` | Raw payer data | Original term as received from the payer |
| `AARETE_DERIVED_SERVICE_TERM` | Post-processing | Canonical standardized term added by this feature |

---

## Running Standalone (Quickstart)

The script can be invoked from **any working directory** — it automatically adds the repo root to `sys.path`:

```bash
# From repo root
python src/scripts/retroactively_standardize_service_term.py \
    /path/to/payer_output.xlsx

# With a separate PC Excel for grouping
python src/scripts/retroactively_standardize_service_term.py \
    /path/to/combined_output.csv \
    /path/to/pc_output.xlsx
```

Outputs will appear alongside the input file:
```
/path/to/payer_output_standardized_services.xlsx
/path/to/payer_output_standardized_services_mapping.csv
```

---

## Related Files

| Path | Purpose |
|---|---|
| `src/pipelines/shared/postprocessing/service_term_standardization.py` | Core standardization logic |
| `src/scripts/retroactively_standardize_service_term.py` | Standalone inspection / retroactive application script |