doczyai-pipelines/documentation/SERVICE_TERM_STANDARDIZATION.md

Service Term Standardization

Overview

The service term standardization feature ensures that raw, inconsistently-named SERVICE_TERM values ingested from payer PC pipeline outputs are normalized into a canonical form stored in a new column: AARETE_DERIVED_SERVICE_TERM. This improves downstream reporting accuracy, cross-payer comparability, and analytics consistency.

---

Problem Statement

Payer PC output files contain a SERVICE_TERM column populated by source systems with no enforced vocabulary. The same concept (e.g., "Medical/Surgical") may appear as dozens of variants: mixed case, abbreviations, typos, or legacy codes. Without standardization, grouping and aggregating by service term produces fragmented, unreliable results.

---

Solution

A post-processing step (standardize_service_terms) is applied after the PC pipeline runs. It:

1. Reads the raw SERVICE_TERM column from the combined pipeline output.
2. Optionally reads a PC_Details sheet from a PC Excel file to enable per-group mapping (different payers or file groups may use different term vocabularies).
3. Maps each raw term to a canonical AARETE_DERIVED_SERVICE_TERM value.
4. Appends the new column to the DataFrame without modifying the original SERVICE_TERM.

The logic lives in:

src/pipelines/shared/postprocessing/service_term_standardization.py

---

Standalone Script

For manual inspection and retroactive application, a standalone script is provided:

src/scripts/retroactively_standardize_service_term.py

Usage

python src/scripts/retroactively_standardize_service_term.py <input_file> [pc_excel_path]

Argument	Required	Description
input_file	Yes	CSV or Excel file with a SERVICE_TERM column
pc_excel_path	No	Separate PC Excel file containing a PC_Details sheet for per-group mode

Auto-detection: If input_file is an Excel file that already contains a PC_Details sheet, per-group mode is enabled automatically — no second argument needed.

Output

Both output files are written to the same folder as the input file:

File	Description
<input>_standardized_services.csv/.xlsx	Full dataset with the new AARETE_DERIVED_SERVICE_TERM column appended
<input>_standardized_services_mapping.csv	Unique SERVICE_TERM → AARETE_DERIVED_SERVICE_TERM mapping table for audit

The output format (CSV vs. Excel) matches the input file format.

Console Summary

The script prints a summary to stdout:

======================================================================
STANDARDIZATION SUMMARY
======================================================================
  Total unique mappings : 142
  Changed               : 89
  Unchanged (pass-thru) : 53

CHANGED MAPPINGS (first 50):
  RAW SERVICE_TERM                                         →  AARETE_DERIVED_SERVICE_TERM
  ...

UNCHANGED TERMS (first 20):
  ...

---

Modes of Operation

Mode	When it applies	Behavior
Global	No PC_Details sheet found	Single mapping applied to all rows
Per-group	PC_Details sheet present (auto-detected or explicitly provided)	Mapping scoped per file group / payer using FILE_NAME and grouping_key from PC_Details

---

Column Reference

Column	Source	Description
SERVICE_TERM	Raw payer data	Original term as received from the payer
AARETE_DERIVED_SERVICE_TERM	Post-processing	Canonical standardized term added by this feature

---

Running Standalone (Quickstart)

The script can be invoked from any working directory — it automatically adds the repo root to sys.path:

# From repo root
python src/scripts/retroactively_standardize_service_term.py \
    /path/to/payer_output.xlsx

# With a separate PC Excel for grouping
python src/scripts/retroactively_standardize_service_term.py \
    /path/to/combined_output.csv \
    /path/to/pc_output.xlsx

Outputs will appear alongside the input file:

/path/to/payer_output_standardized_services.xlsx
/path/to/payer_output_standardized_services_mapping.csv

---

Related Files

Path	Purpose
src/pipelines/shared/postprocessing/service_term_standardization.py	Core standardization logic
src/scripts/retroactively_standardize_service_term.py	Standalone inspection / retroactive application script