jacob-mathison/doczyai-pipelines
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

Merged in stg (pull request #1003)

Browse Source 
Merged in dev (pull request #1001)

* Merged in feature/fixPlaceholder2 (pull request #979)

Updated feature> dev gate to print variables of echo statements

* Updated feature> dev gate to print variables of echo statement

* Reverted placeholder changes


Approved-by: Sujit Deokar

* Merged in bugfix/prov_info_json_fixes (pull request #981)

fix: fall back to PROVIDER_NAME in PROV_INFO_JSON when no TIN/NPI extracted

* fix: fall back to PROVIDER_NAME in PROV_INFO_JSON when no TIN/NPI extracted

When get_prov_info_json short-circuits due to no TIN/NPI regex matches,
PROV_INFO_JSON was left as [] even when PROVIDER_NAME was successfully
extracted via the one-to-one pipeline. This caused inconsistent output
across contracts with the same provider — some files produced a NAME-only
entry (via a false-positive regex hit triggering the LLM), others produced [].

Reconcile at add_group_and_other, the first point where both extraction
streams' results are available. When PROV_INFO_JSON is empty but
PROVIDER_NAME is known, synthesize a NAME-only entry with IS_GROUP:"Y" and
populate PROV_GROUP_NAME_FULL directly — skipping the provider_name_match_check
LLM call since the match is tautological by construction.

Adds 5 unit tests covering the str, list, already-populated, empty-name,
and all-empty-lis…
* Merged in feature/DAIP2-pacificsource-reimbursements-issues (pull request #982)

Feature/DAIP2 pacificsource reimbursements issues

* Tighten PREMIUM_TERM and DISCOUNT_TERM classifier prompts

CARVEOUT_CHECK was misrouting table rate rows into special-case fields,
dropping them from the reimbursement output:

- "110% of CMS allowed" (base fee-schedule rates) was being classified as
  PREMIUM_TERM because the prompt treated "above 100% of reference" as an
  implicit premium. Seen on PacificSource Medicare_Attachment_A1 and A2
  Facility contracts where Inpatient/Outpatient rows were missing (2556)
  or silently fell back to 100% fee-schedule (2557).

- Per-service discount rates like "Progressive Lenses: 15% discount",
  "Contact Lenses: 2% discount", "Frame: 20% discount" were being
  classified as DISCOUNT_TERM because the prompt only required the word
  "discount" to appear. Seen on PacificSource Commercial_Attachment_A2
  and A5 Professional contracts (2558, 2559).

Prompts now require the literal keyword …
* Merged in bugfix/filter-docusign-lines (pull request #983)

remove docusign lines

* remove docusign lines

* add unit tests for clean_header_footer docusign/deleted_lines changes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* black forrmatting


Approved-by: Katon Minhas

* Merged in feature/PC_logic_cleanup_output (pull request #984)

Feature/PC logic cleanup output

* Few tweaks PC_logics

* Fixed orphan_ranking

* black formatting

* Changes on output field and ranking method

* updated few hotfixes

* black format fix


Approved-by: Katon Minhas

* Merged in hotfix/provider_name_group_fix (pull request #988)

Hotfix/provider name group fix

* Fixes done in GROUP column

* black format fix


Approved-by: Katon Minhas

* Merged in bugfix/DAIP2-2524-carveout-code-optimization (pull request #980)

CARVEOUT_CD issue fixed

* CARVEOUT_CD issue fixed

* pipeline error fixed

* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization

* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization

* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization

* trigger cap issue fixed

* trigger cap prompt updated

* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization


Approved-by: Katon Minhas

* Merged in bugfix/exhibit-smart-chunking-cost-improvements (pull request #985)

Bugfix/exhibit smart chunking cost improvements

* Add opt-in instrumentation for per-call token and row-count tracing

Introduce src/utils/instrumentation.py (thread-safe CSV logger) and
src/utils/instrumentation_context.py (ContextVar scope plus
submit_with_context / map_with_context helpers for propagating context
into ThreadPoolExecutor workers).

Emit events at every Bedrock call in llm_utils.invoke_claude, including
in-memory claude_cache hits, with full input/output/cache-read/cache-write
token breakdown. Emit row-count events at each row-mutating stage in the
one-to-N pipeline (clean_reimbursement_primary,
filter_services_without_reimbursements, methodology_breakout,
split_service_terms, carveout, dynamic_code_assignment,
lesser_of_distribution, dynamic_assignment) and chunking / retrieval
events in exhibit smart chunking (chunking_done, retrieval_done) plus
exhibit lifecycle events (exhibit_start, exhibit_gate_skip,
stage_…
* Merged in hotfix/fileextension_issue (pull request #989)

Hotfix/fileextension issue

* fixed strip_ext issue

* black format


Approved-by: Katon Minhas

* Merged in hotfix/exhibit-header-in-tables (pull request #987)

Hotfix/exhibit header in tables

* Merged in feature/FixplaceholderIssue (pull request #977)

Remove curly braces from echo statements in dev->stg

* Remove curly brances from echo statements in dev->stg

* Removed curly braces in echo statements in feature-> dev gate


Approved-by: Sujit Deokar

* Merged in feature/standardized-services (pull request #958)

Feature/standardized services

* service term standardization

* prompt update

* prompt updates for standardization

* only service standardization

* new file

* add supporting files and test scripts for standardization work

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* remove old files

* Merge remote-tracking branch 'origin/dev' into feature/standardized-services

* final fixes

* Merged dev into feature/standardized-services

* additional features

* removed  unwanted files

* remove unwanted files

* Merge branch 'dev' into feature/standardized-services

* Merge remote-tr…
* Merged in hotfix/postprocess_csv (pull request #990)

Hotfix/postprocess csv

* date issue_fix

* black_format

* Merged dev into hotfix/postprocess_csv


Approved-by: Katon Minhas

* Merged in bugfix/molina_ut_dynamic_primary (pull request #991)

Bugfix/molina ut dynamic primary

* Prompt changes for dynamic primary

* Route cover-sheet-only files to ERRORS.csv instead of leaking phantom rows

When every page of a contract was filtered out as a cover sheet / quick-review
form, process_file silently returned a FILE_NAME-only DataFrame. Because the
runner routes by checking for an "error" column, that file landed in
RESULTS.csv as a near-empty row and no ERRORS.csv was generated for the run.

- saas/file_processing.py: raise ValueError when text_dict is empty after
  cover-sheet filtering, so safe_process_file produces a proper error row.
- runner.py: add _is_phantom_result defense-in-depth — promote any result
  with no extracted fields beyond FILE_NAME to error_results with
  error_type=PhantomSuccess.

* Merged dev into bugfix/molina_ut_dynamic_primary

* Tighten PRODUCT prompt: restrict to valid_values, prune LOB/PROGRAM examples

* Merge branch 'bugfix/molina_ut_dynamic_primary' of bit…
* Merged in bugfix/black-format (pull request #994)

Black format for pipeline pass

* Black format for pipeline pass

* Merged in bugfix/DAIP2-2679-fix-nebraska-issues (pull request #995)

incorrect inclusion of CPT4_PROC_CD fixed

* incorrect inclusion of CPT4_PROC_CD fixed


Approved-by: Katon Minhas

* Merged in feature/DAIP2-2314-DAIP2-1687-hybrid (pull request #993)

Feature/DAIP2-2314 DAIP2 1687 hybrid

* remove -files from s3 prefix requirements

* Resolve input paths

* fix: VendorProcessor.process_file returns (df, None) tuple

runner.safe_process_file unpacks the result as (cc_df, dashboard_df), so
returning a single DataFrame caused every vendor/generic file to fail with
"too many values to unpack (expected 2)" — Python iterates DataFrame columns
during unpacking. Vendor pipelines have no dashboard variant; second slot is
None and the existing `dashboard_result is not None` guard in runner.py
already handles it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* DAIP2-2314 + DAIP2-1687: pad DYNAMIC_PRIMARY + DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION over 1024-token cache floor

- Pad DYNAMIC_PRIMARY_INSTRUCTION with three new sections: [SCOPE BOUNDARIES], [SOURCE TEXT INTERPRETATION], [REASONING DISCIPLINE], plus a [WORKED EXAMPLES] block. Estimated tokens: 447 -> 1117 (Sonnet 4.5 …
* Merged in bugfix/postprocessing_date_fix (pull request #996)

Date formatting changes

* Date formatting changes

* Merged dev into bugfix/postprocessing_date_fix


Approved-by: Katon Minhas

* Merged in bugfix/parent-child-rank-orphan-uniqueness (pull request #999)

PC_logic bugfix

* PC_logic bugfix


Approved-by: Katon Minhas

* Merged in feature/document-index (pull request #1005)

Feature/document index

* Add Document Index preprocessing — Layers 1, 2, and 3 wiring

Parse the Textract-emitted Document Index block at the top of each contract
with a single cached LLM call (prompt_document_index) instead of one per-page
call per page. Layer 2 verifies parsed entries via literal string match and
structural regex sweep, escalating suspect pages back to the existing per-page
path. Layer 1+2 failure triggers a full fallback to today's per-page flow.

New symbols:
- preprocessing_funcs.extract_document_index_block — regex slice of index prefix
- preprocessing_funcs.verify_index_against_pages — structural verifier (plain dict return)
- prompt_templates.DOCUMENT_INDEX_INSTRUCTION / DOCUMENT_INDEX — cached prompt pair
- prompt_calls.prompt_document_index — LLM wrapper (usage_label DOCUMENT_INDEX_PARSE)
- config: DOCUMENT_INDEX_PARSE_ENABLED and three threshold flags
- instrumentation: DOCUMENT_INDEX_PARSE mapped to preprocessing segment

one…
* Merged in feature/active-rates (pull request #1004)

Feature/active rates

* initial commit

* Merged in feature/FixplaceholderIssue (pull request #977)

Remove curly braces from echo statements in dev->stg

* Remove curly brances from echo statements in dev->stg

* Removed curly braces in echo statements in feature-> dev gate


Approved-by: Sujit Deokar

* Merged in feature/standardized-services (pull request #958)

Feature/standardized services

* service term standardization

* prompt update

* prompt updates for standardization

* only service standardization

* new file

* add supporting files and test scripts for standardization work

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* remove old files

* Merge remote-tracking branch 'origin/dev' into feature/standardized-services

* final fixes

* Merged dev into feature/standardized-services

* additional features

* removed  unwanted files

* remove unwanted files

* Merge branch 'dev' into feature/standardized-services

* Merge remote-track…
* Merged in feature/one-to-one-confidence-scoring (pull request #1002)

Feature/one to one confidence scoring

* T1 plumbing: capture per-field confidence + retrieved-chunk metadata for 1:1 HSC fields

Prep work for the 1:1 confidence-scoring stage. No scoring logic yet — this
just collects the inputs the next ticket (rule-based scorer) will consume.

- ONE_TO_ONE_SINGLE_FIELD_TEMPLATE: ask the LLM for confidence (0.0-1.0),
  verdict (correct/uncertain/not_found), and supporting_snippet alongside
  the field value. Existing field parser passes the extra keys through
  unchanged.
- prompt_hsc_single_field: now returns a 4-tuple (name, value, field,
  metadata) where metadata holds the confidence/verdict/snippet plus a
  lightweight summary of which chunks the LLM saw (count + ids).
  _extract_hsc_metadata is defensive: clamps out-of-range confidences,
  defaults a missing/garbage verdict, caps the snippet at 500 chars,
  returns _empty_hsc_metadata() on every bail-out path.
- run_hybrid_smart_chunked_fields: opt…
* Merged in bugfix/confidence-flagged-missing-field-col (pull request #1006)

Fix KeyError in compute_flagged when a *_CONF column has no value sibling

* Fix KeyError in compute_flagged when a *_CONF column has no value sibling

Production hit a hard crash at the end of every run:

    KeyError: "['DYNAMIC_PRIMARY_ENTITIES'] not in index"
    src/qc_qa/confidence/summary.py:143

compute_flagged was iterating over *_CONF columns and unconditionally
indexing the dataframe with both the FILE_NAME column and the stripped
value column. That assumed every <FIELD>_CONF column has a sibling
<FIELD> value column in final_df. That isn't always true: dynamic-primary
features carry only the _CONF side (their value side is dropped by
reorder_columns since it isn't in FIELD_FORMAT_MAPPING but its _CONF
suffix matches the explicit _CONF carve-out). When the model produced a
below-threshold score for one of these and the value column was absent,
pandas .loc raised KeyError and the runner crashed.

Fix:
  - Build the .loc colu…
* Merged in feature/active-rates (pull request #1007)

Feature/active rates

* amendment intent tag

* prompt update

* Merge remote-tracking branch 'origin/dev' into feature/active-rates

* Merge remote-tracking branch 'origin/dev' into feature/active-rates

* Merge branch 'dev' into feature/active-rates

* exhibit standardization updates

* use AARETE_DERIVED_EXHIBIT_TITLE for intent

* active rates stuff

* Merge branch 'dev' into feature/active-rates

* prompt update

* amendment intent types

* active rates logic update

* unit tests

* Merged dev into feature/active-rates

* logic updates

* Merge remote-tracking branch 'origin/dev' into feature/active-rates

* Merge remote-tracking branch 'origin/dev' into feature/active-rates

* updated instrumentation cost logs

* cost loging

* caching updates

* Merge remote-tracking branch 'origin/dev' into feature/active-rates

* Merge remote-tracking branch 'origin/dev' into feature/active-rates

* logging fix

* null check issue fixes

* caching fix


Approved-by…
* Merged in bugfix/stg-to-main-prep (pull request #1012)

Bugfix/stg to main prep

* Merged in dev (pull request #1001)

Dev

* Revert premature merge of bugfix/retire_stale_client_file_processing

PR #959 was merged into dev without approval. This reverts commits
5e143c10, 63f32c41, 849aa626, and 927abcae to restore dev to its
pre-merge state. The changes will be re-submitted via a new PR
after proper review.

* Merged in bugfix/retire_stale_client_file_processing (pull request #961)

Return None for dashboard output when dashboard postprocessing is off

* Return None for dashboard output when dashboard postprocessing is off

FINAL_RESULT_DF_DASHBOARD was initialized as an empty DataFrame even
when RUN_DASHBOARD_POSTPROCESSING was False, causing downstream code
to needlessly process it (reorder_columns, etc). Now returns None
when dashboard is not requested, matching the postprocess() contract.

* Merged dev into bugfix/retire_stale_client_file_processing

* Merge dev (with revert) into feature branch

* Re-ap…
* Merged in bugfix/sync-stg-into-dev-20260518 (pull request #1015)

Merged in dev (pull request #1001)

* Merged in dev (pull request #1001)

Dev

* Revert premature merge of bugfix/retire_stale_client_file_processing

PR #959 was merged into dev without approval. This reverts commits
5e143c10, 63f32c41, 849aa626, and 927abcae to restore dev to its
pre-merge state. The changes will be re-submitted via a new PR
after proper review.

* Merged in bugfix/retire_stale_client_file_processing (pull request #961)

Return None for dashboard output when dashboard postprocessing is off

* Return None for dashboard output when dashboard postprocessing is off

FINAL_RESULT_DF_DASHBOARD was initialized as an empty DataFrame even
when RUN_DASHBOARD_POSTPROCESSING was False, causing downstream code
to needlessly process it (reorder_columns, etc). Now returns None
when dashboard is not requested, matching the postprocess() contract.

* Merged dev into bugfix/retire_stale_client_file_processing

* Merge dev (with revert) into fe…
* Merged in dev (pull request #1016)

Dev

* Merged in bugfix/DAIP2-2823-generic-issue-fixes-one-to-one (pull request #1011)

Bugfix/DAIP2-2823 generic issue fixes one to one

* updated auto renewal ind prompt

* updated CONTRACT_AMENDMENT_NUM prompt

* Merged dev into bugfix/DAIP2-2823-generic-issue-fixes-one-to-one


Approved-by: Praneel Panchigar
Approved-by: Siddhant Medar

* Merged in feature/DAIP2-2698-phase-3-program-product-lob-mapping (pull request #1000)

Feature/DAIP2-2698 phase 3 program product lob mapping

* added missing phase 2 modifications

* added phase 3 modifications

* Fixed acronym issues

* black format fix

* Merged dev into feature/DAIP2-2698-phase-3-program-product-lob-mapping

* standardization fixes

* Merged dev into feature/DAIP2-2698-phase-3-program-product-lob-mapping

* added hyphenated suffix names fix

* black format fix

* standardization fix

* dynamic primary fix

* Merge Dev into feature/DAIP2-2698-phase-3-program-product-lob-mapping

* updated Program Product Standardiza…

Approved-by: Praneel Panchigar
This commit is contained in:
Katon Minhas
2026-05-19 20:20:43 +00:00
committed by Praneel Panchigar
parent 0d00d581b1
commit 623799f2a6
146 changed files with 24418 additions and 5986 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/PRE-COMMIT-SETUP.md
+62
View File
@@ -0,0 +1,62 @@
# Pre-commit Local Setup (uv)
## What is pre-commit?
`pre-commit` runs checks before each commit (for example: formatting and file sanity checks).
This repository uses pre-commit locally only. Bitbucket Pipelines does not run pre-commit.
## Install and set up locally (with uv)
1. Install `uv` (if not installed):
```bash
# Windows (PowerShell)
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
```
2. Install all dependencies (including pre-commit):
```bash
uv sync --group dev
```
3. Install git hooks in this repo:
```bash
uv run pre-commit install
```
## How it works
When you run `git commit`, pre-commit automatically runs these checks on your staged files:
| Hook | What it does | Auto-fix? |
|---|---|---|
| `trailing-whitespace` | Removes spaces at end of lines | Yes |
| `end-of-file-fixer` | Ensures files end with a newline | Yes |
| `check-yaml` | Validates YAML syntax | Manual fix |
| `check-json` | Validates JSON syntax | Manual fix |
| `check-merge-conflict` | Detects `<<<<<<<` conflict markers | Manual fix |
| `check-added-large-files` | Blocks files larger than 500KB | Manual fix |
- **Pass** -> commit goes through
- **Fail (auto-fix)** -> commit blocked, files fixed, re-stage and commit again
- **Fail (manual fix)** -> commit blocked, fix the error, re-stage and commit again
## Common commands
Run on all files:
```bash
uv run pre-commit run --all-files
```
Run on staged files (happens automatically on commit):
```bash
uv run pre-commit run
```
documentation/PROGRAM_PRODUCT_CANONICAL_DERIVATION.md
+158
View File
@@ -0,0 +1,158 @@
# AARETE Derived Program & Product Canonical Derivation
---
## Objective
Raw contracts reference the same healthcare program or health plan product using many different names. This feature normalizes them all to a single canonical name per entity, stored in two Dashboard columns:
- `AARETE_DERIVED_PROGRAM` — canonical program name (e.g., CHIP, MMC, STAR)
- `AARETE_DERIVED_PRODUCT` — canonical product/plan name (e.g., HA+, HMO)
The canonical is always **one of the actual values seen in the data** — nothing is invented.
---
## Pipeline (4 Steps)
```
Raw PROGRAM / PRODUCT column (comma-separated strings in Dashboard DF)
│
├─ Step 1 Collect all unique non-empty values across the full DataFrame
│
├─ Step 2 Extract payer context from the DataFrame
│ (PAYER_NAME, PAYER_STATE, FILE_NAME from the first row)
│
├─ Step 3 ONE global LLM batch call with all unique values + payer context
│ + valid-values soft anchor → returns {raw: canonical} dict
│ The LLM groups semantically equivalent variants and assigns one
│ canonical per group, resolving brand-prefix noise vs. meaningful
│ qualifiers in a single pass
│
└─ Step 4 Apply {raw → canonical} map to every row
Output is comma-separated string, e.g. "CHIP, MMC"
```
## Inputs & Outputs
| | PROGRAM | PRODUCT |
|---|---|---|
| **Input column** | `PROGRAM` | `PRODUCT` |
| **Output column** | `AARETE_DERIVED_PROGRAM` | `AARETE_DERIVED_PRODUCT` |
| **Valid-values list** | `constants.VALID_AARETE_DERIVED_PROGRAMS` | `constants.VALID_AARETE_DERIVED_PRODUCTS` |
| **Output format** | `str`, comma-separated | `str`, comma-separated |
**Skip condition:** If the valid-values list is non-empty **and** the derived column is already fully populated, the function returns the DataFrame unchanged.
---
## Exceptions & Edge Cases
| Scenario | Behaviour |
|---|---|
| `PROGRAM` / `PRODUCT` column missing | Warning logged, DataFrame returned unchanged |
| All values empty / null | Info logged, no LLM call made |
| Sentinel values (`N/A`, `UNKNOWN`, `NONE`, `NULL`) | Filtered out before LLM call |
| LLM omits a raw value from its response | Falls back to `raw.upper()` for that key |
| LLM response parse failure | Error logged; entire map falls back to `raw.upper()` for all values |
| Multi-value cell e.g. `"CHIP, MMC"` | Each value mapped independently, re-joined |
| Duplicate canonicals in one cell | Deduplicated before joining |
---
## LLM Prompt Design
Both batch prompts follow the same structure: a **static cached instruction** (registered in `src/prompts/cache_registry.py`) plus a **dynamic template** that injects unique values and payer context at runtime.
The instruction defines 7 rules evaluated in order:
| # | Rule | Description |
|---|---|---|
| 1 | **PAYER PREFIX** | Strip the payer brand name when it appears as a leading prefix (e.g., "Molina Medicaid" → "Medicaid"). For PRODUCT, the base name after stripping must be identical for two values to merge — "Options" ≠ "Options Plus". |
| 2 | **STATE PREFIX NOT NOISE** | A state abbreviation prefix is meaningful if the payer contracts span multiple states; keep it unless context confirms it is redundant. |
| 3 | **DISTINGUISHING QUALIFIERS** | Qualifiers that change meaning (e.g., "Perinatal", "Enhanced") must keep values as separate canonicals. "Plus" is usually distinguishing unless Rule 5 applies. |
| 4 | **ABBREVIATION / FULL-NAME** | Prefer the recognized abbreviation as the canonical (e.g., "CHIP" over "Children's Health Insurance Program"). |
| 5 | **ABBREVIATIONS ABSORB BRAND-PREFIX AND PLUS EXPANSIONS** | A widely-used abbreviation can absorb brand-prefix or "Plus" expansions if it already encodes them (e.g., MMOP absorbs "Molina Medicaid Options Plus"). |
| 6 | **DO NOT STRIP GENERIC SUFFIXES** | Generic trailing words like "Plan" or "Program" are part of the canonical and must be preserved — unless another group in the same batch already uses the suffix-stripped form as its canonical (collision avoidance). |
| 7 | **TIEBREAKER — PREFER SPLINTER** | When uncertain, keep two values as separate canonicals rather than merging them. Over-splitting is safer than over-merging. |
The valid-values list is passed as a soft anchor (canonical selection priority (a)). When a raw value clearly matches a valid value, that exact string is used. Otherwise the LLM selects from the raw input values.
---
## Code Changes
| File | What changed |
|---|---|
| [src/pipelines/shared/postprocessing/aarete_derived.py](../src/pipelines/shared/postprocessing/aarete_derived.py) | Removed clustering helpers (`_merge_acronym_clusters`, `_split_subset_clusters`); rewrote `add_aarete_derived_program_canonical` and `add_aarete_derived_product_canonical` to use a single batch LLM call |
| [src/prompts/prompt_templates.py](../src/prompts/prompt_templates.py) | Added 4 functions: 2 cached batch instructions + 2 dynamic batch templates |
| [src/pipelines/saas/prompts/prompt_calls.py](../src/pipelines/saas/prompts/prompt_calls.py) | Added 2 batch wrapper functions: `prompt_aarete_derived_program_canonical_batch` and `prompt_aarete_derived_product_canonical_batch` |
| [src/prompts/cache_registry.py](../src/prompts/cache_registry.py) | Added 2 cache entries for the batch prompt instructions |
| [test_program_product_canonical.py](../test_program_product_canonical.py) | Updated for batch architecture; removed clustering imports and per-cluster validation checks |
---
## Function Reference
### `aarete_derived.py`
| Function | Visibility | Purpose |
|---|---|---|
| `_split_multi_value_separators(text)` | private | Splits a raw cell value on commas, semicolons, and similar delimiters. |
| `_collect_unique_values_from_list_column(column)` | private | Deduplicated list of non-empty, non-sentinel values from a Series. Handles both comma-string and list formats. |
| `_all_empty_in_list_column(column)` | private | Returns `True` if every cell is empty — used as the skip-derivation guard. |
| `_map_row_list_to_canonicals(raw_val, canonical_map)` | private | Maps one cell to its canonical(s). Splits on `,`, looks up each part, deduplicates, returns comma-separated string. |
| `add_aarete_derived_program_canonical(df, valid_programs)` | **public** | Runs the 4-step batch pipeline for `PROGRAM` → `AARETE_DERIVED_PROGRAM`. |
| `add_aarete_derived_product_canonical(df, valid_products)` | **public** | Same pipeline for `PRODUCT` → `AARETE_DERIVED_PRODUCT`. |
### `prompt_templates.py`
| Function | Purpose |
|---|---|
| `AARETE_DERIVED_PROGRAM_CANONICAL_BATCH_INSTRUCTION()` | Static cached system instruction for batch PROGRAM canonicalization (7 rules). Registered in `cache_registry.py` under label `AARETE_DERIVED_PROGRAM_CANONICAL_BATCH`. |
| `AARETE_DERIVED_PROGRAM_CANONICAL_BATCH(unique_values, valid_programs, payer_name, payer_state)` | Dynamic template. Returns `(context_text, prompt_text, parser)` tuple. |
| `AARETE_DERIVED_PRODUCT_CANONICAL_BATCH_INSTRUCTION()` | Static cached system instruction for batch PRODUCT canonicalization (7 rules). Registered under label `AARETE_DERIVED_PRODUCT_CANONICAL_BATCH`. |
| `AARETE_DERIVED_PRODUCT_CANONICAL_BATCH(unique_values, valid_products, payer_name, payer_state)` | Dynamic template. Returns `(context_text, prompt_text, parser)` tuple. |
### `prompt_calls.py`
| Function | Purpose |
|---|---|
| `prompt_aarete_derived_program_canonical_batch(unique_values, valid_programs, filename, payer_name, payer_state)` | Single LLM call for all PROGRAM values. Returns `dict[str, str]` mapping each raw value to its canonical. Falls back to `raw.upper()` for any key the LLM omits; falls back to `{v: v.upper()}` for all values on parse failure. |
| `prompt_aarete_derived_product_canonical_batch(unique_values, valid_products, filename, payer_name, payer_state)` | Same for PRODUCT. |
---
## Example
**Raw PROGRAM values (all unique values across the DataFrame):**
`CHIP`, `Children's Health Insurance Program`, `Children's Health Insurance Program (CHIP)`, `MMC`, `Medicaid Managed Care`, `Medicaid`
**Batch LLM call (Step 3):**
All 6 values are sent in one call with payer context. The LLM determines:
- `CHIP`, `Children's Health Insurance Program`, `Children's Health Insurance Program (CHIP)` → same program (Rule 4: prefer abbreviation) → canonical: `CHIP`
- `MMC`, `Medicaid Managed Care` → same program (Rule 4: prefer abbreviation) → canonical: `MMC`
- `Medicaid` → distinct broader program; no other group uses `MEDICAID` → canonical: `MEDICAID`
**Resulting `{raw → canonical}` map:**
| Raw value | Canonical |
|---|---|
| CHIP | CHIP |
| Children's Health Insurance Program | CHIP |
| Children's Health Insurance Program (CHIP) | CHIP |
| Medicaid Managed Care | MMC |
| MMC | MMC |
| Medicaid | MEDICAID |
**Final output applied to rows:**
| PROGRAM (raw) | AARETE_DERIVED_PROGRAM |
|---|---|
| CHIP | CHIP |
| Children's Health Insurance Program | CHIP |
| Children's Health Insurance Program (CHIP) | CHIP |
| Medicaid Managed Care | MMC |
| Medicaid | MEDICAID |
| CHIP, Medicaid Managed Care | CHIP, MMC |
documentation/PROMPT_CACHING_ANALYSIS.md
+204
View File
@@ -0,0 +1,204 @@
# Prompt Caching Analysis Report
**Branch:** `feature/DAIP2-2314-expand-caching-for-short-prompts`
**Date:** 2026-04-21
**Comparison:** Feature branch vs `origin/dev`
---
## Executive Summary
This report analyzes the prompt caching implementation in the feature branch, comparing cached prompts against the dev branch and evaluating cost savings from Bedrock prompt caching.
### Key Findings
| Metric | Value |
|--------|-------|
| Total API Calls Analyzed | 1,251 |
| Unique Usage Labels | 35 |
| Total Cache Read Tokens | 1,472,330 |
| Total Cache Write Tokens | 87,167 |
| Overall Cost Savings | **28.1%** ($3.65 saved) |
| Prompts Actively Caching | 22 |
| Prompts Wired but Not Caching | 11 |
| Prompts Not Wired | 2 |
---
## 1. Cache Effectiveness by Usage Label
### Actively Caching (22 prompts) - High Performers
| Usage Label | Calls | Avg Instruction Tokens | Cache Hit Rate |
|-------------|-------|------------------------|----------------|
| OUTLIER_BREAKOUT | 1 | 2,483 | 98.4% |
| SERVICE_ENRICHMENT | 30 | 1,459 | 98.3% |
| CARVEOUT_CHECK | 43 | 1,498 | 95.6% |
| METHODOLOGY_BREAKOUT | 37 | 1,927 | 95.5% |
| LESSER_OF_DISTRIBUTION | 32 | 914 | 94.5% |
| CHECK_PROVIDER_NAME_MATCH | 74 | 957 | 94.2% |
| DYNAMIC_CODE_ASSIGNMENT | 38 | 966 | 94.1% |
| CODE_EXPLICIT | 34 | 871 | 93.9% |
| GROUPER_BREAKOUT | 7 | 939 | 93.6% |
| LESSER_OF_CHECK | 6 | 900 | 92.0% |
| DYNAMIC_ASSIGNMENT | 204 | 1,578 | 89.4% |
| validate_reimbursements_for_llm | 65 | 894 | 89.2% |
| code_implicit_rag | 60 | 896 | 88.4% |
| REIMB_DATES_ASSIGNMENT | 29 | 803 | 87.2% |
| REIMBURSEMENT_PRIMARY | 38 | 1,390 | 85.0% |
| EXHIBIT_LEVEL | 33 | 1,108 | 74.0% |
| EXHIBIT_HEADER | 90 | 922 | 73.7% |
| fill_bill_type | 47 | 968 | 72.5% |
| prompt_provider_info | 24 | 966 | 60.5% |
| DYNAMIC_PRIMARY | 44 | 441 | 49.7% |
### Wired But Not Caching (11 prompts) - Under 1024 Token Threshold
These prompts are wired for caching but have instruction tokens below Bedrock's 1024-token minimum threshold:
| Usage Label | Calls | Avg Instruction Tokens | Issue |
|-------------|-------|------------------------|-------|
| SPLIT_SERVICE_TERM | 10 | 1,600 | Should be caching - investigate |
| prompt_lob_relationship | 56 | 408 | **Needs padding to 1024** |
| SPLIT_REIMB_DATES | 13 | 410 | **Needs padding to 1024** |
| code_implicit_arbitration | 12 | 391 | **Needs padding to 1024** |
| AARETE_DERIVED_PAYER_NAME | 1 | 372 | **Needs padding to 1024** |
| FEE_SCHEDULE_BREAKOUT | 18 | 352 | **Needs padding to 1024** |
| code_implicit_special | 30 | 329 | **Needs padding to 1024** |
| DATE_FIX | 10 | 322 | **Needs padding to 1024** |
| SPECIAL_CASE_ASSIGNMENT | 4 | 251 | **Needs padding to 1024** |
| DERIVED_TERM_DATE | 10 | 247 | **Needs padding to 1024** |
| code_last_check | 23 | 179 | **Needs padding to 1024** |
| EXTRACT_AMENDMENT_NUM_FROM_FILENAME | 3 | 152 | **Needs padding to 1024** |
| EXHIBIT_HEADER_DEDUP | 10 | 35 | **Needs padding to 1024** |
### Not Wired for Caching (2 prompts)
| Usage Label | Calls | Notes |
|-------------|-------|-------|
| prompt_hsc_single_field | 109 | High volume - should investigate wiring |
| SPECIAL_CASE_BREAKOUT | 6 | Low volume |
---
## 2. Prompt Template Changes (Feature Branch vs Dev)
The feature branch includes **649 lines changed** in `src/prompts/prompt_templates.py`. Key expansions:
### Expanded Prompts (Additional Context Added)
| Instruction Function | Change Summary |
|---------------------|----------------|
| **EXHIBIT_LEVEL_INSTRUCTION** | +55 lines: Added detailed guidance, validation rules, examples, common pitfalls, field-specific notes, quality assurance checks |
| **DYNAMIC_PRIMARY_INSTRUCTION** | +18 lines: Added extraction guidance, ambiguity handling, quality checks |
| **LESSER_OF_DISTRIBUTION_INSTRUCTION** | Restructured (net -25 lines): Converted to cleaner decision framework format |
| **DYNAMIC_CODE_ASSIGNMENT_INSTRUCTION** | +10 lines: Added classification guidance with explicit rules |
| **CODE_IMPLICIT_INSTRUCTION** | Reformatted indentation |
| **CODE_IMPLICIT_ARBITRATION_INSTRUCTION** | +43 lines: Added clarifications, deterministic decision rules, calibration examples |
| **FILL_BILL_TYPE_INSTRUCTION** | +49 lines: Added detailed guidance, validation rules, extended examples |
| **TIN_NPI_TEMPLATE_INSTRUCTION** | +22 lines: Added provider/payer distinction clarity |
### Analysis of Prompt Changes
**Positive Impacts:**
- More explicit instructions reduce model ambiguity
- Quality checks and validation rules improve consistency
- Examples help calibrate model responses
- Clearer formatting improves readability
**Potential Concerns:**
- Expanded prompts increase token count (higher cache write cost initially)
- Some prompts may have reduced due to restructuring (LESSER_OF_DISTRIBUTION)
- Need to verify output quality hasn't degraded with expanded context
---
## 3. Cost Analysis
### Bedrock Pricing Applied
- **Input tokens:** $3.00/1M tokens
- **Output tokens:** $15.00/1M tokens
- **Cache read:** $0.30/1M tokens (90% savings)
- **Cache write:** $3.75/1M tokens (25% premium)
### Observed Results
| Metric | Value |
|--------|-------|
| Total Input Tokens | 844,370 |
| Total Output Tokens | 401,926 |
| Total Cache Read Tokens | 1,472,330 |
| Total Cache Write Tokens | 87,167 |
| **Cost Without Caching** | $12.98 |
| **Cost With Caching** | $9.33 |
| **Savings** | $3.65 (28.1%) |
### Per-Prompt Cost Impact (Estimated from previous testing)
| Prompt | Cost Reduction |
|--------|---------------|
| LESSER_OF_DISTRIBUTION | 44% (Best performer) |
| EXHIBIT_HEADER | 7% |
| CHECK_PROVIDER_NAME_MATCH | 5% |
| TIN_NPI_TEMPLATE (prompt_provider_info) | 5% |
| code_implicit_rag | -12% (Negative - needs review) |
---
## 4. Prompts Needing Attention
### Priority 1: Prompts Needing Padding (Under 1024 tokens)
These prompts are wired but not actually caching due to Bedrock's minimum threshold:
1. **prompt_lob_relationship** (408 tokens, 56 calls) - High impact
2. **code_implicit_special** (329 tokens, 30 calls) - Medium impact
3. **code_last_check** (179 tokens, 23 calls) - Medium impact
4. **FEE_SCHEDULE_BREAKOUT** (352 tokens, 18 calls) - Medium impact
5. **SPLIT_REIMB_DATES** (410 tokens, 13 calls)
6. **code_implicit_arbitration** (391 tokens, 12 calls)
7. **SPLIT_SERVICE_TERM** (1600 tokens, 10 calls) - Should be caching, investigate
8. **DATE_FIX** (322 tokens, 10 calls)
9. **DERIVED_TERM_DATE** (247 tokens, 10 calls)
10. **EXHIBIT_HEADER_DEDUP** (35 tokens, 10 calls)
### Priority 2: Investigate Negative Result
- **code_implicit_rag**: Shows -12% cost (increase) - The expanded prompt may be too verbose or structured differently causing cache misses
### Priority 3: Not Wired for Caching
- **prompt_hsc_single_field** (109 calls) - High volume, should consider wiring
---
## 5. Recommendations
1. **Pad short prompts to 1024 tokens**: Add contextual padding to prompts under the threshold, especially high-frequency ones like `prompt_lob_relationship` and `code_implicit_special`
2. **Investigate code_implicit_rag**: The negative cost result (-12%) suggests the expanded prompt may be causing cache fragmentation or misses
3. **Wire prompt_hsc_single_field**: With 109 calls, this is a good candidate for caching
4. **Monitor SPLIT_SERVICE_TERM**: At 1,600 tokens it should be caching but shows 0% cache hit rate - may be a wiring issue
5. **Run testbed comparison**: Use `src/testbed/testbed_metrics.py` to compare output quality between cached and non-cached prompts to ensure no regression
---
## 6. Files Analyzed
- `src/test-PROMPT-CALLS_10.csv` - API call logs with cache statistics
- `src/testbed-usethis.xlsx` - Testbed comparison file (binary, not readable)
- `src/prompts/prompt_templates.py` - Prompt definitions (649 lines changed)
- `documentation/prompt_caching_tracker_v2.csv` - Caching status tracker
---
## 7. Next Steps
1. Run testbed metrics comparison script to validate output quality
2. Add padding to short prompts identified above
3. Investigate and fix code_implicit_rag negative result
4. Wire prompt_hsc_single_field for caching
5. Debug SPLIT_SERVICE_TERM caching issue
documentation/PROMPT_CACHING_SYSTEM_PROPOSAL.md
+154
View File
@@ -0,0 +1,154 @@
# Prompt Caching System Proposal
Branch: `feature/DAIP2-2314-expand-caching-for-short-prompts`
## Goal
Ensure every stable instruction prompt that is intended to be cached is actually cacheable, warmed on the same model used at runtime, and observable in runtime logs.
## Recommended Caching System
1. Introduce a central prompt cache registry.
Each cacheable prompt should be declared once with:
- `usage_label`
- instruction function
- model family used at runtime
- minimum cache token policy
- runtime call sites
- whether the prompt also uses `context_for_caching`
This avoids the current split where cache warming is in `prompt_templates.get_cacheable_instructions()` but runtime model choice is scattered through prompt call sites.
2. Enforce the 1024-token minimum before runtime.
Prompt caching should not be treated as enabled just because `cache=True` is passed. At startup, the registry should validate every cached instruction with the same tokenizer/estimate used by reporting. If a prompt is below the minimum, it should either:
- fail fast in a cache audit mode, or
- automatically append approved static cache padding/instruction context until it crosses the threshold.
The padding must be stable and domain-relevant, not dynamic request data.
3. Warm caches per model, not globally.
Cache entries are model-specific. The runner currently warms all instructions with `sonnet_latest`, but `SPLIT_SERVICE_TERM` runs with `haiku_latest`. The cache warming layer should warm `(usage_label, resolved_model_id)` pairs derived from the registry.
4. Only mark supported models as cacheable.
`_supports_prompt_cache()` currently excludes Haiku. If Haiku prompt caching is not supported in this Bedrock setup, Haiku prompts should not be registered as cached. Either move those prompt calls to `sonnet_latest` when caching matters, or keep them uncached and report them as intentionally uncached.
5. Keep static instruction caching separate from dynamic context caching.
Instruction prompts can be cached predictably. `context_for_caching` is different because it changes by contract, exhibit, or page. It should be tracked as a separate cache class:
- `instruction_cache`: expected to read after warm-up
- `context_cache`: expected to create/read per repeated context
This prevents context variability from making instruction caching look broken.
6. Add a cache contract test.
A local test should inspect the built request body for every registered prompt and assert:
- the instruction block has `cache_control`
- the resolved model supports caching
- the static instruction token estimate is at least 1024
- warm-up model equals runtime model
7. Upgrade runtime reporting.
Add these fields to prompt-call logs:
- `resolved_model_id`
- `cache_policy`: `instruction`, `context`, `instruction+context`, `none`
- `instruction_cache_eligible`
- `context_cache_eligible`
- `cache_miss_reason`
This makes root cause visible without manual investigation.
## Prompt Application Report
### Apply Instruction Caching, Already Over 1024 Tokens
These should remain registered and warmed. They are good candidates for strict cache contract tests.
| Prompt | Estimated Instruction Tokens | Action |
|---|---:|---|
| `OUTLIER_BREAKOUT` | 2483 | Keep cached |
| `METHODOLOGY_BREAKOUT` | 1927 | Keep cached |
| `SPLIT_SERVICE_TERM` | 1600 | Cache only if runtime model supports caching; currently runs on Haiku |
| `DYNAMIC_ASSIGNMENT` | 1578 | Keep cached |
| `CARVEOUT_CHECK` | 1498 | Keep cached |
| `SERVICE_ENRICHMENT` | 1459 | Keep cached |
| `REIMBURSEMENT_PRIMARY` | 1390 | Keep cached |
| `EXHIBIT_LEVEL` | 1108 | Keep cached |
### Apply Instruction Caching After Padding/Expansion
These are wired or partially wired but below the 1024-token minimum. They should be padded or expanded with stable domain guidance before being considered cached.
| Prompt | Estimated Instruction Tokens | Tokens Needed | Action |
|---|---:|---:|---|
| `fill_bill_type` | 968 | 56 | Add small static guidance |
| `DYNAMIC_CODE_ASSIGNMENT` | 966 | 58 | Add small static guidance |
| `CHECK_PROVIDER_NAME_MATCH` | 957 | 67 | Add small static guidance |
| `GROUPER_BREAKOUT` | 939 | 85 | Add small static guidance |
| `EXHIBIT_HEADER` | 922 | 102 | Add static header examples/rules |
| `LESSER_OF_DISTRIBUTION` | 914 | 110 | Add static decision examples |
| `LESSER_OF_CHECK` | 900 | 124 | Add static classification examples |
| `CODE_EXPLICIT` | 871 | 153 | Add static code-system rules |
| `REIMB_DATES_ASSIGNMENT` | 803 | 221 | Add date extraction examples |
| `DYNAMIC_PRIMARY` | 441 | 583 | Expand instruction substantially |
| `SPLIT_REIMB_DATES` | 410 | 614 | Expand instruction substantially |
| `prompt_lob_relationship` | 408 | 616 | Move field prompt into instruction or add static LOB rules |
| `code_implicit_arbitration` | 391 | 633 | Expand arbitration rules |
| `AARETE_DERIVED_PAYER_NAME` | 372 | 652 | Expand payer naming rules |
| `FEE_SCHEDULE_BREAKOUT` | 352 | 672 | Expand fee schedule rules |
| `code_implicit_special` | 329 | 695 | Expand special code rules |
| `DATE_FIX` | 322 | 702 | Expand date normalization rules |
| `SPECIAL_CASE_ASSIGNMENT` | 251 | 773 | Expand special case rules |
| `DERIVED_TERM_DATE` | 247 | 777 | Expand term date rules |
| `code_last_check` | 179 | 845 | Expand or leave uncached if low value |
| `EXTRACT_AMENDMENT_NUM_FROM_FILENAME` | 152 | 872 | Expand or leave uncached if low value |
| `EXHIBIT_HEADER_DEDUP` | 35 | 989 | Do not cache unless redesigned |
### Wire for Caching or Mark Intentionally Uncached
These showed runtime volume but no instruction function match in the audit. They need explicit registry decisions.
| Prompt | Runtime Finding | Action |
|---|---|---|
| `prompt_hsc_single_field` | 109 calls, `cache_enabled=False`, large dynamic prompt | Split stable HSC instructions from dynamic chunk and cache the instruction |
| `SPECIAL_CASE_BREAKOUT` | 6 calls, no instruction cache | Wire only if repeated volume justifies expansion |
| `code_implicit_rag` | Cached in API, audit alias missing | Add registry alias to the actual instruction function |
| `prompt_provider_info` | Cached in API, audit alias missing | Add registry alias to `TIN_NPI_TEMPLATE_INSTRUCTION` or rename usage label |
| `validate_reimbursements_for_llm` | Cached in API, audit alias missing | Add registry alias to `VALIDATE_REIMBURSEMENTS_INSTRUCTION` |
## Immediate Fixes
1. Change `SPLIT_SERVICE_TERM` to use a cache-supported model or mark it intentionally uncached. Current behavior is misleading: `cache=True` is passed, but no Bedrock cache tokens are recorded.
2. Add a registry alias map for usage labels that do not match instruction names:
- `code_implicit_rag` -> `CODE_IMPLICIT`
- `prompt_provider_info` -> `TIN_NPI_TEMPLATE`
- `validate_reimbursements_for_llm` -> `VALIDATE_REIMBURSEMENTS`
- `prompt_lob_relationship` -> its real instruction function
3. Expand the near-threshold instructions first because they need the least work and have current evidence of API cache reads:
- `fill_bill_type`
- `DYNAMIC_CODE_ASSIGNMENT`
- `CHECK_PROVIDER_NAME_MATCH`
- `GROUPER_BREAKOUT`
- `EXHIBIT_HEADER`
- `LESSER_OF_DISTRIBUTION`
- `LESSER_OF_CHECK`
- `CODE_EXPLICIT`
4. Redesign `prompt_hsc_single_field` into `(instruction, prompt)` form. It is high-volume and currently sends all prompt text dynamically.
## Expected Behavior After Fix
For instruction-only prompts, after warm-up the API should report cache reads on normal calls. The cached percentage will still not be 100% of total billed input because the dynamic user prompt remains uncached. The correct target is:
- `cache_creation_tokens` only during warm-up or first uncached use
- `cache_read_tokens > 0` on every eligible runtime call
- `cache_miss_reason` empty for registered instruction cache prompts
For context-cached prompts, 100% cache reads should not be expected unless the same context block repeats. The report should evaluate those separately.
documentation/SERVICE_TERM_STANDARDIZATION.md
+132
View File
@@ -0,0 +1,132 @@
# Service Term Standardization
## Overview
The service term standardization feature ensures that raw, inconsistently-named `SERVICE_TERM` values ingested from payer PC pipeline outputs are normalized into a canonical form stored in a new column: `AARETE_DERIVED_SERVICE_TERM`. This improves downstream reporting accuracy, cross-payer comparability, and analytics consistency.
---
## Problem Statement
Payer PC output files contain a `SERVICE_TERM` column populated by source systems with no enforced vocabulary. The same concept (e.g., "Medical/Surgical") may appear as dozens of variants: mixed case, abbreviations, typos, or legacy codes. Without standardization, grouping and aggregating by service term produces fragmented, unreliable results.
---
## Solution
A post-processing step (`standardize_service_terms`) is applied after the PC pipeline runs. It:
1. Reads the raw `SERVICE_TERM` column from the combined pipeline output.
2. Optionally reads a `PC_Details` sheet from a PC Excel file to enable **per-group** mapping (different payers or file groups may use different term vocabularies).
3. Maps each raw term to a canonical `AARETE_DERIVED_SERVICE_TERM` value.
4. Appends the new column to the DataFrame without modifying the original `SERVICE_TERM`.
The logic lives in:
```
src/pipelines/shared/postprocessing/service_term_standardization.py
```
---
## Standalone Script
For manual inspection and retroactive application, a standalone script is provided:
```
src/scripts/retroactively_standardize_service_term.py
```
### Usage
```bash
python src/scripts/retroactively_standardize_service_term.py <input_file> [pc_excel_path]
```
| Argument | Required | Description |
|---|---|---|
| `input_file` | Yes | CSV or Excel file with a `SERVICE_TERM` column |
| `pc_excel_path` | No | Separate PC Excel file containing a `PC_Details` sheet for per-group mode |
**Auto-detection:** If `input_file` is an Excel file that already contains a `PC_Details` sheet, per-group mode is enabled automatically — no second argument needed.
### Output
Both output files are written to the **same folder as the input file**:
| File | Description |
|---|---|
| `<input>_standardized_services.csv/.xlsx` | Full dataset with the new `AARETE_DERIVED_SERVICE_TERM` column appended |
| `<input>_standardized_services_mapping.csv` | Unique `SERVICE_TERM → AARETE_DERIVED_SERVICE_TERM` mapping table for audit |
The output format (CSV vs. Excel) matches the input file format.
### Console Summary
The script prints a summary to stdout:
```
======================================================================
STANDARDIZATION SUMMARY
======================================================================
Total unique mappings : 142
Changed : 89
Unchanged (pass-thru) : 53
CHANGED MAPPINGS (first 50):
RAW SERVICE_TERM → AARETE_DERIVED_SERVICE_TERM
...
UNCHANGED TERMS (first 20):
...
```
---
## Modes of Operation
| Mode | When it applies | Behavior |
|---|---|---|
| **Global** | No `PC_Details` sheet found | Single mapping applied to all rows |
| **Per-group** | `PC_Details` sheet present (auto-detected or explicitly provided) | Mapping scoped per file group / payer using `FILE_NAME` and `grouping_key` from `PC_Details` |
---
## Column Reference
| Column | Source | Description |
|---|---|---|
| `SERVICE_TERM` | Raw payer data | Original term as received from the payer |
| `AARETE_DERIVED_SERVICE_TERM` | Post-processing | Canonical standardized term added by this feature |
---
## Running Standalone (Quickstart)
The script can be invoked from **any working directory** — it automatically adds the repo root to `sys.path`:
```bash
# From repo root
python src/scripts/retroactively_standardize_service_term.py \
/path/to/payer_output.xlsx
# With a separate PC Excel for grouping
python src/scripts/retroactively_standardize_service_term.py \
/path/to/combined_output.csv \
/path/to/pc_output.xlsx
```
Outputs will appear alongside the input file:
```
/path/to/payer_output_standardized_services.xlsx
/path/to/payer_output_standardized_services_mapping.csv
```
---
## Related Files
| Path | Purpose |
|---|---|
| `src/pipelines/shared/postprocessing/service_term_standardization.py` | Core standardization logic |
| `src/scripts/retroactively_standardize_service_term.py` | Standalone inspection / retroactive application script |
documentation/release-rollback-strategy.md
+177
View File
@@ -0,0 +1,177 @@
# CI/CD Branching, Promotion, Release, and Rollback Strategy
This document describes the current Bitbucket CI/CD implementation and branch governance configured for this repository. It reflects the intended promotion model shown in the attached strategy image.
## 1) Strategy Overview
### Branch model
- `feature/*` and `bugfix/*` are short-lived development branches.
- `dev` is the integration branch.
- `stg` is the pre-production validation branch.
- `main` is the production release branch.
- `hotfix/*` supports emergency fixes with controlled promotion.
### Promotion paths (as implemented)
1. `feature/*` -> `Develop`
2. `bugfix/*` -> `Develop`
3. `dev` -> `Staging`
4. `stg` -> `main`
### Hotfix paths
- `hotfix/*` -> `main` (direct emergency production fix)
- `hotfix/*` -> `dev` (back-merge / forward-fix alignment)
## 2) Pull Request Pipeline Controls
Defined under `pipelines.pull-requests` in `bitbucket-pipelines.yml`.
### A. Source branch gate by pattern
- `feature/*`: pipeline fails unless destination is `dev`
- `bugfix/*`: pipeline fails unless destination is `dev`
- `hotfix/*`: pipeline fails unless destination is `main` or `dev`
- `dev`: pipeline fails unless destination is `stg`
- `stg`: pipeline fails unless destination is `main`
### B. Branch naming validation for all PRs
The `"**"` PR pipeline validates source branch naming convention:
- Allowed: `feature/<description>`
- Allowed: `bugfix/<description>`
- Allowed: `hotfix/<description>`
Any other source branch naming pattern is blocked.
## 3) Repository Branch Restrictions (Bitbucket Settings)
The following controls are enforced through repository settings (in addition to pipeline checks).
## 3.1 `dev`
- Direct write access: restricted
- Deletion: not allowed
- History rewrite: not allowed
- Merge via PR only
- Merge checks:
- minimum 1 approval
- minimum 1 default reviewer approval
- unresolved PR tasks not allowed
- last commit must have successful build
## 3.2 `stg`
- Direct write access: restricted
- Deletion: not allowed
- History rewrite: not allowed
- Merge via PR only
- Merge checks:
- minimum 2 approvals
- minimum 1 default reviewer approvals
- unresolved PR tasks not allowed
- last commit must have successful build
## 3.3 `main`
- Direct write access: restricted
- Deletion: not allowed
- History rewrite: not allowed
- Merge via PR only
- Merge checks:
- minimum 2 approvals
- minimum 1 default reviewer approvals
- unresolved PR tasks not allowed
- last commit must have successful build
## 3.4 `feature/*`
- Write access: broader (Everybody)
- Deletion: not allowed
- History rewrite: not allowed
- Merge to protected branches is still controlled by PR destination branch restrictions and PR pipeline gates.
## 3.5 `bugfix/*`
- Write access: limited to authorized users/groups
- Deletion: not allowed
- History rewrite: not allowed
## 3.6 `hotfix/*`
- Write access: limited to authorized users
- Deletion: not allowed
- History rewrite: not allowed
- PR merges additionally require build success and at least 1 approval (as configured).
## 4) End-to-End Flow (Aligned to Strategy Image)
```mermaid
flowchart LR
FB[feature/* or bugfix/*] -->|PR + checks| D[dev]
D -->|PR + stricter checks| S[stg]
S -->|PR + strictest checks| M[main]
H[hotfix/*] -->|PR + checks| M
H -->|PR + checks| D
```
### Control layering
Each promotion is protected by two layers:
1. Pipeline gate validates source/destination path and syntax/build health.
2. Branch restriction gate enforces approvals, reviewers, task completion, and successful build before merge.
## 5) Release to Production (Custom Pipeline)
Custom pipeline: `release-prod`
### Behavior
1. Enforces execution from `main` only.
2. Reads latest tag matching `v*`.
3. Calculates next version using `RELEASE_TYPE`:
- `major`: increments major, resets minor/patch to 0.
- `minor`: increments minor, resets patch to 0.
4. Creates and pushes annotated tag.
5. Deploys using repository variable `DEPLOY_COMMAND` with `RELEASE_TAG`.
### Inputs
- `RELEASE_TYPE` (`major` or `minor`, default `minor`)
### Required variable
- `DEPLOY_COMMAND` (must consume `RELEASE_TAG`)
## 6) Rollback in Production (Custom Pipeline)
Custom pipeline: `rollback-prod`
### Behavior
1. Enforces execution from `main` only.
2. Validates `ROLLBACK_TAG` is provided.
3. Validates tag exists in repository.
4. Re-deploys by setting `RELEASE_TAG=ROLLBACK_TAG` and executing `DEPLOY_COMMAND`.
### Input
- `ROLLBACK_TAG` (example: `v2.3.0`)
## 7) Operational Notes
- Branch names in pipeline checks are case-sensitive; current gates use `Develop` and `Staging` (capitalized).
- Branch restriction and pipeline branch names must remain consistent to avoid false gate failures.
- Production deployments should be traceable by release tag and deployment record.
## 8) Governance Summary
This setup enforces controlled upward promotion (`feature/bugfix` -> `Develop` -> `Staging` -> `main`), allows emergency hotfix routing, and combines:
- structural branch protections,
- approval/workflow controls,
- automated PR validation,
- and auditable tagged production release/rollback execution.