Merged in feature/new_output_format (pull request #879)

Feature/new output format * bugfix * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * requested postprocessing changes * prompt changes reverted * fix pipeline issues * fix pipeline issues * fix pipeline issues * fixed formatting * fixed formatting * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * save dashboard and cc output separately * save dashboard output in s3 * pipeline error fixed * json list through postprocessing * Merged DEV into Optimize/DAIP2-1474-restructure-postprocess * Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format * Restructure output file organization and add standard field sanitization Output Structure Changes: - Reorganize output files into hierarchical directory structure: - full_outputs/cc_results/ for consolidated CC results - full_outputs/dashboard_results/ for consolidated dashboard results - full_outputs/ for error files - automation_qa-qc/ for QC/QA validated results and statistics - parent-child/ for parent-child mapping outputs - tracking/ for usage and cost tracking data - individual/ for per-file CC results (dashboard individual files removed) - Update file naming conventions to match new structure - Remove QC/QA processing for error files (error files are saved without validation) Post-Processing Changes: - Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.) when they are the only value in a cell (applies before CC/dashboard split) - Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks) - Ensure standard cleaning runs before splitting into CC … * Ran Black for CI * Refactor file splitting logic and add comprehensive tests - Refactored splitting logic in io_utils.py: - Consolidated repeated splitting code into two focused helper functions: - _write_local_split_files() for local file writing with splitting - _write_s3_split_files() for S3 file writing with splitting - Both helpers use shared split_dataframe_by_filename() function - Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py - Added comprehensive test coverage for splitting logic: - Tests for split_dataframe_by_filename() with various scenarios - Tests for local and S3 write operations with single and multiple splits - Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types - Fixed existing test failures (write_s3 error handling, path assertions) - Improved code maintainability and readability * Fix failing tests in test_postprocess.py - Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING (PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped - Added FILE_NAME column to all file structure test DataFrames (required for splitting logic) - Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests - Fixed patch decorators for S3 tests to properly mock logging All 41 tests now passing. * Black for CI * Blank [] and ['[]'] in output instead of displaying them - Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"] - Update format_as_json_list to return blank for empty lists instead of [] - Filter out empty-list placeholder items from list values in format_as_json_list - Add tests for clean_na_values empty list handling and format_as_json_list Co-authored-by: Cursor <cursoragent@cursor.com> * Merged DEV into feature/new_output_format * Pipeline config, parent-child, dashboard, and runner fixes - Parent-child: enable by default, write to run directory, S3 upload via io_utils - Dashboard: optional (CC only by default), run_dashboard=True to enable - Add io_utils.upload_local_file_to_s3 for centralized file uploads - Parent-child returns (row_count, local_path); pipeline handles S3 upload - Add TODO in config for WRITE_PC_TO_S3 removal after approval - Fix indentation errors in runner.py Co-authored-by: Cursor <cursoragent@cursor.com> Approved-by: Katon Minhas
2026-02-12 21:43:34 +00:00
parent 70215bbc69
commit 637d2dea1f
13 changed files with 1941 additions and 156 deletions
@@ -6,6 +6,7 @@ import re
 from datetime import datetime
 import pandas as pd
 from src.utils import string_utils
+import json

 # Determine the base directory for the project
 BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
@@ -40,16 +41,141 @@ def normalize_indicator_field(value: str) -> str:
    """
    Normalize the indicator field value to 'Y' or 'N'.

+    Explicitly handles empty strings, None, NaN, and blank values by setting them to 'N'.
+
    Args:
        value (str): The input value from an _IND field.
    Returns:
        str: 'Y' if the value is 'Y', otherwise 'N'.
    """
-    if isinstance(value, str) and value.strip().upper() == "Y":
-        return "Y"
+    # Handle None, NaN, and empty values
+    if value is None or (isinstance(value, float) and pd.isna(value)):
+        return "N"
+
+    # Handle empty strings and whitespace-only strings
+    if isinstance(value, str):
+        value_stripped = value.strip()
+        if not value_stripped:
+            return "N"
+        if value_stripped.upper() == "Y":
+            return "Y"
+
+    # Everything else becomes 'N'
    return "N"


+def clean_na_values(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Remove cells containing ONLY "N/A", "['N/A']", "[]", "['[]']", "UNKNOWN", or similar placeholder values.
+
+    This function removes placeholder values only when they are the sole value in a cell.
+    If a placeholder value is part of a list with other non-placeholder values, it is kept.
+    Empty list representations (e.g. "[]", "['[]']") are replaced with blank to avoid
+    clutter in the output.
+
+    Args:
+        df (pd.DataFrame): Input DataFrame to clean.
+
+    Returns:
+        pd.DataFrame: DataFrame with placeholder values removed from cells where they are the only value.
+    """
+    if df.empty:
+        return df
+
+    df_cleaned = df.copy()
+
+    # Patterns to match placeholder values (case-insensitive)
+    # Include [] and ['[]'] so empty list representations are blanked instead of shown
+    placeholder_patterns = [
+        r"^N/A$",
+        r"^NA$",
+        r"^NaN$",
+        r"^UNKNOWN$",
+        r"^None$",
+        r"^null$",
+        r"^\['N/A'\]$",
+        r"^\[\"N/A\"\]$",
+        r"^\[N/A\]$",
+        r"^\[\]$",  # Empty list "[]"
+        r"^\['\[\]'\]$",  # Python repr of list containing "[]"
+        r"^\[\"\[\]\"\]$",  # JSON list containing "[]"
+    ]
+
+    def is_placeholder_only(value):
+        """Check if value is only a placeholder."""
+        # Handle None, NaN, and pd.NA
+        if value is None:
+            return True
+        try:
+            if pd.isna(value):
+                return True
+        except (TypeError, ValueError):
+            # pd.isna might raise TypeError for some types, continue checking
+            pass
+
+        if isinstance(value, str):
+            value_stripped = value.strip()
+            if not value_stripped:
+                return True
+
+            # Try to parse as JSON list first
+            if value_stripped.startswith("[") and value_stripped.endswith("]"):
+                try:
+                    parsed_list = json.loads(value_stripped)
+                    if isinstance(parsed_list, list):
+                        # If list contains only placeholder values, consider it placeholder
+                        if len(parsed_list) == 0:
+                            return True
+                        all_placeholders = True
+                        for item in parsed_list:
+                            item_str = str(item).strip()
+                            is_placeholder = False
+                            for pattern in placeholder_patterns:
+                                if re.match(pattern, item_str, re.IGNORECASE):
+                                    is_placeholder = True
+                                    break
+                            if not is_placeholder and item_str:
+                                all_placeholders = False
+                                break
+                        return all_placeholders
+                except (json.JSONDecodeError, ValueError):
+                    # If JSON parsing fails, continue with string matching
+                    pass
+
+            # Check against placeholder patterns for string values
+            for pattern in placeholder_patterns:
+                if re.match(pattern, value_stripped, re.IGNORECASE):
+                    return True
+
+        # Handle list types (Python lists)
+        if isinstance(value, list):
+            if len(value) == 0:
+                return True
+            # If list contains only placeholder values, consider it placeholder
+            all_placeholders = True
+            for item in value:
+                item_str = str(item).strip()
+                is_placeholder = False
+                for pattern in placeholder_patterns:
+                    if re.match(pattern, item_str, re.IGNORECASE):
+                        is_placeholder = True
+                        break
+                if not is_placeholder and item_str:
+                    all_placeholders = False
+                    break
+            return all_placeholders
+
+        return False
+
+    # Apply cleaning to all columns
+    for col in df_cleaned.columns:
+        df_cleaned[col] = df_cleaned[col].apply(
+            lambda x: "" if is_placeholder_only(x) else x
+        )
+
+    return df_cleaned
+
+
 def format_rate_fields_with_commas(value: str) -> str:
    """
    Format the rate with commas and two decimal places.
@@ -69,6 +195,32 @@ def format_rate_fields_with_commas(value: str) -> str:
        return ""


+def normalize_currency(value: str) -> str:
+    """
+    Normalize currency-like strings by removing $ and commas.
+
+    Args:
+        value (str): A string representing a currency value.
+    Returns:
+        str: Normalized numeric string or empty string if invalid.
+    """
+    if string_utils.is_empty(value):
+        return ""
+
+    if isinstance(value, (int, float)) and not pd.isna(value):
+        return str(value)
+
+    if not isinstance(value, str):
+        return ""
+
+    text = value.strip()
+    if text.upper() in {"N/A", "NA", "NAN"}:
+        return ""
+
+    text = text.replace("$", "").replace(",", "").strip()
+    return text
+
+
 def remove_hyphens(value):
    """
    Remove hyphens from the input value.
@@ -238,7 +390,7 @@ def normalize_cpt_fields(value):
    else:
        result = [str(value).strip()]  # Default case for other types

-    return str(result)  # Return the list directly
+    return json.dumps(result)  # Return JSON list string


 def generate_reimb_ids(df: pd.DataFrame) -> pd.DataFrame:
@@ -509,6 +661,30 @@ def add_aarete_derived_amendment_num(df: pd.DataFrame) -> pd.DataFrame:
        pd.DataFrame: The updated DataFrame with the 'AARETE_DERIVED_AMENDMENT_NUM' column.
    """

+    def derive_amendment_num(value):
+        if value is None or (isinstance(value, float) and pd.isna(value)):
+            return "0"
+
+        if not isinstance(value, str):
+            value = str(value)
+
+        text = value.strip()
+        if string_utils.is_empty(text):
+            return "0"
+
+        lowered = text.lower()
+        if lowered in {"n/a", "na", "nan", "unknown", "unspecified", "none", "null"}:
+            return "0"
+
+        if any(token in lowered for token in ["base", "original", "master"]):
+            return "N/A"
+
+        numbers = re.findall(r"\b\d+\b", text)
+        if numbers:
+            return str(int(numbers[-1]))
+
+        return "0"
+
    if "CONTRACT_AMENDMENT_NUM" in df.columns:
        df["AARETE_DERIVED_AMENDMENT_NUM"] = (
            df["CONTRACT_AMENDMENT_NUM"]
@@ -517,6 +693,10 @@ def add_aarete_derived_amendment_num(df: pd.DataFrame) -> pd.DataFrame:
            .fillna(df["CONTRACT_AMENDMENT_NUM"])
        )

+        df["AARETE_DERIVED_AMENDMENT_NUM"] = df["CONTRACT_AMENDMENT_NUM"].apply(
+            derive_amendment_num
+        )
+
    return df


@@ -861,6 +1041,24 @@ def add_aarete_derived_signatory_complete_ind(df: pd.DataFrame) -> pd.DataFrame:
        return df


+def standardize_file_name(file_name: str) -> str:
+    """
+    Standardizes the given file name by applying a series of regex replacements
+    to normalize terms related to time periods.
+
+    Args:
+        file_name (str): The original file name string.
+
+    Returns:
+        str: The standardized file name string.
+    """
+
+    # remove .txt from the end if present
+    file_name = re.sub(r"\.txt$", "", file_name, flags=re.IGNORECASE)
+
+    return file_name
+
+
 def fill_claim_type_from_title(df: pd.DataFrame) -> pd.DataFrame:
    """
    Fill empty AARETE_DERIVED_CLAIM_TYPE_CD values using two strategies:
@@ -978,3 +1176,52 @@ def fill_claim_type_from_title(df: pd.DataFrame) -> pd.DataFrame:
    df["AARETE_DERIVED_CLAIM_TYPE_CD"] = df.apply(infer_from_title, axis=1)

    return df
+
+
+def _is_empty_list_placeholder(item) -> bool:
+    """Return True if item is "[]", "['[]']", or similar empty-list representation."""
+    if item is None or (isinstance(item, float) and pd.isna(item)):
+        return True
+    s = str(item).strip()
+    return s in ("[]", "['[]']", '["[]"]') or s == ""
+
+
+def format_as_json_list(val):
+    """
+    Format a value as a JSON list string. Returns blank for empty lists
+    instead of "[]" to avoid clutter in output.
+    """
+    # 1. Handle actual lists or Nulls
+    if isinstance(val, list):
+        filtered = [
+            str(i).strip()
+            for i in val
+            if i is not None and not _is_empty_list_placeholder(i)
+        ]
+        return json.dumps(filtered) if filtered else ""
+    if pd.isna(val) or val == "" or val == "[]":
+        return ""
+
+    # 2. Cleanup & Parsing
+    text = str(val).strip()
+
+    # Try to parse as JSON if it looks like a list
+    if text.startswith("[") and text.endswith("]"):
+        try:
+            parsed = json.loads(text)
+            if isinstance(parsed, list):
+                filtered = [
+                    str(i).strip()
+                    for i in parsed
+                    if i is not None and not _is_empty_list_placeholder(i)
+                ]
+                return json.dumps(filtered) if filtered else ""
+        except (json.JSONDecodeError, ValueError):
+            text = text.strip("[]")  # Strip brackets if it's a "fake" list like ['A']
+
+    # 3. Handle comma-separated strings & Clean special characters
+    # This regex keeps letters, numbers, commas, and spaces
+    clean_text = re.sub(r"[^a-zA-Z0-9, ]+", "", text)
+    items = [i.strip() for i in clean_text.split(",") if i.strip()]
+
+    return json.dumps(items) if items else ""